P2P over Mobile Ad-Hoc Networks

Antonio Jose Ribeiro da Silva Marques

Dissertacao para obtencao do Grau de Mestre emEngenharia de Redes de Comunicacoes

JuriPresidente: Prof. Doutor Rui Jorge Morais Tomaz ValadasOrientador: Prof. Doutor Fernando Henrique Corte Real Mira da SilvaCo-Orientador: Prof. Doutor Rui Manuel Rodrigues RochaVogais: Prof. Doutor Luıs Filipe Lourenco Bernardo

Novembro de 2009

Acknowledgments

I would like to thank Professor Fernando Mira da Silva and Professor Rui Rocha for their invaluable

guidance and supervision in the course of the long and difficult process of writing this thesis. Their

considerable knowledge and experience provided me with the tools to pursue my own research interests.

I also extend my thanks to my colleague Carlos Andre for helping in setting up the testbed and in defining

the remote monitoring functionality of the proposed middleware. I would like to thank my parents for their

constant support and encouragement as well as my friends for their patience and understanding during

these past few months.

Abstract

Peer-to-peer (P2P) networks and mobile ad-hoc networks (MANET) share many similarities, such as

decentralisation, dynamic topology and auto-configuration, such that all participants must collaborate to

ensure the correct operation of the network. Furthermore, both types of networks share a common goal:

the discovery of nodes, according to specific criteria, along with the routing of messages between such

nodes. The P2P paradigm is therefore well suited for content distribution, collaboration and communi-

cation applications in a MANET. Despite this, existing P2P systems, designed for the Internet, do not

exploit the synergies between P2P networks and MANETs since the P2P overlay network is unaware

of the topology of the underlying physical network. This can lead to inefficient paths between neigh-

bours in the overlay and unnecessary network overhead due to the duplicate functionality implemented

at different layers in the network protocol stack.

This thesis proposes a P2P middleware solution to facilitate the development of P2P applications

optimised for use in MANETs. To achieve this, the process of searching for P2P application content

is combined with the route discovery process of the MANET routing protocol, Hybrid Wireless Mesh

Protocol (HWMP). When multiple sources are available for the same content, the middleware can choose

a peer based on the cost of the path to that peer, as given by the routing protocol. An implementation of

the proposed middleware is shown to improve performance and reduce the network resource use when

compared to a traditional P2P system layered on top of a MANET.

Keywords

P2P, mobile ad-hoc networks, HWMP, middleware

iii

Resumo

As redes peer-to-peer (P2P) e as redes moveis ad-hoc (MANET) apresentam um conjunto de

semelhancas, tais como descentralizacao, topologia dinamica e auto-configuracao, sendo que os par-

ticipantes nas mesmas necessitam de colaborar para garantir o seu correcto funcionamento. Ambos

os tipos de rede partilham um objectivo comum: a descoberta de nos, de acordo com um determi-

nado criterio, e o encaminhamento de pacotes entre nos da rede. O paradigma P2P surge entao como

uma solucao atraente para a construcao de aplicacoes de distribuicao de conteudos, colaboracao e

communicacao em MANETs. Apesar destas semelhancas, os sistemas P2P actuais, pensados para

utilizacao na Internet, nao exploram as sinergias existentes entre as redes P2P e as MANETs dado que

a rede P2P estabelecida nao tem em conta a topologia da rede fısica sobre a qual esta a operar. Tal

facto pode levar ao estabelecimento de caminhos ineficientes entre vizinhos na rede P2P e a geracao

de trafego excedentario devido a duplicacao de funcionalidades entre as varias camadas de rede.

Esta tese propoe um middleware P2P cujo objectivo principal consiste em facilitar o desenvolvi-

mento de aplicacoes P2P optimizadas para MANETs. Esta optimizacao foi conseguida combinando o

processo de pesquisa de conteudo aplicacional com o processo de descoberta de caminhos por parte

do protocolo de encaminhamento da MANET, o Hybrid Wireless Mesh Protocol (HWMP). Os testes rea-

lizados ao middleware demonstraram um melhor desempenho e uma reducao no consumo de recursos

de rede face a um sistema P2P tradicional a operar sobre uma MANET.

Palavras Chave

P2P, redes moveis ad-hoc, HWMP, middleware

iv

Contents

1 Introduction 2

1.1 Motivation and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2 State of the Art 6

2.1 Peer-to-Peer (P2P) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.1 Peer-to-Peer Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.2 Unstructured Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.3 Structured Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.3.A Chord . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1.3.B CAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.1.3.C Tapestry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.2 Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.2.1 Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.2.2 IEEE 802.11s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.3 Peer-to-Peer Systems for Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . . . 22

2.3.1 Unstructured Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.3.2 Structured Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3 A Peer-to-Peer Middleware for Mobile Ad-hoc Networks 28

3.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.3 Peer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.3.1 Extending HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.3.2 P2P-HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.3.3 Storage Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.3.4 Query Routing Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.3.5 Service Registration Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

v

3.3.6 Peer Selection Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

3.3.7 Peer Management Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.3.8 Middleware Communication Service . . . . . . . . . . . . . . . . . . . . . . . . . . 51

4 Implementation 53

4.1 Extending HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.2 P2P-HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

4.3 Middleware Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.4 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5 Testing and Evaluation 64

5.1 Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

5.2 Test Metrics and Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

6 Conclusion 76

A Middleware Search Message Formats 84

B Middleware SNMP MIB Specification 89

C Middleware API 92

D P2P-HWMP Protocol 96

vi

List of Figures

3.1 Comparison between unstructured P2P search and reactive MANET route discovery . . . 31

3.2 Searching for documents in a MANET by leveraging HWMP route discovery . . . . . . . . 33

3.3 Source peer selection using HWMP path metrics . . . . . . . . . . . . . . . . . . . . . . . 34

3.4 Peer architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

3.5 Interactions between middleware components at the requesting peer . . . . . . . . . . . . 37

3.6 Interactions between middleware components at the receiving peer . . . . . . . . . . . . . 38

3.7 IEEE 802.11 management frame containing a middleware search request . . . . . . . . . 39

3.8 HWMP vendor registry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.9 Interactions performed by P2P-HWMP for a received PREQ frame . . . . . . . . . . . . . 42

3.10 Example of a document used by a P2P social networking application . . . . . . . . . . . . 43

3.11 Example of the types of search queries supported by the storage service . . . . . . . . . 44

3.12 Example of a file-backed document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.1 Peer software stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.2 Middleware daemon service dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3 Resource URLs provided by the peer communication interface . . . . . . . . . . . . . . . 61

5.1 Testbed network topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.2 Query forwarding delay for increasing query load . . . . . . . . . . . . . . . . . . . . . . . 69

5.3 Query processing time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

5.4 Query success rate for an increasing query load . . . . . . . . . . . . . . . . . . . . . . . 71

5.5 Overall traffic (bytes) generated for a single query . . . . . . . . . . . . . . . . . . . . . . 72

5.6 Overall traffic (messages) generated for a single query . . . . . . . . . . . . . . . . . . . . 73

5.7 Broadcast traffic (messages) generated for a single query . . . . . . . . . . . . . . . . . . 74

vii

List of Tables

2.1 Classification of peer-to-peer systems and infrastructures . . . . . . . . . . . . . . . . . . 11

5.1 Testbed node configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

viii

List of Acronyms

AODV Ad hoc On-Demand Distance Vector

API Application Programming Interface

ARP Address Resolution Protocol

DHT Distributed Hash Table

GUID Globally Unique Identifier

HTTP Hypertext Transfer Protocol

HWMP Hybrid Wireless Mesh Protocol

IP Internet Protocol

IPC Inter-process communication

JSON Javascript Object Notation

LAN Local Area Network

LKM Loadable Kernel Module

MAC Medium Access Control

MANET Mobile Ad-hoc Network

MIB Management Information Base

MTU Maximum Transmission Unit

NDP Neighbor Discovery Protocol

NMS Network Management System

OID Object Identifier

OUI Organisationally Unique Identifier

ix

P2P Peer-to-Peer

PERR Path Error

PID Process Identifier

PREP Path Reply

PREQ Path Request

REST Representational State Transfer

SNMP Simple Network Management Protocol

TCP Transmission Control Protocol

TTL Time-to-Live

UDP User Datagram Protocol

URI Uniform Resource Identifier

URL Uniform Resource Locator

WMN Wireless Mesh Network

XML Extensible Markup Language

XML-RPC Extensible Markup Language - Remote Procedure Call

1

1Introduction

Contents

1.1 Motivation and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2

The rapid growth of wireless communication and mobile computing technologies have led to the

appearance of a new networking concept — Mobile Ad-hoc Networks (MANETs). These networks

comprise a set of autonomous mobile nodes which are not dependent on existing fixed communication

infrastructure to operate. Nodes communicate using wireless links and rely on intermediate nodes to

forward their messages beyond a node’s radio coverage. MANETs, unlike traditional communications

networks, are fully decentralised and have a highly dynamic topology.

In many situations, users cannot rely on an existing network infrastructure: it may be unreliable, too

expensive or simply inexistent. In these cases MANETs may be the only solution. Various applications

for these types of networks have been identified [1]: spontaneous meetings and unplanned events,

where planning and setting up an infrastructure would be too time consuming; disaster relief scenarios

during which existing infrastructure may be damaged and unreliable; military battlefield communications;

sparsely populated remote areas that lack any existing infrastructure or where existing infrastructure is

too expensive.

Another emerging network concept is that of Peer-to-Peer (P2P) networks which allow direct sharing

of resources (CPU, bandwidth, storage) among a large number of users in a decentralised manner [2].

These networks can serve many purposes, from digital content distribution, popularised by applications

such as Napster, Gnutella and BitTorrent, to Internet telephony in, for example, Skype. Unlike the client-

server model, where a limited number of servers provide services and content to a large number of

clients, P2P networks emphasise direct communication among peers acting as both clients and servers.

In pure P2P networks, such as Gnutella [3], all peers perform the same functions and are capable

of sending and replying to request messages sent by any other peer. No central element is used to

manage and coordinate network interactions. Due to the highly transient population of P2P networks,

their network topology is very dynamic, with peers constantly joining and leaving the network. Current

P2P architectures are mainly designed for the traditional fixed and wired infrastructure and can operate

on a global scale using the Internet.

The synergies between these two networking concepts provide an interesting research topic [4].

Both MANETs and P2P networks follow a P2P paradigm characterised by the lack of a central node or

peer acting as a managing server, all participants having therefore to collaborate in order for the whole

system to work. A key issue in both networks is the process of discovering the requested data or route

efficiently in a decentralised manner. MANETs and P2P networks also have a highly dynamic topology:

nodes are constantly moving in and out of radio coverage of neighbouring nodes in the former and peers

are continuously joining or leaving the network in the latter.

Although similar, the integration of both these concepts presents some important challenges. P2P

networks aim to provide resource sharing on top of an existing communication infrastructure, such as the

Internet, where connectivity is not an issue, by forming an overlay network and maintaining connections

3

among its peers. MANETs focus on providing multi-hop wireless connectivity where no infrastructure

exists or is inadequate. P2P application networks operate at the application layer of the OSI reference

model while MANETs operate at the network layer or data link layer. Routing functionality is therefore

present in both layers. On the one hand, application layer routing is used by search queries to find a

peer containing the desired information within the P2P overlay network. On the other hand, network

layer or data link layer routing is used to discover the route to any given node. Another difference lies in

the fact that nodes in a MANET are normally mobile, constrained in computational power, energy and

communication bandwidth, which is usually not an issue for peers connected via wired infrastructure to

Internet wide P2P applications.

These differences give rise to an important challenge in integrating both concepts — existing P2P

systems form an overlay network that is completely unaware and independent from the underlying phys-

ical topology. Logical proximity differs from physical proximity as neighbours in the P2P overlay network

may in fact be located on opposite sides of the planet. This may lead to extremely long routes and unnec-

essary traffic, while not particularly relevant for the high-bandwidth Internet, presents a larger problem

for MANETs due to their lower bandwidth and reliability.

1.1 Motivation and Goals

Consider the increasing push towards using computing technologies and mobile communications in

developing countries and in remote communities where Internet access is intermittent, expensive and/or

unreliable. A prime example of this scenario is the One Laptop Per Child [5] project which envisions the

use of networked laptops in aiding education in the developing world. These cases, where the ability

to access the Internet is severely hindered, stress the importance of decentralised communication and

collaboration tools and the use of MANETs. These can be used to allow villagers to communicate or

humanitarian aid workers to exchange documents. Given the similarities between P2P systems and

MANETs, it is clear that the P2P paradigm is particularly suited to enable content distribution, commu-

nication and collaboration applications in this environment.

The goal of this thesis consists in studying the common issues faced by traditional P2P applications

operating on top of a MANET and proposing a P2P middleware which is optimised for use in a MANET

environment. The proposed middleware should enable P2P application developers to quickly develop a

wide range of applications while leveraging the unique characteristics of these types of networks.

An emerging IEEE network standard, IEEE 802.11s, a mesh networking amendment to the current

IEEE 802.11 standard, will be used as a basis for the construction of the MANET used in the proposed

solution. Although still a draft, IEEE 802.11s, promises to become the de facto standard for the creation

of multi-hop wireless mesh networks and MANETs.

4

1.2 Structure

This text is structured according to the following chapters. Chapter 2, presents a brief overview of

the state of the art of existing P2P systems, both structured and unstructured, and MANETs, namely

the available MANET routing protocols, together with existing research on optimising P2P systems for

use in MANETs. The current draft of the IEEE 802.11s standard is also described along with its default

routing protocol.

Chapter 3 provides a description of the proposed P2P middleware which is tailored specifically for use

in a MANET scenario. This chapter starts with the definition of the requirements which guided the design

of the middleware as well as the chosen architecture, including other solutions which were considered

during the design process. A detailed description of the functionality provided by each middleware

component follows.

Chapter 4 describes the implementation of the proposed middleware for the GNU/Linux operating

system including the selected technologies and main implementation decisions.

Chapter 5 is dedicated to validating and evaluating the performance of the middleware implementa-

tion. This chapter describes the testbed, test scenarios and methodology employed for both functional

and performance tests. To conclude, the experimental results obtained using the testbed are presented

and discussed.

Finally, Chapter 6 draws some important conclusions and presents some suggestions for further

improvements to the proposed middleware.

1.3 Contributions

A middleware architecture for supporting the execution and development of P2P applications opti-

mised for MANETs is proposed as part of this thesis. In addition to this text, the middleware architecture

is also described in an article presented at the 6th IEEE Annual Communications Society Conference

on Sensor, Mesh and Ad Hoc Communications and Networks Workshop (SECON 2009) in Rome, Italy

on June 23, 2009 [6]. A fully functioning implementation of the proposed middleware architecture was

developed for the GNU/Linux operating system and validated using a MANET testbed.

5

2State of the Art

Contents

2.1 Peer-to-Peer (P2P) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.3 Peer-to-Peer Systems for Mobile Ad-hoc Networks . . . . . . . . . . . . . . . . . . . . 22

6

This chapter provides an overview of the state-of-the-art in the fields of peer-to-peer (P2P) and

mobile ad-hoc networking, as well as an analysis of current research related to their convergence. First

of all, the typical architecture and properties of a P2P network are presented, along with the main design

features of existing P2P systems and their shortcomings. Subsequently, the main aspects and problems

in mobile ad-hoc networks (MANETs) are described, including a brief look at existing routing algorithms.

Finally, the most promising approaches to provide P2P systems tailored for mobile ad-hoc networks are

detailed and compared.

2.1 Peer-to-Peer (P2P)

A distributed network architecture is designated peer-to-peer if it consists of a set of self-organized

and interconnected nodes, forming a topology that enables the sharing of computer resources (content,

network bandwidth, storage capacity, processing power) through direct exchange, without requiring the

intermediation of a central server. A P2P network must provide acceptable connectivity and performance

in the face of failures and a transient population of nodes [7]. Other definitions for the term peer-to-

peer have been proposed [2] with varying degrees of broadness and, as a result, existing systems or

applications are classified as peer-to-peer according to different definitions. For the rest of this chapter,

the terms node and peer are used interchangeably, where appropriate, to refer to the participants in a

P2P network.

One of the defining aspects of a P2P network resides in its ability to share computer resources

through direct exchange between peers, rather than through an intermediary such as a central server or

authority. In spite of this, many existing P2P systems make use of centralised servers, to some extent,

for tasks such as bootstrapping (the process by which a node joins a network), maintaining reputation

ratings and distribution of encryption keys. Other systems, for example Napster go so far as to use

global centralised servers for basic functionality, like indexing and searching, which, although not fitting

the aforementioned definition, are usually considered P2P systems. The lack of centralised coordination

and management of a P2P network requires the active co-operation of all peers with tasks that include

locating content and other peers, routing messages, maintaining connections to neighbouring nodes

and inserting and retrieving content. Another defining aspect is the capacity to deal with, and adapt to,

network instability due to node and connection failures, in addition to the transient population caused by

churn, i.e., nodes constantly joining and leaving the network. Thus, peer-to-peer topologies must pro-

vide fault-tolerant capabilities, adapting themselves to changing network conditions, while maintaining

adequate connectivity and performance.

Peer-to-peer network architectures serve as a basis for a wide range of applications. These appli-

cations can usually be classified in one of the following categories: communication and collaboration,

7

distributed computing, internet service support, database systems and content distribution. For the us-

age scenarios typically envisioned for mobile ad-hoc networks, communication and collaboration as well

as content distribution applications provide the most relevant and interesting services. The former refers

to the infrastructures and systems that enable peers to communicate and collaborate directly, normally

in a real-time fashion, comprehending applications such as instant messaging (e.g. Jabber [8]) and P2P

telephony (e.g. Skype). The latter focuses on systems that target the sharing of digital media or other

types of data among users, from straightforward file-sharing applications (e.g. Napster, Gnutella [3]) to

complex distributed data storage systems (e.g. PAST [9]).

Content distribution systems are the most popular category of P2P applications and the most exten-

sively studied, therefore, these applications shall be used as examples in the following sections in order

to better illustrate the key aspects of P2P networking.

2.1.1 Peer-to-Peer Overlay Networks

As mentioned previously, a P2P system relies on a network of nodes (i.e. peers) and connections

between them for its operation. This application-layer P2P network is formed on top of the underlying

physical network, typically an IP network such as the Internet, and is referred to as an overlay network.

Current P2P overlay networks are implemented independently from, and unaware of, the underlying

physical network topology, whose routing mechanisms are used to establish the virtual links between

nodes in the overlay.

Peer-to-peer overlay networks are characterised by their structure, degree of centralisation and rout-

ing and location mechanisms used to discover information and route messages within the network.

These attributes influence the overlay network’s fault-tolerance and self-organization capabilities, main-

tenance requirements, performance and scalability. Peer-to-peer overlay networks are usually classified

according to their centralisation and their structure.

The centralisation of a P2P overlay network refers to the role played by central entities, if any, in the

coordination of node activities. In [7], three categories for classifying the centralisation of an overlay

network are proposed: purely decentralised, partially centralised and hybrid decentralised.

Purely decentralised networks correspond to the strictest definition of a P2P network in which all

nodes assume the same roles, thus acting as both clients and servers without any central entity coordi-

nating their activities. The term servent has been coined to refer to such nodes, made up of the terms

server and client. Nodes are therefore responsible for initiating requests for information, forwarding

requests on behalf of other nodes and replying to requests directed towards them.

Partially centralised networks are similar to purely decentralised networks with the addition of the

concept of supernodes or superpeers. These nodes assume a more significant role in the network,

functioning as centralised local indexes for a given set of other conventional nodes. Nodes are assigned

8

the role of supernode dynamically by the network according to certain criteria, for example, processing

power and bandwidth. If a supernode fails, its dependant nodes can be assigned to existing supernodes

or to a newly elected supernode.

Hybrid decentralised networks provide a combination of the traditional client-server model and the

P2P model [2]. Peer-to-peer systems based on this type of network architecture rely on a central server

to coordinate interactions between peers. This server is responsible for tasks such as maintaining a list

of peers connected to the network and descriptions of the information shared by each of those peers.

The central server only stores metadata, allowing it to provide search functionality to peers connected

to it, but unlike the client-server model, end-to-end interaction (e.g. file transfer) is accomplished directly

between peers. Hybrid decentralised overlay networks, given the existence of a single point of failure,

the server, are more vulnerable to technical failure.

Another important aspect used in classifying P2P overlay networks is their structure, which refers to

whether the creation of the network is non-deterministic, where links between nodes are established in

a largely ad-hoc manner, or deterministic, where the overlay topology has to follow precise rules. Thus,

overlay networks fall into one of two categories according to their structure: unstructured or structured.

In unstructured overlay networks there is no restriction on which nodes a given peer can select as its

neighbours. Virtual links in the overlay network are therefore established in an ad-hoc manner resulting

in a random overlay topology that does not follow a specific structure. A peer’s routing table, consisting

of its neighbour nodes, requires minimal maintenance, because an unreachable entry can easily be

replaced with any other node. Furthermore, there is no relationship between the placement of an object

or file and the overlay topology. Since the information a peer is searching for can be located anywhere

in the network, and any of its neighbours is equally likely to possess it, query flooding mechanisms

are employed to locate information. A peer starts by sending a query to its neighbours, which in turn

propagate the query to their neighbours, until the desired information is located and a reply is sent back

to the originating peer. A maximum search radius is used to limit the propagation of queries and to

avoid swamping the entire network with queries. More sophisticated and efficient strategies have been

suggested to avoid the flooding behaviour such as the use of random walks [10].

Although query flooding has an undeniable impact on the scalability of the network, unstructured

overlay networks are extremely resilient to the frequent arrival and departure of nodes making them

suitable for highly-transient node populations. Examples of unstructured P2P systems include Napster,

Gnutella [3] [11] and Kazaa [12].

Structured overlay networks appeared in an effort to overcome the scalability issues raised by un-

structured networks. Rather than allowing the overlay network to be created in an ad-hoc manner,

structured overlay networks not only follow a set of specific rules for their formation, but also store ob-

jects at precisely defined locations. This allows the use of efficient and scalable routing algorithms, able

9

to reliably locate a given object in a bounded number of hops, usually O(log(N)), where N is the number

of nodes in the network. Structured overlays provide a mapping between a given object’s identifier and

the location of the object within the network, i.e., the address of the node with the desired object. This

mapping is achieved through the use of a mechanism known as a distributed hash table (DHT), where

each node in the network is assigned a unique identifier and each object stored within the network is

assigned a unique key. Both node identifiers and object keys exist within the same identifier namespace

and a node is responsible for the objects whose keys are closer to its own identifier, among all live nodes

in the network, according to a predefined distance metric (e.g. numerical distance). A peer can then

perform a lookup on a specified key where each node along the path will forward the request to a node

that is progressively closer to the node responsible for that key. This deterministic routing technique is

also known as key-based routing. Prominent examples of structured overlay networks include the P2P

infrastructures Chord [13], CAN [14], Tapestry [15] and Pastry [16], along with P2P systems based on

these infrastructures such as the PAST [9] and Oceanstore [17].

Despite their object lookup efficiency, structured network overlays have two main disadvantages.

The basic lookup interface offered by DHTs only allows for exact-match queries, i.e., queries where the

exact identifier of the desired object is known. This contrasts with the typical rich queries that can be

performed on unstructured networks, for example, through the use of keywords. Some approaches have

been proposed to layer keyword searches on top of DHTs, such as [18], but suffer from issues related

to load balancing, fault tolerance and hotspots caused by popular keywords. This problem represents

an active research area. Another disadvantage is the maintenance requirements incurred by node joins

and departures in the case of highly transient node populations.

A classification of existing P2P systems and infrastructures according to their centralisation and

structure is presented in Table 2.1. In light of the aforementioned taxonomy, some systems can easily

be identified as unsuitable for deployment in the decentralised environment of a MANET. This is the

case of hybrid decentralised P2P systems, which rely on the use of central entities to implement core

functionality. Since nodes in a MANET have limited resources and are more susceptible to failure when

compared to their counterparts in a fixed network infrastructure, it is unreasonable to expect that a given

node should continuously perform the role of a server, serving multiple clients simultaneously. This

node would not only represent a single point of failure, but would also suffer from increased network

and computational load when compared to other nodes. This uneven distribution of resource usage is

clearly at odds with the basic principles of a MANET. Mobile ad-hoc networks are further discussed in

Section 2.2

The following sections will be dedicated to briefly describing the operation and the main design

features of some of the P2P systems listed in Table 2.1, focusing on purely decentralised P2P systems,

given their similarities with MANETs.

10

Purely Partially Hybrid

Decentralised Centralised Decentralised

Unstructured Gnutella (0.4) Gnutella (0.6) Napster

FreeHaven Kazaa BitTorrent (Tracker)

Structured Chord

CAN

Tapestry

Pastry

BitTorrent (DHT)

Oceanstore

PAST

Table 2.1: Classification of peer-to-peer systems and infrastructures

2.1.2 Unstructured Overlay Networks

Gnutella [3] is the most prominent example of a purely decentralised unstructured overlay network

due to its widespread use, open protocol and large scale network topology. It provides a P2P overlay

network with appropriate routing and location mechanisms in order to allow users to share files with

other participants in the network. Given its completely decentralised nature, no central coordination

exists and all nodes perform the same functions, acting as both a client and a server (servent) and

connecting directly to each other.

The Gnutella application layer protocol is used along with the IP routing mechanisms of the under-

lying network to enable communication between peers. The initial versions of the protocol [3] specify

four basic types of messages: ping, pong, query and query hit. The ping message is used to discover

other hosts in the network by requesting that a given node identify itself to the sender. This identifi-

cation, including the IP address and port, is conveyed in a reply message known as pong. The query

message represents a search request and contains the search terms along with parameters specify-

ing quality requirements the replier should satisfy. The response to a query message is conveyed in

a query hit message which contains the identification of the responding node along with the result set.

These message types concern only the act of joining and searching the Gnutella network. File transfer is

done directly between peers, i.e., not through the Gnutella network, using the hypertext transfer protocol

(HTTP).

A node that wishes to connect to the Gnutella network must know the address of at least one other

node that is already connected to the network. This can be done by consulting one of the many well

known GWebCache servers which maintain an updated list of hosts. A client then sends a ping message

to each of its known hosts. Accessible nodes will respond with a pong message, identifying themselves,

11

and propagate the ping request to their neighbours. After obtaining a given number of working host

addresses, the original node can begin issuing search queries. Host addresses that a node learns

through GWebCache, pong replies and successful replies to queries are cached locally, so that future

sessions can be established without consulting a central database.

The search mechanism used in Gnutella reflects the typical approach followed by unstructured over-

lay networks referred in Section 2.1.1. Since there is no relationship between files and the nodes re-

sponsible for storing them, and files can be located anywhere within the network, a non-deterministic

search method is used. The initial versions of the Gnutella protocol, up to and including version 0.4,

specify a pure query flooding mechanism where a requesting node sends its search request, the query

message, to all its neighbours, which in turn forward the message to all their neighbours with the ex-

ception of the node from which they received the message. When the search criteria match one of the

files shared by a given node, a query hit message is sent back along the same path that was followed

by the original query message. Each message contains a time-to-live (TTL) field in its header to avoid

continuously forwarding it and unnecessarily flooding the network. This field is decremented at each hop

until it reaches zero, at which point the message is no longer forwarded. Furthermore, each message

has a unique identifier which allows a node to avoid routing loops and to preserve network bandwidth

by dropping messages whose identifier it has already seen and thus already processed. The identifiers

of both pong and query hit messages are equal to their respective request messages. This allows the

implementation of the reverse path routing previously discussed. Upon receiving a query hit message,

a node will contact the responding host directly and negotiate the transfer of the requested file via HTTP.

Simple query flooding limits the ability for this type of network to scale to an increasing number of

nodes and queries. There is an obvious trade-off between the selected TTL value and the network

traffic generated in addition to the amount of nodes searched. On the one hand, higher TTL values will

result in more nodes being contacted at the expense of increased network traffic, which may swamp the

underlying physical network. On the other hand, reducing the TTL value will reduce the network traffic

but at the expense of less nodes contacted and consequently fewer search results.

To address this problem, the latest version of Gnutella, version 0.6 [11], adopts a two-level hierarchy.

As such, Gnutella should now be classified as a partially centralised P2P system. The main difference

is the introduction of the concept of ultrapeer, a peer that is automatically elected in order to service

a small subset of regular peers, known as leaves, indexing the files contained by each of those peers.

Peers are elected as ultrapeers if they provide higher processing power, network bandwidth or even

higher uptime than other peers. Ultrapeers are responsible for the leaf nodes connected to them. Leaf

nodes will send a list of shared files to their ultrapeers, in order to allow them to index such files. Search

queries are also sent directly to the ultrapeers, which will then proxy the requests on behalf of the leaf

node. Upon receiving a search request, an ultrapeer will forward this request to other ultrapeers using a

12

query flooding method analogous to the one used on the purely decentralised version. An ultrapeer will

only forward a query to one of its leaves if it knows the leaf contains the requested content.

The use of ultrapeers in a partially centralised overlay network brings a few advantages over the

purely decentralised approach. Query latency, network traffic as well as the number of nodes involved

in routing a query are reduced since queries are propagated only among ultrapeers. This is achieved

while still maintaining the robustness and fault-tolerance characteristics of purely decentralised net-

works, since ultrapeers do not constitute single points of failure and can easily be replaced. Another

advantage of this approach is that it exploits the heterogeneity among nodes such as their network

bandwidth, storage capacity and processing power. Ultrapeers bear much of the load of the network,

freeing leaf nodes from costly routing and processing tasks. A more detailed explanation on the opera-

tion of the partially centralised Gnutella network can be found in [11]. Kazaa [12] is another example of

a partially centralised P2P system.

Several solutions have been suggested for improving the performance of search queries in unstruc-

tured overlay networks. These methods focus on reducing query latency, i.e., the time required to find

the specified content, and/or the amount of network traffic generated by search queries. Improving these

metrics thus contributes to increased system performance.

In [10], two alternatives to the standard query flooding approach are suggested: expanding ring

search and random walks. Expanding ring search is proposed as a solution to the problem of selecting

an appropriate TTL value. A node will use successive query floods with an increasing TTL. Starting with

a low TTL value, the node initiates a query flood for the requested content. If the search is unsuccessful,

the node increases its TTL value and tries again, repeating this process until the requested content is

found. Clearly this method presents a performance advantage when used to locate popular content with

higher replication ratios but does not address the issue of message duplication inherent in the flooding

approach. Alternatively, the authors propose the use of multiple parallel random walks, in which each

node will choose a random node from its set of neighbours to forward the query message to. This

message is called a walker. A walk will continue until the requested content is found, the message TTL

is exceeded (this is usually a large value when compared to standard query flooding and is used only to

avoid loops) or the originating node already has enough results (each walker periodically checks back

with the originating node). This method, when combined with proactive replication mechanisms, can

reduce network traffic by an order of magnitude when compared to expanding ring search.

Other more sophisticated search mechanisms are proposed in [19]. These include the use of intel-

ligent forwarding strategies where a node will choose a given neighbour, based on its performance in

past queries, for forwarding its search requests to. Each node maintains a set of simple statistics on its

neighbours allowing it to apply heuristics such as selecting the neighbour through which the most query

results have been received or the neighbour which returns messages with the lowest average hop count.

13

Assuming an intelligent neighbour selection, this method is shown to significantly reduce network traffic

while maintaining the quality of results. Another approach to reduce the number of nodes involved in

query processing and routing relies on the concept of local indices. Each node maintains a local index

of all the data shared by nodes which are located within a fixed number of hops from itself. Thus, a node

which receives a query can process it on behalf of any node within a given radius.

Another approach that relies on a distributed index mechanism is proposed in [20]. The concept

of routing indices is introduced which relies on the existence of an index at each node that provides

an indication of the neighbour that is most likely to be in the “direction” of the requested content rather

than the actual location of such content. The main objective of the routing index is to rank a node’s

neighbours according to their “goodness” for a given query. In the example presented, a node’s routing

index contains the number of documents along each path and the number of documents on each topic

of interest. A node can then select the neighbour whose path contains the highest number of documents

for a particular topic. This method is shown to improve performance by one or two orders of magnitude

over traditional query flooding.

In general, unstructured peer-to-peer systems can be an attractive choice when keyword searching is

desirable, the node population is highly transient, searches focus on content that is sufficiently replicated,

network size is adequate for the search methods used and guaranteed content discovery is not required.

2.1.3 Structured Overlay Networks

Structured overlay networks provide a purely decentralised, efficient, and scalable distributed object

location and routing mechanism. Most of these mechanisms rely on a hash-table interface that serves as

the foundation for a distributed lookup system. This interface places few constraints on the type of data

stored by the system. The only requirements are that the data must be identified by a unique numeric

key and nodes must be willing to cooperate by storing (key, value) pairs for each other. Responsibility

for mapping keys to the nodes that contain their values is distributed among the nodes so as to minimise

the changes required when a node joins or leaves the network.

A DHT must implement a single operation, lookup(key), which allows a node to lookup a given key

and obtain the identity of the node responsible for said key [21]. Consider a distributed file storage

system, a user wishing to store a particular file under a unique name would start by applying a hash

function, such as SHA-1, to the name of the file in order to obtain a numeric key. He or she would then

send the file to the node address that results from calling the function lookup(key) on the file’s key. Later

on, someone wishing to obtain this file, would first obtain its name, apply the same hash function to

obtain the corresponding key, call the lookup(key) function and contact the resulting node address to

obtain the file. This relatively simple example ignores caching and replication strategies, which are also

important in guaranteeing content availability.

14

A DHT based lookup algorithm must address a series of issues for its correct operation. Keys must

be mapped to nodes in a way that balances their distribution evenly among participating nodes. This is

done by mapping nodes and data or data identifiers (e.g. file names) to the same abstract key-space of

n-bit keys, using a standard hash function. A key k is therefore assigned to the node whose identifier

(i.e. key), is closest to k (according to a given metric such as Euclidean distance or numeric distance)

among all nodes within the network overlay. Another issue is the forwarding of lookup queries for a key

to the appropriate node. A node which receives a query for a given key k must be able to forward it to

a node whose identifier is closer to k in order to guarantee that the query will eventually arrive at the

correct node. Thus, each node maintains a small routing table with carefully chosen nodes. In some

systems the query is forwarded to a node whose identifier is larger than the current node but still smaller

than the requested key, while in other systems the query is forwarded to a node whose identifier has a

longer prefix in common with the key than the current node. Finally, routing tables must be constructed

by selecting appropriate nodes. To support routing as described previously, a node must either know

about its sucessor, which is the node with the next largest identifier in the key-space, or a list of nodes

with matching prefixes of different lengths.

This section will provide a brief overview of the design and operation of the most prominent examples

of DHT based peer-to-peer infrastructures: Chord [13], CAN [14] and Tapestry [15]. Although these

infrastructures do not constitute working applications, they provide a powerful abstraction, the hash

table interface, that can be used as a building block for peer-to-peer applications. They provide efficient

and fault tolerant routing and location services through which objects can be efficiently located. Despite

the fact that these infrastructures offer the same abstraction and services, they differ in the algorithms

used for overlay construction and maintenance [21].

2.1.3.A Chord

Chord [13] allocates keys and node identifiers from the same, one-dimensional, identifier space using

a consistent hashing function. This space is organised as a circle and wraps around such that the node

with identifier 0 follows the node with the highest identifier. A node is responsible for a given key k if

its identifier is equal to k or most closely follows k. This node is also known as the successor of k.

Consistent hashing ensures the load is balanced by assigning roughly the same amount of keys to each

node. In order to successfully route a query, a node must maintain a single piece of information, a

pointer to its successor in the identifier circle. A query can then be routed around the circle following

successor pointers until the node responsible for the key is found.

To ensure correct node lookups despite failures, Chord maintains a successor list containing the

addresses of the next r nodes in the identifier circle. This allows incremental progress to be made in

spite of a failed successor node, allowing up to r−1 simultaneous node failures before the node corrects

15

its successor list. Since successive nodes on the identifier circle are likely to be unrelated in the physical

topology, their failures can be considered independent and therefore even with small values of r the

probability of simultaneous failure is negligible.

Routing a query by relying exclusively on the successor list is an inefficient process. In the worst

case, a query will be forwarded by all N nodes in the network. To optimise the routing process, nodes

maintain, in addition to the successor list, a finger table with logN entries. Each entry i of the finger table

corresponds to the successor of the key n + 2i, where n is the current node’s identifier. For that reason,

the finger table maintains the address of a node that is at least half-way around the identifier space, the

address of another node that is at least quarter-way around and so forth. A node that wishes to forward

a query for key k, will select the entry in the finger table with the highest identifier that is less than k.

As a result of halving the distance in identifier space at each hop, Chord requires O(log(N)) time and

messages to resolve lookups.

To join a network, a new node n must ask an existing node in the network to lookup its identifier and

discover its successor. Responsibility for some of the keys assigned to the successor will be transferred,

along with the corresponding data, to the new node and in order to re-establish routing correctness,

both n and its predecessor need to update their successor lists. When n leaves, all its keys must be

transferred to its successor. The number of nodes involved in key reassignments when a node joins and

departs is minimal.

2.1.3.B CAN

Another proposed DHT abstraction is the content addressable network (CAN) [14]. CAN defines a

virtual d-dimensional Cartesian coordinate space to which it maps (key, value) pairs. This space is split

up into a set of hyper-rectangles called zones. Each individual node is responsible for a given zone,

which stores part of the hash table, and is identified by the boundaries of its zone. A key k is thus

mapped deterministically to a point P in the coordinate space and is stored at the node responsible for

the zone that contains P . Each node also maintains a routing table containing the addresses of the

nodes covering adjacent zones, which allows a query to be routed towards the node responsible for the

key.

To retrieve the value corresponding to key k, the same process is used to map k to a point P . If

P does not lie in the zone covered by the requesting node, the query is forwarded along the path that

approximates a straight line in coordinate space from the requesting node to the node storing k. The

lookup time for CAN is O(dN1d ).

In order to join the network, a node uses a bootstrap mechanism to discover a node already on the

network. It then selects a random point P in coordinate space and asks the latter to find the node n

which is responsible for the zone containing P . The new node contacts n, which in turn splits its zone in

16

half and assigns one of the halves to the new node. The new node must then initialise its routing table

using the addresses of n’s neighbours and n itself and advertise the new zone so its new neighbours

can update their routing tables. When a node leaves the network, its zone and corresponding keys

must be handed over to one of its neighbours. A node will periodically inform each neighbour of its

zone coordinates and that of neighbouring zones. This allows neighbours to detect when a node has

failed, through the absence of updates, and initiate a coordinated protocol to allow the neighbour with

the smallest zone to take-over the failed node’s zone.

2.1.3.C Tapestry

Tapestry [15] is based on the lookup mechanisms described in [22], namely the distributed data

structure known as the Plaxton mesh which allows distributed object location and routing using rout-

ing maps of small and constant size. Each node maintains a routing map, known as neighbour map,

consisting of multiple levels. A given level l contains addresses of nodes whose identifiers match the

current node’s identifier up to a certain prefix length of l digits. The number of node addresses stored

for each level is equal to the base of the identifiers used (each entry can contain further backup entries

to increase reliability). Furthermore, each of these entries corresponds to the neighbour closest in the

network whose identifier satisfies the prefix matching condition. As a message is routed, it is forwarded

to neighbour nodes that are progressively closer to the destination. For example, if node with identifier

1872 receives a request for the object with identifier 1833, both identifiers share a common two digit

prefix. The forwarding node must therefore select a node from its neighbour map, such as 1839, whose

identifier shares an additional matching prefix digit with the requested object. A node must therefore

maintain a neighbour map that, for each possible prefix of its own identifier, contains the address of a

node that matches that prefix but differs in the next digit. Tapestry guarantees that, in a stable state,

any node can be reached in logβN overlay hops, where N is the size of the identifier space and β is the

base of the identifiers.

An object with a given identifier o is mapped to a unique live node, its root node nr, which is guar-

anteed to know its location. A server ns, which stores o, will publish the object o by routing a message

toward its root nr. Along the routing path, each node stores a mapping (o, ns), rather than storing the

object itself. In the presence of replicas, each server storing a replica of o will repeat the same operation.

Tapestry then stores the mappings for o’s replicas in sorted order according to the latency measured from

the node holding the mappings. Object lookups are performed by routing a query message towards the

object’s root node nr. Each node along this path will check its mappings for the object, if a mapping

exists for that object, the message is routed directly to the server referred in the mapping, otherwise it

continues in the direction of nr which is guaranteed to know the object’s location. This mechanism is

used to allow nodes to discover object replicas that are close by.

17

The primary advantage in using these structured peer-to-peer systems resides in their object lookup

efficiency. Search performance, in number of hops, is O(logN) for most systems, where N is the network

size, as is the space occupied by the basic routing tables. Unfortunately, efficient operation in a stable

state requires rather costly operations on node join and departure, which is apparent given the afore-

mentioned algorithms. This makes such systems less suitable for highly transient node populations.

Another disadvantage is related to the fact that the above systems only support exact match lookups,

i.e., based on an item’s unique key identifier.

2.2 Mobile Ad-hoc Networks

A mobile ad-hoc network (MANET) is characterised by a set of mobile devices that communicate

among themselves using wireless links. This communication is done in a completely decentralised, self-

organised manner and without requiring any pre-existing fixed communications infrastructure. These

networks are particularly useful in scenarios where the existing infrastructure is unreliable, too expen-

sive, or simply non-existent. This includes unplanned meetings, disaster relief, military battlefield com-

munications, public safety and communications in remote or rural areas. The latter is a particularly

common occurrence in the developing world.

Nodes in a MANET have to rely on each other to forward packets beyond their radio coverage,

forming a multi-hop wireless network. Generally, each of these nodes merges the role of an end-system

and a router. A key aspect that distinguishes MANETs from traditional wired or wireless infrastructures

is that they lack a fixed topology, nodes are mobile and organise themselves arbitrarily. Node mobility

also contributes to the highly dynamic network topology, where nodes are constantly joining and leaving

the radio coverage of other nodes.

2.2.1 Routing Protocols

Routing, given the dynamic nature of MANETs, is one of the most important problems when ensuring

communication between two arbitrary nodes. It is used to determine the path between a source node

and destination node and in forwarding data between them. MANETs exhibit a few important differences

when compared to wired networks that limit the applicability of existing routing algorithms designed for

wired networks. These differences include the possibility of asymmetric links, since each endpoint may

have different antennas and transmit power. Routing information determined for one direction may not be

suitable for the reverse direction. Another issue is interference which can be caused by the broadcast

nature of wireless communications present in MANETs. Nodes may interfere with each other when

transmitting and links tend to be unreliable due to various environmental factors which in turn can lead

to a higher percentage of lost packets. Also, as mentioned previously, the topology is highly dynamic.

18

As nodes move or the environment changes, the set of currently valid routes may change, invalidating

existing routing information and forcing the routing algorithm to recalculate routes. Finally, nodes in a

MANET have limited resources due to their lack of wiring and smaller physical size. Limitations include

processing capacity, memory and battery power, thus, optimising resource usage is a priority.

Mobile ad-hoc networking has attracted much attention from the research community and dozens of

routing protocols have been proposed for MANETs. A detailed analysis of routing protocols for MANETs

can be found in [23]. Rather than describing such algorithms in detail, the rest of this section will highlight

their main design features, advantages and disadvantages. According to [1], MANET routing algorithms

can be classified into three categories: flat ad-hoc routing, hierarchical routing and geographic-position-

assisted routing.

In a flat ad-hoc routing protocol, no node or address hierarchy exists and all nodes perform equivalent

routing roles within the network, operating according to the same routing protocol. This category of

protocols is further subdivided into reactive and proactive routing protocols.

Proactive protocols set up routing tables at each node and keep them updated regardless of the

actual communication needs of the node. In other words, a proactive protocol will maintain routes to

destinations that are not currently used for packets sent by the node. Link-state based algorithms are

commonly used to implement proactive protocols. In this case, routing information is exchanged via

broadcast with neighbouring nodes periodically or as a result of network events. The primary advantage

of proactive protocols is the fast connection setup. When a node needs to transmit a packet to a given

destination node, there is a high probability that a route to that node already exists in the routing table.

Thus, packets do not experience the higher delay that would be caused by an expensive route discovery

process. Furthermore, if changes to the topology are infrequent, a node will have a fairly accurate view

of the current network topology, including the costs (e.g. latency, bandwidth) associated with the paths

it chooses. Unfortunately, their proactive nature represents an unnecessary overhead in lightly loaded

networks, as routing table updates are exchanged regardless of existing communication needs. Given

the mobile nature of nodes and their limited resources, additional traffic will further drain a node’s battery.

It may be difficult to define an adequate trade-off between the frequency of routing table updates, in order

to maintain an accurate view of the current topology, and preservation of already scarce resources.

Prominent examples of proactive protocols for MANETs include DSDV [24] and OLSR [25].

Reactive protocols provide another alternative for flat ad-hoc routing. A reactive protocol avoids

unnecessary maintenance traffic by discovering routes on demand. In other words, when a source node

wishes to send a packet to a given destination node, the underlying reactive routing protocol will initiate

a process to discover the path between both nodes. Discovered routes are cached at each node and

removed from the routing table after a certain period of inactivity. The obvious advantage of reactive

protocols is that they limit maintenance traffic to that which is required by currently used routes. This

19

offers a scalable solution if a node tends to communicate with the same set of nodes and node mobility

is limited, allowing a route to be discovered once and maintained for a long period of time. Additionally,

nodes can better leverage power saving periods in lightly loaded networks given the lack of periodic

route updates. Disadvantages include the time required to setup a route before a packet can be sent

which will impact the latency of the first packet in a connection. This can be problematic for interactive

applications. The overhead incurred by route discovery may also make reactive protocols unsuitable if

a node must communicate with an ever changing set of nodes. AODV [26] and DSR [27] are the most

representative examples of reactive protocols.

Rather than assume that node movement is independent, hierarchical ad-hoc routing protocols group

nodes into clusters of nodes that follow the same movement pattern. These protocols are based on the

idea that members of a group tend to move together and therefore a node will most likely remain within

the same cluster. This allows a node to move freely within its cluster and only inform other cluster

members, abstracting node movement within a cluster so that members of other clusters only need to

know how to communicate with one of its members. One or more nodes are chosen as clusterheads,

these gateway nodes are responsible for routing packets to and from other clusters. Clusters can then

be organised into a hierarchy. Advantages of this approach, alongside scalability, include the ability

to reduce routing table sizes, to shield nodes within a cluster from mobility in other clusters and to

use different routing protocols, with possibly different update frequencies, inside and outside clusters.

Disadvantages include the difficulty in maintaining the structure of clusters in the face of high mobility

(which has a particularly adverse effect when clusterheads change groups), the possible bottleneck

presented by gateway nodes (these nodes also suffer greater resource usage) and the use of sub-

optimal paths. HSR [28] is an example of a hierarchical routing protocol.

Finally, geographic-position-assisted ad-hoc routing protocols make use of a node’s knowledge re-

garding its geographic position to improve routing performance. This position can be obtained through

various location systems, although the most common is the global positioning system (GPS). An exam-

ple of this category of protocols is GPSR [29], where a node periodically exchanges position information

with its neighbours. Routing is then performed by greedily forwarding data packets to the neighbour that

is geographically closest to the destination node.

2.2.2 IEEE 802.11s

IEEE 802.11s is a proposed wireless mesh networking amendment to the IEEE 802.11 standard.

While currently still a draft and the focus of ongoing work by IEEE 802.11 task group s, IEEE 802.11s

specifies a set of mechanisms that enable wireless devices to form ad-hoc networks and communicate

among themselves. It has been designed with flexibility in mind and applicability to a variety of different

networking scenarios. MANETs and wireless mesh networks (WMN) share many similarities [30]: they

20

are both self-organising, decentralised, wireless multi-hop networks and must deal with dynamic network

topologies caused by node mobility and changing environment. WMNs can generally be classified into

two categories: back-haul meshes and client meshes. While back-haul meshes provide a wireless

access network based on an infrastructure of meshed fixed wireless access points, client meshes consist

exclusively of end-user devices (e.g. laptops) which are usually mobile. Client meshes are therefore

equivalent to MANETs providing a flat hierarchy of mobile nodes with identical functionality.

The flexibility of IEEE 802.11s allows it to be applied to either mesh networking scenario and does

not place any constraint on the types of devices used. These can be either fixed infrastructure devices

or mobile devices. It specifies a set of medium access control (MAC) layer enhancements to the IEEE

802.11 standard, focusing on areas such as multi-hop wireless routing, radio-aware metrics, power

management and security. The routing and forwarding mechanisms are of particular interest to this

thesis and will be described in further detail in this section.

The architecture of an IEEE 802.11s network consists of nodes termed mesh points (MPs). Mesh

points are IEEE 802.11 devices that provide basic forwarding functionality within the mesh and there-

fore participate in the routing protocol defined in IEEE 802.11s. These mesh points establish links,

called mesh links, among themselves to form the multi-hop mesh topology. Mesh points may be mobile

nodes or fixed nodes that are part of the infrastructure. Some mesh points may provide access point

functionality to traditional IEEE 802.11 clients. These are called mesh access points (MAPs). Another

type of mesh points, which provide additional functionality, are the mesh portals (MPPs). These act as

gateways, connecting the mesh network to other IEEE 802 networks.

A central component of IEEE 802.11s is its default routing protocol — hybrid wireless mesh protocol

(HWMP). Any device that claims to be compliant with IEEE 802.11s must implement HWMP, although

other protocols may be used in addition to it. HWMP is a hybrid routing protocol that contains a reactive

and a proactive component.

The reactive component is based on a modified version of AODV [26], implemented at the MAC layer

and with radio-aware link metrics. Since it is an on-demand algorithm, path discovery is only performed

for the destination nodes the source node communicates with and discovered paths are cached until they

are no longer needed. To ensure that recent path information takes precedence over older information,

sequence numbers are used. When a source mesh point wishes to discover a path to a destination mesh

point for which it does not have a route in its routing table, it broadcasts a path request (PREQ) message

across the mesh network. A mesh point that receives a PREQ will update its routing information and add

a temporary back pointer to the mesh point it received the message from. It can reply with a path reply

(PREP) message if it is the destination of the request or if it has a more recent path to the destination

mesh point than that which is identified in the PREQ. In this case, the PREP is unicast back to the

source mesh point, otherwise the PREQ is rebroadcast. PREQs that have already been seen by a mesh

21

point are identified and discarded. As the PREP travels backward towards the source mesh point, each

intermediate mesh point adds a forward pointer to the destination. Upon receiving the PREP, the source

can start using the discovered path.

The proactive component is provided as an extension that allows the creation of tree structures based

on specific mesh points, the root mesh points, that usually coincide with MPPs. These root mesh points

will periodically broadcast proactive PREQ messages to the network to maintain a tree of paths leading

to themselves. This allows efficient routing when communicating with nodes outside the mesh.

2.3 Peer-to-Peer Systems for Mobile Ad-hoc Networks

In spite of the popularity of both P2P technologies and MANETs, researchers have only recently

begun to explore the synergies that exist between them. Several common features can be identified

between pure P2P networks and MANETs [4] [31] [32]. Neither a P2P network nor a MANET have any

central peer or node responsible for facilitating the exchange of information or coordinating network ac-

tivity. All nodes or peers must collaborate among themselves, forwarding information on behalf of others

in the network, in order to guarantee basic system operation. Furthermore, all participants assume the

same roles in the network: initiating, forwarding and servicing requests. Another common aspect of both

networks is the primary problem they attempt to solve, that of locating the desired information or route

in an efficient manner, i.e., application and network layer routing respectively. Both networks also have

a highly dynamic topology due to nodes joining and leaving the network, and in the case of a MANET,

the mobility of nodes. Finally, broadcasting is used in both systems to some extent to exchange data or

routing information with neighbouring peers.

Significant differences also exist which present additional challenges when trying to integrate both

approaches. P2P systems rely on overlay networks, implemented at the application layer, while MANETs

are only concerned with the lower layers of the OSI model. Due to this clear separation of layers, current

P2P overlay networks are largely unaware of the underlying physical network topology and assume

the existence of efficient underlying IP routing mechanisms. P2P systems are usually designed for

the Internet where peers are often static, have plentiful resources (e.g. network bandwidth, processing

power, storage capacity) and network links are highly reliable. None of these assumptions hold for

nodes participating in MANETs. Furthermore, overlay broadcasting in a P2P network is done by using

multiple unicast messages but in MANETs any data transmission is inherently broadcast due to the

shared physical medium. Another issue that stems from the mismatch between physical and overlay

topology is the fact that neighbours, nodes one hop apart, in the overlay can be many hops apart in

the underlying physical network. Although not particularly problematic for Internet scale P2P systems,

where the cost of reaching any node is assumed to be similar and high bandwidth links compensate

22

for sub-optimal paths, communicating with distant nodes across multiple hops in a MANET will lead to

much lower throughput and higher network usage.

The aforementioned features of P2P systems and MANETs, including their similarities and differ-

ences, justify further investigation in adapting current P2P technologies and solutions, developed for the

wired Internet, to the unique operating environment of mobile ad-hoc networks. The rest of this section

provides an overview of existing studies, proposals and solutions that discuss the use of P2P systems

for mobile ad-hoc networks.

The use of an unstructured P2P application, a modified version of Gnutella, over three different ad-

hoc routing protocols (DSR, AODV and DSDV) is considered in [32]. The results show that none of

the ad-hoc routing protocols provide a clear performance advantage over the others, with each protocol

performing well under some scenarios for certain metrics and suffering from drawbacks in others. The

characteristics of the application, network and nodes in terms of workload, number of peers, mobility

and network density are shown to have a large influence over the performance of the underlying ad-hoc

routing protocol. In addition, the implemented P2P application makes no attempt to adapt the overlay

topology to the underlying physical topology or to integrate both routing protocols. Further research by

the same authors [33] compares a similar unstructured P2P application with an structured P2P system,

an implementation of Chord, again without any specific optimisations for the MANET scenario. The

results presented show that the unstructured P2P system is more resilient to a transient node population

than the structured system. This comes at the cost of increased network bandwidth usage, energy

consumption and delay. Structured P2P systems on the other hand use less resources but are better

suited for static topologies.

A theoretical comparison of routing approaches that combine P2P search protocols and MANET

routing protocols is presented in [4]. Five different routing approaches are suggested (broadcast over

broadcast, broadcast, DHT over broadcast, DHT over DHT and DHT) and compared in terms of their

routing complexity, scalability, implementation complexity, maintenance complexity, energy efficiency

and use of optimal paths. The authors conclude that the use of cross-layer routing approaches, where

the virtual P2P overlay is eliminated and application-layer searching is integrated with network-layer

routing, provide a significant performance advantage over non-integrated approaches. This applies to

both broadcast based and DHT based routing approaches. The former is an ideal candidate for MANETs

of small size and the latter can provide scalable and efficient search in larger MANETs, so long as the

routing and neighbour tables can be appropriately maintained. This analysis is unfortunately limited to

route discovery and does not refer how routing tables are constructed and maintained when using DHT

based approaches, nor does it refer practical issues such as node mobility and failure.

A comparison of several cross-layer approaches, which try to improve the performance of P2P sys-

tems over MANETs, is provided in [31]. Since the simple application of existing P2P systems over

23

MANETs can lead to poor performance and duplication of routing functionality at both the application

and network layer, as described above, several systems have been proposed that follow cross-layer ap-

proaches, i.e., where application layer routing protocols and network routing protocols are integrated or

interact with each other.

A brief overview of existing P2P solutions designed for use in MANETs is presented in the following

sections. Both unstructured and structured, DHT-based, approaches are considered.

2.3.1 Unstructured Approaches

7DS [34] is a peer-to-peer data sharing and dissemination system for nodes that may have inter-

mittent connectivity to the Internet. It is focused on urban environments with a high spatial locality for

information and enables a user’s information needs to be serviced by nearby nodes when not connected

to the Internet. It can operate in a pre-fetch mode, anticipating those needs, or on-demand, searching

for information among peers only when the device fails to access data directly via the Internet. Col-

laboration can be based on data sharing, forwarding messages for other nodes and caching popular

information such as web pages. 7DS does not assume that all nodes are collaborative or have the same

capabilities and allows the use of fixed infrastructure devices, the 7DS servers. These servers provide

services to non-collaborative clients which clearly contrasts with the MANET paradigm. Furthermore,

the described system relies only on single-hop multicast to obtain data items from nodes that are within

wireless coverage. This usage scenario assumes high node mobility in disseminating information rather

than relying on any routing mechanisms.

Proem [35] provides a mobile P2P platform aimed at facilitating the development and deployment of

collaborative mobile P2P applications in MANETs. It is characterised by its ability to adapt to a rapidly

changing environment, universal applicability to a wide variety of P2P application requirements (from file

sharing to instant messaging), platform independence using web standards and protocol extensibility. It

defines the concept of data space, a collection of data managed co-operatively by a set of peers, and

the concept of communities, providing a way to group entities (e.g. peers, data spaces) and control

access through the use of memberships. Proem defines four protocols, an XML (Extensible Markup

Language) based message oriented transport protocol, a presence protocol to advertise devices and

services, a data protocol for data sharing and synchronisation and a community protocol to manage

community memberships. The Proem platform only defines application layer protocols and does not

provide integration with the services offered by the underlying MANET, namely routing, leading to a

mismatch between virtual an physical topologies.

An unstructured P2P file-sharing system, specifically tailored for MANETs, is presented in [36].

ORION (Optimised Routing Independent Overlay Network) consists of a search algorithm, responsi-

ble for locating files and establishing routes, and a file transfer protocol. An application-layer overlay

24

network is formed by routes which are discovered on-demand by the search algorithm; ORION thus

combines application-layer query processing with the process of discovering network routes. These

routes are maintained only as long as necessary and closely match the underlying network topology.

Redundant paths to each file are also maintained. The ORION search algorithm provides a significant

reduction in control traffic and increases search accuracy when compared to Gnutella layered on top of

DSR. In addition to the search algorithm, the file transfer protocol offers lower overhead and increased

probability of successful file transfers when compared to TCP. Although ORION uses many structures

and mechanisms that are analogous to those used in reactive routing protocols for MANETs, it does

not integrated with, and does not require, any MANET routing protocol to operate. Search queries are

sent using link-layer broadcast messages and replies are sent along the reverse path, similar to AODV.

As a result, ORION does not exploit any synergies with MANET routing protocols that may already be

present in the protocol stack.

A performance analysis of Gnutella when faced with typical MANET conditions such as node mo-

bility, network partitions and node replacements, is described in [37]. Both AODV and OLSR are used

as underlying MANET routing protocols for this evaluation. Despite the fact that Gnutella provides the

necessary capabilities to enable P2P data sharing in an ad-hoc environment, several shortcomings are

identified. When faced with increasing node mobility, Gnutella is unable to satisfy minimum connec-

tivity requirements for the overlay network. Furthermore, network partitions generate traffic bursts due

to the reconfiguration of network topologies, illustrating the negative impact of highly dynamic network

topologies. Bootstrapping in a MANET, in order to enable a node to join the network, is also a problem

since Gnutella assumes the use of bootstrap-servers or prior knowledge of other peers. To address

these shortcomings, the authors propose a cross-layer optimisation of the protocol — XL-Gnutella. In-

teraction between Gnutella and the routing agent is achieved by a cross-layer protocol stack design in

which a vertical interface extends the standard inter-layer primitives enabling both the application and

the network layer to react asynchronously to events generated by the other layer. XL-Gnutella extends

OLSR by allowing the routing agent to spread Gnutella peer credentials along with the routing control

packets, thus enabling efficient peer discovery and the selection of peers closer in the underlying phys-

ical network. The proposed cross-layer approach is able to tolerate the typical dynamic characteristics

of a MANET, providing a solution to the aforementioned shortcomings, while offering lower bootstrap

latency and higher query success rate. Although XL-Gnutella exploits cross-layer interaction, it focuses

on optimising the discovery and selection of neighbour peers and does not consider overlay routing or

search algorithms.

Another unstructured P2P system for MANETs which exploits cross-layer communication is proposed

in [38]. The MPP (Mobile Peer-to-peer Protocol) protocol suite consists of an application layer protocol,

MPP, an inter-layer communication protocol, Mobile Peer Control Protocol (MPCP), and a network rout-

25

ing protocol, EDSR (Enhanced Dynamic Source Routing). Given that MANETs already provide routing

capabilities, allowing the discovery of nodes by their IP address, MPP avoids implementing additional

P2P routing functionality at the application layer. Alternatively, EDSR extends DSR by adding new re-

quest and response types, allowing peers to be discovered based on other criteria besides their IP

address and therefore implementing P2P functionality at the network layer. This allows the P2P network

to automatically take into consideration changes in the physical network topology, concentrate routing

functionality within a single layer and reduce overhead and redundant information. At the application

layer, MPP handles the exchange of data between peers using the HTTP protocol. Additionally, MPP

also aims to provide the user with location aware services using the distance information, i.e. number of

hops, obtained via query replies. Handling the inter-layer communication between the application layer,

i.e. MPP, and the network layer, i.e. EDSR, is MPCP. MPCP provides a communication channel where

the application can register itself with EDSR, allowing it to initiate search requests and to receive, and

reply to, incoming search requests from remote peers. Although the authors do not provide simulation

results, some analytical considerations are presented, demonstrating the improved performance and

lower overhead that result from matching overlay topology with the underlying network topology through

inter-layer communication. Despite leveraging the synergies between the P2P application and MANET

routing algorithm, MPP does not contemplate important mechanisms, such as caching and replication,

used to limit the amount of traffic generated by query flooding. In addition, although EDSR is compatible

with DSR, legacy DSR nodes will drop the new request and response types rather than forwarding them,

which limits its interoperability with existing MANETs.

2.3.2 Structured Approaches

An implementation capable of efficiently supporting a DHT abstraction in a MANET is presented in

[39]. Ekta integrates a DHT, Pastry, with a MANET routing protocol, DSR, at the network layer, exploiting

interactions between both protocols. Integration of the physical and logical namespaces is achieved

through a one-to-one mapping between a node’s IP address and its nodeId which enables the use of

a single integrated routing structure to optimize routing performance. Ekta maintains a routing table

and a leaf set similar to Pastry where routing table entries are chosen based on their physical proximity

among other possible candidates. Entries are continuously updated, based on their proximity, using

overheard routes and active discovery methods. Unlike Pastry, each entry in these routing structures

contains the necessary source route to reach a given nodeId, rather than an IP address. Multiple source

routes per entry are maintained to increase robustness and efficiency. Pastry’s prefix-based routing is

used to deliver a message with a given key k to the node whose nodeId is numerically closest to k. If

no source route exists for the next hop along the overlay route, a route discovery process is initiated by

Ekta. When compared to a traditional layered approach using Pastry and DSR, Ekta provides a higher

26

packet delivery ratio with a lower average delay while incurring comparable overhead. Even with such

advantages, Ekta does not consider physical proximity in the construction of its DHT, it only optimizes

existing routes, replacing them with closer routes that it may overhear. This can still lead to long and

inefficient routes.

Another DHT substrate designed explicitly for MANETs is proposed in [40]. Like Ekta, MADPastry

integrates the functionality of a DHT and an ad-hoc routing protocol at the network layer to provide

efficient indirect routing in a MANET. It combines the Pastry DHT with the AODV routing protocol and

explicitly avoids the use of broadcasts where possible. Since conventional DHTs are unaware of the

underlying physical network and can produce long and inefficient physical paths, MADPastry, unlike

Ekta, considers physical locality in the construction of its DHT. This is achieved by using the concept of

random landmarking to create overlay clusters of physically close nodes. Nodes within a cluster share

a common overlay id prefix. A given node is temporarily considered a landmark node if it is currently

responsible for one of the landmark keys defined in the overlay. Other nodes associate themselves

with the landmark node that is closest to them, adopting its overlay id prefix. Nodes will periodically

check for closer landmark nodes, which, if found, cause a node to rejoin the overlay network with a

newly generated overlay id. Three different routing tables are maintained in MADPastry: a standard

AODV routing table for physical routes, a stripped down Pastry routing table and a standard Pastry leaf

set. The stripped down Pastry routing table is only required to maintain a reference to an arbitrary

node within each overlay cluster. Intra-cluster routing is then performed using the leaf set. This routing

approach sacrifices Pastry routing performance to drastically reduce maintenance overhead. The only

proactive routing table maintenance performed by MADPastry is the periodically checking with the left

and right leaves of the leaf set, which is required to guarantee overlay routing convergence. Results are

shown that demonstrate the significant performance advantage of MADPastry over a similar integrated

approach using Pastry and AODV without random landmarking.

27

3A Peer-to-Peer Middleware for Mobile

Ad-hoc Networks

Contents

3.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.3 Peer Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

28

The proposed P2P middleware is presented and detailed in this chapter. To begin with, an overview

of the main requirements for the middleware is provided, along with a set of basic assumptions about the

scenarios in which it will operate. A comprehensive description of both the network architecture and the

peer, or system, architecture follows, detailing the most important design decisions and the functionality

of the different middleware components.

3.1 Requirements

As discussed in Section 2.1.1, the dynamic and decentralised nature of MANETs, limits the type

of P2P overlays which can be constructed to support P2P applications. Thus, the first and foremost

requirement the middleware and the corresponding P2P overlay should satisfy is the ability to operate in

a decentralised manner, whereby each peer is of equal importance and no central peer or peers exist.

The middleware should be applicable to scenarios, which, due to higher node mobility, churn and/or

network partitions, are unsuitable for use with structured P2P overlays, characterised by a higher over-

lay maintenance overhead in the face of dynamic network conditions [21]. Furthermore, the failure or

departure of a peer from the overlay should have a negligible impact on the remaining peers and should

not require extensive network reconfiguration or a significant increase in signalling traffic.

Considering the fundamental importance of search functionality in P2P applications, the middleware

should provide such applications with the ability to search for a variety of resources available throughout

the network. In order to support a broad set of content distribution, collaboration and communication

applications, these searchable resources should not be limited to files, but also include services, peers

and records, thus providing greater flexibility than the file-centric P2P systems discussed in the previous

chapter. Such search functionality should include an appropriate query language so as to enable P2P

application users and/or developers to easily express the properties of the desired information. Given

the popularity of web search engines among users, the query language should provide a familiar syntax,

supporting keywords, partial match queries and boolean queries.

Although ideally all nodes of a given MANET would be fully aware of the P2P middleware, this

would be impractical in a real-world scenario and would significantly hinder interoperability among nodes

running different software stacks. As a result, the middleware should be able to operate correctly in a

heterogeneous MANET in which it is run on only a subset of the node population. The presence of the

middleware should be transparent to nodes which are not participating in the P2P network. Also, it is

assumed that a standard MANET routing protocol is required in order to support regular, non-middleware

enabled, TCP/IP applications.

A set of Application Programming Interfaces (API) should be provided by the middleware so as to

simplify the process of developing P2P applications. These APIs should not only expose the previously

29

mentioned search functionality, but also provide access to local content storage and indexing, application

registration and communication with other peers. Finally, a modular and extensible architecture should

be employed, where possible, in structuring the middleware to allow components to be easily replaced

or extended in order to improve functionality and the applicability of the middleware beyond the usage

scenarios initially envisioned.

A significant concern related to dynamic multi-hop wireless networks such as MANETs, is the perfor-

mance of large file transfers between nodes. Node mobility and link quality have a significant impact on

the throughput achieved by traditional stream-oriented reliable transport protocols such as TCP. Further-

more, performance and stability of a connection in a MANET degrade as the distance between nodes,

or the number of intermediate nodes, increases, leading to additional retransmissions. Therefore, the

middleware should avoid transferring files from distant peers if other peers, closer by, also provide the

same file. It must also deal with link breakage when transferring said files, as this is common occurrence

in MANETs, and resume file transfers accordingly.

3.2 Network Architecture

The proposed solution combines the use of an unstructured P2P overlay, because of its lower overlay

maintenance overhead in the presence of frequently changing network conditions and support for a

more comprehensive set of search queries when compared to a structured P2P overlay; with a reactive

MANET routing protocol, on account of its greater efficiency in terms of control overhead and increased

suitability for networks with moderate mobility when compared to other flat ad-hoc routing protocols.

HWMP was chosen as the MANET routing protocol due to its inclusion in the IEEE 802.11s draft,

a mesh networking amendment to the current IEEE 802.11 standard [30]. IEEE 802.11s promises

to become the de facto standard for the creation of multi-hop wireless mesh networks and MANETs.

Although HWMP is a hybrid routing protocol, incorporating both reactive and proactive components,

only its reactive component is considered. HWMP’s reactive component is based on a modified version

of the well known AODV [26] routing protocol, adapted for link-layer address-based routing and radio-

aware link metrics, with comparable performance characteristics. In HWMP, link-layer routes are also

referred to as paths to distinguish them from their network-layer counterparts. No distinction is made

throughout the remainder of this document, hence both terms are used interchangeably.

A P2P overlay and the underlying multi-hop MANET share two common objectives: the discovery of

other nodes in the network according to a specific criteria and the routing of packets between nodes.

The MANET routing protocol provides the ability to discover other nodes in the network based on the

destination node’s address. This functionality is integrated in the route discovery process. Similarly, P2P

overlays provide a mechanism to discover peers whose contents match a given query, such as a set of

30

QueryQuery Hit

Overlay linkRoute RequestRoute Reply

Figure 3.1: Searching in an unstructured P2P overlay (left) and route discovery using a reactive MANET

routing protocol (right)

keywords or a key, and route messages between peers.

By considering a typical unstructured P2P overlay and a MANET with a reactive routing protocol, the

similarities are even more pronounced as can be seen in Figure 3.1. In both cases, the discovery of

a peer or node is done on-demand. Controlled broadcasting is used to propagate the search or route

request throughout the network. Each peer or node broadcasts the request to its set of network neigh-

bours, which will in turn broadcast it to their neighbours until a given termination condition is reached.

As the request is propagated throughout the corresponding network, a network path is incrementally

constructed. Upon reaching a peer whose contents match the search query or a node whose address

matches the requested destination, a reply is unicast to the requester using the discovered reverse path.

Given the parallel between both types of networks, the proposed solution avoids duplicating the dis-

covery and routing functionality at different layers in the network stack by integrating the process of

searching for information in the P2P overlay with the discovery of the network routes in the underlying

MANET. Rather than use a P2P overlay network consisting of a set of persistent connections setup be-

tween possibly distant peers in order to route application layer search requests and replies, the MANET

routing mechanisms are used instead. This is achieved by extending the HWMP path request (PREQ)

and path reply (PREP) messages in order to transport, respectively, application layer search requests

and search replies. Therefore, one of the primary services provided by the middleware, which will be

described in further detail in Section 3.3, consists in allowing P2P applications to submit new search

requests to the network routing layer and notifying said applications of received search requests.

31

This integrated approach offers several advantages over the use of two distinct networks. The

MANET is exclusively responsible for the maintenance of the network topology so that any changes

introduced by network participants, such as nodes failing or moving in and out of range of other nodes,

are immediately reflected in the P2P overlay. The P2P overlay does not suffer from a high path stretch,

defined as the ratio of physical hops traversed on the path between two peers in the overlay to the num-

ber of physical hops on the direct unicast path between those peers, because overlay messages are

forwarded along the latter. Thus, no mismatch exists between the P2P overlay network topology and

the underlying MANET topology. Furthermore, traffic generated for overlay maintenance purposes is

reduced, as peers do not need to maintain persistent connections to other peers in the network. Finally,

the packet overhead for application layer searching and the subsequent route discovery is lower than if

both tasks were performed separately.

To illustrate the operation of the proposed solution, consider the multi-hop MANET shown in Fig-

ure 3.2 .Let P = {A,B, C, D, E} be the set of peers in the network and S = {1, 2, 3, 4} the set of

searchable documents stored in the network. Multiple copies of the same document may be available

from different peers. Furthermore, consider an application at peer A which issues a query matching

documents 3 and 4. The application starts by informing the middleware of the search query. An HWMP

PREQ frame containing a search request is generated by the middleware and broadcast using HWMP

to the set of physical neighbours, i.e. the peers which are within radio transmission range: B and C.

Each neighbour peer, upon receiving the PREQ frame, informs the middleware of the received search

request before any further forwarding action is taken. Also, HWMP stores peer A’s address as the next

hop on the reverse path to the requesting peer. The middleware searches its local document repository

for any documents matching the query contained in the search request. Since neither peer has a match-

ing document, the middleware informs HWMP of its forwarding decision and the PREQ frame is again

broadcast to all neighbouring peers. Propagation of the search request throughout the network is limited

by the time-to-live (TTL) value defined for the enclosing PREQ frame and duplicate search requests are

discarded at each peer. Finally, the PREQ containing the search request reaches peer D. HWMP stores

peer B’s address as the next hop on the reverse path to the requesting peer, peer A. Since matching

documents are available at peer D, the discovered reverse path is used to unicast a HWMP PREP frame

containing a search reply back to peer A. This search reply includes information necessary to contact

peer D, such as its network address and transport layer ports, along with the actual search results. It is

important to note that the size of the search results which can be included in this search reply is limited

by the maximum transmission unit (MTU) of the underlying IEEE 802.11 network interface, thus, the re-

questing peer may need to use an alternate mechanism to obtain the complete result set. As the PREP

is unicast to peer B and subsequently to peer A, the forward path between both peers is discovered.

The behaviour of peer E is analogous as it too will generate a PREP containing a search reply which will

32

B

HWMP PREQ with search request

A

D

E

HWMP PREP with search reply

C

Documents: 3, 4Documents: 2

Documents: 4Documents: 1, 2

Figure 3.2: Searching for documents in a MANET by leveraging HWMP route discovery

then be unicast back to peer A through peer C.

Peer A, upon receiving both PREPs containing the search replies, knows of one source peer, peer

D, for document 3 and two source peers, peer D and peer E, for document 4. Additionally, as path

discovery was executed in parallel to the searching process, valid bidirectional network paths are now

available for reaching peer D and peer E from peer A and vice-versa. The application can use the

information contained in the search replies to contact the peers and obtain the requested documents.

Further communication between peers is achieved by using the discovered unicast paths along with

an appropriate transport protocol. If subsequent changes to the network topology cause said paths to

break, HWMP’s path failure notification and discovery mechanisms are used to discover alternate paths.

In the case of document 4, which is available from multiple peers, the application is faced with an

additional problem: deciding which peer it should contact to transfer the document from. Without any

further knowledge of the underlying network, the application would choose randomly between peers D

and E. As discussed in Section 3.1, this is not a desirable solution since the number of hops between

peers and the aggregate link quality over all hops on a given network path affect the achievable transfer

throughput. In general, peers which are physically closer and can be reached with fewer hops are

preferable.

A naive optimisation consists in using the time elapsed between transmitting the search request and

receiving the corresponding search reply from each peer, given that the search reply sent by the peer

closest to the requester would usually be received first. Unfortunately, this value is highly sensitive to

transient queuing delays at intermediate nodes and to the time required to match a query and generate

33

B

A

D

EC

8

3

105

Peer Path metric

15

D

E

11

Figure 3.3: Source peer selection using HWMP path metrics

a reply. The latter depends on a peer’s processing capacity and current load. An alternative optimisation

relies on the hop count, i.e. the number of hops on the path between two nodes. The peer whose path

has the lowest hop count would therefore be selected as the source for the document transfer. Although

unaffected by the problems mentioned for the previous optimisation, the hop count can still be a mis-

leading selection criteria as it does not take into account the characteristics of the links between nodes.

Moreover, in the aforementioned example, the information provided by the hop count is insufficient for

selecting a peer as the paths from peer A to peers D and E have the same hop count, 2.

HWMP uses a link metric when building and selecting network paths [41]. Upon reaching their

destination, PREQ and PREP frames contain the cumulative value of the link metric calculated at each

hop along traversed path. This value is stored along with the remaining forwarding information for each

path. IEEE 802.11s defines a default link metric to ensure compatibility, the airtime link metric, which

defines the cost, i.e. the amount of network resources consumed, to transmit a frame over a given

link [42]. The proposed solution uses the path metric for discovered paths, stored in HWMP’s forwarding

information table, as a criteria for selecting a source peer for a given document.

Consider the network paths and corresponding link metrics shown in Figure 3.3, discovered as a

result of the search process described in the aforementioned example. Furthermore, consider the use

of a bi-directional airtime link metric, that is, the cost of transmitting a frame from peer X to peer Y is

equal to the cost of transmitting the same frame from peer Y to peer X. Peer A, given the availability of

two source peers for document 4, would consult HWMP’s forwarding information table to obtain the path

metric for both destinations: 8 + 3 = 11 in the case of peer D and 10 + 5 = 15 in the case of peer E.

Due to the lower cost of the path from peer A to peer D, the application would opt to transfer document 4

from peer D. The middleware thus provides P2P applications with an optimised, network-aware, source

peer selection service when multiple sources are available for the same content.

34

In the previous example, the set of nodes that form the MANET coincides with the set of peers in the

P2P overlay. As mentioned in Section 3.1, this is rarely the case in real world scenarios. As a result, two

types of nodes can be distinguished for a MANET on top of which the middleware is deployed: the nodes

which execute the middleware, henceforth known as P2P nodes, and the nodes which do not execute

the middleware, henceforth known as legacy nodes. For both types of nodes, HWMP is employed as the

MANET routing protocol. Due to the use of HWMP routing mechanisms to propagate search requests

and replies, the legacy nodes must satisfy only one condition for correct P2P network operation: any

application layer payload contained in received PREQ or PREP frames must be included if the frame

is forwarded. This provides a transparent fallback mechanism, ensuring that legacy nodes forward the

search request or reply toward their destination despite not running the middleware. Satisfying this

condition is dependent on the selected HWMP implementation, hence, this issue will be discussed in

further detail in Chapter 4.

Finally, since a single MANET can have distinct groups of users with different requirements, the set

of P2P nodes can be further divided into subsets known as P2P groups. P2P nodes belonging to the

same group share a common group identifier. The use of P2P groups allows peers to limit the extent

of their search requests, i.e., only peers from the same group will reply to a given request, as well as

establishing private groups of peers with secure searching and communication capabilities.

3.3 Peer Architecture

The P2P middleware provides the functionality required to establish the network architecture de-

scribed in the previous section. An overview of the peer architecture is presented in Figure 3.4, where

the middleware is introduced between the application and the existing operating system services. The

middleware is further divided into a set of interconnected components, each with a well defined set of

responsibilities:

• P2P-HWMP (Section 3.3.2) - Provides an interface for use by the remaining middleware compo-

nents, namely the query routing service, in order to send, and be notified of, search requests and

search replies contained in HWMP messages.

• Storage service (Section 3.3.3) - Provides a local repository for storing and indexing application

content in addition to being responsible for resolving queries, such as those received in search

requests, to determine whether any matching content is available.

• Query routing service (Section 3.3.4) - Responsible for generating new search requests con-

taining application supplied search queries, deciding on how received search requests should be

35

Application

Middleware communication service

Peer Management

Service

Peer Registration

Service

Peer Selection Service

Storage Service

Query Routing Service

Operating System

P2P-HWMP

HWMP

Figure 3.4: Peer architecture

forwarded by the underlying routing protocol, HWMP, and generating search replies in response to

received search requests.

• Service registration service (Section 3.3.5) - Manages the registration of services provided by

P2P applications and the middleware, allowing them to be discovered by other peers in the net-

work.

• Peer selection service (Section 3.3.6) - Performs source peer selection using HWMP path metrics

and maintains a list of known peers in the network.

• Peer management service (Section 3.3.7) - Provides configuration information for other middle-

ware services and enables remote monitoring of the middleware.

• Middleware communication service (Section 3.3.8) - A P2P application interacts with the middle-

ware through an API provided by the middleware communication service. This service is respon-

sible for dispatching application requests to the appropriate middleware service and for enabling

generic middleware to middleware communication, beyond that which is provided by the query

routing service.

Other than these components, HWMP must be extended to support middleware supplied information

and interaction with the middleware.

To exemplify the interactions between the various components, consider a scenario similar to the

one presented in Section 3.2, where a P2P application wishes to search the network for documents

matching a given query string. The application starts by calling the search function provided by the

middleware API and passing an appropriate query string, as illustrated in Figure 3.5. The middleware

36

Application

Middleware communication service

P2P-HWMP

HWMP

Query routing service

QuerySearchRequest

Query

QuerySearchRequest

PathRequest

MANET

search("the quick brown fox")

Figure 3.5: Interactions between middleware components at the requesting peer

communication service dispatches this request to the corresponding service, the query routing service,

which will generate a new search request containing the query string. The newly generated search

request is sent to P2P-HWMP along with the indication of the TTL to use for the enclosing PREQ frame

and the address to which this frame should be sent, typically the broadcast address. P2P-HWMP then

uses HWMP to construct and send a PREQ frame containing the search request.

A neighbour peer, upon receiving a PREQ frame, will contact P2P-HWMP for further processing,

as illustrated in Figure 3.6. P2P-HWMP extracts the search request from the PREQ frame and notifies

the query routing service, which in turn contacts the storage service in order to determine if any locally

stored documents match the query string contained in the search request. If no documents match the

received query, the query routing service notifies P2P-HWMP of its decision to continue forwarding the

search request, along with the address to which the search request should be forwarded. Otherwise,

if any documents match the received query, the query routing service generates a new search reply

containing the peer’s contact information and, depending on their size, the actual search results. The

search reply is sent to P2P-HWMP which will in turn generate and send a PREP frame using HWMP,

containing the search reply, in response to the received PREQ. This PREP frame is unicast back to the

37

P2P-HWMP

HWMP

Query routing service

QuerySearchRequest

PathRequest

MANET

StorageService

QuerySearchRequest

Query

Results

ResultsSearchReply

ResultsSearchReply

PathReply

Figure 3.6: Interactions between middleware components at the receiving peer

requesting peer. Finally, once the PREP frame is received at the requesting peer, the enclosed search

reply travels up to the middleware communication service which ultimately returns the search results to

the P2P application.

A detailed description of each middleware component presented in Figure 3.4, as well as the required

modifications to HWMP, is provided in the following sections.

3.3.1 Extending HWMP

The inclusion of middleware supplied information alongside HWMP control messages, namely path

requests and path replies, requires the extension of HWMP and the frames used to transport its control

messages.

HWMP control messages are transported in IEEE 802.11 management frames of subtype action

[43]. These action frames, which follow the standard format defined for IEEE 802.11 MAC frames,

contain a frame body consisting of an action field, describing the management action, and, optionally,

one or more vendor-specific information elements. An information element refers to the encoding of

a variable length field, in type-length-value format, which is used to store management information.

38

HWMP control messages are stored in information elements. Furthermore, a vendor-specific information

element allows non-standard management information to be included in management frames. Such

information elements are distinguished by an added field containing the IEEE Organisationally Unique

Identifier (OUI) of the vendor or organization responsible for defining the element’s contents.

Given the aforementioned action frame format, two methods are available for including additional

information alongside HWMP control messages: the extra payload can be included in the same infor-

mation element as used for the control message or it can be included in an additional vendor-specific

information element. It is important to note that according to the IEEE 802.11 standard, the information

stored in an information element is limited to 255 bytes. In case of the former method, this severely

restricts the space available for the extra payload, which is already partially filled by the control mes-

sage. Thus, the latter method is employed, whereby one or more vendor-specific information elements

are used to transport the additional information. The number of vendor-specific information elements

which can be included in a given action frame is limited only by the maximum frame size supported by

the underlying network.

In this particular case, the middleware only requires the ability to transport search requests alongside

PREQ messages and search replies alongside PREP messages. Consider the former, if the length of

the search request does not exceed the maximum length supported by a vendor-specific information el-

ement, as mentioned above, a single information element is constructed. Otherwise, the search request

is segmented into multiple information elements. The constructed information elements are inserted into

the corresponding action frame following the PREQ information element as shown in Figure 3.7. The

inclusion of a search reply in a frame containing a PREP message is analogous.

In addition to the format of the management frames used to transport control messages, it is impor-

tant to describe how entities external to HWMP, such as the middleware, interact with HWMP to spawn

and handle such frames. In general, the creation and subsequent transmission of PREQ and PREP

MAC Header PREQAction

(PREQ)

Element ID(Vendor IE)

OUI(XX:XX:XX)

Payload#1

Vendor IE #1

Vendor IE #n FCS

Length

...

24 1 39 257 4

1 1 3 252

6-257Octets:

Figure 3.7: Format of an IEEE 802.11 management frame used to transport an HWMP PREQ control

message and a middleware supplied search request

39

PREQ Handler PREP HandlerVendor OUI

CC:CC:CC

p2p_hwmp_parse_preq()

vendor_a_handler()

p2p_hwmp_parse_prep()

—

99:99:99

Figure 3.8: An HWMP vendor registry with two registered external entities

messages is triggered exclusively by events in the operating system’s network stack, for example, the

detection of a missing route for a certain destination or the reception of a given message. In order to

allow external entities to trigger the creation and transmission of these control messages, the corre-

sponding programming interfaces, used internally by HWMP, are exported. Using such interfaces, an

external entity can define the value of the various parameters, such as the destination or TTL, of the

PREQ and PREP messages it wishes to send and finally trigger the transmission of the corresponding

frame. Furthermore, the functionality of such interfaces is extended, allowing external entities to spec-

ify a set of vendor-specific information elements to be included in the transmitted frame alongside the

control message. These information elements will carry the external entity’s payload.

Similarly, external entities must be notified by HWMP of received PREQ and PREP frames containing

relevant vendor-specific information elements. To achieve this, a vendor registry is added to HWMP

such that each external entity, identified by its vendor’s OUI, registers itself with HWMP, specifying a

handler for each type of control message it is interested in processing. The use of an OUI to identify

external entities provides a convenient and compact solution since it can also be used to identify the

corresponding vendor-specific information elements without any additional fields. As a result, only one

external entity per vendor or organization can be registered with HWMP at a given time.

Figure 3.8 illustrates the structure of a vendor registry with two registered external entities. When

a control message is received by HWMP, the enclosing frame is scanned for any vendor-specific infor-

mation elements. For each information element, if a vendor with a matching OUI exists in the vendor

registry and a handler for that type of control message has been specified for that vendor, then the cor-

responding handler is called, informing the external entity of the contents of the information element and

the enclosing frame.

Handlers can be passive or active. On the one hand, a passive handler simply processes the con-

tents of the information element and returns control to HWMP for further processing and forwarding of

the enclosing frame. Thus, a passive handler does not interfere with the normal operation of HWMP.

Active handlers, on the other hand, assume responsibility for forwarding the enclosing frame to the next

hop or hops and may, if required, modify its contents. Active handlers are useful in cases where the ex-

ternal entity is better suited to decide how and if a frame should be forwarded. As an example, consider

40

a PREQ frame containing a search request which is broadcast to all neighbouring nodes at each hop.

The middleware could decide to forward this frame based on whether or not locally matching results

are available in order to limit the propagation of the search query. On top of that, active handlers allow

the implementation of more advanced query routing mechanisms on top of HWMP in addition to those

based on simple query broadcasting. This characteristic is discussed in further detail in Section 3.3.4.

As a result of aforementioned extensions to HWMP, the middleware at a given node can trigger, as

required, the transmission of PREQ and PREP frames containing, respectively, search requests and

search replies. The middleware will also register a handler for PREQ and PREP frames with the HWMP

vendor registry in order to be notified of any frames containing search requests or replies. Finally, in the

case of PREQ frames, the use of active handlers allows the middleware to carry out its own forwarding

decisions.

It is important to note that these extensions to HWMP are sufficiently generic so as to enable other

uses aside from the P2P middleware solution described in this work. In fact, any application which

benefits from disseminating information as part of the route discovery process can take advantage of

the described functionality.

3.3.2 P2P-HWMP

P2P-HWMP is responsible for sending search requests and replies, received from the query rout-

ing service, to the appropriate peers using HWMP. It is also responsible for notifying the query routing

service of any received requests and replies, and, in the case of search requests, acting on the for-

warding decisions provided by the query routing service. P2P-HWMP uses the mechanisms described

in Section 3.3.1 to interact with HWMP. In spite of this, it abstracts the details of the underlying HWMP

implementation in the functionality it provides to the query routing service.

In order to send a new search request, the query routing service contacts P2P-HWMP with the

request along with a maximum TTL, i.e., the number of hops after which the request should be dropped

by HWMP, and the address or addresses of the next hop to which the request should be sent, such

as, the broadcast address. P2P-HWMP segments the search request by creating a set of one or more

IEEE 802.11 vendor specific information elements according to the method described in Section 3.3.1.

These information elements share P2P-HWMP’s OUI, 99:99:99, which was chosen arbitrarily among

the set of unregistered IEEE OUIs. HWMP is used to create a new PREQ frame, with the specified TTL,

containing the constructed information elements. Unlike a standard path request, which specifies the

address of the destination node to which a path should be discovered, this request uses a destination

address guaranteed to be different from the address of any other node in the network. This ensures that

no node can consider itself to be the final destination of the path request as this would stop the request

from being propagated any further. Finally, the created PREQ frame is sent using HWMP to the address

41

or addresses specified by the query routing service.

HWMP P2P-HWMP Query RoutingService

p2p_hwmp_handle(PREQ)

notify_request(request)

decision(forward, ff:ff:ff:ff:ff:ff)

send(PREQ, ff:ff:ff:ff:ff:ff)

send_reply(reply)

send(PREP, 08:11:14:ef:fa:ff)

Figure 3.9: Interactions performed by P2P-HWMP for a received PREQ frame

P2P-HWMP registers two handlers with the HWMP vendor registry, one for each frame type. These

handlers are called to notify P2P-HWMP of received PREQ and PREP frames containing its information

elements. The PREQ handler is defined as an active handler and the PREP handler as a passive han-

dler. When a PREQ frame, containing information elements matching P2P-HWMP’s OUI, is received by

HWMP, the PREQ handler is called. An example of the interactions performed by P2P-HWMP’s PREQ

handler when processing a received PREQ frame is presented in Figure 3.9. This handler temporarily

stores the PREQ frame and notifies the query routing service of the enclosed search request. No for-

warding action is taken for the PREQ frame until a forwarding decision is received from the query routing

service. The frame can either be dropped or forwarded. To forward the stored frame, P2P-HWMP incre-

ments the hop count, decrements the TTL and sends the frame using HWMP to the next hop address

or addresses specified in the forwarding decision. Due to the use of an active handler for PREQ frames,

P2P-HWMP assumes the responsibility for forwarding such frames if required.

If matching content exists at the current peer, the query routing service will also contact P2P-HWMP

42

with a search reply to be unicast back to the requesting peer. As with the search request, the search reply

is segmented into a set of vendor specific information elements. The previously stored PREQ frame,

namely the address of the requesting peer, is used to construct a new PREP frame containing these

information elements. The PREP frame is sent by HWMP along the discovered path to the requesting

peer.

Lastly, the PREP handler is called for received PREP frames whose vendor specific information ele-

ments match P2P-HWMP’s OUI. This handler merely notifies the query routing service of the enclosed

search reply. If the current peer is not the final destination of the PREP frame, it is forwarded by HWMP

to the next hop on the path to its destination. Since a passive handler is used for PREP frames, P2P-

HWMP does not interfere with the forwarding of PREP frames by HWMP.

3.3.3 Storage Service

P2P applications use the storage service to index, store and search their content. In order to support

a wide range of applications, as mentioned in Section 3.1, the concept of document was chosen as

the primary unit of storage. A document is an object consisting of a set of named fields whereby each

field has an associated value. Documents can have varying fields of different types including strings,

numbers, booleans and dates, among others. The use of documents as a primary unit of storage places

as few restrictions as possible on the type and structure of application content.

To exemplify the use of documents in representing application content, consider a P2P social net-

working application which allows users to create and manage their profile. Users can then search the

network for other users whose profile contains, for example, similar interests. A possible document rep-

resentation of a user’s profile is presented in Figure 3.10. Note that each profile document can have

a different set of named fields. A user might not want to disclose his or her age so the corresponding

field would be omitted. Similarly new fields which weren’t initially envisioned when the application was

developed can easily be added to new and existing documents.

The storage service thus presents P2P applications, as well as the remaining middleware compo-

nents, with a document-oriented database interface for storing and retrieving documents. The provided

"name" : "John Smith"

"age" : 30

"country" : "US"

"state" : "CA"

"interests" : ["Ski", "Fishing", "Cooking"]

Figure 3.10: Example of a document used by a P2P social networking application

43

"name:John AND NOT country:UK"

"Ski John CA"

"interests:Ski AND age:30"

Figure 3.11: Example of the types of search queries supported by the storage service

functionality includes adding new documents, updating and removing existing documents and searching

for documents matching a given search query. The latter relies on the use of an indexing mechanism

when documents are added or modified in order to allow efficient retrieval of matching documents. Each

application can request the creation of a database for storing its own documents and, optionally, specify

a password which is used by the storage service to authenticate future database modification requests.

The created databases are schema-free, i.e., documents of the same type may have a different struc-

ture, and have a flat address space, with each document being identified by a unique identifier.

The interaction between an application and the storage service is carried out through the middleware

API. A typical interaction starts with the creation of a new document database, followed by the addition

of a set of documents, with each newly added document being assigned a unique identifier. This unique

identifier can be used to retrieve, update or remove a specific document from the database. When a

document is added to the database, the indexes for a given subset of its fields are updated, ensuring

fast query resolution. In order to allow other instances of the application, running on remote peers, to

search for documents contained in the database, an application must explicitly mark its database as

being shared. In that case, the query routing service, upon receiving a search query for the specified

application from a remote peer, will contact the storage service in order to determine if matching doc-

uments are available at the current peer and, if so, return the matching results to the requesting peer.

Applications can, at any time, toggle whether or not to share their document databases.

The query syntax supported by the storage service is similar to that which is employed in mainstream

web based search engines. This allows P2P application developers to quickly leverage the search

functionality provided by the document database without having to learn an entirely new syntax. That

being so, a query is composed of a set search terms connected by the boolean operators AND, AND

NOT, OR and XOR. If no boolean operator is specified between two consecutive search terms, the OR

operator is assumed. Optionally, a search term or a set of search terms may be prefixed with a field

specifier, restricting it to a specified field. Recall the document in Figure 3.10 which represents a user’s

profile in a social networking application. Figure 3.11 shows three search queries which could be used

to match this document. The first query contains two field specifiers restricting the search terms ‘John’

and ‘UK’ to the fields name and country respectively. This query would match any users named John

which are not from the United Kingdom. The second query contains three search terms and matches,

since no boolean operators are specified, any documents containing the terms ‘Ski’, ‘John’ or ‘CA’ in

44

"dirname": "/tmp"

"extension": ".iso"

"filename": "debian-502-i386-DVD-1.iso"

"mimetype": "application/octet-stream"

"modification_time": 1246202385

"size": 4697108480

"tags" : "debian linux lenny iso"

Figure 3.12: Example of a file-backed document

any of its indexed fields. The use of this query syntax need not be limited to application developers

and can be exposed to users of such applications, where applicable, to allow users to directly query the

application’s database.

Although the document-oriented storage approach is applicable to a wide range of applications, file

sharing applications represent an important distinction. In such applications, files already exist in the file-

system and storing an additional copy in the document database would not only be inadequate but would

also degrade database performance. As such, a special document type, the file-backed document, is

introduced to describe a file residing in the local file system. A file-backed document consists of a set

of mandatory fields such as the file’s name, location in the file-system, type and size. An example of a

file-backed document is presented in Figure 3.12. Consequently, a file-sharing application will start by

creating a file-backed document for each file it wishes to share. The resulting file-backed documents can

have any number of fields appended, such as the tags field in Figure 3.12, before finally being added to

the application’s document database. Note that file-backed documents are handled identically to other

types of documents throughout the middleware, with the exception of the middleware communication

service which will be discussed in further detail in Section 3.3.8.

3.3.4 Query Routing Service

When an application wishes to search the network for documents matching a given query it interacts

with the query routing service through the middleware API. The query routing service is responsible

for disseminating the search query throughout the MANET, obtaining results from peers which contain

matching documents and processing these results so they can be returned to the application. Further-

more, at each intermediate peer, the query routing service is also responsible for deciding if and how a

received search request should be forwarded to neighbouring nodes, thereby allowing the middleware

to affect the routing of the corresponding HWMP PREQ frame.

An application supplied search query consists of a query string which conforms to the syntax de-

45

scribed in Section 3.3.3. In order to send this query string to another peer, a search request message

is constructed which contains, in addition to the query string itself, information required to contact the

requesting peer, namely its network address and the port used for the middleware communication ser-

vice, as well as a unique message identifier used to detect duplicate messages at receiving peers. This

search request is sent to P2P-HWMP by the query routing service for inclusion in a new HWMP PREQ

frame which will be propagated throughout the network. The length of the query string included in the

search request is limited by the MTU of the underlying IEEE 802.11 network interface used to transmit

the HWMP PREQ frame.

Upon receiving a search request via P2P-HWMP, the query routing service at the receiving peer

must first decide on whether or not it should continue forwarding the search request. For example, if the

receiving peer contains documents matching the enclosed query, it may choose to stop propagating the

query to avoid further network resource consumption. If the search request is to be forwarded to other

neighbouring nodes, be they P2P nodes or legacy nodes, as mentioned in Section 3.3, the query routing

service must also decide to which of these neighbouring nodes the request should be forwarded. This

decision also applies to the requesting peer for the initial transmission of the search request. Both these

decisions, whether to forward the received search request and to which nodes such request should

be forwarded, are collectively referred to as the forwarding decision and constitute the query routing

protocol. Various query routing protocols can be supported by the query routing service without changes

to the underlying MANET routing protocol. Once a decision has been taken, the query routing service

must communicate such decision to P2P-HWMP in order to trigger the appropriate forwarding action.

The query routing service enables a set of query routing protocols similar to the search strategies

described for unstructured P2P overlay networks in Section 2.1.2. Examples of possible query routing

protocols include query flooding, expanding ring search and random walks. Given the lack of correlation

between a peer and the documents it stores, these protocols aim to maximise the probability of locating

matching documents by covering an increasing number of peers.

In the case of query flooding, the behaviour matches that of conventional unstructured P2P systems

and reactive routing protocols, i.e. a search request is broadcast, at each node, to all neighbouring

nodes until a maximum number of hops is reached which is specified by the request’s TTL. The TTL

is used to limit the propagation of the search request and avoid flooding the entire network. Although

providing the fastest way to cover a large number of nodes and yielding the shortest path, this method

strains the available network resources and suffers from the need to select a fixed TTL which can provide

an acceptable trade-off between network coverage and resource consumption.

Expanding ring search consists of an optimisation of the query flooding strategy. A search request

is broadcast, at each node, to all neighbouring nodes but, rather than relying on a fixed TTL, the search

process is split into a set of iterations with increasing TTL values. The first iteration propagates the

46

search request using a low value for the TTL in order to search only within a few hops of the requesting

node. If no reply is received, the TTL is incremented and a new iteration is started with the retransmission

of the search request by the requesting peer. Each iteration covers an increasing number of nodes and

increases the probability of finding the requested documents. This method provides a clear advantage

for popular content, which due to its higher replication rate, can usually be found near the requesting

peer. Expanding ring search limits the query flooding necessary to find such content at the cost of

increasing the query delay and, in the case of rare content, the consumption of network resources.

Finally, a random walk consists in randomly choosing, at each node, a single node from the set of

neighbouring nodes as the next hop for the received search request, excluding the node this request was

received from. A random walk does not rely on the use of broadcast traffic to propagate a search request

throughout the network thereby preserving network resources. Since node coverage only increases

by a single node each time the query is forwarded, query delay is higher than both of the previously

mentioned approaches and longer paths are followed to reach the replying peers. Selecting a query

routing protocol depends on many factors, namely scalability, the trade-off between the query delay

and resource consumption and the popularity of the desired content. To simplify the description of the

remaining functionality of the query routing service, consider the use of query flooding as the query

routing protocol.

In addition to providing P2P-HWMP with forwarding decisions for received search requests, the query

routing service contacts the storage service to determine if the current peer contains documents match-

ing the received query and, if so, generates a search reply which is sent back to the requesting peer.

The search reply adopts a similar structure to the search request by including the information required

to contact the replying peer, necessary in order to transfer documents or further results. The reply also

includes a unique message identifier which is set to the same value as the message identifier of search

request it is replying to. This allows the requesting peer to match replies to the corresponding request.

Lastly, the search reply contains a result set consisting of a list of documents matching the query. In-

stead of including the entire document, as stored by the storage service, in the result set, the query

routing service obtains an abridged copy of the document containing only the set of fields considered

relevant for displaying or processing the results by requesting application. Once constructed, the search

reply is sent to P2P-HWMP which will in turn create the enclosing HWMP PREP frame and send it to

the requesting peer.

As an example, consider a P2P application which allows users to discover music tracks shared by

other users on the network. The developer of this application could choose to include only the artist,

album and track name fields for documents representing tracks contained in the result set, and exclude

other fields, such as the track’s lyrics, which are not relevant when displaying the list of matching tracks to

the user. If the user wishes to obtain further information regarding a selected track, the application would

47

then contact the replying peer to transfer the remainder of the document and all of its fields. By returning

only the relevant fields, significant bandwidth savings can be achieved when searching for documents

which exhibit a large number of fields or fields which are of reduced interest to the requesting application.

Developers can easily define, through the middleware API, the set of fields used to generate copies of

documents stored in an application’s document repository and returned as part of result sets for that

application. If the requesting peer receives an incomplete copy of a document as part of the result set,

it can use the document’s unique identifier, included in any copy of the document, to transfer a complete

copy from the remote peer using the middleware communication service described in Section 3.3.8.

Ideally, all possible result sets would be sent using the search reply but, as is the case with the query

string, the size of the result set is limited by the MTU of the underlying IEEE 802.11 network interface

seeing that the enclosing search reply is transported in an HWMP PREP frame. Rather than impose

a limit on size of the result set, which would severely constrain the number of matching documents

returned, an out-of-band mechanism is employed to obtain the result set in cases which prevent its

inclusion as part of the search reply. Various potential out-of-band mechanisms exist which could be

used to obtain the result set. The query routing service could split the result set into one or more UDP

packets and send these to the requesting peer in parallel with the search reply. Although the requesting

peer would not have to explicitly request the result sets from the replying peers, and thereby suffer an

additional delay, it would be flooded by result set packets if each of these peers were to reply with a

large result set. UDP’s lack of congestion control only exacerbates this problem. To avoid flooding

the requesting peer by implicitly sending the result set, an alternative approach could be employed in

which the requesting peer would explicitly request each segment of the result set from a replying peer.

A description of the number and size of these segments would be included in the search reply, thus

allowing the requesting peer to control the number of inbound segments and stop transferring the result

set once enough matching documents have been received. Despite this, the aforementioned approach

still suffers from the limitations imposed by UDP: the need to reorder segments and reassemble the

result set, lack of congestion control and the loss of segments due the unreliable nature of the transport

protocol.

Since the middleware communication service already provides a reliable request/response protocol,

it is chosen as the mechanism used for obtaining result sets from replying peers. The query routing

service starts by constructing the result set for a given query, if this result set can not be included in

the search reply due to its size, it is temporarily cached and a unique result set identifier is generated

identifying the result set. This identifier is included in the search reply, alongside the number of matching

documents, in place of the actual result set. Upon receiving the search reply, the requesting peer

uses the middleware communication service to obtain the cached result set given the unique identifier

contained in the search reply. Result sets of any size can be reliably transferred using this mechanism.

48

Once enough unique documents have been received, the query routing service at the requesting

peer refrains from obtaining further result sets. If more than one source peer is available for a given

document the list of source peers is sorted using the peer selection service described in Section 3.3.6.

Finally, the matching documents are returned to the application along with the information regarding

their source peers.

3.3.5 Service Registration Service

The P2P middleware is not limited to providing an application with the means to search for documents

shared by other instances of the application on the network. In fact, many applications require the

ability to search for other applications and the services they provide. This service discovery capability is

enabled through the use of the service registration service which allows applications to register services

they provide with the middleware. Once registered, these services can be discovered by other peers in

the network.

Applications interact with the service registration service through the middleware API. The service

registration service maintains a document database destined exclusively for storing service description

documents. An application wishing to enable the discovery of a service would start by creating a service

description document, a special document type with two mandatory fields: the url, indicating the location

of the service on the network per RFC 3986 [44], and the type, used to identify services which share a

common protocol. It is important to note that, although a document can have any number of additional

fields, as mentioned in Section 3.3.3, allowing further service information to be provided, the specifica-

tion of a service description language for describing services is beyond the scope of this document. The

mandatory fields of a service description document, url and type, provide only the essential information

necessary to identify and contact another service on the network.

Once created, the service description document is used to register the service along with a lifetime

during which such registration should remain valid. The document is added to the service registration

service’s document database. Other peers can now search for the service description document, using

the same mechanisms mentioned in Section 3.3.4 and Section 3.3.3, in order to discover the service.

The service registration service periodically checks its database for expired registrations and removes

the corresponding service description documents, preventing other peers from discovering services

whose registration has expired. Alternatively an application can use a unique identifier, received upon

registering the service, to unregister the service at any time.

Discoverable services are not limited to those registered by P2P applications. Other middleware

services, namely the middleware communication service and peer management service, which are re-

motely accessible, are automatically registered with the service registration service and can be easily

discovered by other peers on the network. As a result, a peer can discover other instances of the mid-

49

dleware running on the network and, in the case of the peer management service, determine which of

those instances can be remotely monitored. The remotely accessible functionality provided by these two

services is discussed in Section 3.3.8 and Section 3.3.7 respectively.

3.3.6 Peer Selection Service

Source peer selection, as mentioned in Section 3.2, is achieved through the use of the peer selection

service which is also responsible for maintaining updated information about other peers in the network.

When multiple source peers are available for a given document, the middleware uses information

about the path to those peers, provided by HWMP, in order to determine the best peer to use for obtaining

the document. The best peer among a set of peers is defined as the peer to which the current peer has

the lowest value of the path metric, assuming the path metric represents a cost function. This value is

stored in HWMP’s forwarding information table for each destination to which a valid path exists. The

value of the path metric is calculated during HWMP path discovery by adding up the individual values

of the link metric calculated at each hop on the path between both peers. In this case, the airtime link

metric, which represents the amount of channel resources consumed by transmitting a frame over a

given link, is chosen due to its radio-aware nature and its use as the default link metric for IEEE 802.11s.

It is important to note that any link metric representing the cost of a given link, for example, energy

consumed, could be used instead. The airtime cost [42], ca, of a link is defined by

ca =(

Oca + Op +Bt

r

)1

1− ept

where ept is the bit error rate for a test frame of size Bt, transmitted at a modulation rate of r using

an IEEE 802.11 modulation type characterised by a given channel access overhead, Oca, and protocol

overhead, Op. If two paths have the same path cost, the one with the lowest hop count is preferred as it

usually introduces the lowest queuing and forwarding delay.

By using the path cost, calculated by HWMP for each destination, the peer selection service provides

the ability to sort a list of peers according to their corresponding path costs. This list, sorted in order of

ascending path cost, can be used by applications wishing to obtain a document from a set of available

source peers. The application would typically start by trying to contact the peer with the lowest path cost.

If it is unable to contact this peer, the next peer in the list would be selected and so forth. Whenever the

connection to the currently selected peer is interrupted, the application can either retry this peer or opt

for the next peer. This decision depends on the trade-off between time spent waiting for the currently

selected peer to respond versus using another peer with a higher path cost and, therefore, a greater

network resource usage and in most cases, lower connection throughput. As a result of this sorted peer

list, applications can optimise the transfer of documents for which multiple sources exist by opting for

peers which minimise the amount of channel resources used. This is particularly important in a MANET

50

where such resources are scarce.

Finally, the peer selection service also caches information regarding other peers in the network. For

each known peer, this information includes the peer’s unique identifier, network address and transport

protocol port necessary to contact that peer. The information for a given peer is updated whenever

a search request or reply is received from that peer by the query routing service. Furthermore, the

peer selection service indicates which of the known peers are currently nearby, i.e. which are physical

neighbours to the current peer. This can be used by the query routing service to improve the selection

of the peer or peers to which a search request should be forwarded.

3.3.7 Peer Management Service

The configuration of the middleware’s various services is handled by the peer management service.

This service maintains a global list of configuration parameters, for example, the file system path to the

storage service’s document databases or the network interface used for sending and receiving search

requests and search replies. These configuration parameters can be specified in a configuration file,

which is loaded on middleware start-up, and can, where applicable, be changed during runtime. As

such, other services consult the peer management service in order to obtain the parameters necessary

for their operation and the definition of their behaviour.

Beyond managing the middleware configuration, the peer management service also exposes statisti-

cal information, gathered from the various middleware services, through an embedded SNMP agent.

This allows other peers, and eventually static nodes configured as Network Management Systems

(NMS), to query the current peer’s middleware, via the SNMP protocol, in order to determine, for ex-

ample, its performance in responding to search queries or the number of locally stored documents. A

complete list of the variables exported by the agent, and which can be queried by remote peers, is

provided in Appendix B along with the structure of the corresponding SNMP Management Information

Base (MIB). The middleware services which provide information via the SNMP agent include: the query

routing service, the storage service, the service registration service and the peer selection service.

3.3.8 Middleware Communication Service

The middleware communication service is responsible for providing an interface between applications

and the middleware, in addition to enabling the communication between instances of the middleware

running on different peers. The former is accomplished by presenting applications with an object oriented

API. This API, which can be consulted in its entirety in Appendix C, exposes the functionality provided by

the middleware storage service, query routing service and service registration service to the application.

It allows an application to create, remove and share its document database as well as add, remove,

51

update and search for documents contained in said database. Documents can be searched either locally

or using the network by supplying a query string conforming to the syntax described in Section 3.3.3. If

a search is to be performed locally, the storage service is consulted, otherwise the query routing service

is used. Finally, applications can register services and remove existing registrations through the API.

The middleware also requires the ability to contact other middleware instances in order to obtain

documents matching a given query and, in the case of large result sets, the query results themselves.

In view of this requirement, the middleware communication service provides a RESTful web service

using HTTP, the peer communication interface, which can be accessed by remote peers. Three types of

resources can be obtained through this interface: documents, files and result sets. Any document stored

by the storage service can be retrieved given its unique identifier. File-backed documents, in addition to

the document itself, also have a corresponding file in the peer’s file-system, which can be retrieved using

the unique identifier of any file-backed document which refers to that file. Large result sets, i.e. those

which could not be transported in a search reply due to their size, can be retrieved using the unique

result identifier contained in such search replies.

The web service can be easily extended to support additional resources, and operations on such

resources, in order to address further middleware communication needs. The use of HTTP as the basis

for this service satisfies an important requirement for transferring both documents and files, i.e., transfers

can be resumed if interrupted. Consider a file with multiple source peers, if the source peer from which

the file is being transferred were to fail, another source peer could be chosen to retrieve the remainder of

the file. This is particularly relevant in the context of a MANET, where changes in the underlying network

paths are common and can result in the breakage of established HTTP connections. Partial resource

transfer is achieved through the use of the content range header as defined in HTTP/1.1 [45].

52

4Implementation

Contents

4.1 Extending HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.2 P2P-HWMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

4.3 Middleware Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.4 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

53

An implementation of the middleware described in the previous chapter was developed for the

GNU/Linux operating system. Current versions of the Linux kernel, 2.6.26 and later, include open80211s

[46], the first and foremost open source implementation of the IEEE 802.11s draft, namely of its default

routing protocol, HWMP, and link metric, the airtime link metric. open80211s is part of the new Linux

IEEE 802.11 stack, mac80211, and allows nodes with compatible hardware to establish infrastructure-

less wireless networks such as WMNs and MANETs. A modified version of this HWMP implementation,

included with version 2.6.29 of the Linux kernel, serves as a basis for the developed middleware.

As a result of the in-kernel nature of open80211s, the middleware is comprised of both kernelspace

and userspace components. P2P-HWMP, which is tightly coupled to HWMP, is implemented as a load-

able kernel module (LKM) while the remaining middleware components are implemented in userspace.

Kernel-space components are therefore limited to those which must interact directly with HWMP, i.e.,

P2P-HWMP, so as to limit the amount of code running with elevated privileges as well as to reduce

implementation complexity and avoid compromising the stability of the Linux kernel. The inclusion of fur-

ther middleware functionality in the kernel would not only suffer from restrictions imposed by the kernel

API but also from a lack of flexibility and portability.

The majority of the userspace middleware components are implemented by a daemon which runs as

a background process, independent of the P2P applications. This approach was chosen over the use

of a library, called by each application during its execution, in order to enable persistent functionality,

TCP/IP

Application

MIddleware (Daemon)

mac80211

P2P-HWMP (LKM)

HWMP (open80211s)

Linux Wireless Drivers

cfg80211

Kernel/Userspace boundary

API

Figure 4.1: Peer software stack

54

such as the discoverability of services or remote monitoring, despite the lack of currently running appli-

cations. Furthermore, the use of a single middleware instance reduces the memory footprint required

by applications and improves their start-up time. Given that the storage service stores and manages all

shared application content, an application need not be running in order for its content to be searchable

and retrievable.

Lastly, P2P applications use the middleware API in order to interact with the middleware. This API

corresponds to a thin library which employs an appropriate inter-process communication (IPC) mech-

anism to contact the middleware daemon. The API hides the fact that the remaining userspace mid-

dleware components are running in a separate process as well as the protocol used to interact with

the middleware daemon. Figure 4.1 illustrates the software stack at each peer. The implementation of

the middleware based solution consists of four distinct parts: the modification of the selected HWMP

implementation, the P2P-HWMP kernel module, the middleware daemon and the API library. Each part

is described in further detail in the following sections.

4.1 Extending HWMP

Extending the selected HWMP implementation encompassed the enhancement of the path selection

frame transmission function, the implementation of the HWMP vendor registry and the modification of

the code path used to process received path selection frames. Path selection frames refer to the frames

used by HWMP during the path discovery process, namely PREQ and PREP frames. HWMP is included

in the Linux mac80211 wireless stack which can be compiled into the kernel or as a separate loadable

kernel module. The latter option was used to facilitate development and testing of the aforementioned

functionality.

First off, the function used by HWMP to send new path selection frames, mesh path sel frame tx,

was extended in order to support an additional argument: a buffer containing a set of IEEE 802.11

vendor-specific information elements. This allows the inclusion of additional payload alongside path

selection messages. This function is used exclusively by the HWMP implementation and is therefore

only available within the name-space of mac80211, i.e, other kernel modules, such as P2P-HWMP,

are unable to access this functionality in order to transmit new path selection frames. As a result, the

mesh path sel frame tx function was exported using EXPORT SYMBOL, allowing it to be used anywhere

within the kernel. The caller can specify the type of path selection message, i.e. PREQ, PREP or path

error (PERR), as well as the value of each field in such message, including the TTL, destination address,

path lifetime, sequence number, among others.

The mesh vendor registry, as described in Section 3.3.1, was implemented so as to allow received

path selection frames to be handled outside of mac80211. The mesh vendor registry consists of a table

55

which stores for each registered entity, such as another kernel module, a set of function pointers, one

for each type of path selection frame the entity is interested in processing. Each entity is identified by a

vendor OUI which must coincide with the OUI used in that entity’s vendor-specific information elements.

The OUIs of the vendor-specific information elements contained in received path selection frames are

used to determine the entity and corresponding handler. Two new functions are exported by mac80211,

mesh vendor register and mesh vendor unregister, which can be used anywhere within the kernel

to, respectively, register and unregister an external entity and its handler functions.

The code path followed by received path selection frames, within mac80211, was changed in order

to call the HWMP vendor registry before any forwarding action is taken, or any reply generated, by

HWMP. Each received path selection frame is scanned for vendor-specific information elements whose

OUIs match the OUI of an entity registered with the vendor registry. If such a match occurs, a reference

to the frame, as well as the contents of the matching information element, are passed when calling

the corresponding handler function registered for that frame type. Recall that handlers can be either

active or passive, this being determined by their return value. If at least one active handler was called

for given frame, no further action is taken by HWMP once all vendor-specific information elements for

that frame have been processed. This assumes that the active handler is responsible for forwarding or

replying to the received path selection frame. If on the other hand, no active handler was called for a

given frame, control is passed back to HWMP which will continue processing the frame and follow the

standard forwarding behaviour.

4.2 P2P-HWMP

Since the selected HWMP implementation resides in the Linux kernel, P2P-HWMP, which is tightly

coupled with HWMP, is implemented as a loadable kernel module. P2P-HWMP interacts with HWMP by

calling the path selection frame transmission function and the registration and de-registration functions

of the vendor registry. P2P-HWMP also registers a set of handlers with HWMP in order to be notified of

received frames containing its vendor-specific information elements.

Communication with userspace middleware components, namely the middleware daemon, is achieved

through the use of netlink. Netlink is a Linux IPC mechanism used to exchange messages between the

kernel and userspace processes. For the former, an internal kernel API is provided while for the later,

the standard operating system socket interface is used with the AF NETLINK protocol family. Netlink sock-

ets offer a familiar programming interface for application developers and are similar to UNIX datagram

sockets in that both are datagram oriented, limited to the boundaries of the current host and address

processes rather than other hosts. Netlink is used extensively by recent versions of the Linux networking

stack, including mac80211, for exchanging networking information between the kernel and userspace

56

networking utilities. Alternative userspace to kernelspace interfaces, such as system calls or ioctl, do

not provide the flexibility offered by netlink, namely the ability to asynchronously notify the middleware

daemon of received search requests and replies.

On start-up, P2P-HWMP creates a new netlink socket, with a specific protocol, which listens for any

messages received from userspace. The messages supported by the implemented protocol, used for

the interaction between P2P-HWMP and the middleware daemon, are detailed in Appendix D. When

a middleware registration message is received from userspace, P2P-HWMP records the process iden-

tifier (PID) of the sending process and registers both an active PREQ handler and a passive PREP

handler with the HWMP vendor registry. An unassigned OUI, 99:99:99, is used to identify P2P-HWMP,

which can now be notified by HWMP of any frames containing vendor-specific information elements

which match this OUI. The registration with the HWMP vendor registry is maintained only while the mid-

dleware daemon is also registered with P2P-HWMP. If the connection to the daemon is lost or if the

daemon decides to unregister itself with P2P-HWMP, both handlers will be unregistered. This behaviour

guarantees that, if the middleware daemon is not running, P2P-HWMP will not be called and will not

interfere with the processing and forwarding of PREQ and PREP frames sent by other peers.

When a new search request is received from the middleware daemon, P2P-HWMP segments this

request into a set of vendor-specific information elements identified by P2P-HWMP’s OUI. The size

of the search request is limited to 1000 bytes in order to guarantee that, with the addition of the re-

maining management information and MAC header, the enclosing frame does not exceed the network

interface’s MTU, circa 1500 bytes for a default Linux network interface. The path selection frame trans-

mission function, exported by mac80211, is used to create and send a new PREQ frame, containing the

constructed information elements, to the neighbour nodes specified by the middleware daemon. This

information is included alongside the received search request and is comprised of a set of one or more

node addresses. If the broadcast address, ff:ff:ff:ff:ff:ff, is specified, the frame will be sent to

all neighbour nodes. Another peer, upon receiving this frame, will scan the enclosed vendor-specific

information elements and call P2P-HWMP’s PREQ handler for each information element matching P2P-

HWMP’s OUI. The PREQ handler is passed a reference to the PREQ frame and the contents of the

corresponding information element. These information elements are used by P2P-HWMP to reassem-

ble the original search request which is in turn sent to the middleware daemon, via netlink, using a search

request notification message. The PREQ frame containing the search request is stored by P2P-HWMP

until either a search reply or a forwarding decision is received from the middleware daemon.

After processing the search request, the middleware daemon will communicate its forwarding deci-

sion to P2P-HWMP using a forwarding decision message. This message instructs P2P-HWMP on the

forwarding action it should take for the stored PREQ frame. The frame can either be forwarded, as de-

scribed previously for the requesting peer, or dropped altogether. In addition to a forwarding decision for

57

a given search request, the middleware daemon can also provide a search reply for transmission back to

the requesting peer. In this case, the stored PREQ frame is used to determine some of the fields, such

as the destination address, for the PREP frame which will be created to transport the search reply. The

search reply is segmented in a manner analogous to that described for the search request, inserted into

a new PREP frame and sent to the next hop on the path back to the requesting peer. The stored PREQ

frame can now be discarded. Once received at the requesting peer, the reassembled search reply is

sent to the middleware daemon using a search reply notification message.

Frequently, when the requesting peer receives a reply for a search request, the P2P application will

attempt to contact the replying peer in order to exchange further information. This is achieved through

the use of a transport protocol, such as TCP or UDP, which relies on the network-layer to address the

replying peer. Since HWMP addresses peers at the link-layer, the network address of the replying peer

is not directly available to the requesting peer. Even with knowledge of the network address, the normal

behaviour of the requesting peer would be to determine the mapping from network address to link-layer

address through the use of the Address Resolution Protocol (ARP), for IPv4 network addresses, or

Neighbour Discovery Protocol (NDP), for IPv6 network addresses. In both cases, this entails flooding

the network with an address resolution request, in the case of ARP, and a neighbour solicitation, in

the case of NDP, to which the target node, which matches the network address, would reply. This

additional broadcast traffic further degrades the performance of the MANET. P2P-HWMP provides an

optimisation to avoid executing ARP or NDP if the requesting peer decides to contact the replying peer

immediately after receiving a search reply. Both search requests and search replies contain the network

address of the originating peer in their respective headers, these headers are scanned by P2P-HWMP

upon reception and used to create a new entry, mapping the originating peer’s network address to its

link-layer address, in the kernel ARP or NDP table. Thus, if the application were to establish a TCP

connection to the replying peer, using its IPv4 network address, the kernel would already have an entry

in the ARP cache mapping the peer’s IPv4 address to its link-layer address without the need to generate

further network traffic to discover such mapping.

4.3 Middleware Daemon

The core functionality of the middleware is implemented by the middleware daemon, namely the

communication, storage, query routing, service registration, peer selection and peer management ser-

vices. As mentioned previously, this daemon consists of a background process, typically executed on

system start-up, which is contacted by P2P applications through the middleware API.

The middleware daemon was developed using Python [47], a general purpose dynamic object-

oriented programming language which emphasises code readability and maintainability through clear

58

syntax. Python was chosen due to its ease in rapidly prototyping large applications as well as its com-

prehensive standard library, which implements most common internet protocols and data formats. Given

the middleware deamon’s multi-protocol nature and its need to interact with multiple external processes

simultaneously, both locally and at remote peers, an event-driven network programming framework,

Twisted [48], was chosen as a basis for the implementation. Twisted is particularly suited for complex

network intensive applications and includes support for developing clients and servers using a variety of

protocols, namely TCP, UDP, HTTP, Telnet and XML-RPC (XML Remote Procedure Call), among oth-

ers. Since multiple concurrent network and IPC communication links can be integrated with the Twisted

event-loop, the use of threading and associated synchronisation primitives can be avoided, simplifying

the access to shared data structures.

The middleware daemon was implemented as a Twisted application. A Twisted application is charac-

terised by the use of the Twisted Application Infrastructure to handle application startup and shutdown,

logging, forking to background, selection of the event-loop, or reactor in Twisted terminology, among

other common tasks. This reduces the amount of code required to setup such functionality in the actual

application. The central concept of a Twisted application is the Application object, representing the ap-

plication. This object acts as a container for Twisted Service objects, i.e., entities within the application

which can be started and stopped. Twisted already provides several services, such as, web servers,

Telnet servers, TCP servers and clients, which can be easily extended and on top of which custom pro-

tocols can be constructed. A Twisted application can contain multiple services organised hierarchically.

Each of the previously mentioned middleware components, such as the storage service, which compose

the middleware daemon, are implemented as Twisted services.

Wherever possible, an existing Twisted service was used as a basis for a given middleware service.

The dependencies between the various services are presented in Figure 4.2, where a directed edge rep-

resents the invocation of methods provided by the target service. As expected, all services rely on the

management service in order to obtain configuration information, necessary to configure their respec-

tive behaviours. Moreover, the middleware communication service acts as a proxy between local P2P

applications, or remote peers, and the several services provided by the middleware, such as, storage or

query routing. Beyond implementing the functionality described in Chapter 3, some of the middleware

services have additional implementation details which are worth mentioning.

The storage service relies on the Xapian [49] open source search engine library in order to enable the

indexing and searching of P2P application documents. Xapian provides full text searching capability for

the indexed content, along with, structured boolean queries, unicode support, wildcard searching, large

database support (greater than 2GB) and simultaneous searching and indexing. Despite being written in

C++, bindings for a variety of programming languages, including Python, are available for accessing the

Xapian core. Rather than use the interface provided by the Python Xapian bindings, which deals directly

59

Storage

Management

Service Registration

Query Routing

Peer Selection

Middleware Communication

Figure 4.2: Dependencies between middleware services provided by the middleware daemon

with low-level information retrieval data structures, a wrapper module, Xappy [50], was used instead.

The Xappy module provides a higher-level interface to Xapian, allowing the definition of a field structure,

including the type of information stored at each field, for a given document. This field structure is used to

specify how the document is indexed, based on its fields, and in constructing and performing searches.

A new Xappy database is created in the peer’s filesystem for each individual P2P application. Once a

document has been indexed by Xappy, it is stored in this database alongside the document index. The

Xappy database provides the underlying storage for an application’s document repository. In addition to

the documents themselves, this database is also used for storing metadata, such as, the password for

the document repository, the list of fields which should be indexed for newly added documents and the

list of fields which should be included in search reply messages.

A new datagram-oriented Twisted transport, using netlink sockets, had to be developed in order to

support the interaction between the query routing service and P2P-HWMP. For the purpose of demon-

strating the viability of the designed middleware and comparing it to similar solutions, a query flooding

protocol, with a configurable TTL, was chosen as the query routing protocol for the query routing service.

A search request received at a given peer is therefore broadcast to all neighbour nodes. This is done

regardless of whether or not the peer replies to the enclosed query. Despite the implementation of a

single query routing protocol, it is important to note that the required infrastructure is in place to support

further protocols, such as those described in Section 3.3.4. The forwarding decision message sent to

P2P-HWMP, as well as P2P-HWMP itself, already allow the specification of the addresses the search

request, and corresponding PREQ frame, should be forwarded to. The format of the search messages,

60

http://NODE_ADDRESS:PORT/document/APPLICATION_ID/DOCUMENT_ID

http://NODE_ADDRESS:PORT/document/APPLICATION_ID/DOCUMENT_ID?type=file

http://NODE_ADDRESS:PORT/result/RESULT_ID

Figure 4.3: Resource URLs provided by the peer communication interface

.

namely search requests and search replies, exchanged between the query routing services at different

peers, is detailed in Appendix A.

In order to perform source peer selection, the peer selection service requires a copy of the mesh

path table containing the metrics for the paths discovered by HWMP. This mesh path table is stored,

by mac80211, as an in-kernel memory structure and is therefore not easily available to userspace pro-

cesses. The command-line networking utility, iw, can be used to query the kernel and extract such

information, but, this requires spawning an external process, at regular intervals, to allow the peer se-

lection service to keep an updated copy of the mesh path table. Since mac80211, and the networking

information it stores, can also be queried through netlink, via the nl80211 interface, the peer selection

service periodically polls the kernel in order to obtain an updated copy of the mesh path table. This is

achieved by implementing the protocol necessary to interact with nl80211, and extract the mesh path

table, on top of the previously mentioned Twisted netlink transport.

The web server implementation included with Twisted, and provided as a Twisted service, was used

as a basis for the middleware communication service. This implementation includes support for the

HTTP/1.1 protocol, thus allowing partial retrieval of resources by taking advantage of the Content-Range

header. Two interfaces are provided, on top of HTTP, by the middleware communication service, one

for use by remote middleware instances and another for use by the API library. The first interface,

the peer communication interface, which consists in a RESTful web service, is used for transferring

documents, files associated with file-backed documents and result sets. These resources are retrieved

using the HTTP GET method on the appropriate URL, as shown in Figure 4.3. The first URL, from top

to bottom, is used for retrieving a document based on its unique identifier and the unique identifier of

its application. The latter determines which document repository to retrieve the document from. If the

document is a file-backed document, the associated file, residing in the local filesystem, can be retrieved

by specifying the type=file parameter as shown in the second URL. Finally, the third URL is used for

obtaining a previously cached result set based on its unique identifier. While the representation of a file

consists of its actual content, documents and result sets require an appropriate representation in order

to be transferred. Javascript Object Notation (JSON) was used to encode and represent both types of

resources, enabling their reconstruction at the requesting peer. This JSON representation of the result

set is also used when transferring result sets via the query routing service’s search reply messages.

61

The second interface provided by the middleware communication service, the application interface,

consists in an XML-RPC web service. This web service exposes the functionality provided by the stor-

age, service registration and query routing services, to the API library. While the peer communication

interface is available to remote peers, the application interface is restricted to processes running on the

local peer and intended for internal use by the middleware at that peer. This interface provides an IPC

mechanism between the API library and the middleware daemon in a manner which is transparent to

the P2P applications. These interact exclusively with the API library as discussed in Section 4.4.

Lastly, the SNMP agent functionality, provided by the peer management service relies on Twisted-

SNMP [51], an SNMP protocol implementation for Twisted based on PySNMP, which is not included in

the base Twisted distribution. The SNMP agent provides a read-only MIB, whose content is detailed in

Appendix B, and can be queried by any SNMPv2 compatible client. The OIDs described in the MIB are

mapped to methods at the corresponding middleware service, responsible for calculating and returning

the requested value.

4.4 API

The API library is the only middleware component a P2P application interacts with directly. This

library, which provides an object-oriented API, is also written in Python, allowing application developers

to quickly develop applications with minimal effort. Since most middleware functionality is implemented

by the middleware daemon, the API library is merely a collection of classes for creating new documents

and for wrapping the methods provided by the previously mentioned XML-RPC interface. As a result,

the idiosyncrasies of the communication between the application process and the middleware daemon

process are hidden from the application, resulting in a more flexible interface.

The API itself consists of two Python modules, middleware.types and middleware.api, which must

be imported by applications wishing to take advantage of the middleware. The middleware.types mod-

ule provides a set of classes: Document, File and ServiceDescription, which are used to create

documents, file-backed documents and service description documents respectively. Theses classes

present an interface similar to a Python dictionary for getting and setting document fields. In addition to

this module, the middleware.api module provides a single class, Middleware, which represents a con-

nection to the middleware. This class is used by an application for all interactions with the middleware.

It allows the manipulation of an application’s document repository, namely creating a new repository;

adding, removing, updating and obtaining local documents; removing a repository; setting a repository

password and enabling and disabling sharing for that repository. It also allows the registration and de-

registration of services provided by the application and the ability to specify a query for use in searching

the network for matching documents. Appendix C illustrates the structure of both modules, in addition to

62

providing the source code for an example application using the middleware API. This example applica-

tion searches the network for peers running the middleware agent service and prints the URLs contained

in the service description documents for the discovered services.

Since some of the methods of the Middleware class may block while waiting for results from the

middleware, due to the synchronous nature of XML-RPC invocation, e.g. the method used to search

for remote documents, an asynchronous version of this class is provided in addition to the synchronous

implementation. This asynchronous version is built using Twisted, whereby methods that would block

return immediately. These methods return a special object, a Deferred in Twisted terminology, to which

callbacks can be attached. The callbacks are executed when the actual results of the invocation are

returned. Thus, the API allows for both synchronous and asynchronous call semantics.

63

5Testing and Evaluation

Contents

5.1 Testbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

5.2 Test Metrics and Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

64

In order to validate the implemented middleware solution and evaluate its performance, a set of func-

tional and performance tests were conducted using a MANET testbed consisting of six nodes organised

according to a specific network topology. The primary focus of these tests was the search functionality

offered by the middleware, and the resulting query routing, as this, along with source peer selection,

constitutes the core functionality of the middleware. In the context of the performance tests, the middle-

ware was also compared to a traditional layered solution, using Gnutella, which was not designed for

use in a MANET environment.

5.1 Testbed

The testbed consisted of six nodes equipped with IEEE 802.11 network interfaces forming a multi-

hop MANET using IEEE 802.11s and its default routing protocol, HWMP. The hardware and software

configuration, which is identical for all nodes, is presented in Table 5.1. The hardware used is represen-

tative of the sort of scenarios where the middleware may be employed and therefore provides a good

indication of real world performance. Unless explicitly noted, all test scenarios assume, in addition to the

software listed, that the middleware is installed and running on each node, i.e., that each node is also a

peer.

All nodes in the testbed were connected to a wired LAN, in addition to the MANET, in order to

facilitate the configuration and monitoring of the testbed. This LAN was used to remotely manage all

nodes without interfering with the traffic exchanged via the MANET and for providing access to external

services such as time synchronisation.

System ASUS Eee PC 900A

Processor Intel Atom N270 1.6 GHz

RAM 1 GB DDR2-400

Wireless NIC Atheros AR2425 802.11abg (ath5k)

Operating System Debian GNU/Linux 5.0.1 (”Lenny”)

Software Linux Kernel 2.6.29 (with HWMP extensions)

Python 2.5.4

Twisted 8.1.0

TwistedSNMP 0.3.13

Xapian 1.0.7

Xappy 0.5

gtk-gnutella 0.96.7u-16948

Table 5.1: Testbed node configuration

65

3

1 4

5

2

6

Figure 5.1: Testbed network topology

The topology of the MANET is presented in Figure 5.1, where the edges of the graph represent

bidirectional links. This topology was chosen among the set of possible topologies since it satisfies the

following criteria: it represents a multi-hop network; multiple paths, of differing hop counts, exist between

two nodes and, despite the average node degree, three in the case of the selected topology, the failure

of a single node splits the network into two separate partitions. Thus, the selected topology allows for

the evaluation of both the query routing functionality of the middleware, as well as, the ability to select

an optimal source peer when multiple source peers exist for a given document.

As all nodes were placed near each other, with line-of-sight between each pair of nodes, node visi-

bility, i.e. the ability for a node to detect the presence of another node and establish a link to that node,

was enforced through the appropriate configuration of mac80211. Any frame received from a node to

which the receiving node does not have an edge, according to the graph representing the topology in

Figure 5.1, is dropped at the link-layer, before being passed to HWMP. Therefore, all discovered network

paths conformed to the defined network topology.

5.2 Test Metrics and Scenarios

The assessment of the implemented middleware was achieved through both functional and perfor-

mance testing. On the one hand, functional testing was used to verify the correct operation of the

middleware, namely its ability to coexist with legacy nodes, select source peers and search for, and en-

able the discovery of, documents and services on the network. Performance testing, on the other hand,

was used to determine the delay introduced by various components of the middleware software stack, to

compare the efficiency of the middleware with Gnutella and to evaluate the middleware’s query routing

performance. The terms query and search request are henceforth used interchangeably to refer to the

message containing the application supplied query string.

66

The metrics used consist of the query forwarding delay, query success rate, overall traffic and query

processing time. The query forwarding delay represents the time elapsed between the transmission

of a received search request, by P2P-HWMP, to the query routing service and the reception of the

forwarding decision for that search request. This metric is used to evaluate the overhead introduced by

taking forwarding decisions regarding a search request in userspace. The query success rate is defined

as the fraction of queries for which at least one reply was received over the number of queries sent. The

query success rate only includes queries for which at least one matching document exists at another

peer in the network. The overall traffic, both in terms of total size and number of messages, refers

to the network traffic which is generated as a direct consequence of a search request and includes

the overhead introduced by the IEEE 802.11 MAC, HWMP path discovery, ARP address resolution,

transport and application-layer protocols. Lastly, the query processing time refers to the wall clock time

spent processing a given search request at an intermediate peer, i.e, the time elapsed between receiving

a PREQ frame and forwarding it to the next hop. This processing time is divided according to the

middleware components involved, namely P2P-HWMP, netlink communication, the query routing service

and the storage service. Although similar to the query forwarding delay metric, the query processing time

is more encompassing and provides insight into which middleware components have the greatest impact

on the time required for a search request to traverse a peer.

Four scenarios were created for the purpose of testing the middleware. In the first scenario, a sim-

ple file-sharing application is run at each peer. This application uses the middleware API to create a

document repository and creates a file-backed document for each file in a specified directory. These

file-backed documents are then added to the repository, which is marked as shared, enabling other

peers to discover the files contained in the directory. Each peer shares a directory containing a few

dozen files. To test the correct operation of the middleware search functionality, a peer sends a search

request for a specific file shared by another peer in the network such that at least two intermediate

peers exist between the requesting peer and the replying peer. This constraint is used to exercise query

forwarding over multiple hops. The expected result consists in the reception of a search reply for that

request. Beyond validating the search functionality, this scenario is used to obtain experimental perfor-

mance results using the query forwarding delay, overall traffic, query success rate and query processing

time metrics.

Unlike the first scenario, the following scenarios are dedicated exclusively to functional testing. The

second scenario is a variation of the first, where the intermediate nodes between the requesting peer

and the replying peer do not run the middleware software stack, i.e., the intermediate nodes are legacy

nodes as opposed to the P2P nodes in the first scenario. This scenario is used to validate the correct

operation of the middleware search functionality in the presence of legacy nodes. The expected result

is analogous to that defined for the first scenario.

67

The third scenario is used to test the source peer selection functionality provided by the middleware.

This scenario is also based on the first but, rather than have a single source peer for the requested

document, two source peers, A and B, are available. Although both source peers are chosen to be the

same number of hops away from the requesting peer, peer A is setup adjacent to the requesting peer

while peer B is located in another room. Before executing the test, the throughput achieved on the path

between the requesting peer and each of the source peers is measured to ensure that it is lower in the

case of the path to peer B. As a result of this scenario, peer A should be selected by the middleware as

the first source peer to be contacted, as opposed to peer B, upon the reception of two search replies,

one from each source peer.

Lastly, the fourth scenario verifies the service discovery functionality of the middleware. A given

peer initiates a new search request for a specific service, the middleware communication service, which

is offered by every peer in the network. The expected result is the reception of a search reply from

each peer in the network containing a service description document for the middleware communication

service.

5.3 Experimental Results

The four test scenarios described in the previous section were carried out in order to validate the

operation of the middleware. For each scenario, the results obtained for the transmitted query corre-

sponded to those expected for that scenario. In the case of the third scenario, the source peer selection

functionality was able to select the source peer to which the requesting peer had the lowest value of

the path metric. This was confirmed by consulting the kernel mesh path table. Following the functional

testing, the first scenario was used to obtain experimental results for the query forwarding delay, query

success rate, overall traffic and query processing time metrics. No background traffic was present in the

MANET during each measurement, with the exception of the traffic required by the IEEE 802.11 MAC,

namely periodic beacon transmissions announcing the presence of each node.

The query forwarding delay at a given peer was measured for an increasing query load, i.e. the

number of queries received by that peer during a given unit of time, in order to determine the impact

of the query load on the forwarding delay introduced by the userspace middleware components. The

query forwarding delay was measured through P2P-HWMP, which calculated the time elapsed between

transmitting a received search request, containing a query, to the middleware daemon and receiving

a forwarding decision for that search request. Node 1 was selected as the query generator and node

4 was used to measure the query forwarding delay for the queries received from the query generator.

The query generator was configured to generate queries at different rates: 1, 10, 100 and 1000 queries

per second corresponding to an inter-query spacing of 1000, 100, 10 and 1 milliseconds, respectively.

68

0

7.5

15.0

22.5

30.0

1 10 100 1000

Qu

ery

fo

rwa

rdin

g d

ela

y (

ms)

Queries per second

Without reply With reply

Figure 5.2: Query forwarding delay for increasing query load

Measurements were performed for two cases: when the generated queries match a document stored at

the peer performing the measurements, thus generating a reply, and when no matching documents exist.

Figure 5.2 shows the results obtained for both cases during a 10 second period. For low query loads, i.e.

up to 100 queries per second, the forwarding peer exhibits an almost constant query forwarding delay of

approximately 10.3 ms if the query matches a document stored by the peer and 7.5 ms otherwise. As

expected, the query forwarding delay is higher in the case of the former when compared to the latter,

circa 27%, since the middleware storage service must not only consult the Xapian index to determine

if matching documents exist but also extract those documents from the underlying storage in order

to return the results to the query routing service. Only then can the query routing service deliver a

forwarding decision to P2P-HWMP. If the query load is higher than 100 queries per second, the query

forwarding delay begins to increase since the middleware is unable to finish processing and forwarding

the current query before receiving the next query.

Although the query forwarding delay can be used to analyse the overhead introduced by the userspace

middleware components in forwarding a query, it does not provide a complete overview of the time re-

quired for a query to traverse the middleware at a given peer, i.e., the query processing time. The query

processing time at a given peer is composed of the wall-clock time spent by each middleware compo-

nent, namely P2P-HWMP, netlink communication, the query routing service and the storage service, in

processing a received query. Time-stamps were generated at the handoff points between interacting

components, for example, when the query routing service calls the storage service or when P2P-HWMP

sends a message via the netlink socket. The processing time spent by each component was then

69

0 2.50 5.00 7.50 10.00

Processing time (ms)

P2P-HWMP NetlinkMiddleware (routing service) Middleware (storage service)

Figure 5.3: Query processing time

calculated based on the generated time-stamps.

Figure 5.3 presents the results of measuring the query processing time at a given peer, node 4, for

a received query matching a stored document. Both P2P-HWMP and netlink communication represent

a negligible contribution, 0.78 ms, to the overall processing time. Both these components are executed

in a kernel interrupt handler and therefore do not suffer from scheduling delays. Furthermore, these

components carry out only a limited set of operations on the received search request. The bulk of

the processing time, 8.6 ms, is split between the query routing service, where the search request is

parsed, and the storage service, where the query is resolved by consulting a document repository.

These services represent an opportunity for further optimisation with regard to the code path followed

in order to forward a search request. Furthermore, the priority of the middleware daemon process,

where both services are executed, could be increased to reduce the impact of operating system process

scheduling delays. In spite of this, the time taken by a query to traverse the middleware at a given peer,

9.38 ms, is in line with that estimated for the one-hop packet traversal time for traditional MANET routing

protocols. For AODV, the estimated packet traversal time, including the time required to transmit the

packet to the next hop and any queuing delays, is 40 ms [26].

The query success rate was measured at a given peer to determine how the ability to resolve a query

is affected by increasing query load. Node 1 was chosen to generate queries in a manner analogous

to that described for the query forwarding delay metric. All of the generated queries match at least

one document stored by another peer, node 5. For each query sent, the generating peer waits until a

reply is received or the query times out. The query success rate is then calculated as the number of

queries for which a reply was received over the number of queries sent. The measurements obtained

by the generating peer are presented in Figure 5.4. For query generation rates less than 10 queries

70

0

25

50

75

100

1 10 100 1000

Query

success r

ate

(%

)

Queries per second

Figure 5.4: Query success rate for an increasing query load

per second, the generating peer is able to resolve, on average, 99.5% of the transmitted queries as the

resulting query routing traffic, especially at node 4, is insufficient to cause network congestion. In this

case, the loss of PREQ frames containing search requests and PREP frames containing search replies

is negligible. The query success rate begins to decrease for query generation rates greater than 10

queries per second. Despite this, 76.8% of the queries are resolved in the case of 1000 queries per

second.

A comparison between the middleware and an off-the-shelf solution consisting of Gnutella, TCP/IP

and HWMP was used to analyse the efficiency of the implemented middleware solution with respect to

the overall traffic generated as a result of a single query, i.e., the volume of unicast and broadcast traffic

generated, both in terms of the number of frames as well as the total size. gtk-gnutella was chosen as

the Gnutella client and installed on every node in the testbed alongside the middleware. gtk-gnutella is a

popular Gnutella client for Unix-like operating systems written in C with a GTK+ graphical user interface

and the ability to run in the background, as a daemon, as well as a command-line interface with limited

functionality. The latest version of the Gnutella protocol, version 0.6, is supported by gtk-gnutella as well

as many proposed extensions including IPv6 and Transport Layer Security (TLS) support. gtk-gnutella

is also highly configurable such that most aspects related to the behaviour of the Gnutella protocol can

be customized. For this test, gtk-gnutella was configured in ultra-peer mode, without any leaf nodes,

in order to reflect the query flooding nature of Gnutella and the lack of node hierarchy inherent in a

MANET. Both TLS and compression support were disabled for Gnutella overlay connections to provide

a fair comparison with the middleware which does not support this functionality. Furthermore, most

extensions, such as partial file sharing and swarming, were disabled.

71

0

37500

75000

112500

150000

Middleware Gnutella (warm) Gnutella (cold)

Tra

ffic

(b

yte

s)

MAC HWMPTCP OtherGnutella (Other) Gnutella (Ping/Pong)Gnutella (Query) MIddleware

Figure 5.5: Overall traffic (bytes) generated for a single query

An additional node was used to measure the overall traffic generated. This node was equipped with

an IEEE 802.11 network interface operating in monitor mode to capture all traffic exchanged between the

nodes participating in the MANET. Node 5 was configured to share a given file using both gtk-gnutella

and the P2P file-sharing application described in the previous section. At node 1, each application

was then used to send a query for this file. In the case of the gtk-gnutella, the query was routed

through the Gnutella overlay, constructed by randomly selecting the set of neighbouring peers for each

peer. The topology of the overlay was generated such that the average peer degree, i.e. the number

of neighbours, matched that of the underlying network topology presented in Figure 5.1. The set of

neighbours determined by the generated overlay topology were used to bootstrap each Gnutella peer.

Network traffic was captured during a 30 second interval and analysed using the Wireshark packet

analyser for a query sent according to three scenarios: using the file-sharing application developed

with the middleware API, henceforth known only as middleware, using gtk-gnutella with a previously

established overlay, henceforth known as Gnutella warm, and using gtk-gnutella while establishing the

overlay during the measurement interval, henceforth known as Gnutella cold. It is important to note that

in the case of the middleware scenario, the query results were embedded in the search reply message

sent to the requesting peer rather than being obtained out-of-band using the middleware communication

service.

The overall traffic generated as a result of the query, in terms of total size of the transmitted IEEE

802.11 frames, is displayed in Figure 5.5 for the three scenarios. The composition of the total size is

also shown, where each transmitted frame is classified according to its payload: IEEE 802.11 MAC

management frames, HWMP path selection frames (excluding those with middleware payload), TCP,

72

0

225

450

675

900

Middleware Gnutella (warm) Gnutella (cold)

Tra

ffic

(n

um

be

r o

f m

essa

ge

s)

MAC HWMPTCP OtherGnutella (Other) Gnutella (Ping/Pong)Gnutella (Query) MIddleware

Figure 5.6: Overall traffic (messages) generated for a single query

the various Gnutella protocol messages (query, query hit, ping, pong), frames containing middleware

search requests or search replies and finally other frames used, for example, for ARP address resolution

requests and responses. In the case of the middleware scenario, only two types of frames are generated:

IEEE 802.11 MAC management frames and HWMP path selection frames containing search requests

or search replies. The middleware scenario represents a 74% decrease in traffic when compared to

the Gnutella warm scenario and an 85% decrease when compared to the Gnutella cold scenario. The

overhead introduced by Gnutella is mainly due to the use of Gnutella Ping and Pong messages for

discovering other peers in the overlay and determining the status of connections to neighbouring peers.

Other factors contributing to the overhead include the successive HWMP route discoveries triggered in

order to maintain persistent TCP connections between overlay neighbours, as well as, the overhead

introduced by TCP itself including any packet retransmissions.

The 81% increase in traffic from the Gnutella warm to the Gnutella cold scenario is explained by the

need to establish the P2P overlay before sending the query. In order to establish the overlay, each peer

must initiate a TCP connection to each selected neighbour peer resulting in a TCP 3-way handshake per

connection and a Gnutella protocol handshake on top of the established connection where both peers

exchange the list of supported extensions. Since each peer’s ARP table is initially empty, ARP requests

must be broadcast to the network in order to discover the mapping between each peer’s network address

and its link-layer address. This is not an issue in the middleware scenario, where the transmission of

search requests and replies does not require the knowledge of a peer’s network address, and in the

Gnutella warm scenario since the ARP address resolution process has already been executed as a

73

0

37.5

75.0

112.5

150.0

MIddleware Gnutella (warm) Gnutella (cold)

Bro

adcast tr

affic

(m

essages)

Figure 5.7: Broadcast traffic (messages) generated for a single query

result of establishing the overlay.

Similar observations can be made for Figure 5.6 which presents the overall traffic generated in terms

of protocol messages. Additionally, the obtained results allow for a comparison between the number of

messages broadcast at the link-layer for each scenario. This is a factor in determining the efficiency of

a given solution, as broadcast traffic has a greater impact on network performance than unicast traffic.

Figure 5.7 shows the number of messages broadcast for the three scenarios, excluding IEEE 802.11

MAC beacon frames, as these are always present, with the same frequency, regardless of the scenario.

In the case of Gnutella, broadcasting a query using the overlay results in multiple messages unicast

at the link-layer, therefore, such virtual broadcasts are omitted. Despite this, the number of messages

broadcast for the Gnutella warm and Gnutella cold scenarios represent an increase of 2040% and

2900%, respectively, over the middleware scenario. While the middleware executes a single HWMP

path discovery process to propagate the query to all peers and receive the corresponding reply, Gnutella

uses static TCP connections between overlay peers. Keep-alive traffic, in other words, ping and pong

messages, are constantly exchanged over these connections. Since the paths discovered by HWMP

expire after 5s, each Gnutella peer will trigger a path discovery to each of its current overlay neighbours

in order to maintain the corresponding overlay connection. The PREQ frames which are broadcast as a

result of successive path discovery processes contribute to Gnutella’s poor result.

In short, one can easily conclude that establishing a P2P overlay network on top of a MANET rep-

resents a considerable maintenance overhead, even for an unstructured P2P system, when compared

to a P2P system which takes advantage of the underlying MANET routing protocol to transport queries

and the corresponding results, as is the case of the implemented middleware solution. Furthermore,

the use of static connections between peers in the overlay and keep alive messages to verify that such

74

connections are still valid is inefficient in a MANET as it triggers successive route discovery processes,

and consequent broadcast traffic, if a reactive routing protocol is used. Although not apparent due to the

size of the MANET testbed, the selection of overlay neighbours which are multiple physical hops away

will only exacerbate this problem.

75

6Conclusion

76

P2P networks and MANETs exhibit many similarities, including decentralisation, self-organisation

and a highly dynamic topology. A key issue in both networks is the process of discovering the requested

content or route efficiently in a decentralised manner and routing messages between nodes. As such,

it is clear that the P2P paradigm is particularly suited to enable content distribution, communication and

collaboration applications in a MANET environment. In spite of this, existing P2P systems, designed for

the Internet, do not exploit the synergies between P2P networks and MANETs since the P2P overlay

network is unaware of the topology of the underlying physical network. P2P application networks operate

at the application layer of the OSI reference model while MANETs operate at the network layer or data

link layer. Routing functionality is therefore present in both layers. On the one hand, application layer

routing is used by search queries to find a peer containing the desired information within the P2P overlay

network. On the other hand, network layer or data link layer routing is used to discover the route to

any given node. Logical node proximity differs from physical proximity, resulting in inefficient paths

between neighbours in the overlay and unnecessary network overhead due to the duplicate functionality

implemented at different layers in the network stack.

This thesis proposed a P2P middleware solution to facilitate the development of P2P applications

optimised for use in MANETs. This solution avoids duplicating the discovery and routing functionality

at different layers in the network stack by integrating the process of searching for application content

with the discovery of the network routes in the underlying MANET. Rather than use a P2P overlay

network consisting of a set of persistent connections setup between possibly distant peers in order to

route application layer search requests and replies, the MANET routing mechanisms are used instead.

HWMP was chosen as the MANET routing protocol due to its inclusion in the IEEE 802.11s draft, which

promises to become the de facto standard for the creation of multi-hop wireless mesh networks and

MANETs. Both HWMP path request and path reply messages were extended in order to transport,

respectively, application layer search requests and search replies. As a result, P2P applications can

submit new search requests to the network routing layer and be notified of the reception of new search

requests. When multiple sources are available for the same content, the middleware chooses a peer

based on the cost of the path to that peer, as given by HWMP’s path metric. The middleware therefore

provides applications with an optimised, network-aware, source peer selection service, in addition to the

standard search functionality.

An implementation of the proposed P2P middleware was developed for the GNU/Linux operating

system and based on the HWMP and airtime link metric implementations included with the Linux ker-

nel IEEE 802.11 stack, mac80211. As such, the middleware implementation was divided into both

kernelspace components, which interact directly with HWMP, and userspace components, which com-

prehend the middleware daemon, providing most of the middleware functionality, as well as an object-

oriented Python API to enable developers to easily develop P2P applications which take advantage

77

of the middleware. An appropriate communication channel, using netlink, supports the exchange of

information between both userspace and kernelspace middleware components.

The middleware implementation was evaluated experimentally through both functional and perfor-

mance tests executed on a small multi-hop MANET testbed consisting of nodes equipped with IEEE

802.11 network interfaces. These tests focused on the search functionality provided by the middleware,

and the resulting query routing. The results obtained for the various metrics demonstrated the reduced

overhead introduced by the middleware and the reduction in network resource usage when compared

to Gnutella, a traditional P2P system, layered on top of a MANET.

Several points remain to be addressed as part of future work. First of all, the query routing service

can be used to implement further query routing protocols, such as, random walk and expanding ring

search. The impact of these protocols on the quality of the paths discovered by HWMP, the delay in

resolving queries and the amount of traffic generated could be the focus of further study. Although

experimental performance results were obtained using a testbed, further testing, through simulation,

is required in order to obtain a more detailed characterisation of the performance of the middleware,

namely the impact of mobility and large node populations on the query success rate. Even though

the middleware provides the ability to discover services on the network, only the minimum amount of

information necessary to contact a service is included in the returned service description documents. An

existing service description language could be integrated with the middleware and used in constructing

service description documents. Finally, in spite of the lack of persistent TCP connections between

neighbouring peers, TCP is still used by the middleware in order to transfer both documents and, in

some cases, results from the replying peer. Further reliable transport protocols, designed for wireless

networks, could be used to address the performance issues introduced as a result of TCP operation in

MANETs.

78

Bibliography

[1] S. Jochen, Mobile Communications. Boston, MA, USA: Addison-Wesley Longman Publishing Co.,

Inc., 2003.

[2] R. Schollmeier, “A definition of peer-to-peer networking for the classification of peer-to-peer ar-

chitectures and applications,” in P2P ’01: Proceedings of the First International Conference on

Peer-to-Peer Computing (P2P’01), (Washington, DC, USA), p. 101, IEEE Computer Society, 2001.

[3] The Gnutella Developer Forum, Gnutella protocol specification 0.4, http://rfc-

gnutella.sourceforge.net/developer/stable/index.html.

[4] G. Ding and B. Bhargava, “Peer-to-peer file-sharing over mobile ad hoc networks,” in PERCOMW

’04: Proceedings of the Second IEEE Annual Conference on Pervasive Computing and

Communications Workshops, (Washington, DC, USA), p. 104, IEEE Computer Society, 2004.

[5] One Laptop Per Child project, http://www.laptop.org.

[6] A. Marques, F. Mira da Silva, and R. Rocha, “P2P over Mobile Ad-hoc Networks,” in Sensor, Mesh

and Ad Hoc Communications and Networks Workshops, 2009. SECON Workshops ’09. 6th Annual

IEEE Communications Society Conference on, pp. 1–3, June 2009.

[7] S. Androutsellis-Theotokis and D. Spinellis, “A survey of peer-to-peer content distribution technolo-

gies,” ACM Comput. Surv., vol. 36, no. 4, pp. 335–371, 2004.

[8] Jabber, http://www.jabber.org.

[9] P. Druschel and A. Rowstron, “Past: A large-scale, persistent peer-to-peer storage utility,” in HOTOS

’01: Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, (Washington, DC,

USA), p. 75, IEEE Computer Society, 2001.

[10] Q. Lv, P. Cao, E. Cohen, K. Li, and S. Shenker, “Search and replication in unstructured peer-to-peer

networks,” in ICS ’02: Proceedings of the 16th international conference on Supercomputing, (New

York, NY, USA), pp. 84–95, ACM, 2002.

79

[11] The Gnutella Developer Forum, Gnutella protocol specification 0.6, http://rfc-

gnutella.sourceforge.net/src/rfc-0 6-draft.html.

[12] Kazaa, http://www.kazaa.com.

[13] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan, “Chord: A scal-

able peer-to-peer lookup service for internet applications,” in SIGCOMM ’01: Proceedings of

the 2001 conference on Applications, technologies, architectures, and protocols for computer

communications, (New York, NY, USA), pp. 149–160, ACM, 2001.

[14] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Schenker, “A scalable content-addressable

network,” in SIGCOMM ’01: Proceedings of the 2001 conference on Applications, technologies,

architectures, and protocols for computer communications, (New York, NY, USA), pp. 161–172,

ACM, 2001.

[15] B. Y. Zhao, J. D. Kubiatowicz, and A. D. Joseph, “Tapestry: An infrastructure for fault-tolerant wide-

area location and,” tech. rep., Berkeley, CA, USA, 2001.

[16] A. I. T. Rowstron and P. Druschel, “Pastry: Scalable, decentralized object location, and routing for

large-scale peer-to-peer systems,” in Middleware ’01: Proceedings of the IFIP/ACM International

Conference on Distributed Systems Platforms Heidelberg, (London, UK), pp. 329–350, Springer-

Verlag, 2001.

[17] J. Kubiatowicz, D. Bindel, Y. Chen, S. Czerwinski, P. Eaton, D. Geels, R. Gummadi, S. Rhea,

H. Weatherspoon, C. Wells, and B. Zhao, “Oceanstore: an architecture for global-scale persistent

storage,” SIGARCH Comput. Archit. News, vol. 28, no. 5, pp. 190–201, 2000.

[18] Y.-J. Joung, C.-T. Fang, and L.-W. Yang, “Keyword search in dht-based peer-to-peer networks,”

in ICDCS ’05: Proceedings of the 25th IEEE International Conference on Distributed Computing

Systems, (Washington, DC, USA), pp. 339–348, IEEE Computer Society, 2005.

[19] B. Yang and H. Garcia-Molina, “Improving search in peer-to-peer networks,” in ICDCS ’02:

Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS’02),

(Washington, DC, USA), p. 5, IEEE Computer Society, 2002.

[20] A. Crespo and H. Garcia-Molina, “Routing indices for peer-to-peer systems,” in ICDCS ’02:

Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS’02),

(Washington, DC, USA), p. 23, IEEE Computer Society, 2002.

[21] H. Balakrishnan, M. F. Kaashoek, D. Karger, R. Morris, and I. Stoica, “Looking up data in P2P

systems,” Commun. ACM, vol. 46, no. 2, pp. 43–48, 2003.

80

[22] C. G. Plaxton, R. Rajaraman, and A. W. Richa, “Accessing nearby copies of replicated objects in a

distributed environment,” in SPAA ’97: Proceedings of the ninth annual ACM symposium on Parallel

algorithms and architectures, (New York, NY, USA), pp. 311–320, ACM, 1997.

[23] I. Chlamtac, M. Conti, and J. J. Liu, “Mobile ad hoc networking: imperatives and challenges,” Ad

Hoc Networks, vol. 1, pp. 13–64, July 2003.

[24] C. E. Perkins and P. Bhagwat, “Highly dynamic destination-sequenced distance-vector routing

(dsdv) for mobile computers,” SIGCOMM Comput. Commun. Rev., vol. 24, no. 4, pp. 234–244,

1994.

[25] T. Clausen and P. Jacquet, “RFC3626: Optimized Link State Routing Protocol (OLSR),” RFC Editor

United States, 2003.

[26] C. Perkins, E. Belding-Royer, and S. Das, “RFC3561: ad hoc on-demand distance vector (AODV)

routing,” Internet RFCs, 2003.

[27] D. B. Johnson, D. A. Maltz, and J. Broch, “Dsr: The dynamic source routing protocol for multi-

hop wireless ad hoc networks,” in In Ad Hoc Networking, edited by Charles E. Perkins, Chapter 5,

pp. 139–172, Addison-Wesley, 2001.

[28] G. Pei, M. Gerla, X. Hong, and C. chuan Chiang, “A wireless hierarchical routing protocol with group

mobility,” in In Proceedings of the IEEE Wireless Communications and Networking Conference

(WCNC, pp. 1538–1542, 1999.

[29] B. Karp and H. T. Kung, “Gpsr: greedy perimeter stateless routing for wireless networks,” in

MobiCom ’00: Proceedings of the 6th annual international conference on Mobile computing and

networking, (New York, NY, USA), pp. 243–254, ACM, 2000.

[30] M. Bahr, “Update on the hybrid wireless mesh protocol of IEEE 802.11s,” Mobile Adhoc and Sensor

Systems, 2007. MASS 2007. IEEE Internatonal Conference on, pp. 1–6, Oct. 2007.

[31] M. Bisignano, G. D. Modica, O. Tomarchio, and L. Vita, “P2P over MANET: a comparison of cross-

layer approaches,” in DEXA ’07: Proceedings of the 18th International Conference on Database

and Expert Systems Applications, (Washington, DC, USA), pp. 814–818, IEEE Computer Society,

2007.

[32] L. Barbosa e Oliveira, I. Guimaraes Siqueira, and A. Ferreira Loureiro, “Evaluation of ad-hoc rout-

ing protocols under a peer-to-peer application,” Wireless Communications and Networking, 2003.

WCNC 2003. 2003 IEEE, vol. 2, pp. 1143–1148 vol.2, March 2003.

81

[33] L. B. Oliveira, I. G. Siqueira, D. F. Macedo, A. A. F. Loureiro, H. C. Wong, and J. M. Nogueira,

“Evaluation of peer-to-peer network content discovery techniques over mobile ad hoc networks,”

in WOWMOM ’05: Proceedings of the Sixth IEEE International Symposium on World of Wireless

Mobile and Multimedia Networks, (Washington, DC, USA), pp. 51–56, IEEE Computer Society,

2005.

[34] M. Papadopouli and H. Schulzrinne, “Effects of power conservation, wireless coverage and coop-

eration on data dissemination among mobile devices,” in MobiHoc ’01: Proceedings of the 2nd

ACM international symposium on Mobile ad hoc networking & computing, (New York, NY, USA),

pp. 117–127, ACM, 2001.

[35] G. Kortuem, J. Schneider, D. Preuitt, T. G. C. Thompson, S. Fickas, and Z. Segall, “When peer-

to-peer comes face-to-face: Collaborative peer-to-peer computing in mobile ad hoc networks,” in

P2P ’01: Proceedings of the First International Conference on Peer-to-Peer Computing (P2P’01),

(Washington, DC, USA), p. 75, IEEE Computer Society, 2001.

[36] A. Klemm, C. Lindemann, and O. Waldhorst, “A special-purpose peer-to-peer file sharing system

for mobile ad hoc networks,” in IEEE VehIcular Technology Conference, vol. 4, pp. 2758–2763,

2003.

[37] M. Conti, E. Gregori, and G. Turi, “A cross-layer optimization of Gnutella for mobile ad hoc net-

works,” in Proceedings of the 6th ACM international symposium on Mobile ad hoc networking and

computing, pp. 343–354, ACM New York, NY, USA, 2005.

[38] R. Schollmeier, I. Gruber, and F. Niethammer, “Protocol for peer-to-peer networking in mobile en-

vironments,” in Computer Communications and Networks, 2003. ICCCN 2003. Proceedings. The

12th International Conference on, pp. 121–127, 2003.

[39] H. Pucha, S. Das, and Y. Hu, “Ekta: An efficient DHT substrate for distributed applications in mobile

ad hoc networks,” in Proc. of WMCSA, 2004.

[40] T. Zahn and J. Schiller, “MADPastry: A DHT substrate for practicably sized MANETs,” in Proc. of

ASWN, 2005.

[41] IEEE Task Group S, “IEEE P802.11s/D1. 06, draft amendment to IEEE 802.11: Mesh networking,”

July 2007.

[42] J. Camp and E. Knightly, “The IEEE 802.11 s extended service set mesh networking standard,”

IEEE Communications Magazine, vol. 46, no. 8, pp. 120–126, 2008.

82

[43] “IEEE Standard for Information technology-Telecommunications and information exchange be-

tween systems-Local and metropolitan area networks-Specific requirements - Part 11: Wireless

LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications,” tech. rep., 2007.

[44] T. Berners-Lee, R. Fielding, and L. Masinter, “RFC 3986: Uniform resource identifier (URI): Generic

syntax,” RFC Editor United States, 2005.

[45] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee, “RFC2616:

Hypertext Transfer Protocol–HTTP/1.1,” RFC Editor United States, 1999.

[46] Open 802.11s project, http://www.o11s.org.

[47] Python programming language, http://www.python.org.

[48] Twisted, http://twistedmatrix.com.

[49] The Xapian Project, http://xapian.org.

[50] Xappy, http://code.google.com/p/xappy.

[51] TwistedSNMP, http://twistedsnmp.sourceforge.net.

83

AMiddleware Search Message Formats

84

Two types of messages are exchanged between the query routing services at different peers: search

request and search reply messages. Both message types consist of a common header and a type

specific payload. Further message types can be easily defined by setting the appropriate value for the

type field in the search message header. In order to send a search request and search reply message

to the query routing service at a remote peer, the query routing service at the originating peer contacts

P2P-HWMP so as to include the message in a new PREQ or PREP frame.

Search Message Header

The format of the search message header, used for both search request and search reply messages,

is as follows:

Middleware Port

MessageIDType

Reserved IPv6Addr. Present

Flags

IPv4Addr. Present

1 1 16 2

6 1

Octets: 16

PeerID

IPv4Address

IPv6Address

1

4 16

GroupID

2

Bits:

The header fields have the following meaning:

Flags (1 byte):

Reserved (6 bits)

Reserved for future use. Set to 0 and ignored on reception.

IPv4 Address Present (1 bit)

The IPv4 Address Present bit is set when the message header includes the IPv4 Address

field.

IPv6 Address Present (1 bit)

The IPv6 Address Present bit is set when the message header includes the IPv6 Address

field.

Type (1 byte)

The type of payload included after the message header. This field is set to the value 0 for search

requests and 1 for search replies.

Message ID (16 bytes)

GUID used to uniquely identify this message. A new GUID is generated for each new message

85

sent. The Message ID allows for the detection of duplicate messages at the receiving peer. A

search reply message uses the same GUID as the search request message it is replying to so that

the requesting peer can match the reply to the corresponding request.

Peer ID (16 bytes)

GUID used to uniquely identify the sending peer. The Peer ID can be used to discover the sending

peer if it changes its network address.

IPv4 Address (4 bytes)

The IPv4 network address of the sending peer.

IPv6 Address (16 bytes)

The IPv6 network address of the sending peer.

Middleware Port (2 bytes)

Transport-layer port for the middleware communication service at the sending peer.

Group ID (16 bytes)

GUID used to uniquely identify the group of peers the sending peer belongs to. The Group ID is

used to limit the scope of search requests such that only peers belonging to the same group as

the sending peer will reply to that search request.

The search message header must contain at least one network address field in order to allow the

sending peer to be contacted.

Search Request

The format of the search request message is as follows:

QueryLengthHeader

56-68 16 2Octets:

ApplicationID Query

1-1000

The type specific fields have the following meaning:

Application ID (16 bytes)

GUID used to uniquely identify the P2P application which is the target of the search query con-

tained in this message. The Application ID is used by the receiving peer to determine the docu-

ment respository it should use to search for documents matching the query. When searching for

services, a special GUID, 00000000-0000-0000-b000-000000000001, is used instead.

86

Query Length (2 bytes)

Length of the Query field included in this message. The Query field must be limited to 1000 bytes

in order to be included in a single PREQ frame by P2P-HWMP due to the limitations imposed by

the MTU of the underlying IEEE 802.11 MAC protocol.

Query (1-1000 bytes)

Query string generated by the requesting P2P application in UTF-8 encoding.

Search Reply

The format of the search reply message is as follows:

Embedded Results Present Reserved

56-68

1

Octets:

7Bits:

Header

1

Numberof Hits

2

Flags ResultID

16

ResultLength

2

ResultSet

2-1000

The search reply message is sent in reply to a previously received search request containing a query.

The type specific fields have the following meaning:

Flags (1 byte):

Embedded Results Present (1 bit)

This bit is used to indicate whether or not the query results are included in the message or if

they must be obtained separately, i.e. out-of-band, by the requesting peer. If the Embedded

Results Present bit is set, the message contains a the Result Set Length and Result Set

fields. If the bit is not set, the message contains the Number of Hits and Result ID fields

instead.

Reserved (7 bits)

Reserved for future use. Set to 0 and ignored on reception.

Number of Hits (2 bytes)

Number of documents matching the query.

Result ID (2 bytes)

Temporary GUID used to uniquely identify the query’s result set. The result set is cached by the

replying peer so that it can be obtained by the requesting peer using the this GUID.

87

Result Set Length (2 bytes)

Length of the Result Set field included in this message. The Result Set field must be limited to

1000 bytes in order to be included in a single PREP frame by P2P-HWMP due to the limitations

imposed by the MTU of the underlying IEEE 802.11 MAC protocol.

Result Set (1-1000 bytes)

String containing the JSON representation of the list of documents matching the query.

88

BMiddleware SNMP MIB Specification

89

The P2P middleware exports a set of statistics using an embedded agent which can be queried using

SNMP. The MIB can be found rooted under the Instituto Superior Tecnico arc (.1.3.6.1.4.1.21512) with a

child number of 99. The tree is further subdivided according to the internal middleware services, namely

query routing, storage, peer discovery and service registration. A brief description of each available

Object Identifier (OID) follows.

• .1.3.6.1.4.1.21512.99.1 — Routing service

.1.0 — Search requests sent (INTEGER): Number of search requests sent by the peer.

.2.0 — Search requests received (INTEGER): Number of unique search requests received

from other peers in the network.

.3.0 — Search requests replied (INTEGER): Number of unique search requests the peer

has replied to.

.4.0 — Pending search requests (INTEGER): Number of search requests sent by the peer

which are pending, i.e., search requests for which no reply has been received and which

haven’t yet timed out.

.5.0 — Search replies received (INTEGER): Number of search replies received in response

to search requests sent by the peer.

.6.0 — Bytes sent (INTEGER): Aggregate size in bytes of all the search request and search

reply messages sent by the peer.

.7.0 — Bytes received (INTEGER): Aggregate size in bytes of all the unique search request

and search reply messages received by the peer.

.8.0 — Mean search reply delay (INTEGER): Mean time in milliseconds between sending a

search request and receiving a corresponding search reply.

.9.0 — Mean query result delay (INTEGER): Mean time in milliseconds between accepting

a new query and returning its results.

.10.0 — Resolved queries (INTEGER): Number of queries sent by the peer for which at least

one result was received.

.11.0 — Search replies with embedded results (INTEGER): Number of search replies sent

by the peer containing the complete set of results.

• .1.3.6.1.4.1.21512.99.2 — Storage service

.1.0 — Number of databases (INTEGER): Number of document databases managed by the

storage service.

90

.2.0 — Number of documents (INTEGER): Total number of documents stored in the docu-

ment databases.

.3.0 — Number of shared databases (INTEGER): Number of document databases man-

aged by the storage service which are searchable by remote peers.

.4.0 — Number of shared documents (INTEGER): Total number of documents stored in

shared document databases.

• .1.3.6.1.4.1.21512.99.3 — Peer selection service

.1.0 — Number of neighbours (INTEGER): Number of mesh neighbours of the peer (in-

cludes legacy and P2P nodes).

.2.0 — Number of mesh paths (INTEGER): Number of active mesh paths in the peer’s

HWMP forwarding table.

.3.0 — Number of known peers (INTEGER): Number of peers known by the peer.

• .1.3.6.1.4.1.21512.99.4 — Service registration service

.1.0 — Number of registered services (INTEGER): Number of services registered with the

service registration service and which are discoverable by other peers in the network.

91

CMiddleware API

92

middleware.api UML Class Diagram

+__init__(self, application_name, port = 9001, password = "", async = False)+create_repository(self, password = "", indexed_fields = [], result_fields = [])+set_password(self, password)+set_indexed_fields(self, indexed_fields)+get_indexed_fields(self)+set_result_fields(self, result_fields)+get_result_fields(self)+drop_repository(self)+has_repository(self)+enable_sharing(self)+disable_sharing(self)+register_service(self, service_document)+unregister_service(self, id)+add(self, document)+remove(self, document)+update(self, document)+get(self, id)+get_all(self)+search(self, query, limit = 25, local = False, services = False)

+application_name-_password-_async-_proxy

Middleware

+__init__(self, document, peers, async = False) +downloadURL(self, peer)+get(self, path = None)

+peers+document+local-_async

Result

MiddlewareError

QueryTimeout

AuthenticationError

RepositoryNotFound

DocumentNotFound

CommunicationError

<<throw>><<throw>>

<<instanciate>>

93

middleware.types UML Class Diagram

+__init__(self, fields = None)+__getitem__(self, field)+__setitem__(self, field, value)+__delitem__(self, field)+id(self)+type(self)+complete(self)+fields(self)

-_fieldsDocument

__init__(self, path)

FileDocument

__init__(self, url, type, lifetime = 3600, attributes = {})

ServiceDescriptionDocument

+__init__(self, guid, hwaddr, port, ipv4 = None, ipv6 = None)

+hwaddr+ipv4+ipv6+guid+port

Peer

94

Example Service Querying Application

#!/usr/bin/env python

from middleware.api import Middleware, QueryTimeout

m = Middleware("search")

query = "type:P2P_middleware_agent"

try:

results = m.search(query, limit = 0, services = True)

for result in results:

url = result.document[’url’]

addr, port = url.split("//").pop().split(":")

print "%s %s\n" % (addr, port)

except QueryTimeout:

print "Search query timed out\n"

95

DP2P-HWMP Protocol

96

The P2P-HWMP protocol is used for asynchronous bidirectional communication between kernelspace

and userspace middleware components. More specifically, this protocol consists of a series of messages

exchanged between P2P-HWMP and the query routing service using the netlink IPC mechanism pro-

vided by Linux. As a result, a netlink header is prepended to each of the message types described

below. This netlink header specifies the length of the message, its type as well as the PID of the sending

process (0 in the case of the kernel).

Transmit Search Request

The transmit search request message is sent from the query routing service to P2P-HWMP in order

to request the creation and transmission of a new HWMP PREQ frame containing the enclosed search

request. The format of this message is as follows:

netlink Header(type = 1001)

16 1Octets:

ForwardingType Search Request

1-1200N * 6

Number ofAddresses

1

Next HopAddresses TTL

1

The fields have the following meaning:

Forwarding Type (1 byte)

See forwarding decision message.

Number of Addresses (1 byte)

See forwarding decision message.

Next Hop Addresses (N * 6 bytes)

See forwarding decision message.

TTL (1 byte)

Value of the time-to-live field for the constructed PREQ frame. If set to 0, the default TTL value

configured for the mesh network interface is used.

Search Request (1-1200 bytes)

The search request to include in the constructed PREQ frame.

Search Request Notification

The search request notification message is sent by P2P-HWMP to notify the query routing service

of a newly received HWMP PREQ frame containing a search request. The format of this message is as

follows:

97

netlink Header(type = 1002)

16 6Octets:

SourceAddress Search Request

1-1200

Frame ID

4

The fields have the following meaning:

Source Address (6 bytes)

The link-layer address of the node which generated the received PREQ frame.

Frame ID (4 bytes)

Internal identifier generated by P2P-HWMP when caching received PREQ frames. This identifier

is used by the query routing service to refer to the PREQ frame, and its enclosed search request,

when sending forwarding decision or transmit search reply messages.

Search Request (1-1200 bytes)

The search request contained in the received PREQ frame.

Transmit Search Reply

The transmit search reply message is sent from the query routing service to P2P-HWMP in order

to request the creation and transmission of a new HWMP PREP frame containing the enclosed search

reply. This PREP frame is sent in reply to a previously received PREQ frame. The format of this message

is as follows:

netlink Header(type = 1003)

16Octets:

Search Reply

1-1200

Frame ID

4

The fields have the following meaning:

Frame ID (4 bytes)

The internal identifier of a previously received PREQ frame.

Search Reply (1-1200 bytes)

The search reply to include in the constructed PREP frame.

Search Reply Notification

The search reply notification message is sent by P2P-HWMP to notify the query routing service of a

newly received HWMP PREP frame containing a search reply. The format of this message is as follows:

98

netlink Header(type = 1004)

16Octets:

Search Reply

1-12006

SourceAddress

The fields have the following meaning:

Source Address (6 bytes)

The link-layer address of the node which generated the received PREP frame.

Search Reply (1-1200 bytes)

The search reply contained in the received PREP frame.

Forwarding Decision

The forwarding decision message is sent from the query routing service to P2P-HWMP in order to

drop or trigger the forwarding of a previously received PREQ frame. The format of this message is as

follows:

netlink Header(type = 1005)

16Octets:

Frame ID

41

ForwardingType

N * 6

Number ofAddresses

1

Next HopAddresses

The fields have the following meaning:

Forwarding Type (1 byte)

Determines how the PREQ frame should be forwarded. Possible values include:

0 – Drop the frame.

1 – Broadcast the frame to all neighbours.

2 – Unicast the frame to one or more neighbours whose addresses are specified in the Next

Hop Addresses field.

Number of Addresses (1 byte)

Number of addresses contained in the Next Hop Addresses field.

Next Hop Addresses (N * 6 byte)

Link-layer addresses of the nodes the PREQ frame will be forwarded to.

Frame ID (4 bytes)

The internal identifier of a previously received PREQ frame.

99

Register Middleware

The register middleware message is sent from the query routing service to P2P-HWMP in order to

register the middleware daemon. The process identifier of the middleware daemon, contained in the

netlink header for this message, is used by P2P-HWMP to send subsequent messages to the correct

userspace process. No incoming messages are processed by P2P-HWMP, or notifications sent, until a

valid registration is received. Upon registration, P2P-HWMP will register itself with the HWMP vendor

registry. The format of this message is as follows:

netlink Header(type = 1006)

16Octets: 16

Interface Name

The fields have the following meaning:

Interface Name (16 bytes)

Name of the mesh network interface used to send and receive PREQ and PREP frames on. If no

interface name is supplied, the first valid mesh interface in the kernel’s global device list is used.

Unregister Middleware

The unregister middleware message is sent from the query routing service to P2P-HWMP in order

to unregister the middleware daemon and release associated resources. As a result, P2P-HWMP will

also unregister itself from the HWMP vendor registry. The format of this message is as follows:

netlink Header(type = 1007)

16Octets:

100