Multi-Agent Systems in Mobile Ad hoc Networks

Joseph P. MackerWilliam ChaoRanjeev Mittu

Myriam AbramsonInformation Technology Division

Naval Research Laboratory

ABSTRACTA number of technologies are evolving that willhelp formulate more adaptive and robust networkarchitectures intended to operate in dynamic,mobile environments. One technology area,mobile ad hoc networking (MANET) enables self-organizing, multi-hop heterogeneous networkrouting services and organization. Suchtechnology is important in future DoD networking,especially in the forward edge of the battlespacewhere self-organizing, robust networking isneeded. A second technology area, Multi-AgentSystems (MAS) can enable autonomous, team-based problem solving under varyingenvironmental conditions. Previous work done inMAS has assumed relatively benign wired networkbehavior and inter-agent communicationscharacteristics that may not be well supported inMANET environments. In addition, the resourcecosts associated with performing inter-agentcommunications have a more profound impact in amobile wireless environment. The combinedoperation of these technology areas, includingcross-layer design considerations, has largelybeen unexplored to date. This paper describesongoing research to improve the ability of thesetechnologies to work in concert. An outline ofvarious design and system architecture issues isfirst presented. We then describe models, agentsystems, MANET protocols, and additionalcomponents that are being applied in ourresearch. We present an analysis method tomeasure agent effectiveness and early evaluationsof working prototypes within MANETenvironments. We conclude by outlining someopen issues and areas offurther work.

BACKGROUND AND MOTIVATIONMobile Ad hoc Networking (MANET) technologyis designed to provide effective dynamic InternetProtocol (IP) routing in network environments

where "change is the norm". Change may betopological due to mobility or behavioral due toother wireless environmental effects (e.g., energyconservation, channel fading, etc) [CM99],[POI].In addition, multi-agent system (MAS) technologyis well suited for dynamic, distributed problemsolving in which distributed software agents areable to both sense and act within complexenvironments [W02]. MAS-based architecturesmay be very valuable in the future forwardbattlespace to provide distributed teamwork-basedsolutions to complex problems. However, atpresent, MAS design and performance tradeoffs indisruptive and potentially mobile networks arelargely unexplored.

PROBLEM SPACEMANET is planned for future deployment in theforward edge of the battlespace. There has beensome significant advancement in the developmentof MANET solutions in recent years, but muchwork remains to be done. Overall, there remainslimited experience in using/adapting upper layerprotocols and applications in these environments[B04]. Also more work is needed in developingand adapting multicast routing in theseenvironments to better support group-orientedcommunications. Intelligent multi-agent systemsare also of interest in future DoD systems. These

E Algorithmic M

iTeamwork ..pmixedO

-4

F I e x b I e

Figure 1: Dynamic MAS Benefits

1

systems are composed of autonomous orcooperating software entities; in the latter casethese entities may be goal-driven and able tocoordinate (communicate) as part of a team toreach the goals.Figure 1 shows the potential benefit of MAS formore autonomous and flexible problem solving incomplex environments. Some possible examplesof future MAS applications include distributedcommand and control support software, networkmanagement, sensor networking, and mobilecooperative robotics. When considering MANETdeployment in highly stressed networks, the designof distributed and cooperative agent-based systemsis of interest. This is primarily due to the flexibleproblem solving approach offered by MAS, whichprovides a robust solution to address thechallenges faced in MANET environments.Previous design work has assumed a benign wirednetwork behavior; not MANET environments. InMANET environments peer agents experienceincreased topological dynamics, intermittentconnectivity, and reduced reliability. The cost ofinteragent communications is also high; thereforecommunication resources in a wireless MANETshould be utilized in the most effective meanspossible. For example, multi-party coordinationand communication frameworks should be wellsupported.Separately MAS and MANET encompass twochallenging research areas. Our focus is to beginimproving the performance capability of MASapplications running in MANET networkenvironments. This includes the followingsubgoals:

* improve crosslayer design performancebetween MANET and MAS layers.

* investigate MAS design robustness instressed network environments.

* develop and test new MANET networkservices and modeling where needed tosupport anticipated MAS operation.

ISSUES AND PROJECT APPROACHFigure 2 represents an example of the architecturallayering concept that we are targeting in thisresearch effort. The goals are to identify and filltechnology and protocol gaps that may exist inFigure 2 and improve any flawed designinteractions between the layers. At the same time,

we wish to maintain layered software and networkprotocol abstractions wherever possible.

: i/. .p ID * t //

i II CMiddleware S6-rvic&-tay6r X .

Figure 2: Layered Design Issues

At the MAS layer, we are interested in bettermultiagent designs to improve the potential foroperation in MANET environments. In doing so,we are investigating the network serviceassumptions and robustness requirements at theagent layer. Special attention is being paid tointeragent communication design includingefficiency and flexibility of agent teamwork. Animportant issue is establishing measures ofeffectiveness applied to agent designs within theseenvironments. At present, we are studyingteamwork problem solving effectiveness vs.performance metrics such as increasing networksize, mobility, and decreasing reliability.Figure 2 also shows middleware as a potentialdesign layer between MAS and MANET.Middleware plays a role as an important networkdesign abstraction often used by networkapplication designers, including designers ofnetwork agent software. Dynamic servicediscovery and other higher layer services are alsooften provided by a middleware layer. Examplesof middleware systems include JXTA, Gnutella,and Peer-to-Peer Simplified (P2PS) [W05].Despite their attractiveness as design abstractions,middleware frameworks are not an engineeringpanacea, especially in wireless environments. Forexample, middleware can take advantage ofefficient network communication abstractions suchas multicast but the present designs often assumestatic topology relationships or neighborhoodsubnet assumptions more consistent with wirednetwork deployments. There are also robustnessand efficiency tradeoffs between classes ofmiddleware services that are unique to MANET

2

environment scenarios. Middleware or peer-to-peer layers also often provide some autonomousorganization and identification independent of theunderlying network addresses such as the InternetProtocol (IP) stack provides. We are interested instudying the potential functional redundancy andinefficiencies of such layered abstractions whenoperating in MANET environments. We envisionan enhanced approach in which more efficientlower layer MANET network services (e.g.,multicast) may be utilized by middleware whenoperating in stressed wireless ad hoc scenarios.This will maintain a degree of abstraction desiredof middleware but will provide more efficient andeffective cross-layer components for MANET use.Figure 2 also shows the MANET layer as thebottom layer of the architecture providing ad hocmultiple-hop routing within a wireless network.The figure also depicts the fact that the scenariomay be very heterogeneous between layers. Forinstance, some nodes may only provide networkand possibly middleware services for other nodesthat are operating as agent nodes. In other cases, aphysical node may operate as an agent,middleware, and MANET node simultaneously.As envisioned the MANET protocol layer isproviding the following two main functions:

* Dynamic IP network routing* Some network autoconfiguration support

The MANET dynamic IP routing is performingboth unicast and multicast forwarding of datapackets. We are presently looking at theapplication ofMANET proactive unicast protocolsin this environment. We expect MANETproactive to provide lower average delay betweennodes and more predictable network overheadunder stressed conditions. The dense many-to-many traffic patterns involved in interagentteamwork communications likely favor a moreproactive approach but scaling a proactiveapproach can be challenging and future work inthat area is needed. There is also nothing in ourarchitecture concept to preclude the substitution ofreactive protocols as well.Because of the MAS teamwork model involvinggroup communications, multicast MANET routingis also of particular interest. We are adaptingongoing work on Simplified Multicast Forwarding(SMF) protocol [MDC04] to our evaluation

models. An effective MANET multicast routingcapability will be important since interagent groupcommunications and persistent sharing of teamenvironment or plan information is called for bymany interagent designs. Multicast may also beadjusted and optimized for more localized agentcommunication designs by changing scope rangeof the multicast dissemination.

MODELING APPROACH AND TOOLSDue to analytical complexity, research ofMANETsystems is often done by simulation and sometimesvia emulation. Researchers model the relateddynamic network environments, network protocolstacks, applications, and node mobility withinspecialized simulation or emulation systems. Inour work, we have extended some existingMANET test environments to include theintroduction of non-abstracted agent software andmiddleware components.

nme nA

PS.._

S G n Vat

Figure 3: Example Modeling Approach

Figure 3 depicts part of the modeling approach andtoolset we have developed. As shown, we use across platform prototype code developmentapproach that results in the same software able torun in both simulation and emulation environments[CMW03]. The composite system underexamination includes the agent software at the toplayer, optional middleware, and then associatednetwork stack and MANET protocols below. Atthe MANET layer, we have added the addition ofSMF functionality providing an optional multicastrouting capability within the MANETenvironment. Agents are useful in real worldenvironments because they both sense and react tocomplex environments. To better supportenvironmental modeling we have adapted anexisting set of MANET tools to provide dynamicenvironment control and stimulus to the MASsimulations [D04]. In the past the environment

3

channel was normally used to control MANETnode movement and position information inmobile simulations, but it is now also a channelthat can provide local environmental informationused by the agents (e.g., localized sensor stimulus,human input, etc). Once again the set of scenarios,traffic tools, and visualization tools equally applyto both simulation and emulation testing. InFigure 3, the optional middleware modeling layeris represented by the P2PS protocol layer [T05].We have modified P2PS interface code to run in aportable way in the real world and within the ns2simulation environment. Agents may use themiddleware services or can optionally directlycommunicate through standard network andMANET-optimized network layers. We are alsoworking on adding a multicast pipe capability toP2PS to be able to use the SMF MANET routingand possibly future additional reliable multicasttransport services.

EARLY AD HOC MAS STUDY RESULTSAs we have discussed there are numerous designframeworks and approaches being developed forMAS and the application areas are varied.Because of the rich taxonomy of application areasfor MAS this a "no one size fits all" design space.We have chosen to scope our investigation downto an area of team-based problem solving, withinitial emphasis on the study of distributed roleallocation algorithms and their effectivenesstowards achieving team goals. We accomplishedearly work in adapting existing agent-basesdsystems and frameworks including Machinett[MA], Control of Agent-Based System (CoABS)grid, and REPAST to enable early analysis andrefinement of the algorithms prior to integrationwith MANET environments [AM03].An example of our team-based problem solvingscenario is the prey/predator pursuit problem. Theprey/predator problem is a canonical example ofagent teamwork in the literature and we areadapting this model to further study by including aMANET communications model. We havemeasured coordination (and hence teamwork)quality as a function of interactions between agents(messages and collisions) to accomplish the task ofcapturing the preys.Figure 4 is an example visualization snapshot of anemulation scenario. Within figure 4, the icons ofthe soldiers represent predator and the monsters

represent prey. The circle depicted around theprey represents the capture zone of the preys. Inthis example, it takes 4 predators to successfullycapture a prey within the capture zone. Thereforeagents are required to perform teamwork in orderto satisfy the requirement of four complementaryand unique roles being taken to capture the prey.The links between the predators represents thereal-time network topology being calculated by theMANET routing protocol (in this case a variant ofOLSR). The predators and preys are mobilewithin the scenario and the predators use theMANET to form a mobile network to allow forinteragent coordination and communication. Theagents support the dynamic allocation of roles andagent teamwork by communicating environmentalstate and or local intention across the MANETnetwork.

Figure 4: Emulation Visualization Snapshot

In early studies, we began by examining variousrole allocation algorithms in a more limitedsimulation environment (i.e., REPAST). Weinduced simulated message loss and limitedcommunication range (to crudely represent facetsof MANET) to examine performance of thevarious role allocation approaches [AMC05]. Theinitial results from those experiments showed thatdistributed coordination by sharing state amongagents significantly outperformed algorithmsbased solely on communicating role switchingwith more localized and statistical methods. Theinteresting fact was that this advantage remainedtrue with high network loss rates as well as whenthe system scaled in terms of numbers of agents.

4

0.7

> 0.6

n 0.5a

0.4

~ .3

0.2000 0.1

010 15

Number of Agents

- DCOSDIDSA

20

Figure 5: Role Allocation Simulations

We are now pursuing similar experiments within amore detailed emulation and simulationenvironment. The mobile topology, routingperformance, contention, and congestion are beingmodeling with working protocol code and thewireless network environment and protocol stackwill be more realistic. Figure 5 shows an early setof results demonstrating that the DistributedConstraint Optimization (DCO) based upon theHungarian algorithm continues to outperform themore localized statistical role allocation methods'within the more detailed emulation environment.

Figure 6: Early Simulation and Emulation Results

MANET MULTICAST PROGRESS ANDAGENT USEWe have made significant recent progress infurther developed and applying SMF for multicastrouting within this project [MDC04]. New andimproved relay set algorithms have beenintroduced into SMF and a simulation model hasbeen completed and tested with larger networksthan supported within the emulator. Some agentrole allocation approaches and designs, such as the

' Distributed Stochastic Algorithm (DSA), SimpleDistributed Improvement (SDI)

Hungarian algorithm variant, require significantinteragent sharing of local environmentinformation. These types of algorithms often findmore optimal solutions to a complex problembecause of the more accurate agent view.However, this is accomplished at the cost ofincreased inter-agent communications. SMFreduces network overhead by providing anefficient means of multicast forwarding within adynamic, wireless network area making suchsolutions more viable within MANETs. We arepresently exploring the tradeoffs betweenimproved agent role allocation performance andthe cost of this additional overhead in variousscenarios. The example of Figure 6 uses SMFwithin the emulation experiment and shows thatthe additional network overhead is being utilizedin an effective way. As mentioned, protocols likeSMF can decrease the burden of overhead withinthe MANET operating area and make moreeffective agent communication possible. We planto study this tradeoff more as the network and theMAS scale in size.

SMF -s PEPS d.t. rate. prisio

Figurepipes

7: P2PS Middleware (unicast) vs. SMF

Figure 7 shows some early analysis of networkoverhead with seven agents, two preys, and onescout. All nodes are mobile and the agents andscout use MANET protocols to dynamically routedata within the changing topology. In one case,we use P2PS as a peer-to-peer protocol layerbetween agents. P2PS provides both unicastmessaging pipes and service discovery for theMAS software. In the second case, we appliedSMF as a mobile multicast forwarding directlybetween agents. This is more efficient as a team

5

r r~~~~~

400

c)

c) 30

100 _i

100 200 300

Ti- (s)

400 see 600

communication method and the curve representsthe resulting savings in network overhead fromusing SMF vs. unicast P2PS. In future work, wehope to combine the use of SMF and P2PS so thatthe middleware can take advantage of efficientmulticast forwarding when available in MANETs.

FUTURE WORKPart of the challenge of studying MAS systems isto measure the effectiveness of performance. Thisis even more challenging within highly complexenvironments with competing goals. Using theoutlined simulation and emulation testenvironments, future work will develop a set ofmore comprehensive tests and measures for avariety of agent teamwork scenarios beyond thepresent role allocation problems. Also, MANETenvironment variables can vary greatly andsignificantly affect the outcome of evaluationresults. As a network environment degrades, theremay be no one best answer, so in future work, weplan to investigate the possibility of agentadaptation to network conditions. As the networkdegrades in terms of delay, capacity, orconnectivity the agent teams may adapt theirdistributed decision methods to better matchnetwork conditions.

CONCLUSIONIn summery, we have presented some recent workbeing performed in the area of MAS and MANETtechnology research. The main scope of out workis studying team-based MAS problem solvingusing dynamic role allocation within the context ofMANET environments. Early analysis has shownthat MAS teamwork communications withinMANET can be improved through better designbetween the layers. The use and adaptation ofMANET multicast is one example. We have alsodiscovered design problem with presentmiddleware systems (e.g., JXTA[JX]) often usedby agent to provide abstracted service discoveryand communication services. Furtherimprovements are being examined relating toMANET application scenarios.We have developed and described a set of testtools including both simulation and emulation tofurther study detailed scenarios of combined MASand MANET operation. An environment channelwas developed to better control and exercisedistributed MANET experiments and to provide

agents with simulated external stimulus outside thenetwork domain. We have adapted a lightweightmiddleware framework for study and we areexamining services discovery and other featuresfor possible interagent use in MANET. We havealso adopted the SMF multicast routing prototypefor MANET and we are applying this as a meansfor more efficient inter-agent group multicasting.Presently, we are beginning detailed emulation andsimulation studies involving prey/predator agentscenarios within both the MANET simulation andemulation environments. We plan to study thetradeoffs of various distributed role allocationapproaches and the effectiveness of variousMANET protocol enhancements on agentperformance.Early results confirm our initial hypothesis thatclassic MAS and middleware communicationdesign assumptions may be invalid when appliedin MANET environments. More recently, otherresearchers are examining related areas and havereported similar issues. and Additional researchwill be aimed towards better understanding theperformance tradeoffs and improving designs in avariety of MAS scenarios and MANETenvironments.We would like to acknowledge the contributions ofother NRL scientists within the PROTEANTMResearch Group for their valuable input on thisproject. Justin Dean, Ian Downard, BrianAdamson, and Rick Jones all deserveacknowledgements.

REFERENCES[MDC04] Macker J. P., J. Dean , W. Chao,"Simplified Multicast Forwarding in Mobile Adhoc Networks", IEEE MILCOM 2004Proceedings, November 2004.[W02] Woolridge, M., An Introduction toMultiAgent Systems, John Wiley and Sons, Feb2002.[B04] Basagni, et al, Mobile Ad Hoc Networking,Chapter 9, IEEE Press, Wiley Interscience 2004.[CM99] S. Corson and J. Macker, "Mobile Ad HocNetworking (MANET): Routing ProtocolPerformance issues and EvaluationConsiderations," RFC2501, JAN 1999.[P01] Charles Perkins, editor, Ad Hoc Networking,Addison-Wesley. 2001.

6

[AM03] Abramson, M. Mittu, R. "MultiagentSystems in Open Environments", Workshop onChallenges of large-scale coordination, AAMAS2004, New York, NY. 2004[ACM05] Abramson, M. Chao, W. Mittu, R."Design & Evaluation of Distributed RoleAllocation Algorithms in Open Environments",submitted to IC-Al 2005.[CMW03] W. Chao, J. Macker, J Weston, "NRLMobile Network Emulator," IEEE MILCOM2003, Oct 2003.[D04] Downard, I., "Simulating Sensor Networksin NS-2", NRL Formal Report 5522-04-10, May2004.[TO5] http://www.trianacode.org/p2ps/. P2PS codeand documentation.[W04] Ian Wang. In Proceedings of 13th AnnualMardi Gras Conference - Frontiers of GridApplications and Technologies. Louisiana StateUniversity, pages 54-59, February 2005.[MA] Machinetta Informational website.http://teamcore.usc.edu/doc/Machinetta/[PR] Protean Research Group informationalwebsite. http://cs.itd.nrl.navy.mil/5522/[JX] http://wwwjxta.org/[CDCP05] Marco Conti, et al. Cross-LayerSupport for Group-Communication Applicationsin MANETs. IEEE ICPS REALMAN Workshop,July 14, 2005.

7