bertinorolecturenetworks

The Ant Colony Optimization (ACO) Metaheuristic:

a Swarm Intelligence Framework

for Complex Optimization Tasks

Part IV: ACO for Routing in Networks

Gianni Di Caro

[email protected]

IDSIA, USI/SUPSI, Lugano (CH)

Part IV:

ACO for routing in networks

1

Road map

✦ General structure and characteristics of ACO algorithms forrouting problems in telecommunication networks

✦ AntNet & AntNet-FA [Di Caro & Dorigo, 1998]: ACO forbest-effort routing in wired IP networks✧ Description of the algorithms and discussion of properties✧ Results of extensive simulation experiments and

comparison with several state-of-the-art routing algorithms

✦ AntHocNet [Di Caro, Ducatelle & Gambardella, 2004]: ACO forbest-effort routing in wireless mobile ad hoc networks✧ Characteristics and challenges of mobile ad hoc networks✧ Description of the algorithm and discussion of its properties✧ Results of extensive simulation experiments in realistic

settings, comparison with state-of-the-art

2

ACO well matches network problems

✦ Straightforward mapping:

✧ Decision points→ Network nodes

✧ Decision variables to learn (pheromone)→ Next hops (outlinks)

✧ Each node is like a single colony acting independently and socially

✦ The intrisically distributed nature of ACO (autonomous multipleagents, distributed decision policy, and stigmergic communications)can be fully exploited

✦ Networks are extremely dynamic environments calling for adaptivelearning and control systems

✦ Ants repeatedly construct solutions, realizing a parallel Monte Carlosimulation system which can be “cheaply” implemented online innetworks (but not in other real-world systems!)

✦ Adaptivity, self-organization, and robustness, are focal aspects inTraffic Engineering and Autonomic Communications

3

ACO-routing algorithms

✦ A number of routing algorithms have been designed after ACO andapplied with success to several adaptive routing problems in wiredand wireless networks

✦ Basic mechanisms in typical ACO-routing algorithms:

✧ Ant-like agents are proactively generated at the nodes tofind/check paths toward assigned destinations [probing packets]

✧ Ants move hop-by-hop according to a exploratory routing policybased on the local routing information [Monte Carlo path sampling]

✧ After reaching their destination, ants retrace their path and updatenodes’ routing information according to the quality of the path

✧ Routing information = statistical estimates of the time-to-go to thedestination maintained in pheromone arrays [distance vector]

✧ Data are probabilistically spread over the paths according to theirestimated quality as stored in the pheromone variables

4

Properties of ACO-routing algorithms

✦ Nice properties: good adaptivity, robust to failures, multiplepath routing, automatic load balancing, good scalability

✦ Bad properties: to properly track all the changes in the networkthe frequency of proactive generation should be high and thismight be a problem with scarce bandwidth

5

AntNet & AntNet-FA

✦ First ACO algorithms for datagram networks [Di Caro & Dorigo,1997, 1998] (Schoonderwerd et al. applied ACO totelephone-like networks in 1996)

✦ General architecture: straightforward application of ACO

✦ Careful design of each component

✦ State-of-the-art performance

✦ Reference algorithms for a lot of subsequent algorithms

✦ AntNet-FA is an improvement of AntNet

6

AntNet: algorithm description (1)

✦ Proactive generation of Forward Ants

✦ An ant faithfully simulate a data packet✧ Discover/sample a good path (agents explore. . . )

✧ Update routing information (. . . data packets exploit)

✦ Forward ants maintain a private memory of each visited nodeand of the time of the visit (loops are removed)

Forward Ant

2 40 1 3

T1 T T 3 4T2

Memory

2 40 1 3

7


0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Da

ta-r

ou

ting

ta

ble

Ant-routing table

y=x1.4

y=x

1

2

3

...

L

Network Nodes

........

........

DelayModels

Data ........

....

RoutingTable

Parametric

Link Queues

Pheromone Table

21

L1

1N

2N

LN

22

L2

1211

1 2 N

Network Nodes

Nei

ghbo

r no

des

Network Node

Exponential Mean Exponential Variance Window Best Window Count

MMM

τ

τ τ

τ

τ

ττ

τ

τ

8


✦ Next hop nodes are selected according to a stochasticdecision policy πε parametrized by:✧ Pheromone variables τ ij = Measures of end-to-end delay✧ Heuristic variables ηi = Status of local link queues✧ Memory of the nodes visited so far

ηij(t) ∝1

expected waiting time at j’s queue

πdε (i, j; t) ∝ ατ d

ij(t) + (1− α)ηij(t) Link j

Node i

Table

Pheromone

Queues

Link

Memory

???

9


✦ Reached destination d, the Forward Ant turns into a BackwardAnt and retraces the path back to the source node

✦ At each node i the Backward Ant, coming from neighbor j:

✧ Updates the Parametric Delay ModelMdi

✧ Evaluates the path: rdij = J(Ti→d,M

di )

✧ Updates the pheromone tableand the routing table with rd

ij

✧ Also intermediate nodes✧ Pure Monte Carlo updating

Forward Ant Path

Backward Ant

2 40 1 3

01 23 34

Parametric Delay Model

Memory

Memory

T 12T T T

Pheromone Table

10

Path evaluation and updating: formulas

✦ Non-stationarity and incomplete state information (aliases)make necessary the use of local adaptive models to provide anadvisory evaluation:

rdij = c1

(

W dbest

Ti→d

)

+ c2F (Ti→d, Idsup, I

dinf ), rd

ij ∈ [0, 1]

✦ The pheromone variables associated to the node j theBackward Ant comes from receives a positive reinforcement:

τdij ← τd

ij + rdijα(1− τd

ij)

✦ All the other possible next hops n ∈ N (i) receive, by probabilitynormalization, a negative reinforcement:

τdin ← τd

in − rατdin

11

AntNet-FA: improving AntNet

✦ In AntNet the path trip time Ti→j is the actual timeexperienced by the ant

✦ In AntNet-FA Forward ants make use of high priority queues(they fly!)

✦ The trip time Ti→d is calculated during the backward journey,estimating the current waiting time at the link j:

Ti→j = dj +qj

bj

✦ Ti→j is an up-to-date estimate of the time to hop from i→ j

12

What is the outcome of the ant actions?

✦ Proactive exploration and route adaptation

✦ At each node a bundle of datagram paths are available

✦ Each choice has a goodness value (pheromone) which isonline adapted to the traffic patterns ↓

✦ Data are spread stochastically (multi-path routing)

✦ The less good paths are backup paths

✦ Automatic load balancing

✦ Robust wrt ant failures and to parameter setting

✦ No global propagation of local estimates

✦ Active non-local information gathering

✦ Shortcomings: TCP, short-lived loops, topological adaptivity13

Experimental setup for AntNet(-FA)

✦ Extensive simulation studies

✦ Realistic experimental setup for:

✧ Network topology and physical characteristics

✧ Protocol for data transmission

✧ Spatial and Temporal Traffic Patterns

✧ Algorithms to compare the performances

14

Networks

NSFNET-T1 (14, 21, 1.5) - US backbone (1987)

1

3

10

2

4

56

7

8

9

14

13

11

9

7

15 9

81411

13

9

20

16

5

7

47

7

77

8

5

12

NTTnet (57, 162, 6) - NTT network (?)

Simple Net (8, 9, 10)

8

1 2

43

5

6

7

6x6 Net (36, 99, 10) - (Boyan & Littman, 1994)

Random Nets (100/150, ≈ 3, 10)

15

Traffic patterns

Data Transmission Protocol✦ Best-effort Datagram traffic

✦ IP-like protocol

✦ Discarding packet for no buffer space

✦ Failure situations not considered

✦ No arrival acknowledgment or errornotification packets

✦ Simple Flow control mechanism based ona static production window

Data sessions✦ Negative exp distribution for sessions’

inter-arrival times, global size, and packetsizes

✦ Traffic patterns obtained by thecombination of three basic traffic types:

✧ Poisson (Spatially Uniform (UP) andRandom (RP))

✧ Constant Bit Rate (CBR)

✧ Hot Spots (HS)

16

Algorithms used for comparison

Static - Link-state✦ OSPF: Minimum cost paths, current IGP Internet algorithm

Adaptive - Link-state✦ SPF: Link-state prototype, Adaptive link costs, last ARPANET algorithm

Adaptive - Distance-Vector

✦ BF: Asynchronous Bellman-Ford prototype, Adaptive link costs, ARPANET

✦ Q-Routing: Asynchronous Bellman-Ford with online updates and Q-Learning-like rule

✦ PQ-Routing: Q-Routing with a system to learn a model of the link queues

Ideal✦ Daemon: Access the state of all the net queues, empirical bound on performance

17

Results - NTTnet UP Load

Throughput

0

5

10

15

20

25

30

35

40

45

AntNet OSPF SPF BF Q-R PQ-R Daemon

Th

rou

gh

pu

t (1

06 b

it/s

ec)

3.1 3 2.9 2.8 2.7

End-to-end delay 90-th percentile

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

AntNet OSPF SPF BF Q-R PQ-R Daemon90

-th

pe

rce

ntile

of

pa

cke

t d

ela

ys (

se

c) 3.1 3 2.9 2.8 2.7

18

Results - NTTnet UPHS Load

Throughput

0

5

10

15

20

25

30

35

40

45

50


Th

rou

gh

pu

t (1

06 b

it/s

ec)

4.1 4 3.9 3.8 3.7


0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0


-th

pe

rce

ntile

of

pa

cke

t d

ela

ys (

se

c) 4.1 4 3.9 3.8 3.7

19

Results - NSFNET RP Load

Throughput

0

2

4

6

8

10

12


Th

rou

gh

pu

t (1

06 b

it/s

ec)

2.8 2.7 2.6 2.5 2.4


0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0


-th

pe

rce

ntile

of

pa

cke

t d

ela

ys (

se

c) 2.8 2.7 2.6 2.5 2.4

20

Results - Load Variation

NSFNET: UP Load Variation

6.0

8.0

10.0

12.0

14.0

16.0

Th

rou

gh

pu

t (1

06 b

it/s

ec)

OSPFSPF

BFQ-R

PQ-RAntNet

Daemon

30.0

40.0

50.0

60.0

200 300 400 500 600 700 800 900 1000

Pa

cke

t D

ela

y (

se

c)

Simulation Time (sec)

MSIA=3.0, MPIA=0.3, HS=4, MPIA-HS=0.04

NTTnet: UP Load Variation

15.0

25.0

35.0

45.0

55.0

Th

rou

gh

pu

t (1

06 b

it/s

ec)

OSPFSPF

BFQ-R

PQ-RAntNet

Daemon

0.0

0.2

0.4

0.6

0.8

200 300 400 500 600 700 800 900 1000

Pa

cke

t D

ela

y (

se

c)


MSIA=4.0, MPIA=0.3, HS=4, MPIA-HS=0.05

21

100-Nodes RandomNets - UP Load

Throughput

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

0 100 200 300 400 500 600 700 800 900 1000

Th

rou

gh

pu

t (1

06 b

it/s

ec)


OSPFSPF

BFQ-R

PQ-RAntNet

AntNet-FA

Delay distribution

0.0

0.2

0.4

0.6

0.8

1.0

0 0.25 0.5 0.75 1 1.25 1.5

Em

piric

al D

istr

ibu

tio

n

Packet Delay (sec)

OSPFSPF

BFQ-R

PQ-RAntNet

AntNet-FA

(MSIA = 15.0, MPIA = 0.005)

22

150-Nodes RandomNets - RP Load

Throughput

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0 100 200 300 400 500 600 700 800 900 1000

Th

rou

gh

pu

t (1

06 b

it/s

ec)


OSPFSPF

BFQ-R

PQ-RAntNet

AntNet-FA

Delay distribution

0.0

0.2

0.4

0.6

0.8

1.0

0 0.5 1 1.5 2 2.5 3 3.5 4

Em

piric

al D

istr

ibu

tio

n

Packet Delay (sec)

OSPFSPF

BFQ-R

PQ-RAntNet

AntNet-FA

(MSIA = 10.0, MPIA = 0.005)

23

Routing Overhead

Routing Overhead (10−3) for some of the realized experiments


NSF - UP 2.39 0.15 0.86 1.17 6.96 9.93 0.00

NSF - RP 2.60 0.16 1.07 1.17 5.26 7.74 0.00

NSF - UPHS 1.63 0.15 1.14 1.17 7.66 8.46 0.00

NTT - UP 2.85 0.14 3.68 1.39 3.72 6.77 0.00

NTT - UPHS 3.81 0.15 4.56 1.39 3.09 4.81 0.00

Routing Overhead = Ratio between the generated routing traffic and the total available bandwidth

For all the considered algorithms the routing overhead is quite low

24

AntNet Power Vs. Routing Overhead

Normalized Power Vs. Routing Overhead

for increasing (per-node) rates of ant generation

0.0

0.2

0.4

0.6

0.8

1.0

0.001 0.01 0.1 1 10 100

Ant

Net

Nor

mal

ized

Pow

er V

s. R

outin

g O

verh

ead

Node Ants Launching Rate (sec -1)

25

AntNet: Const. Vs. Non-Const. Reinforcements

Constant Vs. Non-Constant reinforcements

for increasing (per-node) rates of ants production.

4e+06

6e+06

8e+06

1e+07

1.2e+07

1.4e+07

1.6e+07

1.8e+07

2e+07

0.001 0.01 0.1 1 10 100

Tro

ughp

ut /

90-t

h D

elay

Per

cent

ile (

106 b

it/se

c2 )

Node Ants Launching Rate (sec -1)

AntNet with r ≠ Const.AntNet with r = Const.

Non-constant reinforcements give improvements ranging from few percent to more than 40%26

Mobile Ad Hoc Networks (MANETs): Definition

✦ No fixed infrastructure or centralized control

✦ Nodes can move, join, and leave the network at any time

✦ Nodes can communicate each other via wireless interfaces

✦ All nodes are equal and must serve as routers for each other

✦ Data packets are forwarded in a multi-hop fashion27

MANETs: Applications and Challenges

✦ Bring connectivity in infrastructureless areas

✦ A fundamental building block towards flexible, pervasive,ubiquitous, and seamless networking

✦ Constant topology and traffic variations

✦ Access control to the shared wireless medium reduces theeffective available bandwidth (e.g., IEEE 802.11)

✦ Noise and transmission errors

28

Routing in MANETs: Requirements

✦ Work in fully distributed and localized way (Self-organizing)

✦ Adapt to the continual topological and traffic variations

✦ Robust to errors and losses

✦ Scalable to number of nodes and traffic load (limited overhead)

A solution based on Swarm Intelligence looks as a natural choice!

29

AntHocNet: Hybrid, Monte Carlo+Bootstrapping

✦ AntHocNet is a hybrid ACO-routing algorithm:

✧ Reactive: paths are only setup at the starting of a session

✧ Proactive: during the course of a session paths aremaintained and improved proactively

✦ AntHocNet makes use of two learning mechanisms:

✧ Monte Carlo sampling and updating of full paths with ants

✧ Local exchange and boostrapping of routing information

30

AntHocNet: Routing (Pheromone) Tables

. . . . . . .

11

23 73 93

21 51 71 81Q Q Q Q

Q Q Q

Kno

wn

Des

tina

tion

s

Neighbor nodes

1n 8n7n5nQ Q Q Q

Q

✦ Routing tables have an entry for each known destination andeach known neighbor

✦ Each entry is a pheromone variable indicating the estimatedgoodness/quality Qnd of a routing decision n for a destination d

✦ Qnd = F (end-to-end delay, number of hops)

✦ The task of ant agents is to build/update pheromone variables,which are used for routing according to a stochastic policy

31

AntHocNet: Reactive Path Setup

✦ At the start of a session, reactive forward ants are sent fromthe source to the destination

✦ They are broadcast, or follow pheromone whenever possible

✦ Ants are filtered at intermediate nodes to balance number ofpaths and overhead

32

AntHocNet: Stochastic Data Routing

✦ Data packets are routed stochastically according to thepheromone values: links with high pheromone have higherselection probability

✦ This way, data follow paths with lower delay and lower numberof hops, avoiding areas of high congestion

✦ If pheromone values are kept up-do-date, this leads toautomatic load balancing

33

AntHocNet: Dealing with Link Failures

1. Discovered via missed pheromone messages or failed unicast

2. Broadcast message to notify the change in the routing table

3. Try local repair if no alternative is available for data packet

4. Local repair ants travel to destination like reactive ants but withlimited broadcasting

5. If local repair fails the packet is discarded and a notification isbroadcast, otherwise the communication can keep goingwithout the need to notify the source

34

Simulation Scenarios

All test scenarios are obtained by varying the parameters of thefollowing base scenario:

✦ Area: Flat, open space, rectangular of 3000× 1000 m2

✦ Mobility: Random Waypoint (RWP), speed [0,20] m/s,pause time 30s

✦ Data traffic: starting from 20 Constant Bit Rate sources (CBR),64 bytes/s, UDP at transport layer

✦ Radio propagation: two-ray and free space path loss, no fading

✦ MAC: single channel IEEE 802.11b DCF with 2 Mbps

35

Evaluation Methodology

✦ Measures of effectiveness:✧ Delivery ratio✧ Average end-to-end delay✧ Average delay jitter

✦ Measure of efficiency:✧ Routing overhead: control packets vs. correctly delivered data

packets

✦ “Swarm” properties:✧ Scalability: wrt to number of nodes and data sessions✧ Short- and Long-term Adaptivity:

✦ Benchmark algorithm:✧ AODV, a state-of-the art algorithm and common standard for

comparison

✦ Simulation environment:✧ QualNet, a realistic packet-level commercial simulator 36

Mobility (Pause Time)

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0 15 30 60 120 240 480

Pac

ket d

eliv

ery

ratio

Pause time (sec)

AntHocNetAODV

0.04

0.05

0.06

0.07

0.08

0.09

0.1

0.11

0.12

0.13

0 15 30 60 120 240 480

Ave

rage

end

-to-

end

pack

et d

elay

(se

c)

Pause time (sec)

AntHocNetAODV

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

0 15 30 60 120 240 480

Ave

rage

del

ay ji

tter

(sec

)

Pause time (sec)

AntHocNetAODV

18

20

22

24

26

28

30

32

34

36

38

0 15 30 60 120 240 480

Rou

ting

over

head

Pause time (sec)

AntHocNetAODV

Average end−to−end delay

Average delay jitter Routing overhead

Delivery Ratio

37

Node Density (Number of Nodes)

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

50 75 100 125 150

Pac

ket d

eliv

ery

ratio

Number of Nodes

AntHocNetAODV

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

50 75 100 125 150

Ave

rage

end

-to-

end

pack

et d

elay

(se

c)

Number of Nodes

AntHocNetAODV

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

50 75 100 125 150

Ave

rage

del

ay ji

tter

(sec

)

Number of Nodes

AntHocNetAODV

4

4.5

5

5.5

6

6.5

7

7.5

8

8.5

9

9.5

50 75 100 125 150

Rou

ting

over

head

Number of Nodes

AntHocNetAODV



Delivery Ratio

38

Scaling (Number of Nodes)

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

100 200 300 400 500 600 700 800

Pac

ket d

eliv

ery

ratio

Number of Nodes

AntHocNetAODV

0

0.05

0.1

0.15

0.2

0.25

0.3

100 200 300 400 500 600 700 800

Ave

rage

end

-to-

end

pack

et d

elay

(se

c)

Number of Nodes

AntHocNetAODV

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

100 200 300 400 500 600 700 800

Ave

rage

del

ay ji

tter

(sec

)

Number of Nodes

AntHocNetAODV

0

20

40

60

80

100

120

140

100 200 300 400 500 600 700 800

Rou

ting

over

head

Number of Nodes

AntHocNetAODV



Delivery Ratio

39

Scaling (Number of Hot Spots)

0.76

0.78

0.8

0.82

0.84

0.86

0.88

0.9

0 5 10 15 20 25 30 35 40 45 50

Pac

ket d

eliv

ery

ratio

Number of Hot Spots

AntHocNet 20 SessionsAntHocNet 50 Sessions

AODV 20 SessionsAODV 50 Sessions

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

0 5 10 15 20 25 30 35 40 45 50

Ave

rage

end

-to-

end

pack

et d

elay

(se

c)

Number of Hot Spots



0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0 5 10 15 20 25 30 35 40 45 50

Ave

rage

del

ay ji

tter

(sec

)

Number of Hot Spots



6

8

10

12

14

16

18

20

22

24

0 5 10 15 20 25 30 35 40 45 50

Rou

ting

over

head

Number of Hot Spots





Delivery Ratio

40

Short-term Adaptivity (Traffic Burst)

1

1.5

2

2.5

3

3.5

4

4.5

5

0 100 200 300 400 500 600 700 800 900

Thr

ough

put (

pack

ets/

sec)

Time (sec)

AntHocNetAODV

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0 100 200 300 400 500 600 700 800 900

Ave

rage

end

-to-

end

pack

et d

elay

(se

c)

Time (sec)

AntHocNetAODV

Average end−to−end delayDelivery Ratio

41

Basic references

✦ G. Di Caro and M. Dorigo, "AntNet: Distributed Stigmergetic Control for CommunicationsNetworks", Journal of Artificial Intelligence Research (JAIR), Vol. 9, pages 317-365, 1998

✦ G. Di Caro and M. Dorigo, "Two Ant Colony Algorithms for Best-Effort Routing in DatagramNetworks", Proceedings of the Tenth IASTED International Conference on Parallel andDistributed Computing and Systems (PDCS’98), Las Vegas, Nevada, IASTED/ACTA Press,pages 541-546, 1998

✦ G. Di Caro, "Ant Colony Optimization and its Application to Adaptive Routing inTelecommunication Networks", Ph.D. thesis, Faculté des Sciences Appliquées, Université Librede Bruxelles, Brussels, Belgium, 2004

✦ G. Di Caro, F. Ducatelle and L.M. Gambardella, "AntHocNet: an adaptive Nature-inspiredalgorithm for routing in mobile ad hoc networks", European Transactions onTelecommunications, Vol. 16(5), 2006 (to appear)

✦ F. Ducatelle, G. Di Caro and L.M. Gambardella, "Using ant agents to combine reactive andproactive strategies for routing in mobile ad hoc networks", International Journal ofComputational Intelligence and Applications, 2006 (to appear)

You can downloand these papers, as well as other papers related to AntNet and AntHocNet, from:http://www.idsia.ch/~gianni/my_publications.html.

These papers also contain extensive pointers and discussions to the quite vast literature related toACO-routing implementations

42

http://www.idsia.ch/~gianni/my_publications.html

The End

Thanks for listening!

43

bertinorolecturenetworks

Documents

antrouting table

typical acorouting algorithms

multiple path routing

number of routing algorithms

update routing information

datarouting table y

besteffort routing

adaptive routing problems