Linux Traffic ControlLinux Traffic Control

Linux Traffic Control EssentialsLinux Traffic Control Essentials

TCNG OverviewTCNG Overview

Study of a Token Bucket ScenarioStudy of a Token Bucket Scenario

Papadimitriou Panagiotis Papadimitriou Panagiotis 17/06/200417/06/2004

Components of Linux Traffic Components of Linux Traffic ControlControl

The basic components of the Linux The basic components of the Linux QoSQoS

architecture are:architecture are:

• Queuing DisciplinesQueuing Disciplines

• ClassesClasses

• FiltersFilters

Queuing DisciplinesQueuing Disciplines

Queuing Disciplines (qdiscs) have:Queuing Disciplines (qdiscs) have:

• an an enqueueenqueue function, called whenever the network layer of function, called whenever the network layer of the operating system wants to transmit a packet, andthe operating system wants to transmit a packet, and

• aa dequeuedequeue function, called when the device is able to function, called when the device is able to transmit the next packettransmit the next packet

The available qdiscs can be divided into two groups:The available qdiscs can be divided into two groups:

• The simple qdiscs which have no inner structure, known as The simple qdiscs which have no inner structure, known as queuesqueues. These can be used to shape traffic for an entire . These can be used to shape traffic for an entire interface, without any subdivisions.interface, without any subdivisions.

• The qdiscs which have classes, known as The qdiscs which have classes, known as schedulersschedulers. These . These are very useful when there are different kinds of traffic which are very useful when there are different kinds of traffic which should have differing treatment.should have differing treatment.

Path of a Data Socket through Path of a Data Socket through the Linux Network Stackthe Linux Network Stack

QueuesQueues

The first group of The first group of queuing disciplinesqueuing disciplines includes: includes:

• pfifo_fastpfifo_fast: a 3-band priority FIFO queue (default): a 3-band priority FIFO queue (default)

• sfqsfq: a stochastic fair queuing discipline: a stochastic fair queuing discipline

• tbftbf: a Token Bucket Filter queue: a Token Bucket Filter queue

• redred: implements the Random Early Detection Behavior : implements the Random Early Detection Behavior (RED)(RED)

• gredgred: a generalized RED implementation used for : a generalized RED implementation used for DiffServDiffServ

supportsupport

• ingressingress: a queue used for policing ingress traffic: a queue used for policing ingress traffic

SchedulersSchedulers

The second group of The second group of queuing disciplinesqueuing disciplines includes: includes:

• cbqcbq: implementation of the class based queuing link-: implementation of the class based queuing link-sharingsharing

schemescheme

• atmatm: a special qdisc which supports the re-direction of : a special qdisc which supports the re-direction of flowsflows

to ATM virtual channelsto ATM virtual channels

• cszcsz: a Clark-Shenker-Zhang scheduling discipline: a Clark-Shenker-Zhang scheduling discipline

• dsmarkdsmark: qdisc for DiffServ support (uses DSCP): qdisc for DiffServ support (uses DSCP)

• wrrwrr: a Weighted Round Robin scheduler: a Weighted Round Robin scheduler

Sample qdisc which has innerSample qdisc which has innerclasses, filters and qdiscsclasses, filters and qdiscs

Sample Scenario for Traffic ControlSample Scenario for Traffic Control

• A small company has a 10 Mbit/s link which connects its workstations and one FTP server to an Internet service provider.

• Since bandwidth is a scarce resource the company wants to limit the share of the FTP traffic to 20% and at times where less bandwidth is needed by FTP the rest should be available for the workstations. On the other hand FTP traffic must never exceed its 20% share even if the rest of the bandwidth is currently unused because the companies.

• ISP charges extra for any bandwidth consumed above a rate of 2 Mbit/s. In order to solve this problem, a Linux router is installed at the edge of the corporate network.

Traffic Control ConfigurationTraffic Control Configuration

The first Ethernet interface of the Linux router (eth0) is connected to the ISP,

the second interface (eth1) is the link towards the internal network. Since it is

only possible to limit outgoing traffic, the setup consists of two parts:

• the CBQ configuration limiting the outgoing traffic on eth0 (the downstream” traffic from the internal network’s point of view), and

• a second part limiting outgoing traffic on eth1 (the “upstream”).

Introduction to TCNGIntroduction to TCNG

The Traffic Control Next Generation (TCNG) project focuses on:

• providing a compact and user-friendly configuration language, in which traffic control systems can be expressed in an intuitive way

• supporting hardware accelerators in traffic control

TCNG is comprised by two major components:

• the Traffic Control Compiler (TCC)

• the Traffic Control Simulator (TCSIM)

Traffic Control CompilerTraffic Control Compiler

• TCNG language is closely modelled after common programming languages, such as C, Perl or Java.

• Consequently, learning effort is reduced for anyone who is familiar with one of these languages.

• Traffic Control Compiler translates configuration scripts from the TCNG language into a multitude of output formats used to configure traffic control subsystems.

TCC in OperationTCC in Operation

Traffic Control Compiler:

• gets its input from a script program

• invokes the appropriate input parser to translate the configuration data into a common internal data structure

• invokes one or more output generators (named “targets”) to issue commands to the corresponding output processor(s)

Finally, output processors translate the output from tcc into actions

understood by lower-level components.

TCC Internal Structure & InterfaceTCC Internal Structure & Interface

Traffic Control SimulatorTraffic Control Simulator

Traffic Control Simulator is used to simulate the behavior of Linux

Traffic Control at a very high level of detail.

Traffic Control Simulator has been developed mainly for thefollowing purposes:

• validation of configurations generated by tcc

• development of configuration scripts

• testing of traffic control components

TCSIM in Operation (1)TCSIM in Operation (1)

Traffic Control Simulator:

• directly supports configuration using the standard traffic control language (tc), and

• it supports the new TCNG language by automatically invoking TCC, and integrating its output

Furthermore, Traffic Control Simulator:

• combines the original traffic control code from the Linux kernel with the user-space code of the configuration utility tc, and

• adds the framework for communication among them, plus an event-driven simulation engine

TCSIM in Operation (2)TCSIM in Operation (2)

• The resulting program runs entirely in user space, but executes almost exactly the same code as a “real system”, approximating the behavior of traffic control in a Linux system much more accurately than a more general simulator (e.g. NS-2) would.

Traffic Control Simulator:

• processes a script defining the system configuration and the data to send, and

• generates a message trace, which can then be processed to obtain statistics or graphs

TCSIM Internals and Helper TCSIM Internals and Helper ProgramsPrograms

TCNG Example: Steps 1-2TCNG Example: Steps 1-2

Step 1: We write the following TCNG code in the file: example.tc

dev eth0 { egress {

drop if tcp_sport != PORT_HTTP; }

}

Step 2: We run tcc to convert the TCNG configuration to tc commands. We save the output in the file:

example.sh

tcc example.tc > example.sh

TCNG Example: After Step 2TCNG Example: After Step 2

After Step 2 the file example.sh contains the following tc configuration:

tc qdisc add dev eth0 handle 1:0 root dsmark indices 1 default_index 0

tc filter add dev eth0 parent 1:0 protocol all prio 1 handle 1:0:0 u32 divisor 1

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u8 0x6 0xff at9 offset at 0 mask 0f00 shift 6 eat link 1:0:0

tc filter add dev eth0 parent 1:0 protocol all prio 1 handle 1:0:1 u32 ht 1:0:0match u16 0x50 0xffff at 0 classid 1:0

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u8 0x6 0xff at9 classid 1:0 police index 1 rate 1bps burst 1 action drop/drop

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0 at

0 classid 1:0

TCNG Example: Step 3TCNG Example: Step 3

Step 3: We define a simulation scenario in the file: example.tcsim

with one interface called eth0, running at 100 Mbps. The

simulation scenario consists of sending two packets.

#include “packet.def”

#include “ports.tc”

dev eth0 100 Mbps { #include “example.tc”

}

send TCP_PCK($tcp_sport = PORT_HTTP);send TCP_PCK($tcp_sport = PORT_SSH);

end

TCNG Example: Step 4TCNG Example: Step 4

Step 4: We run the simulation with tcsim:

tcsim –s 22 example.tcsim

The output looks like this:

0.000000 E : 0x80bd560 40 : eth0: 45000028 00000000 40060000

0a000001 0a000002 0050 ...0.000000 D : 0x80bd560 40 : eth0: 45000028 00000000

400600000a000001 0a000002 0050 ...0.000000 E : 0x80bd870 40 : eth0: 45000028 00000000

400600000a000001 0a000002 0016 ...0.000000 * : 0x80bd870 40 : eth0: enqueue returns POLICED (3)

TCNG Example: Steps 5-6TCNG Example: Steps 5-6

Step 5: We verify that the configuration did indeed work: The first packet was enqueued (“E”), and then dequeued (“D”). When trying to enqueue the second packet, it is rejected.

Step 6: We can try this example on a live system. We execute the tc commands to create the configuration in the kernel:

sh example.sh

A more comprehensive TCNG example A more comprehensive TCNG example (1)(1)

This example illustrates most of the elements found in a typical TCNG configuration:

dev "eth0" { egress {

class (<$high>) if tcp_dport == PORT_HTTP; class (<$low>) if 1;

prio { $high = class (1) { fifo (limit 20kB); }

$low = class (2) { fifo (limit 100kB); } } }}

A more comprehensive TCNG example A more comprehensive TCNG example (2)(2)

The dev and egress lines dev "eth0" {

determine what is being configured: egress {

i.e. the egress (outbound) side of the network interface eth0.

The configuration consists of two parts:

• the classification: class (<$high>) if tcp_dport == PORT_HTTP;

class (<$low>) if 1;

• the setup of the queuing system: prio {

$high = class (1) { fifo (limit

20kB);} $low = class (2) { fifo (limit 100kB);}

In this example, we use a priority scheduler with two classes for the

priorities “high” and “low”.

A more comprehensive TCNG example A more comprehensive TCNG example (3)(3)

In this configuration, packets:

• with TCP destination port 80 (HTTP) are sent to the high priority class,

• while all other packets (if 1;) are sent to the low priority class

The queuing part defines the queuing discipline for static priorities,

with the two classes:

• Inside the high priority class, there is another queuing discipline: a simple FIFO with a capacity of 20 KB.

• Likewise, the low priority class contains a FIFO with 100 KB.

A more comprehensive TCNG example A more comprehensive TCNG example (4)(4)

The compilation of this TCNG code results in the following tc configuration:

tc qdisc add dev eth0 handle 1:0 root dsmark indices 4 default_index 0

tc qdisc add dev eth0 handle 2:0 parent 1:0 prio

tc qdisc add dev eth0 handle 3:0 parent 2:1 bfifo limit 20480

tc qdisc add dev eth0 handle 4:0 parent 2:2 bfifo limit 102400

tc filter add dev eth0 parent 2:0 protocol all prio 1 tcindex mask 0x3 shift 0

tc filter add dev eth0 parent 2:0 protocol all prio 1 handle 2 tcindex classid 2:2

tc filter add dev eth0 parent 2:0 protocol all prio 1 handle 1 tcindex classid 2:1

tc filter add dev eth0 parent 1:0 protocol all prio 1 handle 1:0:0 u32 divisor 1

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u8 0x6 0xff at 9

offset at 0 mask 0f00 shift 6 eat link 1:0:0

tc filter add dev eth0 parent 1:0 protocol all prio 1 handle 1:0:1 u32 ht 1:0:0

match u16 0x50 0xffff at 2 classid 1:1

tc filter add dev eth0 parent 1:0 protocol all prio 1 u32 match u32 0x0 0x0 at 0

classid 1:2

Simulation OutputSimulation Output

By default, TCSIM prints a message:

• whenever a packet is enqueued or dequeued, or

• when some exceptional condition (e.g. an error) occurs.

This output can be post-processed:

• to extract statistical data,

• or to generate a graphical representation of traffic characteristics

TCSIM can also provide more detailed information on the inner workings

of the traffic control subsystem, which is useful:

• for testing configurations, and

• the development of new traffic control elements

Pretty-printing Traces (1)Pretty-printing Traces (1)

The script tcsim_pretty can be used to format traces in a morehuman-readable way.

Running the simulation script example.tcsim with the command syntax:

tcsim example.tcsim

produces the following output:

0.000000 E : 0x93a87c8 40 : eth0: 45000028 00000000 40060000

0a000001 0a000002 00500000 00000000 00000000 50000000 00000000

0.000000 D : 0x93a87c8 40 : eth0: 45000028 00000000 40060000

0a000001 0a000002 00500000 00000000 00000000 50000000 00000000

0.000000 E : 0x93a88c0 40 : eth0: 45000028 00000000 40060000

0a000001 0a000002 00160000 00000000 00000000 50000000 00000000

0.000000 * : 0x93a88c0 40 : eth0: enqueue returns POLICED (3)

Pretty-printing Traces (2)Pretty-printing Traces (2)

Running the same simulation script with the following command syntax:

tcsim example.tcsim | tcsim_pretty

produces a more readable output:

----- 0.000000 ------------------------------------------------------------------------

0x9d207c8 E 40: eth0: 45000028 00000000 40060000 0a000001 0a000002

+ 00500000 00000000 00000000 50000000 00000000

= D 40: eth0: 45000028 00000000 40060000 0a000001 0a000002

+ 00500000 00000000 00000000 50000000 0000000

0x9d208c0 E 40: eth0: 45000028 00000000 40060000 0a000001 0a000002

+ 00160000 00000000 00000000 50000000 00000000

= * eth0: enqueue returns POLICED (3)

Output Filtering (1)Output Filtering (1)

Enqueue and dequeue records can be selected in trace output

with the tcsim_filter script.

Additional filtering is supported, according to a selection of fields.

The following fields are recognized:

• tos: TOS byte

• len: Total length field

• src: Source IP address

• dst: Destination IP address

• sport: Source port (TCP or UDP)

• dport: Destination port (TCP or UDP)

• dev: Device name (e.g. eth0)

Output Filtering (2)Output Filtering (2)

When printing records, each line contains:

• the time

• the ID string

• the packet length in bytes

The tcsim_filter script supports counting the results instead

of printing data points on standard output.

In this case, the records with the same ID string are counted.

Examples of Output FilteringExamples of Output Filtering

Running the simulation script dsmark+policing with the command syntax:

tcsim dsmark+policing | tcsim_filter -c tos

produces the following output: D:00 201 D:b8 139

E:00 201 E:01 201

Likewise,

tcsim dsmark+policing | tcsim_filter -c tos=0xb8

produces the output: D 139

Graphical OutputGraphical Output

Filtered output can be further processed with the script tcsim_plot,

which uses gnuplot to generate plots.

The following plot types are available:

• rate: Bit rate (based on the inter-arrival time)

• iat: Packet inter-arrival time

• cumul: Cumulative amount of data

• delay: Queuing delay, measured at dequeue time

Token Bucket ScenarioToken Bucket Scenario

#define RATE 1Mbit#define BURST 3kB#define LIMIT 20kB

#define NOTHING#define PACKET /* 100-sizeof(iphdr) = 80 bytes. */ IP_PCK(NOTHING) 0 x 80

dev eth0 10000 /* 10 Mbps */

tc qdisc add dev eth0 root handle 1:0 tbf limit LIMIT rate RATE burst BURST

every 0.0005s send PACKET /* 1.6 Mbps */

time 1send

Packet LossesPacket Losses

Scenario 1 Scenario 2 Scenario 3 Scenario 4

Rate = 1 Mbps Rate = 1 Mbps Rate = 1 Mbps Rate = 1.1 Mbps

Burst=3KB Burst = 3KB Burst = 5KB Burst = 3KBLimit = 20KB Limit = 10KB Limit = 20KB Limit = 20KB

Running the simulation script tbf with the command syntax:

tcsim tbf | tcsim_filter -c

which counts the packets enqueued and dequeued, produces thefollowing outputs for each scenario:

D: 1590 D: 1488 D: 1612 D: 1727E: 2002 E: 2002 E: 2002 E:

2002

Time of 1Time of 1stst Packet Loss (1) Packet Loss (1)

Scenario 1

0.356500 E : 0x9ecf858 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.357000 E : 0x9ed3c58 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.357000 * : 0x9ed3c58 100 : eth0: enqueue returns DROP (1)0.357500 E : 0x9ed3c58 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.357500 * : 0x9ed3c58 100 : eth0: enqueue returns DROP (1)

Scenario 2

0.189500 E : 0x82b0ee8 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.190000 E : 0x82b0ff8 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.190000 * : 0x82b0ff8 100 : eth0: enqueue returns DROP (1)0.190000 D : 0x82aece8 100 : eth0: 45000064 00000000 40000000 0a0000 ...

Time of 1Time of 1stst Packet Loss (2) Packet Loss (2)

Scenario 3

0.388000 E : 0x96dbb48 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.388500 E : 0x96dbc58 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.388500 * : 0x96dbc58 100 : eth0: enqueue returns DROP (1)0.389000 E : 0x96dbc58 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.389000 * : 0x96dbc58 100 : eth0: enqueue returns DROP (1)

Scenario 4

0.448500 E : 0x8583b48 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.449000 E : 0x8583c58 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.449000 * : 0x8583c58 100 : eth0: enqueue returns DROP (1)0.449500 E : 0x8583c58 100 : eth0: 45000064 00000000 40000000 0a0000 ...0.449500 * : 0x8583c58 100 : eth0: enqueue returns DROP (1)

Cumulative Amount of Data for Cumulative Amount of Data for Scenarios 1-4Scenarios 1-4

1 2

3 4

Queuing Delay for Scenarios 1-4Queuing Delay for Scenarios 1-4

1 2

3 4

TCSIM Restrictions & ExtensionsTCSIM Restrictions & Extensions

• TCSIM only includes a small part of the network stack, and does not support full routing or firewalling. Therefore, the route classifier is not available in tcsim, and the usability of the fw classifier is limited.

• tc bugs may crash TCSIM.

• TCSIM supports only the simulation of constant bit-rate flows (using the every keyword) and the sending of single packets at a specified point in time.

• In order to support the simulation of Poisson distributed and bursty flows, a simple tool, the TCSIM Traffic Generator (TrafGen) was developed, which creates trace files to be used in a simulation.

ReferencesReferences

• TCNG HomePage, URL: http://tcng.sourceforge.net

• Linux Advanced Routing & Traffic Control, URL: http://lartc.org/

• L. Wischhof and J. W. Lockwood, “Packet Scheduling for Link-Sharing and Quality of Service Support in Wireless Local Area Networks”, November 2001

• Linux IP, URL: http://linux-ip.net/

• Practical QoS, URL: http://www.opalsoft.net/qos/index.html