In containers

Alban Crequy

Testing applicationswith traffic control

ContainerCon North America - August 2016https://goo.gl/ZLLlv3

Alban Crequy

∘ Worked on rkt the last 1.5 years∘ Currently tech lead on rkt∘ In 2014, worked on traffic control for multimedia

applications in cars (tcmmd)

https://github.com/alban

Berlin-based software company building foundational Linux technologies

Some examples of what we work on...

OSTreegit for operating system binaries

Find out more about us…

Blog: http://kinvolk.io/blog

Github: https://github.com/kinvolk

Twitter: https://twitter.com/kinvolkio

Email: hello@kinvolk.io

∘ What is traffic control and how does it work on Linux∘ How it can be used for testing a microservices application∘ Demo

∘ With CoreOS Linux, Kubernetes, Weave Scope

Plan

What is traffic control?

How does it work on Linux?

Traffic control, why?

web server client

client

client

THEINTERNET

∘ fair distribution of bandwidth

∘ reserve bandwidth to specific applications

∘ avoid bufferbloat

∘ Network scheduling algorithm∘ which packet to emit next?∘ when?

∘ Configurable at run-time:∘ /sbin/tc∘ Netlink

∘ Default on new network interfaces: sysctl net.core.default_qdisc

Queuing disciplines(qdisc)

eth0 THE INTERNETqdisc

Stochastic FairnessQueueing (sfq)

eth0

THE INTERNET

FIFO n

FIFO 1

FIFO 0

...

round robin

Traffic control for testing?

Network emulator(netem)

eth0 THE INTERNETnetem

bandwidth

latency packet loss

corrupt...

Testing with containers

container 1 container 2

eth0eth0

Testing framework

configure “netem” qdiscs:bandwidth, latency, packet drop...

The demo application

microservices-demo

https://github.com/microservices-demo/microservices-demo

Some micro-services

front-end Firefox

catalogue

ordersorders-db

payment

Kubernetes

Kubernetes objects- Kubernetes “Pods”

- Group of container(s) running together

- Kubernetes “Replication Sets”- Control the number of “Pods” replicas running

- Kubernetes “Service”- Directing the traffic to “Pods”

pod

pod

pod

service

Testing with traffic control in Kubernetes

Kubernetesnode 1

pod

pod

Kubernetesnode 2

pod

pod

tc tccontrols∘ Latency∘ Bandwidth∘ Packet drop

∘ configure network simulator

∘ play scenarios

Weave Scope

Weave Scope

Testing with Weave Scope

Kubernetesnode 1

tcScopeProbe

pod pod

pod pod

KubernetesNode 2

tcScopeProbe

pod pod

pod pod

ScopeApp

Demo

Reproduce the demo yourself: https://github.com/kinvolk/demo

Plugins in Scope

tc pluginScopeProbe

ScopeApp

report

reportcontrol

control

- Unix socket in /var/run/scope/plugins/- Protocols: report and control- Write your own plugins for your testing needs

Testing frameworkfor web apps

Selenium

Testing more complex scenarios

(my “wishlist”)

Add latency on a specific connection

front-end Firefox

catalogue

ordersorders-db

payment

latency=100ms

How to define classes of traffic

eth0

netem

interface

latency=100ms

dest_ip=10.0.4.* dest_ip=10.0.5.* other

u32: filter on contenteth0

HTB

HTB

HTBHTB HTB

netemnetem netem

interface

root qdisc (type = HTB)

root class (type = HTB)

leaf qdiscs (type = netem)

leaf classes (type = HTB)

filters (type=u32)

otherip=10.0.5.*ip=10.0.4.*

latency=10ms

Filtering with cBPF/eBPF

eth0

BPF

netemnetem

kernel

userspace

BPF_JMP...BPF_LD...BPF_RET...

if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation

clang... -march=bpf

uploadin the kernel:

- bpf()- Netlink

x86_64 codeJIT compilation

eBPF maps

eth0

BPF

netemnetem

kernel

userspace

x86_64 code

eBPF map

tc

∘ Build statistics∘ Make them available to

the testing framework

The EndTry the demos yourself: https://github.com/kinvolk/demo

Read more: https://kinvolk.io/blog/

The slides: https://goo.gl/ZLLlv3

Questions?