![Page 1: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/1.jpg)
In containers
Alban Crequy
Testing applicationswith traffic control
ContainerCon North America - August 2016https://goo.gl/ZLLlv3
![Page 2: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/2.jpg)
Alban Crequy
∘ Worked on rkt the last 1.5 years∘ Currently tech lead on rkt∘ In 2014, worked on traffic control for multimedia
applications in cars (tcmmd)
https://github.com/alban
![Page 3: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/3.jpg)
Berlin-based software company building foundational Linux technologies
Some examples of what we work on...
OSTreegit for operating system binaries
![Page 4: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/4.jpg)
Find out more about us…
Blog: http://kinvolk.io/blog
Github: https://github.com/kinvolk
Twitter: https://twitter.com/kinvolkio
Email: [email protected]
![Page 5: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/5.jpg)
∘ What is traffic control and how does it work on Linux∘ How it can be used for testing a microservices application∘ Demo
∘ With CoreOS Linux, Kubernetes, Weave Scope
Plan
![Page 6: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/6.jpg)
What is traffic control?
How does it work on Linux?
![Page 7: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/7.jpg)
Traffic control, why?
web server client
client
client
THEINTERNET
∘ fair distribution of bandwidth
∘ reserve bandwidth to specific applications
∘ avoid bufferbloat
![Page 8: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/8.jpg)
∘ Network scheduling algorithm∘ which packet to emit next?∘ when?
∘ Configurable at run-time:∘ /sbin/tc∘ Netlink
∘ Default on new network interfaces: sysctl net.core.default_qdisc
Queuing disciplines(qdisc)
eth0 THE INTERNETqdisc
![Page 9: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/9.jpg)
Stochastic FairnessQueueing (sfq)
eth0
THE INTERNET
FIFO n
FIFO 1
FIFO 0
...
round robin
![Page 10: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/10.jpg)
Traffic control for testing?
![Page 11: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/11.jpg)
Network emulator(netem)
eth0 THE INTERNETnetem
bandwidth
latency packet loss
corrupt...
![Page 12: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/12.jpg)
Testing with containers
container 1 container 2
eth0eth0
Testing framework
configure “netem” qdiscs:bandwidth, latency, packet drop...
![Page 13: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/13.jpg)
The demo application
![Page 14: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/14.jpg)
microservices-demo
https://github.com/microservices-demo/microservices-demo
![Page 15: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/15.jpg)
Some micro-services
front-end Firefox
catalogue
ordersorders-db
payment
![Page 16: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/16.jpg)
Kubernetes
![Page 17: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/17.jpg)
Kubernetes objects- Kubernetes “Pods”
- Group of container(s) running together
- Kubernetes “Replication Sets”- Control the number of “Pods” replicas running
- Kubernetes “Service”- Directing the traffic to “Pods”
pod
pod
pod
service
![Page 18: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/18.jpg)
Testing with traffic control in Kubernetes
Kubernetesnode 1
pod
pod
Kubernetesnode 2
pod
pod
tc tccontrols∘ Latency∘ Bandwidth∘ Packet drop
∘ configure network simulator
∘ play scenarios
![Page 19: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/19.jpg)
Weave Scope
![Page 20: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/20.jpg)
Weave Scope
![Page 21: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/21.jpg)
Testing with Weave Scope
Kubernetesnode 1
tcScopeProbe
pod pod
pod pod
KubernetesNode 2
tcScopeProbe
pod pod
pod pod
ScopeApp
![Page 22: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/22.jpg)
Demo
Reproduce the demo yourself: https://github.com/kinvolk/demo
![Page 23: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/23.jpg)
Plugins in Scope
tc pluginScopeProbe
ScopeApp
report
reportcontrol
control
- Unix socket in /var/run/scope/plugins/- Protocols: report and control- Write your own plugins for your testing needs
![Page 24: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/24.jpg)
Testing frameworkfor web apps
Selenium
![Page 25: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/25.jpg)
Testing more complex scenarios
(my “wishlist”)
![Page 26: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/26.jpg)
Add latency on a specific connection
front-end Firefox
catalogue
ordersorders-db
payment
latency=100ms
![Page 27: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/27.jpg)
How to define classes of traffic
eth0
netem
interface
latency=100ms
dest_ip=10.0.4.* dest_ip=10.0.5.* other
![Page 28: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/28.jpg)
u32: filter on contenteth0
HTB
HTB
HTBHTB HTB
netemnetem netem
interface
root qdisc (type = HTB)
root class (type = HTB)
leaf qdiscs (type = netem)
leaf classes (type = HTB)
filters (type=u32)
otherip=10.0.5.*ip=10.0.4.*
latency=10ms
![Page 29: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/29.jpg)
Filtering with cBPF/eBPF
eth0
BPF
netemnetem
kernel
userspace
BPF_JMP...BPF_LD...BPF_RET...
if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation
clang... -march=bpf
uploadin the kernel:
- bpf()- Netlink
x86_64 codeJIT compilation
![Page 30: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/30.jpg)
eBPF maps
eth0
BPF
netemnetem
kernel
userspace
x86_64 code
eBPF map
tc
∘ Build statistics∘ Make them available to
the testing framework
![Page 31: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/31.jpg)
The EndTry the demos yourself: https://github.com/kinvolk/demo
Read more: https://kinvolk.io/blog/
The slides: https://goo.gl/ZLLlv3
![Page 32: with traffic control Testing applications · In containers Alban Crequy Testing applications with traffic control ContainerCon North America - August 2016File Size: 617KBPage Count:](https://reader034.vdocuments.us/reader034/viewer/2022042220/5ec6c1f7a89cc52f307c6501/html5/thumbnails/32.jpg)
Questions?