devops guide to container networking
Post on 08-Jan-2017
1.516 Views
Preview:
TRANSCRIPT
DevOps Guide toContainer Networking
Dirk Wallerstorfer DevOpsSummit New York, June 8th
2
Technology Lead SDN, OpenStack
dirk.wallerstorfer@dynatrace.com@wall_dirkblog.ruxit.com
3Dirk Wallerstorfer, @wall_dirk
5Dirk Wallerstorfer, @wall_dirk
6Dirk Wallerstorfer, @wall_dirk
SDN
7Dirk Wallerstorfer, @wall_dirk
http://systematicrelativestrength.com/2013/11/12/your-plan-vs-reality/
9Dirk Wallerstorfer, @wall_dirk
10Dirk Wallerstorfer, @wall_dirk
11Dirk Wallerstorfer, @wall_dirk
12Dirk Wallerstorfer, @wall_dirk
13Dirk Wallerstorfer, @wall_dirk
web:$ docker run -itd wordpress
14Dirk Wallerstorfer, @wall_dirk
web:$ docker run -itd wordpress
15Dirk Wallerstorfer, @wall_dirk
web:$ docker run -itd wordpress
user:wordpress$ ping 8.8.8.8
iptables –t nat –A POSTROUTING –o eth0 –j MASQUERADE
16Dirk Wallerstorfer, @wall_dirk
web:$ docker run –itd –p 8080:80 wordpress
17Dirk Wallerstorfer, @wall_dirk
web:$ docker run –itd –p 8080:80 wordpress
iptables –t nat –A PREROUTING ... –j DOCKERiptables –t nat –A DOCKER --dport 8080 --redirect-to 172.18.0.2:80
18Dirk Wallerstorfer, @wall_dirk
SDN
Dirk Wallerstorfer, @wall_dirk 19
Three reasons for SDN• Permanent connectivity• Virtualization of everything• Paradigm shift in software development
Dirk Wallerstorfer, @wall_dirk 20
Three reasons for SDN
Networking had to keep up somehow!
Continuous delivery
Virtualize everything
Permanent connectivity
Dirk Wallerstorfer, @wall_dirk 21
SDN• Classic SDN
• SD WAN
• Network Overlay
Dirk Wallerstorfer, @wall_dirk 22
SDN• Classic SDN
• SD WAN
• Network Overlay
Dirk Wallerstorfer, @wall_dirk 23
SDN• Classic SDN
• SD WAN
• Network Overlay
Dirk Wallerstorfer, @wall_dirk 24
SDN• Classic SDN
• SD WAN
• Network Overlay
Dirk Wallerstorfer, @wall_dirk 25
26Dirk Wallerstorfer, @wall_dirk
Dirk Wallerstorfer, @wall_dirk 27
Multi-host Container NetworkingNo SDNdb:$ docker run -itd –p 3306:3306 mysql
web:$ docker run -itd –p 8080:80 –e WORDPRESS_DB_HOST=172.16.198.248:3306 wordpress
Dirk Wallerstorfer, @wall_dirk 28
Multi-host Container NetworkingPrerequisites• Underlying network
• Distributed K/V store
• Accessible ports
Dirk Wallerstorfer, @wall_dirk 29
Multi-host Container Networking
Overlay No overlay
http://s568.photobucket.com/user/LMG_09/media/CrowdSurfftw.jpg.html Ocean’s Eleven, Warner Bros, 2001
Dirk Wallerstorfer, @wall_dirk 30
Multi-host Container NetworkingOverlay Protocols
• VXLAN
OuterEthernet
OuterIP
OuterUDP VXLAN Ethernet IP TCP Payload
Dirk Wallerstorfer, @wall_dirk 31
Multi-host Container NetworkingOverlay Protocols
• VXLAN
OuterEthernet
OuterIP
OuterUDP VXLAN Ethernet IP TCP Payload
Ethernet IP UDP VXLAN Ethernet IP TCP Payload
Dirk Wallerstorfer, @wall_dirk 32
Multi-host Container NetworkingOverlay Protocols
• VXLAN
OuterEthernet
OuterIP
OuterUDP VXLAN Ethernet IP TCP Payload
Ethernet IP UDP VXLAN Ethernet IP TCP Payload
Flags Reserved VXLAN Network Identifier (VNI) Reserved
Dirk Wallerstorfer, @wall_dirk 33
Multi-host Container NetworkingOverlay Protocols
• VXLAN
OuterEthernet
OuterIP
OuterUDP VXLAN Ethernet IP TCP Payload
Ethernet IP UDP VXLAN Ethernet IP TCP Payload
14 bytes 20 bytes 8 bytes 8 bytes
+ 50 bytes
Dirk Wallerstorfer, @wall_dirk 34
Multi-host Container NetworkingOverlay Protocols
• VXLAN• Ethernet in UDP, defacto standard, won the overlay war
• NVGRE• Ethernet in IP, Microsoft’s answer to a question nobody asked
• STT• Ethernet in fake TCP, to utilize TSO of NIC
• Geneve• Ethernet in UDP, best of breed approach• A+ for extensibility• https://packetpushers.net/podcast/podcasts/pq-show-68-geneve-data-center-overlay-update/
Dirk Wallerstorfer, @wall_dirk 35
Multi-host Container NetworkingOverlay
• Docker Libnetwork• WeaveNet• Flannel
Dirk Wallerstorfer, @wall_dirk 36
Docker libnetwork
https://blog.docker.com/2015/04/docker-networking-takes-a-step-in-the-right-direction-2/
Dirk Wallerstorfer, @wall_dirk 37
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 38
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 39
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 40
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 41
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 42
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 43
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 44
Docker libnetwork
Dirk Wallerstorfer, @wall_dirk 45
Why UDP?
46Dirk Wallerstorfer, @wall_dirk
47Dirk Wallerstorfer, @wall_dirk
48Dirk Wallerstorfer, @wall_dirk
Departmentof
RedundancyDepartment
Dirk Wallerstorfer, @wall_dirk 49
Multi-host Container NetworkingNo overlay
• Project Calico • Flannel host-gw• Romana• Contiv• MACVLAN/IPVLAN
Dirk Wallerstorfer, @wall_dirk 50
Project Calico
https://www.projectcalico.org/docker-libnetwork-is-almost-here-and-calico-is-ready/
51Dirk Wallerstorfer, @wall_dirk
52Dirk Wallerstorfer, @wall_dirk
Dirk Wallerstorfer, @wall_dirk 53
© http://de.slideshare.net/grkvlt/metaswitch-project-calico
Dirk Wallerstorfer, @wall_dirk 54
© http://de.slideshare.net/grkvlt/metaswitch-project-calico
Host Host
Containers Containers
Dirk Wallerstorfer, @wall_dirk 55
Project Calico• Host is a router for the workloads• BGP to distribute routes• etcd backed• Pure Layer 3, no encapsulation
Dirk Wallerstorfer, @wall_dirk 56
Project Calico
Dirk Wallerstorfer, @wall_dirk 57
Project Calico
Dirk Wallerstorfer, @wall_dirk 58
Project Calico
Dirk Wallerstorfer, @wall_dirk 60
Location of services
k8s pods, marathon application groups, swarm constraints, fleet units
61Dirk Wallerstorfer, @wall_dirk
Dirk Wallerstorfer, @wall_dirk 62
Connectivity Problemsnf_conntrack: table full, dropping packet.
dirk@fueldev:~$ sudo sysctl –a | grep conntrack...net.netfilter.nf_conntrack_buckets = 8192net.netfilter.nf_conntrack_count = 0net.netfilter.nf_conntrack_max = 31760...
• Large number of iptables rules
Dirk Wallerstorfer, @wall_dirk 63
Connectivity Problems• The notorious MTU• https://www.youtube.com/watch?v=H2lBkj5zbYs
dirk@fueldev:~$ ip addr show enp0s32: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:f3:4e:5d brd ff:ff:ff:ff:ff:ff inet 172.16.99.14 brd 172.16.11.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fef4:4e56/64 scope link valid_lft forever preferred_lft forever
Dirk Wallerstorfer, @wall_dirk 64
TCP/IP over VXLAN Overhead
Ethernet IP UDP VXLAN Ethernet IP TCP Payload
14 bytes 20 bytes 8 bytes 8 bytes
+ 50 bytes
Send 1MB of data
1,000,000 bytes = 710 packets á 1410 bytes710 x 50 bytes = 35,500 bytes overhead
1,035,500 bytes are transmitted
3.55 %
Dirk Wallerstorfer, @wall_dirk 65
Send 1MB of data
1,000,000 bytes = 736 packets á 1330 bytes736 x 100 bytes = 73,600 bytes overhead
1,073,600 bytes are transmitted
TCP/IP over VXLAN over VXLAN Overhead
Ethernet IP UDP VXLAN Ethernet IP UDP VXLAN Ethernet IP TCP Payload
14 bytes
20 bytes
8bytes
8bytes
14 bytes
20 bytes
8bytes
8bytes
+ 100 bytes
7.36 %
66Dirk Wallerstorfer, @wall_dirk
Dirk Wallerstorfer, @wall_dirk 67
68Dirk Wallerstorfer, @wall_dirk
YOU WERE SO PREOCCUPIED WITH WHETHER OR NOT YOU COULD
YOU DIDN’T STOP TO THINK IF YOU SHOULD
Dirk Wallerstorfer, @wall_dirk 69
1460
1410136013101260
12101160
MTU overhead
25,9%20,7%
15,9%11,5%
7,4%3,6%
0%
70Dirk Wallerstorfer, @wall_dirkFebruary 20, 2016http://machinezone.github.io/research/networking-solutions-for-kubernetes/
Performance Comparison of Networking
Solutions for Kubernetes
71Dirk Wallerstorfer, @wall_dirk
Performance Comparison of Networking
Solutions for Kubernetes
with --net=host
aws-vpc
vxlan
host-gw
IPvlan
February 20, 2016http://machinezone.github.io/research/networking-solutions-for-kubernetes/
72Dirk Wallerstorfer, @wall_dirk
Performance Comparison of Networking
Solutions for Kubernetes
with --net=host
aws-vpc
vxlan
host-gw
IPvlan
libnetwork
February 20, 2016http://machinezone.github.io/research/networking-solutions-for-kubernetes/
73Dirk Wallerstorfer, @wall_dirk
https://github.com/machinezone/tcpkali
serving 350 byte responsemaking 250,000 requests per second
Performance Comparison of Networking
Solutions for Kubernetes
Different network options - latency?
74Dirk Wallerstorfer, @wall_dirk
250,000 requests per second, 350 bytes response
February 20, 2016http://machinezone.github.io/research/networking-solutions-for-kubernetes/
75Dirk Wallerstorfer, @wall_dirk
> 3 sec
46 %response time
will leave the page
76Dirk Wallerstorfer, @wall_dirk
+0.5 s
-11 %response time
in revenue
keep it manageable
keep it simple
keep it fast
78Dirk Wallerstorfer, @wall_dirk
http://i.coastingfish.com/image/3M
Volume-oriented network metrics
Quality-oriented network metrics
79Dirk Wallerstorfer, @wall_dirk
http://i.coastingfish.com/image/3M
82Dirk Wallerstorfer, @wall_dirk
83
Technology Lead SDN, OpenStack
dirk.wallerstorfer@dynatrace.com@wall_dirkblog.ruxit.com
Image sources:pixabay.com (3, 4, 5, 7, 9, 10, 23, 41, 57, 59, 60, 61)
top related