networking approaches in a container world
TRANSCRIPT
![Page 2: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/2.jpg)
Disclaimer● There a many container engines, I’m going to focus on Docker
● Multiple networking solutions are available:
– Introduce the core concepts
– Many projects → cover only some of them
● Container orchestration engines:
– Tightly coupled with networking
– I’m going to focus on Docker Swarm and Kubernetes
● Remember: the container ecosystem moves at a fast peace, things can suddenly change
2
![Page 3: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/3.jpg)
The problem● Given:
– Containers are lightweight
– Containers are great for microservices
– Microservices: multiple distributed processes communicating
→ Lots of containers that need to be connected together
3
![Page 4: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/4.jpg)
Single host
4
![Page 5: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/5.jpg)
Reuse the host network
5
host
container-01
eth0lo ...
Container has full access to host’s interfaces!
![Page 6: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/6.jpg)
Reuse the host network
6
$ docker run --rm --name container-01 --net=host -ti busybox /bin/sh/ # ifconfig docker0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::x/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:19888 errors:0 dropped:0 overruns:0 frame:0 TX packets:19314 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3063342 (2.9 MiB) TX bytes:29045336 (27.6 MiB)
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx inet addr:192.168.1.121 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::x/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:135513 errors:0 dropped:0 overruns:0 frame:0 TX packets:109723 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:102680118 (97.9 MiB) TX bytes:22766730 (21.7 MiB)
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:230 errors:0 dropped:0 overruns:0 frame:0 TX packets:230 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:37871 (36.9 KiB) TX bytes:37871 (36.9 KiB)
Warning: the container can see and control all the host interfaces
![Page 7: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/7.jpg)
Network bridge
7
host
container-01
eth0
docker0
container-02
172.17.0.0/16
● An internal, virtual switch● Containers are plugged in that switch● Containers on the same bridge can talk to each other● Users can create multiple bridges
![Page 8: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/8.jpg)
Network bridge: as seen by the host
8
$ ifconfig docker0docker0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::42:a2ff:fe10:ccf7/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:7 errors:0 dropped:0 overruns:0 frame:0 TX packets:30 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:480 (480.0 B) TX bytes:5025 (5.0 KB)
$ ip route default via 192.168.1.1 dev wlan0 proto static metric 600 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1...
route handling traffic from host to containers
docker0 is by default at 172.17.0.1
![Page 9: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/9.jpg)
How to expose a service
9
host
container-01 container-02
172.17.0.0/16
eth0
docker0
port 80
port 8080
● Port 80 of container-02 is mapped to port 8080 of the host● Risk: port exhaustion on the host
![Page 10: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/10.jpg)
Multi-host networking
10
![Page 11: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/11.jpg)
Multi-host networking scenarios
11
host-A host-B host-C
container-02
container-01
container-03
container-04
container-05
container-06
frontendnetwork
applicationnetwork
databasenetwork
eth0 eth0 eth0
![Page 12: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/12.jpg)
Multi-host networking scenarios
12
a big host-A
frontendnetwork
container-02
container-01
container-03
container-04
container-05
container-06
applicationnetwork
databasenetwork
![Page 13: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/13.jpg)
Multi-host networking scenarios
13
a big host-A
frontendnetwork
container-02
container-01
container-03
container-04
container-05
container-06
applicationnetwork
databasenetwork
VM-1 VM-2 VM-3
![Page 14: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/14.jpg)
Routing solutions
14
![Page 15: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/15.jpg)
Routing approach● Create a common IP space at container level
● Assign a /24 subnet to each host
● Setup IP routes between the hosts
● Main projects:
– Calico
– Flannel
– Romana
15
![Page 16: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/16.jpg)
Routing approach
16
192.168.1.2
host-A
container-01
eth0
docker0
10.0.9.0/24
10.0.9.4 10.0.9.5
10.0.9.1
container-02
host-B
container-03
eth0
docker0
10.0.10.0/24
10.0.10.8 10.0.10.9
10.0.10.1
container-04
Routing rule: 10.0.10.* goes through eth0
192.168.1.3
Routing rule: 10.0.10.* goes through docker0
![Page 17: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/17.jpg)
Calico's approach
17
192.168.1.2
host-A
container-01
eth0
docker0
10.0.9.0/24
10.0.9.4 10.0.9.5
10.0.9.1
container-02
host-B
container-03
eth0
docker0
10.0.10.0/24
10.0.10.8 10.0.10.9
10.0.10.1
container-04
192.168.1.3
BGP● One of the protocols used to
build the Internet● Used to advertise routes
● Felix agent● Uses kernel’s L3 forwarding
capabilities● Handles ACLs
![Page 18: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/18.jpg)
Flannel's approach
18
192.168.1.2
host-A
container-01
eth0
docker0
10.0.9.0/24
10.0.9.4 10.0.9.5
10.0.9.1
container-02
host-B
container-03
eth0
docker0
10.0.10.0/24
10.0.10.8 10.0.10.9
10.0.10.1
container-04
192.168.1.3
etcd● flanneld process● Keep routes up-to-date
● Network configuration● Network topology
![Page 19: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/19.jpg)
Calico + flannel = Canal● Collaboration announced on May 9th 2016
● Use Calico and flannel together
● Project still in its early days
19
![Page 20: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/20.jpg)
Overlay solutions
20
![Page 21: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/21.jpg)
Overlay network approach● Create a parallel network for cross communication
● Connect hosts with encapsulation tunnels
● Connect containers to the virtual networks
● Main projects:
– Docker (native)
– Flannel
– Weave
21
![Page 22: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/22.jpg)
Overlay network
22
192.168.1.2
host-A
container-01
eth0
docker0
10.0.9.0/24
container-02
host-B
container-03
eth0
docker0
10.0.9.0/24
container-04
192.168.1.3
capture traffic leaving to some other container in 10.0.9.X
outerEther
Header (src/dst)
outerIP
Header (src/dst)
outerUDP
Header
VXLAN Header
Inner Ether FrameEncapsulated traffic (eg. VXLAN)
Overlay traffic
![Page 23: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/23.jpg)
Overlay network and k/v storeNetwork state and configuration can be saved into a k/v store:
● Docker < 1.12: supports etcd, consul and zookeeper via libkv
● Docker >= 1.12: no external dependency, built-in component
● Flannel: etcd
● Weave: no external dependency, doesn’t use k/v store at all
23
![Page 24: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/24.jpg)
Overlay backends
● VXLAN:
– faster than UDP, traffic doesn't go to userspace
– Some hardware acceleration available
● UDP: can add encryption easily24
VXLAN UDP
Docker X -
Weave X X*
Flannel X X*
* backed by custom protocol
![Page 25: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/25.jpg)
Routing vs Overlay
25
Good Bad
Routing
● Native performance● Easy debugging
● Requires control over the infrastructure
● Hybrid cloud more complicated (requires VPN)
● Can run out of addresses
Overlay
● Easier inter-cloud● Doesn’t require control over
the infrastructure
● Inferior performances● Debugging more
complicated● No IP multicast (except for
weave)
![Page 26: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/26.jpg)
How to use these projects● Container Network Module (CNM):
– Specification used by Docker
– Plugins for: calico, weave, …
– Note well: Docker 1.12+ Swarm mode works only with the native overlay network driver
● Container Network Interface (CNI):
– Derived from rkt networking proposal
– Supported by rkt, kubernetes, Cloud Foundry, Mesos,…
– Support for: calico, flannel, weave,...
26
![Page 27: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/27.jpg)
More troubles...
27
![Page 28: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/28.jpg)
Are we done?● Now we can:
– Connect containers running on different hosts
– React to network changes
● Is that enough? Unfortunately not...
28
![Page 29: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/29.jpg)
Service discovery● A container runs a service: producer
● A container accesses this service: consumer
● The consumer needs to find where the producer is located (IP address, in some cases even port #)
29
![Page 30: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/30.jpg)
Challenge #1: find the producer
30
host-A host-B
web-01 redis-01
eth0 eth0
Where is redis?
host-A host-B
web-01 redis-01
eth0 eth0
![Page 31: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/31.jpg)
Challenge #2: react to changes
31
host-A host-B
web-01 redis-01
eth0 eth0
host-C
eth0
“web” is already connected to “redis”
![Page 32: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/32.jpg)
Challenge #2: react to changes
32
host-A host-B
web-01 redis-01
eth0 eth0
host-C
redis-02
eth0
“web” points to to the old location → it’s broken
“redis” is moved to another host → different IP
![Page 33: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/33.jpg)
Challenge #2: react to changes
33
The link has to be reestablished
host-A host-B
web-01
eth0 eth0
host-C
redis-02
eth0
Containers can be moved at any time:● The producer can be moved to a different host● The consumer should keep working
![Page 34: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/34.jpg)
Challenge #3: multiple choices
34
Multiple instances of the “redis” image
host-A host-B
web-01 redis-01
eth0 eth0
host-C
redis-02
eth0
Which redis?
Workloads can be scaled:● More instances of the same producer● How to choose between all of them?
![Page 35: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/35.jpg)
Addressing service discovery
35
![Page 36: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/36.jpg)
Use DNS● Not a good solution:
– Containers can die/be moved somewhere more often
– Return DNS responses with a short TTL → more load on the server
– Some clients ignore TTL → old entries are cached
Note well:
● Docker < 1.11: updates /etc/hosts dynamically
● Docker >= 1.11: integrates a DNS server
36
![Page 37: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/37.jpg)
Key-value store● Rely on a k/v store (etcd, consul, zookeeper)
● Producer register itself: IP, port #
● Orchestration engine handles this data to the consumer
● At run time either:
– Change your application to read data straight from the k/v
– Rely on some helper that exposes the values via environment file or configuration file
37
![Page 38: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/38.jpg)
Handing changes &multiple choices
38
![Page 39: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/39.jpg)
DIY solution● Use a load balancer
● Point all the consumers to a load balancer
● Expose the producer(s) using the load balancer
● Configure the load balancer to react to changes
→ More moving pieces
39
![Page 40: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/40.jpg)
Rely on the orchestration engine● Service has an unique and stable IP address
● Consumers are pointed to the service
● Service redirects the request to one of the containers running the producer
● Traditional DNS can be added on top of it → no changes to legacy applications
● Feature offered by Kubernetes and Docker >= 1.12
40
![Page 41: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/41.jpg)
Kubernetes and Swarm services
41
host-B
redis-01
eth0
host-C
redis-02
eth0
host-A
web-01
eth0
redisservice
VIP
● User declares a service● Orchestration engine allocates a virtual IP address for it● On each container node:
– iptables rules to handle VIP ↔ container IP translation– A process keeps the iptables rules up-to-date
![Page 42: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/42.jpg)
Are we really done?
42
![Page 43: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/43.jpg)
Ingress traffic● Your production application is running inside of a container
cluster
● How to route customers’ requests to these containers?
● How to react to changes (containers moved, scaling,…)?
43
![Page 44: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/44.jpg)
Kubernetes’ approachServices can be of three different types:
● ClusterIP: virtual IP reachable only by containers inside of the cluster
● NodePort: ClusterIP + the service is exposed on all the nodes of the cluster on a specific port → <NodeIP>:<NodePort>
● LoadBalancer: NodePort + k8s allocates a load balancer using the underlying cloud provider. Then it configures it and it keep it up-to-date
44
![Page 45: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/45.jpg)
Docker 1.12 approach● Define a service using the `--publish` flag
● The service is exposed on all the nodes of the cluster on a specific port → <NodeIP>:<ServicePort>
45
![Page 46: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/46.jpg)
Ingress traffic flow
46
Load balancer
http://guestbook.com
host-B
guestbook-01
8081
blog-01
8080
host-A
guestbook-01
80818080
host-C
8081
blog-01
8080
● Load balancer picks a container host● Traffic is handled by the internal service● Works even when the node chosen by the load balancer is not running the
container
![Page 47: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/47.jpg)
Recap
47
Calico Docker built-in Flannel Weave
Approach routing overlay routing, overlay overlay
Specification CNI, CNM CNM CNI, CNM CNI, CNM
It’s not just a matter of connecting containers:● Service discovery● Handling changes & multiple choices● Handling ingress traffic
![Page 48: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/48.jpg)
Questions?
48
![Page 49: Networking approaches in a container world](https://reader034.vdocuments.us/reader034/viewer/2022042611/58a2e8661a28abac578bed45/html5/thumbnails/49.jpg)