Container Orchestration Integration:
OpenStack Kuryr and
Apache Mesos@takufukushima
@takufukushima• MidoNet contributor sometimes
• OpenStack Kuryr contributor these days
Agenda1. Introduction to Docker and Apache
Mesos
2. The history of Docker and Apache Mesos Networking
3. OpenStack Kuryr as the building block
4. Summary
1. Introduction to Docker and Apache Mesos
We need the cluster manager• We distribute workloads to containers on
hosts or VM instances
• Docker and other containers are building blocks
• We want to manage them from the bird’s-eye view
Container cluster managers• Docker native
• Docker Swarm
• Apache Mesos
• Docker based
• Google Kubernetes
rocks
• Blazing fast (VM? Huh?)
• Great ecosystem
• e.g., DockerHub, Meetups
• Golang dev hipstersAnd nice art works
(seriously)
The dark side of
• “fundamentally flawed”
• “It’s The Future”
• “So I just need to split my simple CRUD app into 12 microservices, each with their own APIs which call each others’ APIs but handle failure resiliently, put them into Docker containers, launch a fleet of 8 machines which are Docker hosts running CoreOS, “orchestrate” them using a small Kubernetes cluster running etcd, figure out the “open questions” of networking and storage, and then I continuously deliver multiple redundant copies of each microservice to my fleet. Is that it?”
rocks• The core of Mesosphere DCOS
• Originally research project of UCB RAD (AMP) lab
• Great ecosystem and use cases
• Twitter, Apple, Airbnb, eBay and so on
• Pluggable frameworks
• Apache Aurora, Chronos, Marathon
architecture
Retrieved from http://radar.oreilly.com/2015/10/swarm-v-fleet-v-kubernetes-v-mesos.html
and
• Mesos has few containerizers
• cgroups and namespace based containerizer
• Docker containerizer
• External containerizer
2. The history of Docker and Apache Mesos Networking
Docker networking• docker0 bridge
• veth pairs and netns
• --icc and --link
• --net
• bridge, container, host, none
• NAT by iptables
Extended Docker networking• CoreOS flannel
• For Kubernetes
• Weave
• SocketPlane
• pipeworks
libnetwork• Networking component as a plugin
• docker network command
• Drivers separated from Docker core
• bridge
• overlay
• none
• Remote driver opened up for everyone
overlay driver• SocketPlane
• Container communication over the hosts
• VXLAN
• libkv for storing the network state in the distributed datastore
• --cluster-store and --cluster-advertise
• etcd, Consul and ZooKeeper
network.CreateEndpoint()
controller.NewNetwork()
InterfaceInfo
endpoint.Join() endpoint.Leave()
endpoint.Delete()
network.Delete()
sandbox.Info
e.g., netns
Network Controller
Driver
Built-in Remote
Endpoint
BridgeHostNone
OverlayCalicoKuryrWeave
Network Network
Endpoint Endpoint
IP addressesMAC addressesRoutesDNS entries
Container Container
OptionLabel
CLI
Sandbox Sandbox
Container Container
libnetwork CNM
networking
• Almost the same as Docker
• especially if you’re using Docker as the containerizer
• Containers share the IP of the slaves
• NAT and netns
integration point• External Containerizer Program (ECP)
• Slaves delegate the containerising to ECP
• It’s just building the Docker command
• Protobuf data is passed through stdin and stdout
• ENV vars can be used for additional data
new networking
• IPAM server
• IPAM client on masters and slaves
• Network Isolator Module (NIM) on slaves
• Cleanup Module on masters
new networking
Retrieved from https://github.com/apache/mesos/blob/master/docs/networking-for-mesos-managed-containers.md
3. OpenStack Kuryr as a building block
OpenStack? Why is it
related with Docker?
OpenStack and Docker• OpenStack and Docker are exclusive for each other at
this point
• Multi tenancy
• Strict resource isolation
• OpenStack Magnum
• Docker managed by OpenStack
• Docker containers on VM instances
• OpenStack Kolla
Revisiting OpenStack Neutron• Neutron is a networking component of OpenStack
• Networking resource allocation through the API
• Vendor agnostic APIs
• Many network controllers supporting these APIs
• The model of libnetwork is getting close to Neutron’s one
Maximizing the developers effects: Investment for the most effective way
OpenStack Kuryr• A new component in “Neutron Stadium”
• A translator between Neutron and libnetwork
• Map the API calls on the remote driver into Neutron’s API calls
OpenStack Kuryr• A new component in “Neutron Stadium”
• A translator between Neutron and libnetwork
• Map the API calls on the remote driver into Neutron’s API calls
OpenStack Kuryr• A new component in “Neutron Stadium”
• A translator between Neutron and libnetwork
• Map the API calls on the remote driver into Neutron’s API calls
Kuryr architcture
Neutron
Daemon
Host
DistributedDatastore
DistributedDatastoreDistributed
Datastore
ContainerContainerContainer
Daemon
Host
ContainerContainerContainer
Daemon
Host
ContainerContainerContainer
Kuryr Kuryr Kuryr
Keystone
Kuryr as a translator
Kuryr as a translator
4. Summary
Container networking made easy• Container networking had some issues
• The new networking models and APIs are emerging
• OpenStack Kuryr can be the common building block
Kuryr as a translator
The end of slides. Any questions?