Medallia © Copyright 2015. 1
PLACE IMAGE OVER THIS SPACE
Docker IP RoutingHaving your first-hop load-balancer on Docker
Medallia © Copyright 2015. 2
Who are you?
Medallia:“Software to improve the customer experience”“Aggregating 1B documents in 1s or less”
This talk: Infrastructure to run the crunching application
Medallia © Copyright 2015. 3
• Docker for everything!• Docker for applications!• Docker for load-balancers!• Docker for zookeeper! And DNS!
Problem to solveWant a reliable, flexible data-center
Medallia © Copyright 2015. 4
Problem to Solve TodayMoving non-movable services
DataCenter Firewall
Host: 10.1.2.3:80
Host: 10.1.2.5:80
172.17.0.3:80 nginxHost: 10.1.2.4:2181
172.17.1.0:2181 zookeeper
172.17.1.2:80 application
Medallia © Copyright 2015. 5
• Buy the cheapest possible servers○ Optimize for performance○ Sacrifice redundancy○ Service Contract: “Unrack and ship by mail”
• Solve the redundancy at a higher level○ Run multiple instances of everything○ Rapidly restore status quo
Design for FailureIt will fail sooner or later; choose sooner
Medallia © Copyright 2015. 6
• No special snowflakes• Commodity Components & Supported Open Standards• Fully automated provisioning and reinstall• Cheap• Scalable• “Simple”• Every component must be able to run anywhere
Design Principles
Medallia © Copyright 2015. 7
Going to talk about Network today.
(Storage next time)
Two ProblemsStorage and Network
Medallia © Copyright 2015. 8
• VLAN○ Can’t do leaf/spine; need monstrous “core switches”
• VXLAN○ Network Island: How do you exit it?○ Vendor interoperability for forwarding DB (VXLAN-to-IP)
Time to leave layer 2 behind?
Things we triedThat didn’t work for us
Medallia © Copyright 2015. 9
Docker Bridged Networking Model
host1
container 1
eth0
10.1.2.3/24
172.16.1.1172.16.1.2
docker0
veth0 veth1
Medallia © Copyright 2015. 10
Default (Bridged) Strategy
● Creates a pair of veth.● Moves one to the container
namespace.● Renames the container veth to
eth0● Attaches the host veth to the
docker0 bridge● Configure port forwarding in
iptables
Routed Strategy
● Creates a pair of veth.● Moves one to the container
namespace.● Renames the container veth to
eth0.● Add route to 0.0.0.0/0 via
eth0 in container.● Add route to container IP via
veth0 in the host.
Docker* new strategy
Medallia © Copyright 2015. 11
OSPF Area
host1% ip route10.4.5.6 dev veth0
...
Docker* Routed Networking Model
eth0
10.1.2.3/24
veth0
container-A
% ip routedefault eth0
10.4.5.6/32
eth0
host2
host3
hostN
Medallia © Copyright 2015. 12
Route to 10.1.2.3/32Infrastructure
Spine
Leaf
Server
10.1.2.3/32
10.1.2.3/32
Medallia © Copyright 2015. 13
Servers, Network, OSPFInfrastructure
Spine
Leaf
Server
Medallia © Copyright 2015. 14
OSPF: 1998
• Open Shortest Path First○ Propagated Link State Database○ Supported by every vendor
• OSPF is computationally expensive○ On a 1998-style embedded controller: Yes○ On a 2015-style Intel Atom 64-bit: No
• Everything is point-to-point L3 links• Switches and Servers run OSPF (Quagga)• Cumulus! OSPF unnumbered
Old and boring is the new sexy
Medallia © Copyright 2015. 15
Running a Container(technically, create and start a container)
% docker run -it --name=foo --net=routed --ip-address=10.2.3.4/32 ubuntu /bin/bash
(Will likely change to use labels)
Medallia © Copyright 2015. 16
demo!
Medallia © Copyright 2015. 17
Pros and ConsAwesomeness SucksiesIP MobilityQuick failoverNo special snowflakesEverything in docker
Future: CRIU?
T2 routing limit: 128K entries
Medallia © Copyright 2015. 18
Leaf /etc/network/interfaces Server /etc/network/interfacesauto loiface lo inet loopback address 10.225.10.245/32%for v in range(1,17):auto swp${v}iface swp${v} mtu 9000 address 10.225.10.${v*8+1}/30%endfor%for v in range(17,33):auto swp${v}iface swp${v} mtu 9000 address 10.225.10.245/32%endfor
auto loiface lo inet loopback
auto data0iface data0 inet static mtu 9000 address 10.225.10.10 netmask 255.255.255.252 gateway 10.225.10.9
How difficult is the network config?
To Servers
To Spines
To Leaf
Medallia © Copyright 2015. 19
Leaf/Spine Switch ospfd.conf Server ospfd.confrouter ospf ospf router-id 10.225.10.245 network 10.224.0.0/12 area 0.0.0.0!interface swp1 ip ospf network point-to-point!interface swp2 ip ospf network point-to-point!….
! Bootstrap Configrouter ospf ospf router-id 10.225.10.10 redistribute kernel passive-interface default no passive-interface data0 network 10.224.0.0/12 area 0.0.0.0!log syslog!interface data0 ip ospf network point-to-point!
How difficult is the network config?
Medallia © Copyright 2015. 20
“Good enough”
• 24-39 Gbit/s (core affinity)• 13us ICMP ping
Performance
Medallia © Copyright 2015. 21
Local Development With Style
IP Mobility on Local Laptop
Allow for easy and rapid development
Boot2Docker
Medallia © Copyright 2015. 22
Front-End (On 10.10.2.1)
What day is today as 2 servicesBack-End (On 10.10.2.2)while true; do
date | nc -l 9999done
while true; doecho Today is $(nc 10.10.2.2 9999) | nc -l 8080
done
Medallia © Copyright 2015. 23
Boot2Docker on OSX
My MacBookBoot2Docker VM
10.10.0.0/16
10.10.0.0/16
10.10.2.1/32 frontend on 8080
10.10.2.2/32 backed (date) on 9999
lo0: 10.10.2.2/32 backend
My Shell
Medallia © Copyright 2015. 24
demo!
Medallia © Copyright 2015. 25
Next Steps for us
• Share with the world!○ Everything you’ve seen today is (or will be) open source
• Storage with CEPH○ Already works in medallia-container (think systemd-nspawn)○ Porting to docker
• SSH Hot Redirect○ SSH to container => SSH to host with automatic “docker exec”
• Docker Labels in Aurora/Mesos
In the next sprint...
Medallia © Copyright 2015. 26
Checkout and have fun
www.github.com/medallia/dockerwww.github.com/medallia/boot2docker-iso
Medallia © Copyright 2015. 27
Questions?