drupalcon 2014: mo' servers, mo' problems - betting on containers
DESCRIPTION
Servers have approximately about (ls /etc/; ps aux; netstat -nltp) | wc -l things that can go awry. Find out how you can safely and securely run fewer servers to achieve operational and cost-of-goods efficiencies using different Open Source ‘containerization’ technologies. We’ll go over some of the foundational technologies including ‘cgroups’ and ‘kernel namespaces’, as well as tools to deploying and managing containers, including Let Me Containerize That For You from Google, Docker from Docker.io, Systemd from the Red Hat community, and LXC from Linux core.TRANSCRIPT
Mo Servers, Mo ProblemsMo Servers, Mo Problems
Really, containers vs. VMs
What is a problem?
How using containers instead of VMs can help you increase uptime and decrease problems requiring human intervention and decision-making
We are living in the future
We will get our handsdirty
Chapter 1
In which you begin to believe me
when I tell you, “Mo’ Servers Mo
Problems”
Let’s bundle optimized hosting along with amazing workflow tools, team management, and runtime analytics, stick it on a VM, and charge $$$!
Bro!
We did it!300 Clients300 Virtual Machines300 Problems
● We can now support Freemium!● Each site has 3+ environments● Containers for PHP-FPM, Nginx, Mount processes, MySQL DB and Redis● To 300 30GB VMs, 100,000 LAMP stacks, ~750,000 containers (TODO count?)
Problems = Infrastructrue*Sites
PITA Coefficient (O)
PITA Coefficient (O)
ODrupal Developer ~= 0.27
OPage View ~= 2.5e-7
OContainer ~= 0.005
ODrupal User ~= 0.025
OVM/Server ~= 8.3
With two Containers on one VMRisk = ½Likelihood * 2xConsequences
With two, single-tenant VMsRisk = 2xLikelihood * ½Consequences
Risk = Likelihood * Consequence
Self-healing Problems
Problems Requiring Basic Manual Intervention
Problems Requiring Decisions
Problems Requiring Coding
Problems Requiring Hard Decisions
Easy Hard
Humans Decisions Compound
5 servers means 10 (network) problems6 servers means 15 (network) problems
1 more server bought you 5 problems
http://aphyr.com/posts/288-the-network-is-reliable
O(N2) Network Failure Paths
If you want fewer Problems
● Increase Mean Time Between Failure○ You could get more reliable things….where?○ You can get fewer things!
● Decrease Mean Time To Resolution○ You can speed-up detection, insight, resolution○ You can reduce reliance on human decisions
“Chief Chirpa Sucks”
[nick@endpoint9a71a1ef ]$
vs.
[nick@ChiefChirpa ~]$
Chapter 2
In which we use English to describe
WTF containers are, and why people
might want to use them.
This is what our marketers say we built
Resource-constrained, system-isolated, metered processes.
Containers are simply....
Time to container$: systemd-nspawn -D /srv/debian/ date
Spawning namespace container on /srv/debian.
Init process in the container running as PID 9159.
Tue Jun 3 17:32:14 UTC 2014
real 0m0.007suser 0m0.001s
real 0m0.007s
Even if you just run one server...
OS Upgrades SuckCloud VMs get ‘weird’Container migration FTW.
End of lifeis a
way of life!
OS upgrade dropsavg server life
Container Migration to MariaDB
One-click migration to convert thousands of MySQL containers to MariaDB
Chapter 3
In which we plumb the depths of the
/proc filesystem, in search of clues
about CGroups and namespaces
Containersare based on the
CGroups and Namespacesfunctionality on the Linux kernel
cgroups is merely a hierarchy ofprocesses All processes
Development processes
PHP-FPM Drush
Production processes
Drush Rsync
75% 25%
cgroups is merely a hierarchy ofprocesses All processes
Processes for people I don’t like
PHP-FPM Drush
Processes forpeople I like
Drush Rsync
2%98%
cgroups submodules aka Controllers
● memory: Memory controller● cpuset: CPU set controller● cpuacct: CPU accounting controller● cpu: CPU scheduler controller● devices: Devices controller● blkio: I/O controller for block devices● net_cls: Network Class controller● ...
Kernel Interaction: /proc, /sys/fs # Inspect ip forwarding setting
$: cat /proc/sys/net/ipv4/ip_forward
# Turn ip forwarding off/on
$: echo "0" > /proc/sys/net/ipv4/ip_forward
$: echo "1" > /proc/sys/net/ipv4/ip_forward
# Examine file descriptors used by nginx..
$: ls -l /proc/$NGINX_PID/fd/
lrwx------ 1 root Jun 3 13:48 0 -> /dev/null
lrwx------ 1 root Jun 3 13:48 10 -> socket:[64376]
l-wx------ 1 root Jun 3 13:48 2 -> /var/log/nginx-access.log
# Nuke logs
$: rm -rf /var/log/nginx-access.log
# Read log (even after you rm -rf’d it!)
$: tail /proc/$NGINX_PID/fd/2
62.211.78.166 - - [05/May/2014:10:00:54 +0000] "GET /vtiger.php
Kernel Interaction: /proc, /sys/fs
# Create a Control Group named “AA”
$: mkdir /sys/fs/cgroup/memory/AA
# New directory magically contains...
$: ls /sys/fs/cgroup/memory/AA
cgroup.clone_children
memory.kmem.usage_in_bytes memory.limit_in_bytes
cgroup.procs memory.max_usage_in_bytes … ...
Managing cgroups: manually
# Limit AA’s memory to 100 bytes
$: echo 100 > /sys/fs/cgroup/cpu/AA/memory.limit_in_bytes
Managing cgroups: manually
Creating cgroups: libcgroups# Create a Control Group named “AA”
$: cgcreate -g cpu:AA
# Set the ‘cpu.shares’ to 100 for “AA”
$: cgset -r cpu.shares=100 AA
# Run a python script in the “AA” control group
$: cgexec -g cpu:AA python test.py
# Limit teensy’s memory to 100 bytes
$: cgcreate -g memory:teensy
$: cgset -r memory.limit_in_bytes=100 teensy
# Associate current shell’s PID with “teensy”
$: echo $$ > /sys/fs/cgroup/memory/teensy/tasks
# Any command will exhaust memory
$: ls
Killed
memory.limit_in_bytes in action
cpu.shares in action
PID USER PR NI VIRT RES SHR S %CPU 9693 root 20 0 107908 624 532 R 60.08 9692 root 20 0 107908 624 532 R 6.307
cpu.shares = 100
cpu.shares = 10
# Run script within each cgroup
$: cgexec -g cpu:AA python test.py &
$: cgexec -g cpu:BB python test.py &
$: top
● Mount● IPC● PID● User● UTS● Network
Kernel Namespaces
“Before one can share, one must first unshare” - Share Bear
# Run a shell with isolated
# network namespace:
$: unshare --net /bin/bash
Chapter 4
In which we agree that nobody (here)
wants to care about /proc, /sys/fs,
and we investigate alternatives
Container Managers
https://github.com/containers/container-rfc
LXC
● The liblxc library● Several language bindings (python3, lua,
ruby and Go)● A set of standard tools to control the
containers● Container templates
Let Me Contain That For You (lmctfy)
● Created by Google● Open Source(ish)● Every process at Google runs within lmctfy● Supports nested containers
systemd-nspawn
● From systemd project “PID EINS!”● Will ship with all Fedora, RHEL, Ubuntu1
[1] It will ship even with you on boardhttps://speakerdeck.com/joemiller/systemd-for-sysadmins-what-to-expect-from-your-new-service-overlord
# Launch Vagrant
$: vagrant ssh
# Install a base debian tree
$: debootstrap unstable /srv/debian/
# Launch a debian container
$: systemd-nspawn -D /srv/debian/
systemd-nspawn
Container Inception
# Launch a read-only debian container
$: systemd-nspawn --read-only -D /srv/debian/
systemd-nspawn
Docker“In its early age, the dotCloud platform used plain LXC (Linux Containers)....The platform evolved, bearing less and less similarity with usual Linux Containers.”1
[1] http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part
[2] https://prague2013.drupal.org/session/automate-drupal-deployments-linux-containers-docker-and-vagrant
Check out @ricardoamaro’s Drupalcon Prague session2
Containerizeralater SpectrumDocker nspawn lxc lmctfy
And once you get containers….
http://coreos.com/blog/cluster-level-container-orchestration/
● Servers solve and create problems● Containers yield agile portability● Containers = CGroups + namespaces● Use tools to manage containers● The future is now
Pantheon, a platform for the content web, running 10s of Ks of LAMP CMS installshttps://www.getpantheon.com/customers IMAGES
Thanks!Nick [email protected]/nstielau/containerz
Image CreditsContainers: https://flic.kr/p/4o3Ria
Clouds: https://flic.kr/p/hHRdBL
Back to the Future (Lego): https://flic.kr/p/fbThy5
Dirty Hands: https://flic.kr/p/8G3aM5
Risk: https://flic.kr/p/81nfaV
Pita Equation: http://www.codecogs.com/latex/eqneditor.php
Pita Evil Eyes: http://www.clipartbest.com/cliparts/7ia/4eL/7ia4eL9iA.png
Containers http://bighugelabs.com/onblack.php?id=6764705137&size=large
CGroups http://fbcg.com/small-groups/
Pengiun Container: http://2.bp.blogspot.com/-47sakFH6uSw/UXgrhNqYF8I/AAAAAAAAHzQ/0W8zFVgR--w/s1600/lxc.png
No Logo: http://static.tumblr.com/i4bgb5d/Uzblps3wo/no-logo-1.jpg
Book sprectrum: https://flic.kr/p/k5jmja
Bottles: https://flic.kr/p/nj8jMn
Mac: https://flic.kr/p/auKEX2
Corn: https://flic.kr/p/6NVL68