Cassandra and DockerTwo buzzwords or a match made in heaven?

instaclustr.com @Instaclustr

Who am I and what do I do?• Ben Bromhead

• Co-founder and CTO of Instaclustr -> www.instaclustr.com

<sales>

• Instaclustr provides Cassandra-as-a-Service in the cloud.

• Currently in AWS, Azure and Google Cloud in private beta with more to come.

• We currently manage 50+ nodes for various customers, who do various things with it.

</sales>

Objectives

• A quick intro on docker.

• Why docker matters and how it works.

• Working with Cassandra and docker.

• Running C* in a constrained env w/ docker

The ChallengeStatic website

Web frontend

DBQueue

Background workers

API endpoint

nginx 1.5 + modsecurity + openssl + bootstrap 2

Java + Cassandra + Spark

Ruby + Rails + sass + Unicorn

Redis + redis-sentinel

Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs

Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client

Development VM

QA server

Public Cloud

Disaster recovery

Contributor’s laptop

Production Servers

Mul$p

licity

*of*S

tacks*

Mul$p

licity

*of*

hardware*

environm

ents* Production Cluster

Customer Data Center

Do*services*and*apps*interact*

appropriately?*

Can*I*migrate*

smoothly*and*quickly?*

Static website

Web frontend

Background workers

DB

Analytics

Queue

Development VM QA Server Single Prod

ServerOnsite Cluster Public Cloud Contributor’

s laptopCustomer

Servers

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

Dependency madness

Pre 1960’s transportMul$p

licity

*of*G

oods*

Mul$p

ilicity*of*

metho

ds*fo

r*tran

spor$n

g/storing*

Do*I*worry*about*

how*goods*interact*

(e.g.*coffee*beans*next*to*spices)*

Can*I*transport*quickly*and*sm

oothly*(e.g.*from

*boat*to*train*to*truck)*

Also a dependency mess? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

? ? ? ? ? ? ?

Solution: Intermodal Shipping containersMul$p

licity

*of*G

oods*

Mul$p

licity

*of*

metho

ds*fo

r*transpor$n

g/storing*

Do*I*worry*about*

how*goods*interact*

(e.g.*coffee*beans*next*to*spices)*

Can*I*transport*quickly*and*sm

oothly*(e.g.*from

*boat*to*train*to*truck)*

…in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another!

A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.!

Docker, shipping containers for codeStatic website Web frontend User DB Queue Analytics DB

Development VM

QA server Public Cloud Contributor’s laptop

Mul$p

licity

*of*Stacks*

Mul$p

licity

*of*

hardware*

environm

ents*

Production ClusterCustomer Data Center

Do*services*and*apps*interact*

appropriately?*

Can*I*migrate*

smoothly*and*quickly*

…that can be manipulated using standard operations and run consistently on virtually any hardware platform !

An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…!

Solves the deployment dependency matrixStatic website

Web frontend

Background workers

User DB

Analytics DB

Queue

Development VM QA Server Single Prod

ServerOnsite Cluster Public Cloud Contributor’

s laptopCustomer

Servers

Why docker matters• Finally Developers have a solution to build once and deploy

anywhere

• Finally Ops/Admin has a solution to configure anywhere

• Finally DevOps is easy

• Dev == Test == Staging == Production

• Move with speed

Docker, how it works.• Runs anywhere (Linux kernel 2.6.32+)

• Uses lightweight VMs:

• Own process space (namespace)

• Process isolation and resource control (cgroups)

• Own network adapter

• Own filesystem (chroot)

• Linux Analog to Solaris Zones, *BSD jails

Docker, how it works.• Difference between a container and a VM

Virtual Machine Container

Docker, how it works.• What about the packaging component?

• Uses Union filesystem to create a git like workflow around your deployed code:

!!

Docker!Container!Image!Registry!

Push%

!!!!

Bins/!Libs!!!!!

App!A!

App!Δ!!

!!!!Bins/!

Docker'Engine' Docker'Engine'

Update'

Host'is'now'running'A’’''

App'Δ''

''''Bins/'

''''

Bins/'Libs'''''

App'A'

''''Bins/'

''''

Bins/'Libs'''''

App'A’’'

Host'running'A'wants'to'upgrade'to'A’’.'Requests'update.'Gets'only'diffs''

Docker is it production ready?

Cassandra and Docker• So how do we get on board the hype train? Without killing

performance or stability?

• Build Cassandra in a docker container, run it, then test.

• Run in dev to get comfortable with it.

• Talk to others who use it in production

• https://github.com/docker/docker/issues - You will spend a lot of time here

Docker + Networking• 1st attempt, throughput dropped in half!

• Writes sucked, streaming sucked, what was going on?

• Quick check with iperf showed a 50% hit in throughput

• Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput.

• Use the host network stack instead (—net=host), only saw a ~10% hit on network performance

Docker + Networking

• Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput.

• Use the host network stack instead (—net=host), only saw a ~10% hit on network performance

• Also solves NAT issues in an AWS like networking environment.

Docker + Filesystem• Don’t want to throw it out when you upgrade/stop container.

• Use volume mount folders to the underlying host!

Docker + Filesystem

• The filesystems (AUFS, BTRFS etc) that bring great benefits to Dockers workflow around building and snapshoting containers are not very good for databases.

• UnionFS (AUFS) is terrible for writing lots of big files.

• BTRFS is a pain to use from an ops point of view.

• Hooray volume mounts use the underlying filesystem.

Docker + Process Capabilities

• Mlockall permission denied? A process needs CAP_IPC_LOCK & RLIMIT_MEMLOCK in order to perform this operation. By default docker doesn't assign this to a running container…

• Can’t use native memory. Cassandra becomes slooow.

• Can use --privileged and be done with it. Kind of lazy though

• Use --cap-add instead

Docker + SIGTERM propagation• When stopping the process docker will send a SIGTERM.

• PID 1 does not have default signal handlers!

• Bad if you use a bash script to launch Cassandra

Images shameless copied from https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/

Docker + SIGTERM propagation

• Java to the rescue!

• Make sure you run the cassandra bash script with -f (foreground)

• exec causes the JVM to replace the bash process… making the world a happier place

Docker + SIGTERM propagation• Tools like OpsCenter Server will have trouble with this.

• Can be fixed using a wacky combination of trap and wait stanzas in your OpsCenter Server script (see http://veithen.github.io/2014/11/16/sigterm-propagation.html)

• But now you have a bash script that duplicates init/systemd/supervisord

• The debate rages on…

Docker + CoreOS• Docker + fav OS + CM?, CoreOS + etcd, Swarm + Machine, Deis

etc

• We chose CoreOS (Appeared to be sane, etcd is cool, systemd if you are into that kind of thing)

Docker + CoreOS

• Disable automatic updates + restarts (seriously do this)

• Fix logging, otherwise you will log to 3 locations (/var/log/cassandra, journalctl and dockers json based log

• JVM will exit with error 143 (128 + 15 for SIGTERM). Need to ignore that in your systemd service definition.

Docker + Dev Env

• Docker relies on Linux kernel capabilites… so no native docker in OS X

• We use OSX for dev, so we run vagrant and the CoreOS vagrant file https://github.com/coreos/coreos-vagrant

• Look at https://github.com/tobert/cassandra-docker for something more off the shelf

Docker + C* + Dev Env

• How do I run lots of C* instances on a VM or my dev laptop without it falling over?

• Make it run as slowly, but as stable as possible!

• This is actually a great learning exercise as you discover a lot about how Cassandra works under the hood.

Docker + C* + Dev Env• Set Memory to be super low, edit your cassandra-env.sh:

MAX_HEAP_SIZE="128M"  HEAP_NEWSIZE="24M"

Docker + C* + Dev Env• Tune compaction to have free reign and to smash the disk

concurrent_compactors:  1  in_memory_compaction_limit_in_mb:  2  compaction_throughput_mb_per_sec:  0

Docker + C* + Dev Env• Let’s use HSHA thrift server as it reduces the memory per thread

used.

rpc_server_type:  hsha

Docker + C* + Dev Env• The HSHA server also lets us limit the number of threads serving in

flight requests, but still have a large number of clients connected.

concurrent_reads:  4  concurrent_writes:  4  rpc_min_threads:  2  rpc_max_threads:  2

• You can play with these to get the right numbers based on how your clients connect, but keep them low.

Docker + C* + Dev Env• This is Dev! Caches have no power here!

key_cache_size_in_mb:  0  reduce_cache_sizes_at:  0  reduce_cache_capacity_to:  0

Docker + C* + Dev Env

• How well does this work?!?!

• Will survive running the insane workload in the c* 2.1 new stresstest tool.

• We run this on AWS t1.micro instances

• Sign up at https://www.instaclustr.com and give our new Developer nodes a spin!

Go forth and conquer!

Questions?