vincit teatime 2015.2 - niko kurtti: saasiin pa(i)nostusta
TRANSCRIPT
Building containersTurbocharged!
Shopify
• All the commerce
• 165k+ merchants, 8 Billion GMV
• 300M+ unique visits/month
• 4000+ containers
• RoR, MySQL, Redis, Memcache, Elasticsearch, Chef, Go, Podding, Multi-DC,...
Docker at Shopify• Serving production traffic for over a year
• Gives developers more freedom (and responsibility) around how applications are being ran
• Offers super fast and reliable deploys1
1 When it works. Warstories are plenty
Building containers
Dockerfiles
Super simple (Dockerfile is essentially shell commands + a handful of specialized instructions)
Super efficient (Detects changes and skips things that have not changed)
Super inefficient and complicated for anything nontrivial
Simple, eh?FROM ruby:2.1.3RUN apt-get updateRUN apt-get upgradeRUN apt-get install -qq -y libmysqlclient-devRUN useradd --create-home rails_is_funWORKDIR /home/rails_is_funCOPY . /home/rails_is_funRUN bundle installCMD bundle exec unicorn -p 3000
NOPE NOPE NOPE
FROM ruby:2.1.3RUN apt-get updateRUN apt-get upgradeRUN apt-get install -qq -y libmysqlclient-devRUN useradd --create-home rails_is_fun
WORKDIR /home/rails_is_funCOPY Gemfile /home/rails_is_fun/GemfileCOPY Gemfile.lock /home/rails_is_fun/Gemfile.lockRUN bundle install
COPY . /home/rails_is_funCMD bundle exec unicorn -p 3000
Even with this you have no concept of Gem cache, so any changes to Gemfile essentially means building them from scratch
Not to mention the burden of apt-get update/upgrade
What about secrets?When building application containers its pretty common to have secrets (API keys, DB urls etc.).
Dockerfile doesn't support any temporary/external files. While building you can't use VOLUMES either
No real solutions. Everything is bad (secrets in plaintext in layers, external wrappers etc)
#!/bin/bashgem install ejsonejson decrypt config/config.production.ejson > config.jsondocker build -t rails_is_fun .rm -rf config.json
The things we needed• Performance (not only for building a lot of containers but
building a single one quickly)
• Security. We handle money and peoples private information, so having a secure system is kinda important
• Coordinator for builds
Solution: Write your own builder
Our custom builder Locutus is a Golang app that:
• Listens to webhooks (every commit to GH triggers a build)
• Builds containers
• Pushes to our private registry
• Has a webgui showing the build step-by-step
• Sends hooks about build statuses
• Scales horizontally, eg. we can have N builders
The interesting part, buildingWe replaced Dockerfiles with prepare, precompile and compile scripts (vanilla Bash)
These scripts actually represent a different build phase and live in the app repo
Each phase does a docker run with the script and saves the container with docker commit.
prepareis used for your system packages (eg. apt-get install libmysqlclient-dev).
In this phase we only copy the script itself to our baseimage
We cache this image based on the md5sum of the prepareDuring Shopify builds we really rarely have to run this phase since system level depencies doesn't change often.
precompileis used for application dependencies and asset building (eg. bundle install)Used for creating a folder of wanted artifacts. Only thing that is persisted is /artifactsRun on every build, but build upon the previous cache of the specific app (and branch). So we never end up pulling all gems because of a Gemfile change etc.
compileis used for moving the generated arfifacts to correct folders (eg. .bundle) and finishing setup
Run on every build, but super fast since we are just doing bunch of mv and rm (to cleanup caches etc. to slim the image)
The result of this phase is the final image we actually can deploy to be run.
Build times around 40-50 seconds thanks to heavy caching
Containers never see the keys that are used for decrypting secrets
Questions?ps. http://www.shopify.com/careers