nixcon berlin 2015 - nix at logicblox
TRANSCRIPT
INTRODUCTION
©2014. LogicBlox. All Rights Reserved.
▪ Rob Vermaas
▪ IRC: ikwildrpepper▪ Email: [email protected]
MY NIX HISTORY
©2014. LogicBlox. All Rights Reserved.
▪ Contributed to:▪ Nixpkgs, Hydra, NixOps, Nix
▪ First Nix related commit▪ August 2004, adding Octave
▪ Infrastructure support
▪ NixOS Foundation
LOGICBLOX
▪ Develops a state-of-the-art database system that is used to develop applications for many top retailers and banks
▪ 150 people, all over the world▪ USA, Canada, Tunisia, The Netherlands, Greece, United Kingdom, Belarus, Sweden
▪ LogicBlox Nix-ers:▪ Eelco Dolstra
▪ Rob Vermaas
▪ Martin Bravenboer
▪ Shea Levy (former employee)
▪ Evgeny Egorochkin (contractor)
©2014. LogicBlox. All Rights Reserved.
WHY DO I WANT TO USE NIX?
©2014. LogicBlox. All Rights Reserved.
▪ Nix protects me against me
▪ Nix let’s me do things multiple times consistently, even on different machines
▪ Nix exposes the things I forget
▪ Nix, the one language to rule them all
HOW TO NIXIFY A COMPANY
©2014. LogicBlox. All Rights Reserved.
Step 1:Find a nice company
Step 2:Apply Nix everywhere
Step 3:Profit
▪ 3 step plan to Nixify a software company
HOW TO NIXIFY A COMPANY
©2014. LogicBlox. All Rights Reserved.
Step 1:Find a nice company
Step 2:Apply Nix everywhere
Step 3:Profit
▪ 3 step plan to Nixify a software company
BuildTest
Deploy
NixNixOps / NixOS
NIX - THE BIG THREE
©2014. LogicBlox. All Rights Reserved.
Hydra
Build Test Deploy
Nix
NIX - THE BIG THREE
©2014. LogicBlox. All Rights Reserved.
Hydra
Build Test Deploy
▪ How to start? Too much to do! Can’t change everything immediately!
NixOps / NixOS
NIXIFYING EXISTING BUILDS
▪ Closed source software
▪ Buildscripts everywhere!
▪ Hardcoded locations
▪ Binary files/libraries
©2014. LogicBlox. All Rights Reserved.
▪ Long builds
▪ Huge builds
▪ Network access
▪ Not just 1 language
HYDRA - CONTINUOUS BUILDS
▪ Why Hydra? ▪ All builds consolidated in one system, with one language to describe all▪ Less maintenance▪ Better traceability▪ Reproducibility of build▪ Caching of builds, very useful when composing builds▪ Integrates with our deployment tools
▪ Terminology▪ Projects▪ Jobsets / Jobs▪ Builds
▪ Features▪ Easy to clone jobsets and build branches▪ Ability to compose multiple builds as one logical build
©2014. LogicBlox. All Rights Reserved.
HYDRA - WHAT DO WE BUILD
▪ Hydra structure:▪ 19 active projects
▪ 205 active jobsets
©2014. LogicBlox. All Rights Reserved.
HYDRA - BUILDING PLATFORM
©2014. LogicBlox. All Rights Reserved.
▪ What is built?▪ Builds, debug builds, regressions tests,
benchmarks, applications, binary releases
▪ Continuous builds▪ Each commit -> ~ 90 builds, ~ 3 buildhours
▪ Nightly integration builds▪ Each commit -> ~ 300 builds, ~ 120
buildhours
HYDRA - BUILDING CLIENT APPLICATIONS
{ platform_release ? "3.9.5", increment ? ./.}:with (import <config> {});let platform = getPlatform platform_release;in genericAppJobset { build = buildLBConfig { name = "increment-${version increment}"; src = increment; buildInputs = [ logicblox bloxweb ]; }; extraNixPath = { inherit increment; }; }
Everything needed for deployment▪ LB application (installer)▪ Software dependencies (logicblox, bloxweb, nixpkgs)▪ Operating system (nixpkgs)▪ System configuration libraries for Logicblox (lbdevops)▪ Machine definitions (increment)
HYDRA - CHARON_CLOSURE (EHHHM, NIXOPS_CLOSURE)
HYDRA - BUILDING MACHINE IMAGES
©2014. LogicBlox. All Rights Reserved.
▪ Also used for NixOS EC2/GCE/VB images:
▪ <nixpkgs/nixos/lib/make-disk-image.nix>
with (import <nixpkgs> {});
vmTools.runInLinuxVM (
stdenv.mkDerivation {
name = "need-root";
buildCommand = ''
echo Do stuff only root can do
'';
})
HYDRA - TESTING ON OTHER DISTRIBUTIONS
©2014. LogicBlox. All Rights Reserved.
fedora19x86_64 =
testDownloadPackage
(diskImageFuns.fedora19x86_64 {
size = 2048;
extraPackages = [
"bash"
"java-1.7.0-openjdk"
"python"
"procps-ng"
];
});
HYDRA - BENCHMARKING
©2014. LogicBlox. All Rights Reserved.
▪ Dedicated benchmarking teamusing nix as benchmark composer
▪ Tracking performance over time
▪ Evaluate performance of PR’s
▪ Improvements▪ Scheduler
▪ Hydra EC2 provisioner
▪ Prioritization of builds
Softw
are
Hardware
Data
CPU x Memory x Storage
Systemx
Versionx
Benchmark
Scale x Data Set
BOB / HYDRA
▪ 2011 - 2013: Started out with 3 linux machines, up to 8 machines▪ On-premise hardware in the main office in Atlanta
▪ High initial investments
▪ Need IT people to handle hardware issues
▪ Need fast internet connection to scale out
▪ 2013-2015: 21 x Linux, 2 x Mac OS X▪ 8 general purpose
▪ 12 dedicated benchmark
▪ Can scale up on Amazon EC2, when needed▪ https://github.com/NixOS/hydra-provisioner
©2014. LogicBlox. All Rights Reserved.
NIXOPS
Tool to deploy networks of NixOS machines
▪ Single formalism for packaging, provisioning and configuration management▪ Easy to add software to deployment▪ Based on Nix package manager
▪ Expressive configuration language
▪ Supports separation of logical and physical aspects of a deployment▪ Reuse spec for multiple cloud backends
▪ Combined configuration and provisioning▪ Interaction between config and infrastructure
©2014. LogicBlox. All Rights Reserved.
NIXOPS
©2014. LogicBlox. All Rights Reserved.
+ Mixed spec
Logical spec
+ VirtualBox physical spec
+ EC2 physical spec
+ GCE physical spec
EC2
Physical hardware
EC2
NIXOPS
▪ What to deploy?
▪ Where to deploy?
©2014. LogicBlox. All Rights Reserved.
{ machine =
{ config, pkgs, ... }:
{ imports = [ ./nix-homepage.nix ]; };
}
{ machine =
{ deployment.targetEnv = "ec2";
deployment.ec2.accessKeyId = "ec2";
...
};
}
NIXOPS - ARGUMENTS
{ account ? "logicblox-dev"
, region ? "eu-west-1"
, instanceType ? "r3.xlarge"
, name
, enableBatch ? false
, production ? false
, ...
}:
{
…}
©2014. LogicBlox. All Rights Reserved.
NIXOPS
▪ Do we use NixOps?▪ Hell yeah!
▪ Networks/clusters with up to 54machines per cluster
©2014. LogicBlox. All Rights Reserved.
NIXOPS
▪ Do we use NixOps?▪ Hell yeah!
▪ Networks/clusters with up to 54machines per cluster
▪ Last 30 days▪ 100 ‘nixops create’
▪ 88 ‘nixops delete’
▪ 1700 ‘nixops deploy’
▪ 300 ‘nixops destroy’
▪ 40 different people running deployments
©2014. LogicBlox. All Rights Reserved.
NIXOPS
▪ Do we use NixOps?▪ Hell yeah!
▪ Networks/clusters with up to 54machines per cluster
▪ Last 30 days▪ 100 ‘nixops create’
▪ 88 ‘nixops delete’
▪ 1700 ‘nixops deploy’
▪ 300 ‘nixops destroy’
▪ 40 different people running deployments
©2014. LogicBlox. All Rights Reserved.
Type # Where?
Internal servers ~ 50 EC2, GCE, Hetzner, on-premise hardware
Client applications > 500 EC2, GCE
Dynamically scaled 0-1500 EC2
NIXOPS - OUR CURRENT SETUP
▪ Deployment servers▪ Log in via ssh, switch to specific deploy user via sudo, containing a group’s deployments
▪ Different deployment server for development / production
©2014. LogicBlox. All Rights Reserved.
NIXOPS - OUR CURRENT SETUP
▪ Deployment servers▪ Log in via ssh, switch to specific deploy user via sudo, containing a group’s deployments
▪ Different deployment server for development / production
▪ Issues▪ Hard to use for non-technical users (project-managers, sales)
▪ Easy access to provider (AWS/GCE) credentials
▪ Inconsistent deployments (e.g. managing code clones)
▪ Hard to track what is going on
©2014. LogicBlox. All Rights Reserved.
DEPLOYMENT - NIXOPS - OUR CURRENT SETUP
▪ Deployment servers▪ Log in via ssh, switch to specific deploy user via sudo, containing a group’s deployments
▪ Different deployment server for development / production
▪ Issues▪ Hard to use for non-technical users (project-managers, sales)
▪ Easy access to provider (AWS/GCE) credentials
▪ Inconsistent deployments (e.g. managing code clones)
▪ Hard to track what is going on
▪ New project: NixOps Dashboard▪ Web UI for NixOps , started November 1st
▪ Will be open-sourced once we have a usable version
▪ Created by Oussama Elkaceh and me.©2014. LogicBlox. All Rights Reserved.
DEPLOYMENT - NIXOPS DASHBOARD
▪ Improved security▪ By limiting the direct access to environments that have keys to access Cloud resources. Users
will use the Dashboard (RESTful API) to perform any deployment actions.
▪ Deeper operational visibility▪ Centralized, consistent and detailed view of the real-time status of all machines and
deployments.
▪ Audit trail▪ Easy access and insight into what is happening and what has happened with regards to
deployments. Also, all performed operations are tracked and visible to administrators.
▪ Usability, Consistency, Predefined Actions and Operations▪ Anyone can access and view the state of the existent deployments / historical deployments.
©2014. LogicBlox. All Rights Reserved.
SUMMARY - LOGICBLOX AND NIX
▪ Nix / Hydra▪ Reproducible, composable builds, traceability
▪ NixOS▪ Reproducible system configuration
▪ NixOps▪ Automation of provisioning, creating reproducible networks
of machines, one click deployment!
▪ Nix helps us, so we help Nix!
▪ Giving back by contributing to:▪ Nix, Nixpkgs, Hydra, NixOps▪ Supporting the infrastructure running Nix*
▪ Binary caches, EC2 capacity
©2014. LogicBlox. All Rights Reserved.