the nixos project and deploying systems declaratively
TRANSCRIPT
The NixOS project and deploying systemsdeclaratively
Sander van der Burg
March 12, 2016
Sander van der Burg The NixOS project and deploying systems declaratively
On being declarative
A declarative sentence makes a statement. It is punctuated by aperiod:
The dog in the neighbor’s yard is barking.
(Source: http://www.slideshare.net/luigi a97/parts-of-a-sentence-8862361)
Sander van der Burg The NixOS project and deploying systems declaratively
On being imperative
An imperative sentence is a command or polite request:
(Source: https://xkcd.com/149/)
Sander van der Burg The NixOS project and deploying systems declaratively
On being declarative in programming
A style of building the structure and elements of computerprograms – that expresses the logic of a computation withoutdescribing its control flow(Source: https://en.wikipedia.org/wiki/Declarative programming)
Sander van der Burg The NixOS project and deploying systems declaratively
On being declarative in programming
Declarative: describing ”what” is to be computed rather than”how” to compute the result/behavior
Imperative: a description of a computation that involvesimplicit effects, usually mutable state and input/output.
(Source:
http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html)
Sander van der Burg The NixOS project and deploying systems declaratively
On being declarative in programming
Declarative: describing ”what” is to be computed rather than”how” to compute the result/behavior
Imperative: a description of a computation that involvesimplicit effects, usually mutable state and input/output.
(Source:
http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html)
Sander van der Burg The NixOS project and deploying systems declaratively
Declarative
“declarative” is a spectrum – hard to draw a hard linebetween “what” and “how”.
Imperative is not necessarily the opposite ofdeclarative.
Example: HTML and CSS
<!DOCTYPE html>
<html><head>
<title>Test</title><link rel=”stylesheet” href=”style.css” type=”text/css”>
</head><body>
<div id=”outer”><div id=”inner”>
<p>HTML and CSS are declarative and so cool!</p></div>
</div></body>
</html>
#outer {margin−left: auto;margin−right: auto;width: 20%;border−style: solid;
}
#inner {width: 500px;
}
Sander van der Burg The NixOS project and deploying systems declaratively
Example: HTML and CSS
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment: What do we want?
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment: Activities
Building
Packaging
Transferring packages from producer to consumer site
Activating
Deactivating
Modifying configuration files
Upgrading
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment complexity
Diverse technology imposes many kinds of deployment procedures:
Different operating systems, different dependencies, manyvariants
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment complexity
Deployment may need to be done on a large scale:
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment complexity
How to update the deployment frequently?
How not to break the system while upgrading?
How to minimize downtimes?
How to roll back in case of a failure?
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment automation
To deal with deployment complexities automation is needed!
Many automated deployment solutions available
Automation is typically driven by a specification
Some solutions have been developed for specific kinds oftechnology:
Apache Felix (for OSGi components)
Some solutions are general:
ChefPuppetCFEngineNix
Some solutions use declarative deployment specifications
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment automation
To deal with deployment complexities automation is needed!
Many automated deployment solutions available
Automation is typically driven by a specification
Some solutions have been developed for specific kinds oftechnology:
Apache Felix (for OSGi components)
Some solutions are general:
ChefPuppetCFEngineNix
Some solutions use declarative deployment specifications
Sander van der Burg The NixOS project and deploying systems declaratively
On being declarative in deployment
Declare what system you want to run in the consumer environment,not the activities that need to be executed to accomplish it!
Sander van der Burg The NixOS project and deploying systems declaratively
Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
remote_file wordpress_latest do
source "http://wordpress.org/latest.tar.gz"
mode "0644"
end
directory node["phpapp"]["path"] do
owner "root"
group "root"
mode "0755"
action :create
recursive true
end
execute "untar-wordpress" do
cwd node[’phpapp’][’path’]
command "tar --strip-components 1 -xzf " + wordpress_latest
creates node[’phpapp’][’path’] + "/wp-settings.php"
end
(Source: http://gettingstartedwithchef.com/first-steps-with-chef.html)
Sander van der Burg The NixOS project and deploying systems declaratively
Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
remote_file wordpress_latest do
source "http://wordpress.org/latest.tar.gz"
mode "0644"
end
directory node["phpapp"]["path"] do
owner "root"
group "root"
mode "0755"
action :create
recursive true
end
execute "untar-wordpress" do
cwd node[’phpapp’][’path’]
command "tar --strip-components 1 -xzf " + wordpress_latest
creates node[’phpapp’][’path’] + "/wp-settings.php"
end
(Source: http://gettingstartedwithchef.com/first-steps-with-chef.html)
Sander van der Burg The NixOS project and deploying systems declaratively
Declarative
The specification captures the outcome of a set ofchanges as a fixpoint. Chef converges to the outcome.
Specification applies to set of machines – but does notguarantee that an entire machine’s configuration canbe reproduced elsewhere
How to roll back to a previous configuration?
How to mimimize downtime?
NixOS
NixOS: A GNU/Linux distribution using the Nix package manager
Sander van der Burg The NixOS project and deploying systems declaratively
NixOS configuration
/etc/nixos/configuration.nix
{pkgs, ...}:
{
boot.loader.grub.device = "/dev/sda";
fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ];
swapDevices = [ { device = "/dev/sda1"; } ];
services = {
openssh.enable = true;
xserver = {
enable = true;
desktopManager.kde4.enable = true;
};
};
environment.systemPackages = [ pkgs.mc pkgs.firefox ];
}
Sander van der Burg The NixOS project and deploying systems declaratively
NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages and generates all configuration files, e.g.OpenSSH configuration
Upgrades are (almost) atomic
Components are stored safely next to each other, due to hashesNo files are automatically removed or overwritten
Users can switch to older generations of system configurationsnot garbage collected yet
Sander van der Burg The NixOS project and deploying systems declaratively
NixOS bootloader
Sander van der Burg The NixOS project and deploying systems declaratively
Nix store
Main idea: store all packagesin isolation from each other:
/nix/store/rpdqxnilb0cg...-firefox-3.5.4
Paths contain a 160-bitcryptographic hash of allinputs used to build thepackage:
Sources
Libraries
Compilers
Build scripts
. . .
/nix/storel9w6773m1msy...-openssh-4.6p1
bin
ssh
sbin
sshdsmkabrbibqv7...-openssl-0.9.8e
lib
libssl.so.0.9.8c6jbqm2mc0a7...-zlib-1.2.3
lib
libz.so.1.2.3im276akmsrhv...-glibc-2.5
lib
libc.so.6
Sander van der Burg The NixOS project and deploying systems declaratively
Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetchurl {
url = http://.../openssh-4.6p1.tar.gz;
sha256 = "0fpjlr3bfind0y94bk442x2p...";
};
buildCommand = ’’
tar xjf $src
./configure --prefix=$out --with-openssl=${openssl}
make; make install
’’;
}
Sander van der Burg The NixOS project and deploying systems declaratively
Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
openssl = import ../development/libraries/openssl {
inherit fetchurl stdenv perl;
};
stdenv = ...;
openssl = ...;
zlib = ...;
perl = ...;
nix-env -f all-packages.nix -iA openssh
Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1package in the Nix store.
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
ssh
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
42
43
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env -u openssh)
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
pp56i0a01si5...-user-envbin
firefoxssh
l9w6773m1msy...-openssh-4.6p1bin
sshrpdqxnilb0cg...-firefox-3.5.4
binfirefox
aqn3wygq9jzk...-openssh-5.2p1bin
sshi3d9vh6d8ip1...-user-env
binsshfirefox
(nix-env --remove-generations old)
Sander van der Burg The NixOS project and deploying systems declaratively
User environments
I Users can havedifferent sets ofinstalled applications.
I nix-env operationscreate new userenvironments in thestore.
I We can atomicallyswitch between them.
I These are roots of thegarbage collector.
PATH
/nix/.../profiles
current
43
/nix/store
rpdqxnilb0cg...-firefox-3.5.4bin
firefoxaqn3wygq9jzk...-openssh-5.2p1
binssh
i3d9vh6d8ip1...-user-envbin
sshfirefox
(nix-collect-garbage)
Sander van der Burg The NixOS project and deploying systems declaratively
NixOS
In NixOS, all packages including the Linux kernel andconfiguration files are managed by Nix.
NixOS does not have directories such as: /lib and /usr
NixOS has a minimal /bin and /etc
Sander van der Burg The NixOS project and deploying systems declaratively
Distributed deployment
NixOS has good properties for deployment of a single system
Can we extend these properties to distributed systems?
Sander van der Burg The NixOS project and deploying systems declaratively
Motivating example: Trac
Sander van der Burg The NixOS project and deploying systems declaratively
Motivating example: Trac
Trac can be deployed in a distributed environment:
Subversion server
Database server
Web server
Sander van der Burg The NixOS project and deploying systems declaratively
Distributed NixOS configuration
network.nix
{ storage = {pkgs, ...}:
{
services.nfsKernel.server.enable = true; ...
};
postgresql = {pkgs, ...}:
{
services.postgresql.enable = true; ...
};
webserver = {pkgs, ...}:
{
fileSystems = [
{ mountPoint = "/repos"; device = "storage:/repos"; } ];
services.httpd.enable = true;
services.httpd.extraSubservices = [ { serviceType = "trac"; } ]; ...
};
...
}
Sander van der Burg The NixOS project and deploying systems declaratively
Distributed deployment
$ nixops create network.nix -d production$ nixops deploy -d production
Build system configurations by the Nix package manager
Transfer complete system and all dependencies to targetmachines in the network
Efficient: only missing store paths must be transferredSafe: Existing configuration is not affected, because no filesare overwritten or removed
Activate new system configuration
In case of a failure, roll back all configurationsRelatively cheap operation, because old configuration is storednext to new configuration
Sander van der Burg The NixOS project and deploying systems declaratively
The Nix project
Tools part of the Nix-project: http://nixos.org:
Nix. A purely functional package manager
NixOS. Nix based GNU/Linux distribution
Hydra. Nix based continuous build and integration server
Disnix. Nix based distributed service deployment
NixOps. NixOS-based multi-cloud deployment tool
Sander van der Burg The NixOS project and deploying systems declaratively
The Nix project
Automated deployment using declarative specifications with thefollowing properties:
Generic. Can be used with many programming languages,component technologies, and operating systems.
Reproducible. (Almost) no impurities – if inputs are the same,result should be the same regardless of its location
Reliable. Dependency completeness, (almost) atomicupgrades and rollbacks.
Efficient. Only the required deployment activities areexecuted.
Sander van der Burg The NixOS project and deploying systems declaratively
Nix-related tools: how declarative are they?
Nix-related tools solve problems in a technical domain:
e.g. deployment of packages, machines, services, ...
What about your domain?
Sander van der Burg The NixOS project and deploying systems declaratively
A real world example: Conference Compass
Conference Compass provides a service to improve the waypeople experience events
Most visible part of the service: apps for conference attendees
Each customer basically gets “their own” app.
Sander van der Burg The NixOS project and deploying systems declaratively
A real world example: Conference Compass
We have a product-line using a Nix-based build infrastructure,including Hydra, driven by simple app specific configurations:
{
name = "wroclove.rb 2016";
homepage = "http://www.wrocloverb.com";
iconSet = ./icons;
backgroundImage" = ./background.png;
...
}
Sander van der Burg The NixOS project and deploying systems declaratively
A real world example: Conference Compass
The app’s contents is customizable with a configurator serviceallowing organizers to create and update their content
Apps connect to a configurator to retrieve the data to bedisplayed and other configuration settings
Integration with third party information systems is alsopossible
Sander van der Burg The NixOS project and deploying systems declaratively
A real world example: Conference Compass
{
wrocloverb = {
eventName = "wroclove.rb 2016";
domain = "http://www.wrocloverb.com";
channels = [ "wrocloverb" ];
};
otherevent = ...;
yetanotherevent = ...;
...
}
We have developed a formalism to concisely model suchconfigurations and to automatically deploy them
Tool figures out which machines to configure, what services todeploy etc.
If underlying implementation and technology evolves,specifications (probably) remains the same.
Sander van der Burg The NixOS project and deploying systems declaratively
Conclusions
I have illustated a declarative deployment vision
I have demonstrated NixOS and the Nix package manager
I have explained that domain specific deployment tools can bebuilt on top of tools from the Nix project
Sander van der Burg The NixOS project and deploying systems declaratively
References
NixOS project homepage: http://nixos.org
Software available under free and open-source licenses(LGPL/X11)
Nix package manager can be used on any Linux system, MacOS X, and (in some extent) Cygwin and FreeBSD.
Sander van der Burg The NixOS project and deploying systems declaratively
Questions
Sander van der Burg The NixOS project and deploying systems declaratively