hydra: continuous integration and testing for demanding people: the details

Hydra: Continuous Integration and Testing forDemanding People: The Details

Sander van der Burg

Conference Compass

July 15, 2014

Sander van der Burg Hydra: Continuous Integration and Testing for Demanding People: The Details

Continuous integration

We want to deliver and test software rapidly

We quickly want to see the impact of changes to the sourcecode and its dependencies.


Hydra

Hydra: A Nix-based continuous integration server:

Generic. Supports multiple programming languageenvironments and component technologies.

Deployment. Build and test environments are deployedautomatically and all dependencies are ensured to be presentand correct.

Variability. Multiple versions/variants of dependencies cansafely coexist.

Multi platform support. Builds can be easily delegated tomachines with a different operating system.

Scalability. Builds are transparently delegated to any machinein a cluster capable of building it.


Hydra

How to use Hydra to build or test stuff?


Hydra overview


Hydra overview


Components

Queue runner: Regularly checks what has changed andwhat to build

Evaluator: Builds the jobs

Server: Web application making builds and test resultsavailable

Nix: Package mananger responsible for the actualbuilds and depedency management

Hydra overview


The Nix package manager

A package manager borrowing concepts from purely functionalprogramming languages.

x = y ⇒ f (x) = f (y)


Nix store

Main idea: store all packagesin isolation from each other:

/nix/store/40awryfqzp46m...-disnix-0.3

Paths contain a 160-bitcryptographic hash of allinputs used to build thepackage:

Sources

Libraries

Compilers

Build scripts

. . .

/nix/store40awryfqzp...-disnix-0.3

bin

disnix-env

disnix-manifest

disnix-servicekjlv4klmra...-getopt-1.1.4

bingetopt

am13rq9ka...-dbus-glib-0.102

liblibdbus-glib-1.so.2

94n64qy99...-glibc-2.19

lib

libc.so.6


Nix expressions

disnix.nix

{ stdenv, fetchurl, pkgconfig, dbus_glib

, libxml2, libxslt, getopt, nix, dysnomia }:

stdenv.mkDerivation {

name = "disnix-0.3";

src = fetchurl {

url = http://.../disnix-0.3.tar.bz2;

sha256 = "1jjmzdd7fac6isq5wdaqjbwwnsnzjag5s4...";

};

buildInputs = [ pkgconfig dbus_glib libxml2 libxslt

getopt nix dysnomia ];

buildCommand = ’’

tar xjf $src

./configure --prefix=$out

make; make install

’’;

}


Nix expressions

disnix.nix

{ stdenv, fetchurl, pkgconfig, dbus_glib

, libxml2, libxslt, getopt, nix, dysnomia }:

stdenv.mkDerivation {

name = "disnix-0.3";

src = fetchurl {

url = http://.../disnix-0.3.tar.bz2;

sha256 = "1jjmzdd7fac6isq5wdaqjbwwnsnzjag5s4...";

};



buildCommand = ’’

tar xjf $src

./configure --prefix=$out

make; make install

’’;

}


Environments

Expression defines a function that composes anenvironment in which a build is executed

Nearly any type of build can be performed inside it,e.g. C/C++, Java, Perl, Python...

We can also run tests inside these environments

The buildInputs parameters are used to configure allsettings to make a build find its dependencies, e.g.setting PATH, PYTHONPATH, CLASSPATH ...

There are also function abstractions for different kindsof packages

If no buildCommand is given, it executes the defaultGNU Autotools build procedure: ./configure;make; make install.

Nix expressions

all-packages.nix

{system ? builtins.currentSystem}:

rec {

stdenv = ... { inherit system; };

fetchurl = ...;

pkgconfig = ...;

dbus_glib = ...;

libxml2 = ...;

libxslt = ...;

getopt = ...;

nix = callPackage ../pkgs/tools/package-management/nix { };

dysnomia = import ../pkgs/tools/package-management/dysnomia {

inherit stdenv fetchurl getopt;

};

disnix = import ../pkgs/tools/package-management/disnix {

inherit stdenv fetchurl pkgconfig dbus_glib;

inherit libxml2 libxslt getopt nix dysnomia;

};

...

}Sander van der Burg Hydra: Continuous Integration and Testing for Demanding People: The Details

Nix expressions

all-packages.nix

{system ? builtins.currentSystem}:

rec {

stdenv = ... { inherit system; };

fetchurl = ...;

pkgconfig = ...;

dbus_glib = ...;

libxml2 = ...;

libxslt = ...;

getopt = ...;

nix = callPackage ../pkgs/tools/package-management/nix { };

dysnomia = import ../pkgs/tools/package-management/dysnomia {

inherit stdenv fetchurl getopt;

};

disnix = import ../pkgs/tools/package-management/disnix {

inherit stdenv fetchurl pkgconfig dbus_glib;

inherit libxml2 libxslt getopt nix dysnomia;

};

...

}Sander van der Burg Hydra: Continuous Integration and Testing for Demanding People: The Details

Environments

Composes packages by calling them with the requiredfunction arguments.

Function invocations are lazy – they are only evaluatedif needed.

Previous expression for Disnix that defines a functionis imported here.

All dependencies are composed in the same expressionas well.

Building Nix expressions

Building a Nix package:

$ nix-build all-packages.nix -A disnix/nix/store/40awryfqzp46mjzm1rwy9qa8vxscjhgx-disnix-0.3

The Nix package manager builds disnix and all itsdependencies that have not been built yet.

Hash component is derived from all build inputs used to buildthe package.


Finding runtime dependencies


bin

disnix-env

disnix-manifest


bingetopt



94n64qy99...-glibc-2.19

lib

libc.so.6


Building Nix expressions

During a build of package many side effects are removed:

Most environment variables are initially cleared or set todummy values, such as PATH.

Environment variables, such as PATH, are configured to onlycontain the specified dependencies.

Nix store paths prevent packages to be implicitly found inmany cases (unlike “traditional” systems using /usr/lib,/usr/bin or C:\WINDOWS\System32).Timestamps are set to 1 second after the epoch

Files in the Nix store are made read-only.

Optionally, builds can be performed in a chroot()environment, improving purity


User environments

I Users can havedifferent sets ofinstalled applications.

I nix-env operationscreate new userenvironments in thestore.

I We can atomicallyswitch between them.

I These are roots of thegarbage collector.

PATH

/nix/.../profiles

current

42

/nix/store

pp56i0a01si5...-user-envbin

firefoxdisnix-env

b9w6q73mqm...-disnix-0.2bin

disnix-envmr8f62946...-firefox-30.0

binfirefox


User environments





PATH

/nix/.../profiles

current

42

/nix/store


firefoxdisnix-env



binfirefox

40awryfq...-disnix-0.3bin

disnix-env

(nix-env -u disnix)


User environments





PATH

/nix/.../profiles

current

42

/nix/store


firefoxdisnix-env



binfirefox


disnix-envi3d9vh6d8ip1...-user-env

bindisnix-envfirefox

(nix-env -u disnix)


User environments





PATH

/nix/.../profiles

current

42

43

/nix/store


firefoxdisnix-env



binfirefox




(nix-env -u disnix)


User environments





PATH

/nix/.../profiles

current

43

/nix/store


firefoxdisnix-env



binfirefox




(nix-env --remove-generations old)


User environments





PATH

/nix/.../profiles

current

43

/nix/store

mr8f62946...-firefox-30.0bin

firefox40awryfq...-disnix-0.3

bindisnix-env

i3d9vh6d8ip1...-user-envbin

disnix-envfirefox

(nix-collect-garbage)


Hydra expression

release.nix

{ nixpkgs ? <nixpkgs>, systems ? [ "x86_64-linux" "x86_64-darwin" ]

, dysnomia ? { outPath = ./.; rev = 1234; } }:

let pkgs = import nixpkgs {}; in

rec {

tarball = pkgs.releaseTools.sourceTarball {

name = "dysnomia-tarball";

version = builtins.readFile ./version;

src = dysnomia;

buildInputs = [ pkgs.getopt ];

};

build = pkgs.lib.genAttrs systems (system:

let pkgs = import nixpkgs { inherit system; }; in

pkgs.releaseTools.nixBuild {

name = "dysnomia";


src = tarball;


});

...

}


Hydra expression

release.nix


, dysnomia ? { outPath = ./.; rev = 1234; } }:


rec {

tarball = pkgs.releaseTools.sourceTarball {

name = "dysnomia-tarball";


src = dysnomia;


};


let pkgs = import nixpkgs { inherit system; }; in

pkgs.releaseTools.nixBuild {

name = "dysnomia";


src = tarball;


});

...

}


Jobs

An Hydra expression is a function returing an attributeset: rec{attr1 = value1; ...; attrn = valuen; }Function parameters define variability points:

Locations of the Dysnomia, Nixpkgs collection GitrepositoriesTarget system architectures

Each attribute corresponds to a job.

Each value refers to a function performing a build ortest.

File is typically placed in the root folder of a sourcepackage.

Building jobs from the command-line

Building a source tarball:

$ nix-build release.nix -A tarball

Building Dysnomia for 64-bit AMD Linux:

$ nix-build release.nix -A build.x86 64-linux

Building Dysnomia for Mac OS X (Nix delegates the build toa Mac machine if the build is run on Linux and an externalmachine is configured):

$ nix-build release.nix -A build.x86 64-darwin


Building jobs from Hydra

Create a project:



Create a jobset:



Create a jobset:


Jobs

All (but one) inputs are provided as functionarguments to the release expression.

One input is the package itself (dysnomia) thatcontains the release.nix expression


Evaluation results (job names correspond to those defined inrelease.nix):


Hydra expression referring to other jobsets

release.nix


, dysnomiaJobset ?

import ../dysnomia/release.nix { inherit nixpkgs systems; }

, disnix ? { outPath = ./.; rev = 1234; } }:


rec {

tarball = ...


let dysnomia = builtins.getAttr system (dysnomiaJobset.build); in

with import nixpkgs { inherit system; };

releaseTools.nixBuild {

name = "disnix";

src = tarball;



};

...

}



Use an input of type: ’Previous Hydra evaluation’:


Hydra jobs I typically write

Source packages. Jobs that assemble tarballs, Zip files orother archives containing the source code.

Binary packages. The actual builds for a variety ofarchitectures, such as i686-linux, x86 64-linux,x86 64-darwin.

Program manuals. For example, building the manual fromDocbook.

Program documentation catalogs. Generating adocumentation catalog from the source code, e.g. usingjavadoc, doxygen or JSDuck.

Unit tests. Running Unit tests, for example with JUnit ormocha and producing a coverage report.

System integration tests. Composing NixOS Linux VMs withall environmental dependencies, e.g. DBMS, web server etc,and run tests inside them.


Nix channel


Nix channel

Adding a channel:

$ nix-channel --add http://localhost/jobset/disnix/disnix-master/channel/latest

$ nix-channel --update

When running:

$ nix-env -i disnixinstalling ‘disnix-0.3pre174e883b7b09da822494876d2f297736f33707a7’these paths will be fetched (0.31 MiB download, 0.91 MiB unpacked):/nix/store/70rkq38r69fwrz90ayc4fyg823z92nmf-disnix-0.3fetching path ‘/nix/store/70rkq38r69fwrz90ayc4fyg823z92nmf-disnix-0.3’...

The build gets downloaded from the Hydra server, instead of beingbuilt from source code.


Nix package manager: Exercises

Check it out yourself!!!


Availability

Nix and Hydra are available as free and open source software underthe LGPLv2 and the GPLv3 licenses:

Nix: http://nixos.org/nix

Hydra: http://nixos.org/hydra

NixOS’ Hydra server: http://hydra.nixos.org

Nix can be used on any Linux distribution, NixOS, Mac OS X,FreeBSD, and Windows (through Cygwin)

Hydra can be used on any Linux distribution


http://nixos.org/nix

http://nixos.org/hydra

http://hydra.nixos.org

Related work

Using Nix while doing development:Deploy development packages and composing an environmentin which they can be found

NixOS: http://nixos.org/nixosDeploy an entire system configuration (Linux distribution) withNix.

System integration testing with NixOSEfficiently compose networks of NixOS machines within a buildin which system integration tests can be performed

Disnix: http://nixos/disnix(Re)deploy service-oriented systems into networks of machines

NixOps: http://nixos/nixopsDeploy networks of NixOS configurations to physical machinesor into the cloudAutomatically creates VM instances if needed

NiJS: https://www.npmjs.org/package/nijsCompose Nix packages in JavaScriptInvoke JavaScript functions from Nix expressionsVery primitive stand-alone package manager


http://nixos.org/nixos

http://nixos/disnix

http://nixos/nixops

https://www.npmjs.org/package/nijs

Questions


hydra: continuous integration and testing for demanding people: the details

Software

burg hydra

continuous integration

testing fordemanding

test environments

test resultsavailablenix

test stuff

delegated tomachines

detailshydra overviewsander