deploying npm packages with the nix package manager

Deploying NPM packages with the Nix packagemanager

Sander van der Burg

Feb 4, 2017

Sander van der Burg Deploying NPM packages with the Nix package manager

The Nix project

Family of declarative software deployment tools:

Nix. A purely functional package manager

NixOS. Nix based GNU/Linux distribution

Hydra. Nix based continuous build and integration server

Disnix. Nix based distributed service deployment

NixOps. NixOS-based multi-cloud deployment tool


The Nix project

Non-functional properties:

Generic. Can be used with many programming languages,component technologies, and operating systems.Reproducible. (Almost) no impurities – if inputs are the same,result should be the same.Reliable. Dependency completeness, (almost) atomicupgrades and rollbacks.Efficient. Only the required deployment activities areexecuted.


NixOS configuration

/etc/nixos/configuration.nix

{pkgs, ...}:

{

boot.loader.grub.device = "/dev/sda";

fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ];

swapDevices = [ { device = "/dev/sda1"; } ];

services = {

openssh.enable = true;

xserver = {

enable = true;

desktopManager.kde4.enable = true;

};

};

environment.systemPackages = [ pkgs.mc pkgs.firefox ];

}


NixOS configuration

nixos-rebuild switch

Nix package manager builds a complete system configuration

Includes all packages and generates all configuration files, e.g.OpenSSH configuration

Upgrades are (almost) atomic

Components are stored safely next to each other, due to hashesNo files are automatically removed or overwritten

Users can switch to older generations of system configurationsnot garbage collected yet


NixOS bootloader


The Nix project

Basis of all tools: The Nix package manager


Nix store

Main idea: store all packagesin isolation from each other:

/nix/store/rpdqxnilb0cg...

-firefox-3.5.4

Paths contain a 160-bitcryptographic hash of allinputs used to build thepackage:

Sources

Libraries

Compilers

Build scripts

. . .

/nix/store

l9w6773m1msy...-openssh-4.6p1

bin

ssh

sbin

sshd

smkabrbibqv7...-openssl-0.9.8e

lib

libssl.so.0.9.8

c6jbqm2mc0a7...-zlib-1.2.3

lib

libz.so.1.2.3

im276akmsrhv...-glibc-2.5

lib

libc.so.6


Nix expressions

openssh.nix

{ stdenv, fetchurl, openssl, zlib }:

stdenv.mkDerivation {

name = "openssh-4.6p1";

src = fetchurl {

url = http://.../openssh-4.6p1.tar.gz;

sha256 = "0fpjlr3bfind0y94bk442x2p...";

};

buildCommand = ’’

tar xjf $src

./configure --prefix=$out --with-openssl=${openssl}

make; make install

’’;

}


Nix expressions

all-packages.nix

openssh = import ../tools/networking/openssh {

inherit fetchurl stdenv openssl zlib;

};

openssl = import ../development/libraries/openssl {

inherit fetchurl stdenv perl;

};

stdenv = ...;

openssl = ...;

zlib = ...;

perl = ...;

nix-env -f all-packages.nix -iA openssh

Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1

package in the Nix store.


User environments

I Users can havedifferent sets ofinstalled applications.

I nix-env operationscreate new userenvironments in thestore.

I We can atomicallyswitch between them.

I These are roots of thegarbage collector.

PATH

/nix/.../profiles

current

42

/nix/store

pp56i0a01si5...-user-envbin

firefoxssh

l9w6773m1msy...-openssh-4.6p1bin

sshrpdqxnilb0cg...-firefox-3.5.4

binfirefox


User environments





PATH

/nix/.../profiles

current

42

/nix/store


firefoxssh



binfirefox

aqn3wygq9jzk...-openssh-5.2p1bin

ssh

(nix-env -u openssh)


User environments





PATH

/nix/.../profiles

current

42

/nix/store


firefoxssh



binfirefox


sshi3d9vh6d8ip1...-user-env

binsshfirefox



User environments





PATH

/nix/.../profiles

current

42

43

/nix/store


firefoxssh



binfirefox



binsshfirefox



User environments





PATH

/nix/.../profiles

current

43

/nix/store


firefoxssh



binfirefox



binsshfirefox

(nix-env --remove-generations old)


User environments





PATH

/nix/.../profiles

current

43

/nix/store

rpdqxnilb0cg...-firefox-3.5.4bin

firefoxaqn3wygq9jzk...-openssh-5.2p1

binssh

i3d9vh6d8ip1...-user-envbin

sshfirefox

(nix-collect-garbage)


Nix expressions

openssh.nix

{ stdenv, fetchurl, openssl, zlib }:


name = "openssh-4.6p1";

src = fetchurl {

url = http://.../openssh-4.6p1.tar.gz;

sha256 = "0fpjlr3bfind0y94bk442x2p...";

};


tar xjf $src

./configure --prefix=$out --with-openssl=${openssl}

make; make install

’’;

}

Nix complements existing build tools with a pure environment

You can also run other build tools in an environment, e.g.cmake, ant, scons


Pure environments

Precautions in service of making build results bit-identical(regardless on what machine the package has been built):

Clearing (most) environment variables or setting them todummy values

Modifying search environment variables to contain Nix storepaths to only specified dependencies (e.g. PATH, PYTHONPATH,CLASSPATH, PERL5LIB)

Using designated temp folders and output folders

Making packages immutable by making their files read-only

Resetting timestamps to 1

Chroot environments/namespaces

Restricting network access


What about Node.js/NPM projects?

NPM is Node.js’ ubiquitous deployment tool.

NPM also does dependency management in addition to buildmanagement

NPM’s dependency management conflicts with Nix’sdependency management


package.json configuration

{"name" :"nijs",

"version" :"0.0.23",

"description" :"An internal DSL for the Nix package manager in JavaScript",

"repository" :{"type" :"git",

"url" :"https://github.com/svanderburg/nijs.git"

},"author" :"Sander van der Burg",

"license" :"MIT",

"main" :"./lib/nijs",

"dependencies" :{"optparse" :">= 1.0.3",

"slasp": "0.0.4"

}}

$ npm install$ ls node modules/optparse slasp


Dealing with a conflicting dependency manager

Solution: substitute the conflicting dependency manager byproviding the dependencies with Nix first!


Solution: substitute NPM’s dependency management

{ stdenv, fetchgit, nodejs, optparse, slasp }:

let

nodeSources = ...

in


name = "nijs-0.0.23";

src = fetchgit {

src = https://github.com/svanderburg/nijs.git;

rev = "...";

sha256 = "...";

};

buildInputs = [ nodejs ];


# Compose node_modules/ folder from the dependencies

...

# Perform the build by running npm install

# Since a node_modules/ folder with the dependencies already exists,

# NPM will not obtain them

npm --registry http://www.example.com --nodedir=${nodeSources} install

’’;

}




let

nodeSources = ...

in


name = "nijs-0.0.23";

src = fetchgit {


rev = "...";

sha256 = "...";

};




...





’’;

}


Substituting dependencies

We may be able to automatically generate a Nix expressionfrom a package.json file since they capture similar proper-ties!



let

nodeSources = ...

in


name = "nijs-0.0.23";

src = fetchgit {


rev = "...";

sha256 = "...";

};;




...





’’;

}


Substituting dependencies

Appears to be straight forward! Is it really that straightforward?

Generating Nix expressions

It is actually much more complicated!


NPM dependency classes

Dependencies:

Run-time dependencies of a package

Development dependencies:

Build-time dependencies, e.g. transpilersShould only be installed if they have been requested

Peer dependencies:

Check whether a shared dependency does not conflict.In old versions of NPM, missing peer dependencies were alsoinstalled.

Bundled dependencies:

Statically bundled with a package. No need to install.

Optional dependencies:

Dependencies that are allowed to break.Typically used to bundle platform-specific code.Permitting errors (especially non-deterministic ones)incompatible with Nix deployment model


NPM dependency classes

Dependencies:

Run-time dependencies of a package

Development dependencies:

Build-time dependencies, e.g. transpilersShould only be installed if they have been requested

Peer dependencies:

Check whether a shared dependency does not conflict.In old versions of NPM, missing peer dependencies were alsoinstalled.

Bundled dependencies:

Statically bundled with a package. No need to install.

Optional dependencies:

Dependencies that are allowed to break.Typically used to bundle platform-specific code.Permitting errors (especially non-deterministic ones)incompatible with Nix deployment model


Interpreting dependency classes

We only need to consider dependencies and development de-pendencies (if they are requested) and substract the bundleddependencies

Version specifiers: NPM registry

NPM version specifiers are nominal and semantically versioned:

Precise version numbers: e.g. 1.0.0, 2.1.1

Version ranges and wildcards: e.g. 1.0.x, *, >= 1.0.1

Tags: latest, beta

The above version specifiers refer to packages obtained from theNPM registry.


Version specifiers: external packages

Local filesystem paths, e.g. /home/sander/nijs

External URLs, e.g.http://example.com/packages/nijs-0.0.23.tgz

Git URLs, e.g.https://github.com/svanderburg/nijs.git

GitHub, BitBucket, GitLab, e.g. github:svanderburg/nijs

Above specifiers obtain packages from other sources than theregistry. They indirectly resolve to a package with a name andversion number.


/home/sander/nijs

http://example.com/packages/nijs-0.0.23.tgz

https://github.com/svanderburg/nijs.git

github:svanderburg/nijs

Version specifiers: translation to Nix

Nix version specifiers are exact (any point of variation is reflectedin the hash prefix):

/nix/store/rpdqxnilb0cg...-firefox-3.5.4

To make a translation from NPM version specifiers to exactversion specifiers:

We must snapshot a version range and pinpoint the resolvedversion (version ranges are unsupported in Nix)

We must supply the output hash to make it deterministic:

src = fetchurl {

url = "https://registry.npmjs.org/nijs/-/nijs-0.0.23.tgz";

sha1 = "dbf8f4a0acafbe3b8d9b71c24cbd1d851de6c31a";

};


Private, shared and cyclic dependencies

When including a package as a CommonJS module:

./node2nix/node modules/nijs/lib/execute/index.js

var slasp = require(’slasp’);

The module loader searches recursively upwards in node modules/

sub folders for the corresponding package:

./node2nix/nijs/node_modules/slasp

./node2nix/node_modules/slasp

./node_modules/slasp



Private dependency: a package residing in a package’snode modules/ sub folder.

Shared dependency: a package residing in a parent directory’snode modules/ sub folder.



When installing dependencies with NPM, NPM will not install adependency privately if a conforming shared dependency exists.Some undesired implications:

A version range does not always yield the latest version thatfits in a version range.

Cyclic dependencies are permitted. Bad practice, as packagesare supposed to be units of reuse.


Flat-module installations

npm < 3.x installs dependencies privately by default (unless aconforming dependency exists in any parent node modules/

folder):

webapp/

...

package.json

node_modules/

express/

...

package.json

node_modules/

accepts/

array-flatten/

content-disposition/

...

ejs/

...

package.json


Flat-module installations

npm ≥ 3.x implements flat-module installations (moving packagesas high as possible in the node modules/ hierarchy):

webapp/

...

package.json

node_modules/

accepts/

array-flatten/

content-disposition/

express/

...

package.json

ejs/

...

package.json

Because no packages conflict, they all appear in the top levelnode modules/ folder. Implications:

Shorter path names, some degree of deduplicationThe order in which packages are installed matters


Simulating flat-module installations

We cannot build each NPM dependency as a Nix package, with aseparate store path:

CommonJS module resolves its own symlink location. Somerelative paths may not work:

var slasp = require(’../slasp’);

Flattening the module hierarchy is imperative.

Nix packages are made immutable after they have been built.

Solution: compose the entire dependency tree statically ahead oftime in one Nix package


Replacing impure version specifiers

Some version specifiers (e.g. tags and Git URLs), trigger NPM toalways check the remote locations for changes in each npm

install run:

Replace these version specifiers by: ’*’. Downside: itsometimes confuses NPM, in particular flat moduleinstallations.

Better solution: create fake processes (e.g git) giving adeterministic result. This is still an open issue.


The solution1: node2nix

1Sort of, but not entirely :-)Sander van der Burg Deploying NPM packages with the Nix package manager

Example: using node2nix on project-level

Generating Nix expressions from a package.json file:

$ node2nix

Building the project as a Nix package:

$ nix-build -A package

./result/bin/node2nix --help

Opening a development shell:

$ nix-shell -A shell

$ node bin/node2nix.js --help


Example: using node2nix on a package set

custom-packages.json

[

"node2nix"

,"bower"

,{"nijs": "https://github.com/svanderburg/nijs.git" },{"grunt-cli": "1.2.0" }]

Generating Nix expressions:

$ node2nix -i custom-packages.json

Installing an NPM package with Nix:

$ nix-env -f default.nix -iA node2nix

$ node2nix --help


Example: simulating global dependencies

Global packages do not exist in Nix build environments. This isproblematic for certain kinds of projects, such as Grunt projects,requiring the grunt-cli to be installed globally:

package.json

{"name": "grunt-test",

"version": "0.0.1",

"private": "true",

"devDependencies": {"grunt": "*",

"grunt-contrib-jshint": "*",

"grunt-contrib-watch": "*"

}}

$ npm install

$ grunt build



We can supply global packages as extra packages:

supplement.json

[

"grunt-cli"

]

And generate the expressions as follows:

$ node2nix -d -i package.json \

--supplement-input supplement.json



override.nix

{ pkgs ? import <nixpkgs> {}

, system ? builtins.currentSystem

}:

let

nodePackages = import ./default.nix {

inherit pkgs system;

};

in

nodePackages // {

package = nodePackages.package.override {

postInstall = "grunt";

};

}

$ nix-build override.nix


Conclusion

I have explained Nix and NPM’s deployment concepts

I have described node2nix generating Nix expressions frompackage.json configurations, making it possible to deployNPM packages with the Nix package manager.

Besides the package manager, it also becomes possible todeploy entire systems using Node.js with NixOps and Disnix.


Lessons

Naming packages: a name and version number typically doesnot suffice!

Nix uses hash codes derived from all build inputs.NET global assembly cache: strong names

Organization: isolate dependencies

Cyclic dependencies: disallow them – packages are supposedto be units of reuse


Related work: other generators

npm2nix: Original generation attempt done by Shea Levy.Composes Nix store paths for each dependency and symlinksthem.

Faster, but less accurate when dealing when shareddependenciesDoes not support flat module installations or cyclicdependencies

nixfromnpm: Implementation of npm2nix’s Nix concepts inHaskell. Can partially regenerate expressions.


Related work: NPM alternatives

yarn: parallelization for speed improvements, version locking

ied: parallelization, content-addressable store, atomicity,maximal sharing of dependencies (some of its concepts areheavily inspired by Nix according to its author :-) )


References

Nix project: http://nixos.org.

Nix package manager: http://nixos.org/nix.

node2nix: https://github.com/svanderburg/node2nix.


http://nixos.org

http://nixos.org/nix

https://github.com/svanderburg/node2nix

Questions


deploying npm packages with the nix package manager

Software