弁当 (BENTO)A packaging solution for python software

WHAT’S PACKAGING ?•Whatever that has to happen between svn/git/etc... and the

end-user

•Different end-users, different workflows: not just tarballs

(maybe those guys don’t need packaging)

DISTUTILS

•De-facto solution since ±2000

•Works well for simple packages:

• Setuptools/distribute

• add package dependencies

DISTUTILS ISSUES

•Make one’s head hurt:

•Hard to extend

•Fragile codebase

•Reliability issues

BENTO

•Declarative description of package: metadata inspection is simple

• Flexible installation scheme: install any file anywhere

• Layered internal architecture: easy to extend

• Scale down and up: simpler for small packages, more flexible for big ones

A SIMPLE BENTO FILE

Name: fooVersion: 0.1Description: Package description.

Library: Packages:! foo, foo.core

BENTOMAKER

• Command-line interface to bento

$ bentomaker configure$ bentomaker build$ bentomaker install# Or more simply$ bentomaker install

ADDING DATA-FILES

Datafiles: pdf_documentation TargetDir: $pdfdoc Files: main.pdf

$ bentomaker configure --pdfdoc=/usr/pdfdoc

# ant-like glob format supportedDatafiles: pdf_documentation TargetDir: $pdfdoc Files: doc/**/*.pdf

CUSTOM INSTALL PATHS

Path: foodir Description: foo directory Default: $datadir/foo

Datafiles: fubar-doc TargetDir: $foodir Files: main.foo

$ bentomaker configure --foodir=/usr/pdfdoc

HOOK FILES

• Can “hook” python files into the build process

from bento.commands import hooks@hooks.commanddef hello(context): print "hello"

• Hook are pure python: no auto-import, etc...

$ bentomaker hellohello

FITTING IN THE PYTHON ECOSYSTEM

# Convert a setup.py to bento.info$ bentomaker convert

# setup.pyimport setuptoolsimport bento.distutilsbento.distutils.monkey_patch()# grab metadata from bento.infosetuptools.setup()

Convert distutils package to bento

Distutils compatibility layer (pip-installable !)

INTERNALS

DESIGN PRINCIPLES

• Usable as a library (still a few singleton to remove...)

• Low-coupling of commands: new package format should not care about building phase

•Driven by real packages (numpy, scipy, twisted, ipython, etc...)

CODE ORGANIZATION• bento/core

• ply-based parser (bento/core/parser)

• python representation of bento.info

• bento/commands

• commands and contexts

• bento/distutils

• distutils compat layer

COMMANDS AND CONTEXTS

Context Commandself.run(context)

• Contexts encapsulate a command ‘environment’• Context-command relationship is overridable (hook)• e.g. mechanism to interact with 3rd party build tools

self.pkgself.cmd_argvself.run_node

MORE ABOUT COMMANDS

• Command options defined externally: any command can query any other’s options

• Commands dependencies resolved at runtime

@hooks.commanddef hello(context): print "hello"@hooks.commanddef goodbye(context): print "goodbye"@hooks.startupdef startup(context): context.set_before("goodbye", "hello")

BUILD MANIFEST

• Goal: decouple build and install

• format: a json file containing info about built parts

• Inspired by cabal (‘distutils’ for haskell)

• build command produces a build manifest

• install only knows about build manifest

• Install only install built files

NODE• Every file is represented internally as a node (waf concept)

• nodes are a in-memory representation of the fs

• gives reliable relative paths between files

sdir = os.getcwd()bdir = os.path.join(os.getcwd(), "build")root = create_root_with_source_tree(sdir, bdir)top_node = root.find_node(sdir)build_node = root.find_node(bdir)print top_node.path_from(build_node)

SOME NEAT FEATURES (1)

• Using metadata in the packaged software

Name: fooVersion: 1.0MetaTemplateFile: foo.py.in

# foo.py.inname = $NAMEversion = $VERSION

# foo.py.inname = "foo"version = "1.0"

SOME NEAT FEATURES (2)

• Sane way to load data files

Name: fooVersion: 1.0ConfigPy: __config.py

# __config.pyBINDIR = r”/usr/bin”...

SOME NEAT FEATURES (3)

•Out of tree build (bye bye source tree pollution)

# Convert a setup.py to bento.info$ bentomaker --bento-info=../source_dir/bento.info

A FEW NUMBERS

• bento: ± 8000 LOC (not including ± 2500 LOC for tests)

• numpy.distutils: ±10000 LOC

• distutils: ± 10000 LOC

• numpy: ±2000 LOC for distutils vs ±1000 LOC for bento

• scipy: ±1600 LOC vs ±1100 LOC

DEMO ON NUMPY

WHERE TO GET IT ?

• Getting bento on github: http://github.com/cournape/Bento.git

• Bento doc: http://cournape.github.com/Bento

• Bento mailing-list: bento@librelist.com