Dynamic Language Dev Process at LTS:An attempt at a taxonomy of modern web development tooling.

Dave Mayo

Thursday, June 4, 2015 (2015-06-04)

Précis

This presentation is aimed at getting across two things:

A set of categories of tools/components that are commonto current "reasonable practices" in dynamic languagedevelopment. Not necessarily best practices, but somecollection of stuff that looks like the hammer and wrenchof "modern" development practice.Examples, mostly Ruby/Python, of tools in each of thesecategories, and some attempts at explaining their use.

"Reasonable," not "best?"

Best Practices - Optimum way to do things, basicallyhygiene. If you’re not doing them, you’re wrong."Reasonable Practices" - Ways to do things, whichreasonable people have selected, across multiplelanguages and environments.

Some reasonable practices are kind of terrible over someproblems.

I’ll be covering some general overviews, and comparing thetools in the field available in the Ruby and Python ecosystems,with some mention of PHP/JS/Clojure.

This isn’t supposed to be utterly unstructured - but wheredeparting from structure has let me go on for a minute or soabout useful tools/practices, I have done so. Because tools!

Basic anatomy of the stacks we target

Most deployments of systems in dynamic languages involve themore or less the following setup, from the outside in:

Web ServerApplication ServerDatabase

Web Server

This handles static traffic, and then proxies requests fordynamic stuff to an application server.

Application server

This is what actually runs your Ruby/Python/PHP/whateverprocesses.

Can be managed directly by the web server (think mod_php,passenger, mod_wsgi) or external (unicorn, gunicorn,candycorn1, php-fpm)

1Not real, just checking if you’re all awake

Database

Or databases. A common pattern is, SQL by default, mixedSQL/NoSQL if you have a combination of mostly staticstructured data and fast-churning unstructured data, andNoSQL if you’re under thirty and wear t-shirts to work. I’msemi-arbitrarily throwing in-memory caches like Redis ormemcached in here, we’re not interested in them, anyway.

Our Taxonomy

External management toolsversion control (as first class citizen)Dependency Isolation (This is NOT virtualization)Package managementDeployment/Remote operations toolProject Generators

Internal language conveniencesEnhanced REPL/Shell/ConsoleWeb server abstraction layerStandalone development serverOwn-language task runnerAsset management

Subordinate language conveniencesHTML/CSS/JS preprocessing

External Management Tools

Version Control

Using version control is a best practice. Do it or be ashamed.What I mean by version control here is really integration ofother tools with your version control, i.e. your other toolsinteract with your version control system.

Examples

Capistrano pulls software (in most common cases) fromdistributed VCs.Code review tools pull lists of commits from VCS toattach reviews to.Continuous Integration stuff runs integration testsper-commit (or per commit to master repository)

Dependency Isolation

Tools to isolate your project from the OS-installed version ofyour language and its libraries.

Enables complex versioning of language and libraries on aper-project level, really handy when developing many projectswith different deps on same dev system.

Examples include:RVM, rbenv, Bundler (Ruby)virtualenv (Python)leiningen (Clojure)also exists for PHP, according to Google

Dependency Isolation

For Ruby projects at LTS, we use:RVM to manage the ruby interpreter, and the BundlergemBundler to manage all per-project dependencies

For Python:Virtualenv and wrappers around virtualenv are basicallythe only game in town. . .

Package Management

Specifically, repository-based project management. Throw outa name and a version/version spec, get package (anddependencies).

Upgrade path as you get more serious basically involves eithersupport for external/local packages, or support for running therepo software.

At this point, for anything nontrivial, you’re probably dealingwith multiple layers of package management.

Package Management

Bundler in Ruby processes a GemfileVirtualenv runs pip in the virtual environmentnpm for node manages JS packages on the serverBower manages client side packages for JS. Alsosometimes server-side.This is ALL on top of OS package manager (e.g.apt/RPM) and third party OS-level package managers(e.g. Homebrew on OSX)

Deployment/Remote Operations

Some piece of software that gets you to single-step deploysand single step running of admin tasks on remote systems.There’s a non-zero value in having this be a tool in thelanguage you’re deploying, but it’s not really important.

Deployment/Remote Operations Examples

Capistrano (Ruby) This is what we’re using across ourRails/Django projects at LTS. It’s fairly mature, it has a lot ofstuff built in for a standard deploy flow that looks like this:

1 Clone software from git-archive to host2 Symlink configuration files3 Run system update tasks (DB migrations, asset

compilation and preprocessing, etc)4 Restart system.

Deployment/Remote Operations Examples

Fabric (Python) We don’t use this at LTS, but it seems fine.

Lower level, doesn’t have prebuilt workflow for deploy

Deployment/Remote Operations Examples

The main thing is, you should be able to enter a command onyour local dev box, and deploy to/run command on X target

Project Generators

This is an idea that’s already pervasive in frameworks.

A command that generates a somewhat customized projectskeleton is not news.

Yeoman tries to generalize this, provides essentially a genericproject skeleton generator.

Internal Language Conveniences

Enhanced REPL/Shell/Console

Interactive shells with much of the functionality of the editor,and which are capable of maintaining state over a session.The built-in shells for some langs suffice, but mostly this willbe a third party extension for reasons. Some reasons:

Python’s interactive shell doesn’t default to havingReadline support, or tab completion, or highlighting, oranything reallyRuby’s irb doesn’t highlight codeNeither of them have good built in introspection tools.

Enhanced REPL/Shell/Console

Fortunately:Ruby has ‘pry‘ and its various pluginsPython has ‘ipython‘, ‘bpython‘, etc.

Web Server Abstraction Layer

A lot of stuff these days is sitting on top of a simplified HTTPlayer that can plug into multiple servers and stack on top ofmiddleware.

Examples:Rack (Ruby)WSGI (Python)Ring (Clojure)

Standalone Server Mode

"Just for development" HTTP server mode, so you don’t haveto set up Apache on your dev laptop.

Webrick (Ruby)There are a bunch of different answers for Python, mostlyframework level.

Side note

It’s good to have this, but sometimes it’s better to put in alittle effort.For LTS projects, my local dev environment is nginx, with aserver block (virtual host) per project, and an entry in myhosts file to simulate DNS lookup.

Own Language Task Runner

Because who likes makefiles?

Own Language Task Runner

Own language is important, because introspection ofapplicationExamples:

Rake, Thor (Ruby)Django has manage.py tasks, fabric can be used moregenerally in PythonGrunt, Gulp, a bunch of other contenders for JS

Asset Management

For frameworks, this is usually baked in.

Rails asset pipelineDjango does via manage.py

Asset Management

Outside of frameworks, usually handled via the task runner

Preprocessing, moving assets into directories, setting e-Tags,etc.

Subordinate Language Conveniences

HTML Preprocessing

Templates. Stuff better have em’

CSS Preprocessing

HUGE uptake, and with good reason.Done even moderately well, MASSIVE deduplication ofrepeated work.Examples:

LessSass/SCSSStylus

JS Preprocessing

TBH, I don’t really do this.

JS is a high level language.That being said, CoffeeScript and similar have sane iterationprimitives, do some evening out of scope stuff.

Miscellanea

If you work on the command line, run out and grab this rightnow:

ack - http://beyondgrep.com

A grep that understands programming languages and VCSReturns order of magnitude faster than recursive fgrep or mostother search methods, despite being written in Perl instead ofC

Miscellanea

If you use an editor or IDE and write markup(HTML/XMLish), go grab this:

http://emmet.ioIt’s an in-editor template language. Write a CSS-selectorlooking thing, hit expand, markup!