Common LispIn Practical Usage

Contents● A walk through Lisp history● What the man in the street seems to think● Some highlights of Common Lisp● Lisp in production

History

The Beginnings● Lisp grew from a research paper by John McCarthy● He showed how you could build a language from basic operators● And a simple data structure - the list● He called it List Processor - Lisp● Importantly, this data structure was used for data and code

Academia● Lisp grew as a research language● Used as a language with which to research languages● Its academic and theoretical roots made it ideal for fundamental research● And it grew into a practical language with multiple university-supported

implementations● But having a such a high-level language with

○ First-class functions○ Garbage-collection○ Many other abstractions (bignums for example)

● Was a major resource hog on early computers - leading to:

Lisp Machines- Custom hardware designed to run Lisp- Solved many problems such as slow garbage collection- And lead to operating systems written entirely in Lisp- Lisp machines were commercialized (Symbolics etc)- Which lead to many Lisp users outside of Academia

Lisp Machines (2)

Lisp Machines (3)

Lisp Machines (4)● An O/S written almost to the metal in Lisp● Everything was editable● Hyperlinked documentation● A version of Lisp now featuring

○ A full object system○ Threading○ A large collection of data structures○ Graphics and windowing system

● All of this led to a demand for a standard, Common Lisp was born

Nothing good ever lasts● Commodity hardware became much cheaper● US government funding cuts removed a large part of the LM market● In the case of Symbolics, bad real-estate investments

● The result being that cheap and good enough won● But (Common) Lisp didn’t die as various free and commercial

implementations grew from all of this work and running on all common platforms

Today● There are multiple Common Lisp implementations running on commodity

hardware and under development● Commercial

○ Lispworks○ Franz Allegro○ Clozure (Former MCL) - free but backed by a consulting company

● Open Source○ SBCL○ Clisp○ ABCL - runs on the JVM○ And various other projects

Libraries● Common Lisp has plenty of libraries● But until recently was very badly organized● Now we have https://www.quicklisp.org/beta/

○ Perennial BETA

Resources● https://common-lisp.net/ a good gateway to resources● The various provider websites● Books

○ Practical Common Lisp, free or in book form○ On Lisp○ Assorted other books

● #lisp on freenode● https://www.reddit.com/r/lisp/

Conferences● The next European Lisp Symposium will be

○ April 16th, 17th 2018○ In Marbella

What the man on the street thinks

General criticisms that we hear● Lisp is slow

○ Original implementations, particularly of scheme (used in teaching) were entirely interpreted○ All modern implementations are compiled○ GC technology has advanced hugely

● Too many parenthesis○ It doesn’t look like C or Java, this is not a problem○ There’s a reason why Lisp code looks like an abstract syntax tree

■ Because it was, originally. Alternative syntax projects all stalled■ And now it’s recognised as an important strength■ Different != bad

Fun features

Basic syntax● Lisp’s syntax is about as simple as can be● The magic is that code and basic data structures share notation● Which makes playing games in code very easy

EXAMPLE

Basic data structures● Lists, for code and data● Some formalised List structures with API

○ PLISTS (:a 1 :b 2) with getf○ ALISTS ((:a . 1) (:b . 2)) with assoc

● Arrays● Hash tables● Structures● Classes (CLOS)

About the language● Let’s look at a few strengths of Common Lisp

○ Lambda○ Macros○ CLOS○ Conditions

● We’ll see what still sets the language apart● And why it has been so influential● And why we might still want to use it

Lambda● Lambda is possibly Lisp’s most famous export to the programming world

○ Know to some as the “anonymous function”

● Lisp Lambdas can be passed around and called just as in any other typical functional programming language

● Nothing here will be new to functional programmers

EXAMPLE

Macros● Macros take advantage of Lisp’s regular syntax to allow the writing and

rewriting of code● We can use this to do things like

○ Create DSLs○ New control structures○ Manage resources○ Just make some code prettier

● They have their downsides○ Hygiene○ Compilation issues○ Complexity

CLOS● The Common Lisp Object System works from the point of view of Generic

Functions and not classes● We have multimethods and can specialise on mutiple arguments● Around, Before and After methods are part of the language● We can customize method how methods are applied● CLOS is implemented in CLOS so we can rewrite the object system if we

wanted to.

EXAMPLE

Conditions● The Lisp conditions system is a superset of the traditional try/catch construct● Conditions are objects and can be subclassed● We can unwind the stack if we want, or not● We can restart computations in arbitrary places, changing values as we go● For development we can catch and retry areas of code that we’re working on

EXAMPLE

Real life usage

RavenPack● We process and extract structured information from unstructured text● We operate mostly on news sources from various providers● Speed is important as well as maintaining a large archive● Customers are mostly large institutions and universities

RavenPack infrastructure● Running 24/7● Data is streamed through the system● When reclassifying historical data we can be running ~1000 machines● Each machine needs immediate access to Gigs of metadata● All of this makes certain demands that Lisp helps with● Not the only solution of course

Development● Keep your new app running all day● Working on individual functions● Start and stop threads as necessary● Protect parts of the program as you develop● Doing all of this well requires a functional mindset

○ Restarting function calls requires no global state and immutable arguments○ Closures can also be problematic○ Functions must be insulated from their exterior loops

Deployment● We can start a Lisp image from zero

○ But that can take a few minutes to compile, load and load data○ And requires repositories on the server○ And access to the data in a DB or AWS

● Or we can ship a prebuilt Lisp image○ With all application data already loaded (2GB+)○ Starts in 5 seconds

● Either way we have a running REPL○ So we can connect and debug the image while it runs○ Load updates○ Fix problems at the REPL

When things go wrong● So connecting to a production server for debugging allows

○ We can stop individual threads○ Examine the stack, inspect the objects○ Change their values○ Restart the threads

● Usually this is just a precursor to developing the fix in development○ Then shipping it out○ And loading it into the running Lisp image○ Or a restart if necessary

● But it means that errors don’t result in a corpse, it’s still alive

Downsides● Limited ecosystem if you’re doing something typical● Encourages is some a “cowboy” mindset● And individualism● Too much fun? :)

Questions?● Ask me now● I’m not here tomorrow● Or email me alawson@ravenpack.com● And we are hiring, attitude is more important than experience