the holistic programmer

The Holistic Programmer

Adam Keyshttp://therealadam.com

OSCON 2007

Hi, I’m Adam. I hail from Dallas, Texas; a place not known for its holism. I am a software developer. Its a profession that doesn’t have a lot of holism to it. Or at least, we don’t usually come about it that way. But I think we should give it a try.

Holistic?

Ye olde’ dictionary tells me holistic means “characterized by comprehension of the parts of something as intimately interconnected and explicable only by reference to the whole”.

Programming?

Applied to programming, I like to think that means we should focus on the whole picture and resist the temptation to lose ourself within the intricacies of the abstract levels of thought.

Let’s talk about one reason you need to take a holistic view on life.

Abstractions


AbstractionsThey leak


“All non-trivial abstractions, to some degree, are leaky.”

Joel Spolsky

He’s says they “fail”, but I think that’s a little harsh. I’ve found the abstractions that try too hard are those that are just barely useable. Or they just aren’t fun to use.

Some leaky abstractions

• TCP/IP

• SQL

• Templates

• OpenGL

• C

• Lisp

• Video games (?)

• C++ strings

• Sockets

• ORMs

You are here

User experience

HTML/CSS/JS

Ruby, PHP, Python, Perl, etc.

MySQL, PostgreSQL, Apache, Nginx, etc.

Linux, FreeBSD, etc.

Userland

Kernel API

Kernel module

GCC, ld, etc.

Processor, devices

What I want to suggest is that a profession programmer should be very familiar with the layer directly above and below them in whatever stack they are programming to. Further, a working knowledge of the layers above and below those is a distinguishing factor between the average programmer and the great programmer.

A two island tour

Originally I’d hoped to give a whirlwind tour through about eight topics that reside either above or below the typical developer. But in reality, there’s no way I could do any topic justice in five minutes. The original goal was you would walk out of here with enough background to have a conversation with someone who works on the components above or below you and understand their jargon. Since I can’t cram all that information into your head without a mechanism like in the Matrix, I’m going to cover two of them and instead try to convey what it is to have a holistic view on understanding the layers above and below you in the stack where you live.

Compilers:Epicenter of Computer

Science

The first stop on our tour is the compiler. That scary bit of code that yells at you when you do wrong and occasionally does weird things to your loops so they go faster.

Steve Yegge recently wrote about how important it is to understand how compilers work. His reason, which I’m going to expand upon, is that, as an undergraduate CS course, they bring the room together. Studying a compiler means you’ve got to understand all sorts of stuff from the previous courses you took.

Data structuresProgram text

Tokens

Syntax tree

Intermediate code

Machine code

Tokenizer

Parser

Code gen

Assembler

Lexical analysis

Semantic analysis

So here we’ve got an egregiously colorful slide. Let me start ‘splaining at the center. Compilers start with programs encoded as text, almost always in files (ahem, Smalltalk). They then run a tokenizer to perform what uppity folks will call lexical analysis. From there you have a bunch of tokens; constants, literals, operators, symbols, etc.

Next you run a parser over those symbols to get your semantic analysis on. This is where the compiler actually makes sense of your program. So here a compiler might catch you trying to put a function definition where an expression should go. If the language doesn’t allow that. The end result of all that is a tree showing how the tokens all relate to each other.

From that tree, you typically do type checking, if that’s your thing. If that goes alright then you get to generate some intermediate code. Runtimes like Ruby don’t yet get here; it just executes the syntax tree. But a classic compiler now generates some form of code that is a little less tree-like than the syntax tree but a little closer to real instructions that one can execute on some sort of machine.

Finally, you take that intermediate code and you turn it into real code. Sometimes this takes assembler that converts to machine code for some manner of machine, whether it be hardware or software. This is also where you can do many sorts of optimizations.

The main take-away here is that there are a lot of data structures here; if you kinda slacked off on learning how trees are created, you’re going to fall behind a little bit. I’m not saying we should all rush out and read everything Knuth ever wrote on data structures, but you do need at least a working knowledge.

Algorithms

OK, so now that we’ve got a basic grasp of what is going on inside ‘ye ‘olde compiler. But the processes that get us from one sort of data from another, well you could spend months studying each one.

First you’ve gotta turn the program text into tokens. That’s where you need to know about Deterministic Finite Automata and Non-deterministic Finite Automata. And of course there’s an algorithm to create the former from the latter.

So then you’ve got tokens and you need an abstract syntax tree. So you’ve gotta write a recursive descent parser from your context-free grammar. But that’s only one choice. You’ve got your LL(k) parsers, you’ve got your LALR parsers, etc. But at the end, at least you’ve got an AST.

From there you can do type checking, generate code and even do some optimization. Those are all algorithms unto themselves, built on top of algorithms and patterns like syntax trees and visitors.

Algorithms

NFA





Algorithms

NFA

DFA





Algorithms

NFA

DFA

CFG





Algorithms

NFA

DFA

CFG

Recursive descent parsing





Algorithms

NFA

DFA

CFG

Visitor






Algorithms

NFA

DFA

CFG

Visitor

Tree traversal






Algorithms

NFA

DFA

CFG

Visitor

Tree traversal


Type checking





Algorithms

NFA

DFA

CFG

Visitor

Tree traversal

Peephole optimization


Type checking





Algorithms

NFA

DFA

CFG

Visitor

Tree traversal

Peephole optimization

Code generationRecursive descent parsing

Type checking





Computer architecture

Once you get past all the analysis, you start getting pretty close to the machine. To some extent, writing code-gen tools is a matter of just chaining together a bunch of assembly recipes. Here, you need to know at least the instruction set of the machine.

But as you go even lower, you need to know how to make the machine go fast, or how to express logic in a smaller space. For that you need to about how the machine works. And for that, you need to know its architecture. Its a fun topic that I can’t really get into, but I’ll point out a couple books that I found really useful in learning about it.

Theory of computation:YAGNI (sorta)

Adam Keys secret: I never took theory of computation. My university didn’t offer it. I’ve heard it helps in grokking compilers, languages and various uppity topics. It amuses me that one of the use cases for the Ragel compiler is that you can use it to check your theory of computation homework.

The important thing to note is that I didn’t let “the man” slow me down. Just because it wasn’t offered doesn’t mean I ignored it. I’m probably totally preaching to the crowd here. I bet that half of ya’ll never stepped foot in a CS course, or ran away from them as fast as you could at some point. Point is, you’re here and you’re getting your knowledge on. That’s the important part about being a “Holistic Programmer”. You peak under the covers of the stuff around you to enhance your knowledge of what’s going on around you in your programming world.

The Realm of Possibility

Allow me to use this venue as a support group, briefly. I have a co-worker who has an approach to problem solving that drives me insane. First, he likes to explore the entire problem space, finding all the possible alternative solutions and exploring their pros and cons. Then he proceeds to make judgements on what solutions are best for solving the problem.

Whereas I prefer to operate in a more intuitive manner. I do like to ask a few questions, but mostly just to triangulate and make sure I’m on the same page as the problem. Then I like to narrow down the problem space even further so that I can implement something easily and quickly. This lets me get back to looking at LOLcats, which is obviously the end goal of any activity.

I’ll attempt to not pass judgement on either approach; they certainly each have their place. The important thing to note about my way is that its predicated on knowing a bit about a whole lot. My “intuition”, if you can call it that, is informed by poking just below the covers on many topics. Sometimes this causes me to just complain, like when a Java compiler whines about not knowing the type of something when I know its possible it could infer it, as in Ocaml. But sometimes it does pay off, like looking at the problem of extracting names, zip codes and addresses from a document and seeing a really lax compiler. True story.

CSS: Design and code at

loggerheads

Now I’d like to jump way up the stack, at least for most of us. CSS is where much of the UI magic happens in web sites and applications. It is pretty much the visual language of web design.

All that said, let me get a show of hands of those who enjoy writing CSS. Be honest, keep those hands up if you look forward to problems whose solutions lies only in CSS.

A lot of hands went down there. I’ve got theories as to why that is, which I’ll come to in a moment. But in the meantime, I’d like to explore that what and why of CSS so I can get to how a holistic programmer approaches the challenges of CSS.

Syntax and rules#foo, div.bar, a, div#quux div.urf { /* A comment */ color: #fff; background-color: #000; float: right; display: block;}

We’ll start our little foray into CSS with our programmer hats on. Here’s a chunk of CSS, albeit somewhat exaggerate to make it more nerdy. The general format of a stylesheet is a selector followed by a bunch of rules that apply to elements in the HTML document which match that selector. So we’ve got a declarative little language with something of a query syntax and then properties to set for nodes that match the query.

To quickly fly though the query language, the first line will match any element whose ID is “foo”. The second will match any DIV element whose class is “bar”. I keep the hash and the dot notation straight by thinking of the Smalltalk notation for distinguising instance and class methods – hash refers to instance methods, specific IDs in this case, while dot refers to class methods, or any node of the class in this case. Next we specify that any A element will match this selector, as well as any DIV with an ID of “quux” followed by a child DIV with an “urf” class.

Within the rules, we first see a comment. The format should be familiar to anyone who has done C, C++, Java, JavaScript, et. al. It is worth noting that CSS doesn’t support the // comment syntax.

Finally we see a couple declarations of the visual style that will be set for elements matching this node. We’ll set the color of the text to white and the background color of the element to black, as specified by the hex color triplets.

So that’s skimming across the surface. There’s a lot more to know related to cascades, inheritance and arcane selector syntax, but let’s get more to the point. What’s up with those “float” and “display” rules?


Selector







Selector

<p id=”foo”>







Selector

<p id=”foo”> <div id=”foo”>







Selector

<p id=”foo”><div class=”bar”>

<div id=”foo”>







Selector


<a>

<div id=”foo”>







Selector


<a><div id=”quux”> <div class=”urf”>

<div id=”foo”>







Rules

Selector


<a><div id=”quux”> <div class=”urf”>

<div id=”foo”>






The box model

Courtesy Hicks Design

So one of the basic notions of CSS is that each element in the HTML document a stylesheet is applied to generates a rectangular box where the content within the element is placed. Now, some elements don’t actually generate their own box by default; these are called inline elements, such as the A or SPAN tags. Elements like DIV and P however, do get their own boxes.

So the browser decides for us the height and width of the boxes, depending on the content inside. And it also displays how to put those boxes on the screen, how they bump into each other or not, etc.

From a programmer’s point of view, its all kind of a mess. We’re used to just dragging controls onto a form and then wiring up the population of the controls or handling of events they may generate. Failing that, at least we can write some code creating horizontally and vertically aligned boxes and then throw the controls in there. And if we’re even lower-level than that, at least we can draw lines to the framebuffer or something right?

Not so with CSS. But there is a reason to the madness. Let me show you what led to my “aha!” moment for CSS.

http://www.hicksdesign.co.uk/boxmodel/

http://www.hicksdesign.co.uk/boxmodel/

Detour: print

So to really get this stuff, you kinda need to play around with print layout. So I’ve fired up Pages and I put three paragraphs of text in a basic document. Its just “lorem ipsum” stuff – the “Hello World!” of typography.

The important part is there are three paragraphs and you can sort of imagine the three boxes they live in. The layout engine, be it Pages or a web browser, decides how to throw the text content into those boxes, how to hyphenate text, etc.

Now let’s look at a slight modification. I’ve injected myself into this page and done something called “floating an image”. Basically, I’ve told the layout engine to flow the original text around this image. Its like a rock in the middle of a stream of text. The layout engine just has to figure out how to get the water aka text to go around it.

There’s actually a really involved part of the CSS specification that gets down to nuts and bolts about how this works in a web browser. But until I had the context of doing this in print, it didn’t really click.

Detour: print

Boxes





Detour: print

Boxes

Floated





Detour: typography

Typography: the art or process of setting and arranging types and printing from them. These two examples are the same text, even the exact same font face and size. But one is clearly larger than the other and reads vastly differently. I accomplished this by dragging a few sliders around in Pages, but you can do this sort of thing in CSS too.

I’m pulling this in mainly because its sort of my current fascination as far as visual design goes. There’s all sorts of interesting stuff you can do to the visual design of a web page without having to push one pixel in the GIMP or Photoshop. Here I’ve increased the character spacing and also the distance between the baselines for each line of the text. I’m not yet sure how I apply those tools tastefully, but I’m assuming it can be done.

Why we can’t have nice things

So all of this print stuff works out really well. Until you come to most versions of IE and early versions of Netscape.

Can I get a hallelujah?

And so there are myriads of “hacks”, which I sometimes think are just little “open sesames”. Your friendly neighborhood “front-end developer” lives amongst these. You should recruit them to help you when you get a frantic call about your web site or application looking “all funny” on the client’s Windows 98 ME machine.

Barring that, well you’ll just have to keep your ear to the grindstone. The eccentricities of various browsers are widely discussed on lists and weblogs and you can probably consult your favorite broad-web index to find any discrepancy you may come across.

But keeping your ear to the grindstone, while not the best way to solve a problem that’s right in your face and strange, is the best way to get accustomed to the often strange jargon of the print-turned-web-designers that are doing the really interesting work with CSS. Learning the local jargon, that’s holistic.

Why we can’t have nice things

Icons courtsey I Love Jack Daniels

So all of this print stuff works out really well. Until you come to most versions of IE and early versions of Netscape.

Can I get a hallelujah?

And so there are myriads of “hacks”, which I sometimes think are just little “open sesames”. Your friendly neighborhood “front-end developer” lives amongst these. You should recruit them to help you when you get a frantic call about your web site or application looking “all funny” on the client’s Windows 98 ME machine.

Barring that, well you’ll just have to keep your ear to the grindstone. The eccentricities of various browsers are widely discussed on lists and weblogs and you can probably consult your favorite broad-web index to find any discrepancy you may come across.

But keeping your ear to the grindstone, while not the best way to solve a problem that’s right in your face and strange, is the best way to get accustomed to the often strange jargon of the print-turned-web-designers that are doing the really interesting work with CSS. Learning the local jargon, that’s holistic.

http://www.ilovejackdaniels.com/css/box-model/2/

http://www.ilovejackdaniels.com/css/box-model/2/

Erudition pays off

So one day, while fooling around with the notion of trying to really understand LaTeX, all this print layout and typography business suddenly made CSS click. “OH! Its all print.” Which is probably obvious to those less stubborn than I, but I digress.

Being widely read paid off though. Its certainly not necessary to my duties as a professional programmer to have any interest in how you take words and print them out in an automated fashion. Even as someone who bangs words together, its not really something I need to know the nuances of. But pulling the covers back just a little bit gave me the missing piece I needed to pull it all together and finally get it.

I forget where I read it, but at some point I came across something noting that all the interesting action is on the edges of topics these days. Its not enough to be an expert; you’ve gotta see how things can interact and fit together to make the really interesting progress these days. I think that’s a big part of what I think of as holism in programming.

Erudition pays off

Print layout, typography, etc




Erudition pays off


CSS, visual design, user interface




Erudition pays off


CSS, visual design, user interface

Aha! moment




Communicating

So we’ve covered the first part of my vision for holistic programming – understanding. Now I’d like to spend some time on the axis of holistic programming: communication.

I’m a fella who considers programming as fundamentally an act of writing. I’m either writing for a computer to understand me or I’m writing for a person to understand me. Often, when I’m writing code, its one in the same. Fun notion: programs are oddly phrased letters to people which are, coincidentally, executable by machines.

(Wo)man

Humans. We’re irrational and emotional. Often illogical. Didn’t we get into programming to avoid all this?

Unfortunately, no. The days of the cowboy coder building his own hardware, system and applications are long since past. Involving yourself with Open Source mandates that you find adeptness in “programming” people to understand what you’re thinking, or were at some point thinking.

Ambiguity FTW

Let’s talk about language for a second. Human language is full of uncertainty, riddle and doubt. You say potato, I say potato. You check under the bonnet of your car, I check the hood.

Did you ever think, what was the guy who invented synonyms like? Don’t you think he may have been a jerk? “Hey, here’s a way for us to talk about the exact same thing and not even realize it!” And what about the guy who invented homonyms, you know, words that sound the same but have different meanings? He must have been a total dick! But I digress.

So its worth noting that human language is all about nuance. Its cool, but sometimes, well, it leads to misunderstandings, flamewars and angry dance-offs.

As an aside, I really like referring to angry dance-offs and using that image.

Ninja move:get people on the same

page

So the ninja move here is that communicating with humans is all about working through the ambiguity and getting people on the same page. Once your team has a design aesthetic, a shared goal or a common jargon, then you can proceed to conquering the world.

OK, so what are some tools to get folks on the same page? Semi-literate programming like JavaDoc, PHPDoc, PODs, RDoc, etc. Then you can step up the stack (can you tell I’m all about moving up and down the stack?) to design docs, highly technical blog posts, mailing list posts and issues in a tracker. Finally, you move up even higher and you interact with people, get in their face, check out their facial expressions, huddle around a whiteboard and become one mind. Then: world domination!

Just FYI, I’m doing a whole talk tomorrow called “PeopleHacks” about how to better interoperate with humans.

Machine

Cold and emotionless. Exacting, patient and never forgetting. Clearly, man created the computer so that he’d never again have to remember his anniversary or carry trigonometry tables with him.

But it sure is a kick when he can make the machine show silly pictures with cats in them. And to do so, we need to understand all the different levels at which we might tell the computer what we want it to do.

Rules FTW

Computers thrive on rules. Its how us humans can even begin to deal with the pure thought-puzzles that we’re presented when programming in the absence of actual physical constraints. So we built machines in that image; you put a certain pattern of bytes in the instruction stream, and you will always get the same results.

Unlike society, computers don’t give you a break when you violate the rules. You divide by zero, that machine will throw you right out of the method you were in, with only an exception to get you back on your feet.

Ninja move:bend machines to the

whim of human language

So the ninja move here is to take the rigor that the machine wants and use it to benefit us as we try to communicate solutions to problems, both to people and the computer.

One example of this is type checking, if you’re into that sort of thing. We can create a compiler that puts into place a set of rules preventing us from committing a whole range of errors. To some, solutions expressed with that level of rigor are much easier to understand than those without.

But there are those aren’t big into type checking, bless them. But if you probe enough, even they like the rules. You can embrace the limitations of a language’s rules to give you something like an internal DSL or fluent interface. And thus, you push the machine’s rules closer to the human’s rules and get the machine’s cold devotion to rules as a bonus.

Further still, you have something like Perl. All the rules seem to be optional. Especially if you can hold your head the right way and chant Larry Wall’s name in the right obscure dialect of an extinct language. Again, I digress.

Your take-away: bend machine languages, whether they are assembly or JavaScript, to make the most of the rules in place so that you can better express thoughts and solutions to people, in the medium of code.

In conclusion

Well its getting close for me to let dozens of new holistic programmers bloom out in the real world. But let me leave you with a couple parting words.

Practice, practice, practice!

OK, this one bears repeating. Holistic programming, unfortunately, is not available as a summer class offered at your local community college. Unless it is, in which case I’d love to hear from you.

Otherwise, you just embrace continuous learning. Read a lot, talk to folks. You’re already at a conference, why not start here. I dare you to go to the keynote presentation for a language you’re not programming in. Except the Python guys, its all just Monty Python silliness with them.

Always with the digressions with me. Anyway, just strive to learn a bit about a whole lot. As you keep on doing it, you’ll grow stronger and wiser every day.

The magic moment, for me, is like this cube transition I’m about to abuse. Most days, you go about your life, sort of milling about along a two dimensional surface. And then one wonderful day, you somehow manage to peer around the edge. Hey, there’s a whole other realm of interesting going on over there! So you get the cube rotated all the way around and you keep learning. Repeat that several dozen times and you’re nearing Holistic Nirvana!

Canonical tomes

Here are some books that will greatly help you in your practice.

First off, the Dragon Book. The definitive book on compilers. Also, horribly dry.

Let me give you a tip. With the exception of the last two, these are all text books. So the chapters are too long and not very exciting, with little cohesive narrative. So don’t get down when you can’t tear through a thousand page textbook as fast as you did with Harry Potter. Believe me, the pages are a lot wider. Even though there are more pictures.

Next we have Patterson and Hennesey, a great text on computer architecture. Take that with a dose of the dinosaur book on operating systems and the eternally wise W. Richard Stevens and you will know a damn lot about how all the gizmos underneath your application work. And that knowledge is power, friends.

Bulletproof Web Design is just a great book on CSS. It helped to demystify a lot of things and made me confident I could tackle larger UI problems with just CSS and my wits.

Finally, we have the Pragmatic Programmer. Simply a great book, an easy read and the reason why I’m here. If you haven’t read it yet, please do. Reading it was an inflection point in my career; its only improved since then.

The questions and the answers

end

Thank you! Now, I must go check out this “Death Star” my new master has been telling me about.

the holistic programmer

Technology