““Why Python and Why Python and other dynamic other dynamic

languages are a new languages are a new way of thinking way of thinking

about programming about programming and why it’s and why it’s

interesting to me interesting to me and possibly you ifand possibly you if

you’re at all like me you’re at all like me or if you’re different or if you’re different

than me but just than me but just happen to share that happen to share that

same specific same specific interest as me or if interest as me or if you like progress you like progress

and new ideas and and new ideas and stuff.”stuff.”by Chris Gahan

First of all…First of all…

Python and other dynamic languages Python and other dynamic languages (LIKE RUBY) are super-hot right now.(LIKE RUBY) are super-hot right now.

How hot?How hot? IMAGINE A JULY DAY IN HELL AT HIGH IMAGINE A JULY DAY IN HELL AT HIGH

NOON AND THE WEATHER ALSO NOON AND THE WEATHER ALSO HAPPENS TO BE ESPECIALLY HUMID HAPPENS TO BE ESPECIALLY HUMID THAT DAY FOR SOME REASONTHAT DAY FOR SOME REASON

That’s hot.That’s hot.

You want some proof?You want some proof?

I’ll give ya proof!!I’ll give ya proof!!

Cool Python Sh…stuff.Cool Python Sh…stuff.

FuseFSFuseFS PyGamePyGame PyOpenGLPyOpenGL PyMediaPyMedia PaiMeiPaiMei Pyro (Python Robotics)Pyro (Python Robotics) Matplotlib, SciPyMatplotlib, SciPy FannFann PIL (Python Imaging Library)PIL (Python Imaging Library) PyGTK, Tkinter, wxPython, PyGTK, Tkinter, wxPython,

waxwax Google’s clusterGoogle’s cluster sCrAmBlEd?HaCkZ!sCrAmBlEd?HaCkZ! Orange (component-based Orange (component-based

data mining software)data mining software)

Scapy (packet masticator)Scapy (packet masticator) Twisted (everything, a kitchen Twisted (everything, a kitchen

sink, and a side of fries)sink, and a side of fries) CherryPy, SQLObject (web CherryPy, SQLObject (web

applications)applications) BeautifulSoupBeautifulSoup ElementTreeElementTree Psyco (optimizer)Psyco (optimizer) PyPyPyPy PyrexPyrex Maya, blenderMaya, blender PlonePlone AmarokAmarok Subversion, TracSubversion, Trac

And many, many, many more…

Why is it being used for so Why is it being used for so many different applications?many different applications?

It’s It’s

AWESOME GLUE!AWESOME GLUE!

Mmm.. delicious glue…

What makes a good glue What makes a good glue language?language?

High-level-ousityHigh-level-ousity You spend more time thinking about what you want to do than You spend more time thinking about what you want to do than

how you want to do ithow you want to do it Python does the least to get in the way of you solving the Python does the least to get in the way of you solving the

problemproblem Excellent for text masticationExcellent for text mastication

Text manipulation was well thought out in the language designText manipulation was well thought out in the language design Strings have handy methods like “.startswith()” and “.endswith()” Strings have handy methods like “.startswith()” and “.endswith()”

and “.split()”and “.split()” They can be They can be slicedsliced List comprehensionsList comprehensions

doing transformations on large chunks of data is efficientdoing transformations on large chunks of data is efficient DictionariesDictionaries

fast lookups of stringsfast lookups of strings Powerful regular expression libraryPowerful regular expression library

No more hundreds-of-lines-of-IF-statements!No more hundreds-of-lines-of-IF-statements!

What makes a good glue What makes a good glue language?language?

The built-in libraries let you do almost everything you want… The built-in libraries let you do almost everything you want… easily!easily!

Any computer that has Python installed contains almost all the Any computer that has Python installed contains almost all the modules you need to do whatever you want!modules you need to do whatever you want!

No sysadmin dependency, no installing 9832749823742 packagesNo sysadmin dependency, no installing 9832749823742 packages Can call (and wrap) code written in other languagesCan call (and wrap) code written in other languages Can combine disparate systemsCan combine disparate systems

Connectors for every db formatConnectors for every db format Full access to Win32 API (COM, registry, all that…)Full access to Win32 API (COM, registry, all that…) Excellent network protocol supportExcellent network protocol support Easy to write C extensionsEasy to write C extensions

MaturityMaturity Has been used in tons of industries so that there are a slew of libraries Has been used in tons of industries so that there are a slew of libraries

floating around that let you do whatever you wantfloating around that let you do whatever you want Stability (New versions don’t break all your code)Stability (New versions don’t break all your code) Speed! (Python’s built-in functions are Speed! (Python’s built-in functions are heavily heavily optimized in C)optimized in C)

What makes a good glue What makes a good glue language?language?

DynamicicityDynamicicity You can load code modules at runtimeYou can load code modules at runtime You can execute strings that came from You can execute strings that came from

somewhere else as Python programssomewhere else as Python programs You can do generative programmingYou can do generative programming You can introspect your codeYou can introspect your code

This is a very cool feature. I think it needs its This is a very cool feature. I think it needs its own slide.own slide.

INTROSPECTIONINTROSPECTION

You’d better have a good You’d better have a good reason for wasting a slide reason for wasting a slide

there, buddy!there, buddy! Introspection is an incredibly powerful Introspection is an incredibly powerful

feature when combined with a dynamic feature when combined with a dynamic languagelanguage

Introspection allows a program to look at Introspection allows a program to look at itselfitself

If the program can look at itself, and the If the program can look at itself, and the program can create itself dynamically, program can create itself dynamically, then you can write a program that then you can write a program that programs itself!programs itself!

Why the hell would you want to Why the hell would you want to make a program that writes make a program that writes

itself?itself? Because you’re lazy!Because you’re lazy!

Isn’t laziness bad?Isn’t laziness bad?

Short-term laziness is badShort-term laziness is bad Long-term laziness is goodLong-term laziness is good

Why?Why?

Good-lazy Good-lazy programmers are lazy in that programmers are lazy in that they invent tools to do their work for themthey invent tools to do their work for them

Achieving laziness means improving your Achieving laziness means improving your technology, or toolstechnology, or tools

Improving your Improving your toolstools improves your improves your productivityproductivity

“The reasonable man adapts himself to the world; the unreasonable man persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man.”

-- George Bernard Shaw

Variation in ProductivityVariation in Productivity

“In a low-tech society you don't see much variation in productivity. If you have a tribe of nomads collecting sticks for a fire, how much more productive is the best stick gatherer going to be than the worst? A factor of two? Whereas when you hand people a complex tool like a computer, the variation in what they can do with it is enormous. […]

(cont’d)

Variation in ProductivityVariation in Productivity

[…] Productivity varies in any field, but there are few in which it varies so much. The variation between programmers is so great that it becomes a difference in kind. I don't think this is something intrinsic to programming, though. In every field, technology magnifies differences in productivity. I think what's happening in programming is just that we have a lot of technological leverage. But in every field the lever is getting longer, so the variation we see is something that more and more fields will see as time goes on. And the success of companies, and countries, will depend increasingly on how they deal with it.”

-- Paul Graham

Technology is what made this Technology is what made this country great!country great!

If you’ve read the book If you’ve read the book Guns, Germs, and Guns, Germs, and SteelSteel, you know that the only reason white , you know that the only reason white people conquered the world is because of people conquered the world is because of all the awesome technologies they stoleall the awesome technologies they stole Gunpowder, writing, math, metallurgy, the Gunpowder, writing, math, metallurgy, the

centralized state, horses, livestock, spaghetti…centralized state, horses, livestock, spaghetti… basically, everything you need in the game basically, everything you need in the game

CivilizationCivilization to get to the nuclear age. to get to the nuclear age.

The only thing they didn’t steal was germsThe only thing they didn’t steal was germs although, we did acquire them from livestock… although, we did acquire them from livestock…

so I guess white people really are useless!so I guess white people really are useless!

BUT ANYWAYS…BUT ANYWAYS…

When do you need a program When do you need a program to program itself?to program itself?

A good example is when you do the same A good example is when you do the same thing over and over and over and over.thing over and over and over and over.

DRYnessDRYness

A good rule of thumb is the DRY A good rule of thumb is the DRY principle:principle:

DDon’ton’t RRepeatepeat YYourselfourself

Okay, yes, you can fix the repetition by Okay, yes, you can fix the repetition by just putting all the “things” in a list and just putting all the “things” in a list and iterating over it like this:iterating over it like this:

But there an easier way than manually But there an easier way than manually specifying every element…specifying every element…

If you can see a pattern, then you can If you can see a pattern, then you can program a computer to see it as well.program a computer to see it as well.

Couldn’t you fix that without Couldn’t you fix that without introspection?introspection?

You could use You could use locals()locals()

locals() returns all of the variables in locals() returns all of the variables in the local scope as a dictthe local scope as a dict

Instead of specifying each “thing”, Instead of specifying each “thing”, you could automatically grab you could automatically grab allall “things” like this:“things” like this:

You could also improve debug You could also improve debug messagesmessages

Debugging by putting “print” statements Debugging by putting “print” statements everywhere is tedious, and if you want to change everywhere is tedious, and if you want to change how they’re all displayed, you have to change how they’re all displayed, you have to change every single one!every single one!

And what if you only want debug sometimes?And what if you only want debug sometimes? Why not make a debug function that figures out Why not make a debug function that figures out

the name of the function it was used in?the name of the function it was used in?

You could also improve debug You could also improve debug messagesmessages

Here’s what it looks like in the code:Here’s what it looks like in the code:

(rest of the code omitted)

You could also improve debug You could also improve debug messagesmessages

$ python snooch.py -v http://slashdot.org/=========================================== snooch-o-matic 9000 pro elite pro v9.1 xp-------------------------------------------

Getting all links from: http://slashdot.org/

[snooch_urls_from_page] page url = http://slashdot.org/[snooch_urls_from_page] url_is_url(found_url): False[snooch_urls_from_page] Found: 'http://slashdot.org/http://www.ostg.com'[snooch_urls_from_page] URL IS FILE![same_directory] base = ('http', 'slashdot.org', '/', '', ''), other = ('http','slashdot.org', '/http://www.ostg.com', '', '')[snooch_urls_from_page] url_is_url(found_url): False[snooch_urls_from_page] Found: 'http://slashdot.org/http://sf.net'[snooch_urls_from_page] URL IS FILE![same_directory] base = ('http', 'slashdot.org', '/', '', ''), other = ('http','slashdot.org', '/http://sf.net', '', '')

etc..

And here it is in action

How else can you DRY up your How else can you DRY up your code?code?

Mr. BotMr. Bot

What if you wanted to write an IRC bot?What if you wanted to write an IRC bot? IRC bots have a bunch of actions that you IRC bots have a bunch of actions that you

execute like this: !do_somethingexecute like this: !do_something When you’re writing a program, you could When you’re writing a program, you could

have a whole crapload of “if” statements have a whole crapload of “if” statements that check for the command, or a series of that check for the command, or a series of switch statements.switch statements.

But what if the program could figure out But what if the program could figure out what commands the bot supports by what commands the bot supports by looking at the names of the methods in looking at the names of the methods in the bot’s class?the bot’s class?

Mr. BotMr. Bot Here’s a chunk of code from Mr. Here’s a chunk of code from Mr.

Bot’s “logic core” (the class where all Bot’s “logic core” (the class where all !actions are coded):!actions are coded):

UnittestUnittest The unittest module also discovers methods by name and The unittest module also discovers methods by name and

runs them as tests with pretty output:runs them as tests with pretty output:

Creating new Classes Creating new Classes automaticallyautomatically

Also known as “metaclasses”Also known as “metaclasses” A metaclass is a class that can change what A metaclass is a class that can change what

happens when you subclass another classhappens when you subclass another class A good example is SQLObjectA good example is SQLObject

SQLObject creates a new classes that let you access SQLObject creates a new classes that let you access your database as if it’s a Python objectyour database as if it’s a Python object

These classes are created on the fly, at runtimeThese classes are created on the fly, at runtime This saves a lot of typing – you just supply the data This saves a lot of typing – you just supply the data

needed and no more.needed and no more.

Now that the computer is Now that the computer is writing most of the code, what writing most of the code, what

do us humans do?!do us humans do?!

Now that you’ve become a meta-Now that you’ve become a meta-programming ninja, you don’t have programming ninja, you don’t have to code as much. You’ve gotta spend to code as much. You’ve gotta spend all your time all your time THINKING!THINKING!

You bastard! We had it so You bastard! We had it so good! Why’d you have to go good! Why’d you have to go

and ruin it?and ruin it? I know, I know, it seems unfair, but I know, I know, it seems unfair, but

it’s a cruel world out there.it’s a cruel world out there. If If wewe don’t become more productive, don’t become more productive,

someone else is going to, like those someone else is going to, like those sneaky JAPANESE! Then we’re sneaky JAPANESE! Then we’re screwed.screwed.

So, how do we improve So, how do we improve Thinking?Thinking?

What about What about Lean Thinking!Lean Thinking!

See how the letters are leaning?That’s called a “visual aid”.

What’s Lean Thinking?What’s Lean Thinking?

The basic concept is this:The basic concept is this: Do less activities that are a wasteDo less activities that are a waste Do more activities that are valuableDo more activities that are valuable

Examples of things that are Examples of things that are a wastea waste:: PaperworkPaperwork

A bureaucratic organization produces mostly A bureaucratic organization produces mostly paperwork and little value. Paperwork generally gets paperwork and little value. Paperwork generally gets in the way of actually doing things.in the way of actually doing things.

Surplus inventorySurplus inventory Produce products Just-in-Time™ instead of making Produce products Just-in-Time™ instead of making

tons and tons of them, transporting them thousands tons and tons of them, transporting them thousands of miles to a warehouse, waiting 3 months, then of miles to a warehouse, waiting 3 months, then transporting them all back.transporting them all back.

What’s Lean ThinkingWhat’s Lean Thinking

Another thing that’s not valuable is Another thing that’s not valuable is overly detailed program specificationsoverly detailed program specifications The only The only true true representation of a representation of a

program is its source code.program is its source code. Eventually, all specifications must be Eventually, all specifications must be

transcribed as code, and that’s the thing transcribed as code, and that’s the thing that gets run.that gets run.

It’s easier to use the code as your It’s easier to use the code as your One One True ReferenceTrue Reference instead of constantly instead of constantly converting between code and converting between code and specifications.specifications.

Wait, keeping it all in your Wait, keeping it all in your head?head?

Well, okay, not ALL of it…Well, okay, not ALL of it… Don’t document everything – just the things Don’t document everything – just the things

that the code is bad at describing.that the code is bad at describing. Use casesUse cases

Keep only what you need on paper. Don’t Keep only what you need on paper. Don’t overdo it.overdo it.

Planning is good, but a lot of the time, actually Planning is good, but a lot of the time, actually implementing something changes your plans.implementing something changes your plans.

Only plan things that you’re absolutely certain will Only plan things that you’re absolutely certain will not change.not change.

For everything else, learn as you go!For everything else, learn as you go!

Cool! I like lean thinking!Cool! I like lean thinking!

Lean Thinking is just the beginning… Some Lean Thinking is just the beginning… Some prolific software design process dudes got prolific software design process dudes got together in 2001 and created the “Agile together in 2001 and created the “Agile Manifesto”.Manifesto”.

The Agile Manifesto contains the current The Agile Manifesto contains the current ideas on the best way to produce software.ideas on the best way to produce software.

It’s goals areIt’s goals are maximal qualitymaximal quality maximal programmer satisfactionmaximal programmer satisfaction minimal timeminimal time equal costequal cost

Agile DevelopmentAgile Development

Agile tries to keep focus on creating a process Agile tries to keep focus on creating a process that allows each individual in each team to find that allows each individual in each team to find and use their unique strengthsand use their unique strengths FlexibilityFlexibility DynamismDynamism LearningLearning Trying new ideasTrying new ideas

Adaptability and learning are essentialAdaptability and learning are essential Continuous improvement Continuous improvement of: of:

the people the people the processthe process the source codethe source code

Agile DevelopmentAgile Development

Agile has a lot to do with optimizing Agile has a lot to do with optimizing the efficiency of the efficiency of peoplepeople Humans Humans are the bottleneck, not are the bottleneck, not

computers!computers! Maximizing human efficiency is trickyMaximizing human efficiency is tricky

Humans are pattern recognizers, not Humans are pattern recognizers, not state machinesstate machines

This means that people kinda suck at This means that people kinda suck at reading and imagining codereading and imagining code

We can understand small pieces at a time, We can understand small pieces at a time, but not an entire system in all its gory detail.but not an entire system in all its gory detail.

Agile DevelopmentAgile Development

Techniques to help improve the efficiency of programmersTechniques to help improve the efficiency of programmers Writing testsWriting tests

Tests let you use your module before you’ve written itTests let you use your module before you’ve written it Keeps you secure in your code so you don’t have to worry about itKeeps you secure in your code so you don’t have to worry about it

Short release cyclesShort release cycles Get your program running and interfacing with real-world data as soon as Get your program running and interfacing with real-world data as soon as

possiblepossible Helps keep you focussed on what’s importantHelps keep you focussed on what’s important

Keep in constant communication with team members and customerKeep in constant communication with team members and customer If the every member of the team doesn’t know the state of the program, If the every member of the team doesn’t know the state of the program,

then they’re going to be afraid to change parts that they’re not responsible then they’re going to be afraid to change parts that they’re not responsible for, and they’ll also make mistakes when integrating their code with the for, and they’ll also make mistakes when integrating their code with the code of otherscode of others

If the customer isn’t kept in the loop, then the project could start going in If the customer isn’t kept in the loop, then the project could start going in the wrong direction. The customer is the end-user and ultimate judge of the the wrong direction. The customer is the end-user and ultimate judge of the finished product – their feedback is the most important.finished product – their feedback is the most important.

Pair programmingPair programming Two brains each doing different specialized tasksTwo brains each doing different specialized tasks

brain #1: tactical [writing code]brain #1: tactical [writing code] brain #2: strategic and verification [thinking about architecture, verifying what brain #2: strategic and verification [thinking about architecture, verifying what

brain #1 is doing]brain #1 is doing] Keeps the programmers completely synchronized – two people will always Keeps the programmers completely synchronized – two people will always

be 100% up to speed on a certain part of the programbe 100% up to speed on a certain part of the program

PyPy has taken Agile to the PyPy has taken Agile to the extreme: Sprints!extreme: Sprints!

A PyPy Sprint is a two or three-day focused A PyPy Sprint is a two or three-day focused development session in which developers development session in which developers pair off together to focus on building a pair off together to focus on building a particular subsystemparticular subsystem

It ends up being incredibly productive, and It ends up being incredibly productive, and is really the best way to produce high-is really the best way to produce high-quality codequality code

The alternative is to work slowly, The alternative is to work slowly, unfocussedly, individually.unfocussedly, individually.

Humans don’t work well like that! Our Humans don’t work well like that! Our memories fade over time.memories fade over time.

Why does nCircle exist?Why does nCircle exist?

Because software has tons of flaws!Because software has tons of flaws! Thank Thank the maker the maker for bad development for bad development

practices and tools.practices and tools.

Why does software have tons Why does software have tons of flaws?of flaws?

Because people are stupid!Because people are stupid!

Don’t be glib…Don’t be glib…

Well, okay, programming beyond a Well, okay, programming beyond a certain level of complexity is too certain level of complexity is too mentally taxing for humans.mentally taxing for humans.

Fundamental Problems with Fundamental Problems with SoftwareSoftware

With the advent of faster computers, With the advent of faster computers, and with society pushing inexorably and with society pushing inexorably towards the refinement and towards the refinement and automation of all things, software automation of all things, software complexity has shot through the roofcomplexity has shot through the roof

Programmers are digging themselves Programmers are digging themselves into holes all over the placeinto holes all over the place Making their lives more complicated Making their lives more complicated

than it needs to bethan it needs to be

Basically…Basically…

We know how we want software to We know how we want software to behave, but we just don’t have the behave, but we just don’t have the mental hardware (or the software mental hardware (or the software tools) to allow us to continuously tools) to allow us to continuously juggle a massive nest of juggle a massive nest of interconnected logical constraints and interconnected logical constraints and remember every single one.remember every single one. We get tired! We get hungry! We get We get tired! We get hungry! We get

distracted by that videos of the monkey distracted by that videos of the monkey peeing into his mouth!peeing into his mouth!

The Future of ProgrammingThe Future of Programming

Jonathan Edwards says:Jonathan Edwards says:“I discovered the creative exhilaration of programming at a “I discovered the creative exhilaration of programming at a formative age. Yet all along I have also felt stifled by primitive, ill-formative age. Yet all along I have also felt stifled by primitive, ill-suited languages and tools. We are still in the Stone Age of suited languages and tools. We are still in the Stone Age of programming. My quest is to better understand the creative act of programming. My quest is to better understand the creative act of programming, and to help liberate it. Reflection upon my own programming, and to help liberate it. Reflection upon my own practice of programming has led me to three general observations:practice of programming has led me to three general observations:

Programming is mentally overwhelming, even for the smartest of us. Programming is mentally overwhelming, even for the smartest of us. The more I program, the harder I realize it is.The more I program, the harder I realize it is.

Much of this mental effort is laborious maintenance of the many mental Much of this mental effort is laborious maintenance of the many mental models we utilize while programming. We are constantly translating models we utilize while programming. We are constantly translating between different representations in our head, mentally compiling them between different representations in our head, mentally compiling them into code, and reverse-engineering them back out of the code. This is into code, and reverse-engineering them back out of the code. This is wasted overhead.wasted overhead.

We have no agreement on what the problems of programming are, We have no agreement on what the problems of programming are, much less how to fix them. Our field has little accumulated wisdom, much less how to fix them. Our field has little accumulated wisdom, because we do not invest enough in the critical self-examination needed because we do not invest enough in the critical self-examination needed to obtain it: we have a reflective blind-spot.to obtain it: we have a reflective blind-spot.

(cont’d)(cont’d)

The Future of ProgrammingThe Future of Programming

““From these observations I infer three corresponding From these observations I infer three corresponding propositions:propositions: Usability should be the central concern in the design of Usability should be the central concern in the design of

programming languages and tools. We need to apply programming languages and tools. We need to apply everything we know about human perception and cognition. everything we know about human perception and cognition. It is a matter of “Cognitive Ergonomics”.It is a matter of “Cognitive Ergonomics”.

Notation matters. We need to stop wasting our effort juggling Notation matters. We need to stop wasting our effort juggling unsuitable notations, and instead invent representations that unsuitable notations, and instead invent representations that align with the mental models we naturally use.align with the mental models we naturally use.

The benefits of improved programming techniques can not The benefits of improved programming techniques can not be easily promulgated, since there is no agreement on what be easily promulgated, since there is no agreement on what the problems are in the first place. The reward systems in the problems are in the first place. The reward systems in business and academics further discourage non-incremental business and academics further discourage non-incremental change.”change.”

Ruby + Domain LanguagesRuby + Domain Languages

Managing complex projects is about abstractionsManaging complex projects is about abstractions Your program’s API is like its own domain-languageYour program’s API is like its own domain-language Good abstractions make writing programs easier because we can mentally Good abstractions make writing programs easier because we can mentally

manipulate them bettermanipulate them better ““A change of perspective is worth 80 IQ points.” – Alan KayA change of perspective is worth 80 IQ points.” – Alan Kay

Intuitive paradigms are the most efficient way to allow a human to interface with a Intuitive paradigms are the most efficient way to allow a human to interface with a computercomputer

The best programming paradigms are very intuitive:The best programming paradigms are very intuitive: ListsLists StacksStacks DictionariesDictionaries Memory “space”Memory “space” Objects with ActionsObjects with Actions ThreadsThreads

The implications of calling a function should be intuitiveThe implications of calling a function should be intuitive Ruby allows you to make your API easier to understand by making it easy to create Ruby allows you to make your API easier to understand by making it easy to create

domain languagesdomain languages Ruby almost allows you to extend the language itselfRuby almost allows you to extend the language itself

eg: ActiveRecord, Choiceeg: ActiveRecord, Choice A standard API will have a lot of boilerplate. In Ruby, it’s easy to make all that boilerplate A standard API will have a lot of boilerplate. In Ruby, it’s easy to make all that boilerplate

disappear.disappear.

Intentional ProgrammingIntentional Programming

This is probably going to be the first This is probably going to be the first step on a road towards an evolving step on a road towards an evolving system that allows humans to system that allows humans to represent programs on their own represent programs on their own terms.terms.

Think of the Star Trek computerThink of the Star Trek computer