djangocon 2011 talk jesper noehr

8/6/2019 Djangocon 2011 Talk Jesper Noehr

1/25

Bitbucket - Lessons learned@jespern

1Tuesday, June 7, 2011


2/25


3/25

Who am I?

Founder of Bitbucket, now the CTO Started programming back in 1998 Been using Django for 4 years (magic

removal branch, anyone?) @jespern on Twitter


Opening .exe les -> Basic -> PHP (.php3) -> C -> Perl -> Python


4/25

Bitbucket in summary

First commit was May 2008, launch ~1

month after that. 2 people, part time, for 2 years. Ran entirely on AWS. Acquired in August 2010.


Eirik left after 2 years, had a baby, just me after that.

Started from my blog.


5/25

Bitbucket in summary

Now 9 people, 6 engineers, one designer. 2 in Sydney, 7 in San Francisco. Only 2 people knew Django, only 3 Python! 1 is a Mercurial core developer. Incubated at Atlassian.


Mehmet and I knew Django, Brodie knew Python.


6/25

Behind the scenes

Its all Python. 73934 lines of code, 86% is Django. We use Mercurial (surprisingly.) PostgreSQL, virtualenv/pip, etc.


The entire website is 100% django, only parts that arent are background stu f like scripts,various SSH stu f , hooks.

Deploys semi-contained in virtualenvs, individual Pythons, most system-wide packagesinside the env as well (statically linked.)


7/25

Behind the scenes

On production: HAProxy, nginx, gunicorn, puppet. Roll out once a week, hotxes go out

instantly.


Didnt use to be like this, but more details on that later.


8/25

Behind the scenes

Testing: Selenium for functional tests. Kong for headless tests (@ericholscher) Bamboo for CI (just getting there.)



9/25

Lessons learned


Experiences over the years, things that helped us and things that didnt.


10/25

Stay idiomatic


Probably the most important point Im gonna make.What I mean is to follow Django idioms, models, forms, dont step too much outside thecommon way of doing things.

Doing this has helped us *so* much. Its not always the most intuitive thing to do, but itcomes in very handy in so many situations.


11/25

From our codebase (actual code):


Basically everythings like this, from the core concept of Repositories to everything else wedo.


12/25

Everythings an app

Repositories

Issue tracker User accounts (SSH, etc.) Compare view ...


They all have models.py, forms.py, views.py, etc.Can perhaps run standalone but wouldnt make sense.


13/25

Benets & Disadvantages



14/25

Free upgrades, little backporting

Getting/nding help is usually easy --awesome community

Easy for new people to pick up...


...lead into where you can point them for the documentation (next slide.)


15/25

https://docs.djangoproject.com/Wherere the docs?


Really nice to be able to just point people there. They can go there and learn all the Djangoidioms.
http://docs.djangoproject.com/http://docs.djangoproject.com/


16/25

Kinda feels like youre shoe-horningsometimes

Unusual application requirements (werequite transactional)

..


CNAME is one thing we had to really twist Django into doing for us.

Transactional = involving the disk (we obviously do this a lot), handling errors, rolling back,etc. The Django ORM isnt *that* great. Certainly su ces, but does dumb things like

redundant joins and the transactional stu f (we use Postgres -- mention isolation levels.)


17/25

For better or worse.. Usually better.We stick to idioms


Next up, things change.


18/25

Things change


Dont do vendor lock-in! We did (AWS - S3/cloudfront, but mostly EBS), and it hurt is quite abit down the line. The places in which we didnt, came in *really* handy.

We took the pain of making sane technology choices and decisions early -- often a good

idea. AWS great for bootstrap, but we didnt make use of anything we couldnt replace later(EBS looks like a disk).


19/25

Embrace it!


Design your app in such a way that youre not bound to any specic technology or vendor.

Changes *will* happen, whether its to replace a bad technology, or more often, to makesomething scale better.


20/25

Good choices we made

Linux

Django

WSGI

No need to pick a DB Everything else we left

open-ended


WSGI was awesome, as theres a ton of WSGI hosts out there -- weve been throughmod_wsgi, uwsgi, and now gunicorn. Hell, we even ran FastCGI.We could rip out postgres for mysql or whatever any day.


21/25

Open source


Piggyback on existing e f orts, stand on the shoulders of giants. Dont fear NIH.


22/25

We use stuff like... Celery, South, etc.

-(social)registration,-compressor, ...

A lot of really well-donestuff out there!



23/25

And try to give back django-piston

django-dogslow

tipper


Explain what dogslow is, how it relates to tipper.


24/25

Be transparent& humble


Explain outages, dont think youre better than your users.


25/25

Thanks!


djangocon 2011 talk jesper noehr

Documents