anatomy of a large django site
DESCRIPTION
TRANSCRIPT
Anatomy of a large Django site
Andy McKay Mozilla
mozilla
mozilla
Vancouver
PythonZope and Plone...now at Mozilla
mozilla
mozilla
Using Djangohttp://www.djangoproject.com
Credit: http://www.flickr.com/photos/abiavati/3110357974/
1. About the site2. Performance3. Localisation4. Reuse
mozilla
1. About the site
mozilla
mozilla
All code is open:https://github.com/jbalogh/zamboni
mozilla
All* bugs are open:https://bugzilla.mozilla.org
mozilla
Convert from CakePHP (remora)
to Django (zamboni)
mozilla
mozillaCredit: http://www.flickr.com/photos/improbcat/4177702580/
Changing one URL at a time from CakePHP to Django
mozilla
General trend to move away from PHP and do more Python and Django
How large?250k+ addons
150 mn views month500 mn API hits day
mozilla
Lines of code
PHPPython
40k18k
mozilla
Lines of code
PHPPython
Unit tests
40k18k15k
mozilla
No pages go out until they are faster
mozilla
running both php and python side by side. a few issues on that
3 zeus load balancers24 django (and php) 1 mysql + 4 slaves3 memcached3 sphinx1 rabbitmq + 2 celeryd1 redis master + 1 slave
Credit: http://www.flickr.com/photos/tbridge/15300843/ mozilla
2. Performance
mozilla
mozilla
As usual, database bottleneck
Cache machinehttp://bit.ly/cache-machine
mozillaCredit: http://www.flickr.com/photos/mwichary/4063534688/
mozilla
from django.db import modelsimport caching.base
class Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager()
available as a mixin
need to addin the custom manager
mozilla
>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13
mozilla
>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13
>>> Addon.objects.filter(status=public)>>> len(connection.queries) 13
mozilla
Invalidation
mozillamozilla
md5(‘select... a’) [addon 3615]
mozillamozilla
md5(‘select... a’) [addon 3615]
addon 3615 md5(‘select... a’)
mozillamozilla
md5(‘select... a’) [addon 3615]
md5(‘select... b’) [addon 3615, addon 1685]
addon 3615 md5(‘select... a’)md5(‘select... b’)
mozilla
Memcachedrules = cache.get(3615)rules.add('select...')cache.set(3615, rules)
mozilla
Redisredis.SADD(3615, ‘select...’)
mozilla
Home page20+ addons
400+ sql queries
mozilla
standard answer in django is select-related
add-on version
files
version
version
files
files
mozilla
django: select_related()http://bit.ly/select-related
mozilla
Transformerhttp://bit.ly/queryset-transform
simonw
mozilla
@staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version
slightly outdated example
mozilla
Home page20+ addons
~14 sql queries
big SQL statements... :(
14313 character
UpdateCalled on startup
about:config ☞ extensions.update.url
mozilla
IncomingUncached
8,000 req/sec1,600 req/sec
Im used to Plone in the bad old days
mozilla
IncomingUncached
PHP
8,000 req/sec1,600 req/sec550 req/sec
Im used to Plone in the bad old days
mozilla
PHP Python
v1Plain
Django
mozilla
PHP Python
v2Min. SQL
queries
mozilla
oh god
mozilla
PHP Python
v3Django and
raw SQLmax-requests 200, actually we hit 210
mozilla
PHP Python
v4WSGI
no Django
mozilla
PHP Python
v5Pooling,
optimised queries
Thats 700 req/secwhich translates into
mozilla
Reducing the SQL queries...doesn’t always help
mozilla
mySQL query cache is fast
mozilla
Celeryhttp://celeryproject.org
mozillaCredit: http://www.flickr.com/photos/chiotsrun/3843988392/
Push things asyncemail
image processingadd-on validation
specifically fixing data changes between php and python
mozilla
from celeryutils import task
@taskdef update_tag(tag, **kw): tag.update_stat()
mozilla
from celeryutils import task
@task(rate_limit='60/h')def update_tag(tag, **kw): tag.update_stat()
mozilla
from tasks import update_tag
update_tag.delay(tag)
mozilla
mozilla
Measurement
Timing Middlewarehttp://bit.ly/timing-ware
mozillaCredit: http://www.flickr.com/photos/wwarby/3297205226/
mozilla
3. Localization
mozilla
mozilla
40+ languagesincluding rtl
show site in arabic?
mozilla
Database strings
content translated
and
templates
mozilla
class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation)
mozilla
addon.name = 'name'addon.save()
mozilla
addon.name = 'name'addon.save()
addon.name = {'fr': 'la nomme'}addon.save()
mozilla
Templates
mozilla
Django{% blocktrans with app=request.APP %}
Add-ons for {{ app }} {% endblocktrans %}
mozilla
Jinja2http://jinja.pocoo.org/
{{ _('Add-ons for {0}')|f(request.APP) }}
mozilla
Python Unicode hellUnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 16: ordinal
not in range(128)
4. Reuse
mozilla
Bleachhttp://pypi.python.org/pypi/
mozillaCredit: http://www.flickr.com/photos/maisonbisson/3350954463/
>>> bleach.clean('an <script>evil()</script> example')
'an <script>evil()</script> example'
mozilla
>>> bleach.linkify('an http://ex.com url')
'an <a href="http://ex.com" rel="nofollow">http://ex.com</a> url'
mozilla
Javascript tests
mozilla
django-qunithttp://bit.ly/django-qunit
mozilla
kumar
mozilla
test('English', function() { z.refreshL10n('en-us'); equals($('textarea:visible', this.sandbox).text().trim(), 'Firebug integrates with Firefox to put ' + 'a wealth of development tools...');});
mozilla
test('Japanese', function() { z.refreshL10n('ja'); equals($('textarea:visible', this.sandbox).text().trim(), 'Firebug は、Web ページを閲覧中にクリック一つ ' +
' で使える豊富な開発ツールを Firefox' + ' に統合します。あなたはあらゆる');});
Use HudsonJenkinshttp://bit.ly/jstestnet
mozilla
So we use hudson for CI, but haven’t got the automated tests in yet
Hoping to do this via jstestnet
Flake8http://bit.ly/flake8
mozillaCredit: http://www.flickr.com/photos/nebarnix/357779131/
pep 8py flakesMacCabe
~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.pyapps/editors/tasks.py:1: 'datetime' imported but unusedapps/editors/tasks.py:3: 'stat' imported but unused
mozilla
Playdohhttp://bit.ly/mozilla-playdoh
mozilla
fred wenzel
Credit: http://www.flickr.com/photos/ahmee/97960570/
Basis for new projects
mozilla
Celery supportJinja2 supportSimple migrations
By default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly flags on cookies
mozilla
fred wenzel
Take inspiration from...but not the best for you
mozilla
for example jinja2 which makes integration with lots of django addons possible, but a bit harder