the life of a web request - techniques for measuring and improving django application performance
DESCRIPTION
The connection between sub-second web application performance and revenue is becoming more and more apparent with established companies regularly reporting the benefits of reducing page load times. This talk will cover: * Designing for performance * Approaches to instrumenting and measuring application performance * Areas of focus for both front-end and back-end improvement * Techniques, tools and modules available in Django-land for improving performance * New and emerging technologies, for example SPDY protocol and Django 1.5's StreamingHttpResponseTRANSCRIPT
The Life of a Web RequestMeasuring and improving
Django application performance
PyCon Australia 2013
Roger Barnes@[email protected]://slideshare.net/mindsocket/
What are you here for
Aspects of site performance
Measuring everything
Fixing the right thing
AND HOW
Performance focus for this talk
✔ user perspective✔ seconds per request✗ benchmarks✗ users per server✗ requests per second
Why should we care?
● Web application performance affects– engagement
Why should we care?
● Web application performance affects– engagement– bounce rate
http://www.slideshare.net/joshfraz/sept-2012rumtalk
Why should we care?
● Web application performance affects– engagement– bounce rate– conversion/abandonment
Why should we care?
● Web application performance affects– engagement– bounce rate– conversion/abandonment– search engine ranking
Why should we care?
● Web application performance affects– engagement– bounce rate– conversion/abandonment– search engine ranking– revenue
Why should we care?
Web sites are
larger&
than ever
Why should we care?
Average web page:
1.3Mb85 requests
- HTTPArchive.org
Why should we care?
Users are increasingly mobile
Their expectations are up
Many "responsive" sites are no smaller than their desktop version
Let's look at aweb request!
There's a redirect!
How about some CSS?
Let's get some
JavaScript and images
Start Render: 1.6s
Document complete: 8.6s
Fully loaded: 9.0s
Requests: 177
Bytes in: 2,689kb
This is on a good connection
The Performance Golden Rule
"80-90% of the end-user response time is spent on the frontend.
Start there."
- Steve Souders
Source: http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/
Front-end Measurement
● Instrument your live application for Real User Monitoring (RUM)– Google Analytics– New Relic
Front-end Measurement
● Synthetic testing● Lots of tools available● 3 in particular...
Chrome Developer Tools
webpagetest.org
Browser navigation timing
http://www.w3.org/TR/navigation-timing/
django-debug-toolbar
Not in PyPi yet,see GitHub version
Back end Front end
Front-end offenders
● Static resources – size and number● Lack of caching and compression● 3rd-party resources● Overdownloading
Static Resources
● Minification/combination of CSS/JavaScript– django-compressor django-pipeline
● Images– Appropriate format and compression level– Correct size– Remove metadata
● jpegoptim, optiPNG
– Consider sprites
Static Resources
● Web server/accelerator config– Caching/expires/compression headers
● Static files to CDN– see Django's storage hooks
● Deferred loading● Silver bullet (?)
– mod_pagespeed (apache)– ngx_pagespeed (nginx)
Ok, now what?
Back-end measurement
Production instrumentation with New Relic
django-statsd and graphite
django-debug-toolbar
runprofileserver indjango-extensions
Web server
WSGI application (WSGIHandler.__call__)
Initialise Request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template Rendering
Backend Services
Typical Django Request
Caching Dynamic Requests
● Potentially easy wins● Beware cache invalidation● Different levels
– Caching headers– Web Server/Web Accelerator– Caching middleware– Per view caching– Template fragment caching– Low level object caching
Web server / Reverse proxy
WSGI application (WSGIHandler.__call__)
Initialise request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template rendering
Backend services
Typical Django Request
Caching
Browser
Compressing Dynamic Content
Web server
Web accelerator
gzip middleware
Query Performance
● Use debug toolbar● Avoid unnecessary queries
– eg exists followed by separate get
● select_related to get FKs in same query– Be wary of optional foreign keys
● prefetch_related to get collections– 1 additional query instead of N+1
Defer work
● Defer user-irrelevant workloads● Target
– Views– Signals (eg post_save)– Middleware
Defer work with celery
def login(request): # Your login code here # ... user = form.save()
upload_to_nsa.delay(user.id) return HttpResponse(...)
Ok, now what?
Web server
WSGI application (WSGIHandler.__call__)
Initialise request
Get Response (BaseHandler.get_response)
URL Resolution
Request Middleware Response Middleware
View Middleware Template Response Middleware
View Template rendering
Backend services
Django Works Then Responds
Bytes on the wire(Time to First Byte)
Back end
Can we stream?
● StreamingHttpResponse in 1.5● Patch for streaming templates (#13910)● Caveats
– Middleware compatibility– Exception handling– YAGNI?
What about eager streaming?
Hack: View middleware that streams the top of a page before even calling the view
Proof of concept:
https://github.com/mindsocket/django-perf-example
Regular
Streaming
Eager streaming
Visual appearance
Future context
● APIs, websockets and JavaScript, oh my!● SPDY● Mobile● Full stack ownership
Summary
● Measure, measure, measure● Don't ignore the front-end● Beware diminishing returns● Technology is changing
Resources
● General web performance– http://www.slideshare.net/Strangeloopnet/37-lessons-
ive-learned-on-the-performance-front-lines– http://www.webpagetest.org/– https://developers.google.com/speed/pagespeed/– http://www.mnot.net/cache_docs/– http://newrelic.com/– https://dvcs.w3.org/hg/webperf/raw-
file/tip/specs/NavigationTiming/Overview.html– https://www.varnish-cache.org/
Resources
● Django – https://docs.djangoproject.com/en/dev/topics/cache/– https://code.djangoproject.com/wiki/ProfilingDjango– https://code.google.com/p/django-command-extensions/wiki/RunProfileServer– https://developers.google.com/chrome-developer-tools/– https://docs.djangoproject.com/en/dev/howto/static-files/deployment/#staticfiles-
from-cdn– https://docs.djangoproject.com/en/dev/howto/custom-file-storage/– https://github.com/django-debug-toolbar/django-debug-toolbar/issues/319– https://github.com/sorl/sorl-
thumbnail/commit/e5f493a340dea8a33703c05fd66ee76d2f019881– https://github.com/aaugustin/django-c10k-demo– https://github.com/mindsocket/django-perf-example– https://www.djangopackages.com/grids/g/asset-managers/
Questions?
http://slideshare.net/mindsocket
Thank You!