tricks & challenges developing a large django application
Post on 18-Oct-2014
8.930 views
DESCRIPTION
Presented at EuroPython 2011 on the 22nd of June.TRANSCRIPT
Tricks & challengesdeveloping a large Django application
Simon WillisonEuroPython 2011, 22nd June
http://lanyrd.com/sfwcq
Lanyrd.com
Lanyrd.com
Definitive databaseof Professional Events
and Speakers
Lanyrd.com
Social event recommendationEvent networking tools
Archive of slides, notes and video
Definitive databaseof Professional Events
and Speakers
Part 1: Tricks
Trick #1: Signing
Lanyrd.com
Pass data through an untrusted source with confidence that it hasn't been tampered with
Lanyrd.com
Signing usesUnsubscribe" links in emails
lanyrd.com/un/ImN6VyI.ii0Hwm7p71DEcGfaVzziQaxeuu
Signed cookies "You are logged in as simonw" without hitting the database
Lanyrd.com
Signing in Django 1.4from django.core import signingsigning.dumps({"foo": "bar"})signing.loads(signed_string)
response.set_signed_cookie(key, value...)request.get_signed_cookie(key)
Trick #2: cache_version
add a conferenceadd a conference you are signed in as simonw, do you want to sign out?
calendarcalendar conferencesconferences coveragecoverage profileprofile
searchsearch
ON NOW
Django Plone Pyramid Python Twisted
EuroPython 2011Italy / Florence
19th–26th June 2011
SEPTEMBER
2011Django Open Source Python
Django Python
DjangoCon US 2011United States / Portland
6th–8th September 2011
PyCON FR 2011France / Rennes
17th–18th September 2011
OCTOBER PyCon DE 2011
Django events looking for participants1 Django event is looking for participants
Djangocoverage
By countryIreland 1
Django conferencesDjango conferences
19
6
17
4
52 videosMost recent added 3weeks ago
52 slide decksMost recent added 4hours ago
3 audio clipsMost recent added 1week ago
27 write-upsMost recent added 1week ago
11 handoutsMost recent added 18hours ago
3 notesMost recent added 10hours ago
Lanyrd.com
class Conference(models.Model): ... cache_version = models.IntegerField(default = 0)
def save(self, *args, **kwargs): self.cache_version += 1 super(Conference, self).save(*args, **kwargs)
def touch(self): Conference.objects.filter( pk = self.pk ).update( cache_version = F('cache_version') + 1 )
Lanyrd.com
{% cache 36000 conf-topics conference.pk conference.cache_version %} <ul class="tags inline-tags meta"> {% for topic in conference.topics.all %} <li><a href="{{ topic.get_absolute_url }}">{{ topic }}</a></li> {% endfor %} </ul>{% endcache %}
Lanyrd.com
from django.models import F
topic.conferences.all().update( cache_version = F('cache_version') + 1)
Bulk invalidation
Trick #3: NoSQL for denormalisation
Lanyrd.com
NoSQL = Not Only SQL
Lanyrd.com
Slides from a session on Lanyrd show up... Under "coverage of this conference"Under "slides by Simon Willison"Under "slides about Django"
But we like joins!
Lanyrd.com
Solution: denormalise data to Redis and Solr
Lanyrd.com
Try it
Ready for a test drive? Check this interactiveinteractive
tutorialtutorial that will walk you through the most
important features of Redis.
Redis is an open source, advanced key-value store. It is often
referred to as a data structure server since keys can contain
stringsstrings, hasheshashes, listslists, setssets and sorted sorted setssets.
Learn more Learn more →→
Download it
Redis 2.2.10 is the latest stable version.Redis 2.2.10 is the latest stable version.
Interested in legacy or unstable versions?
Check the downloads page.Check the downloads page.
What people are saying
More...More...
Comparison of CouchDB, Redis,MongoDB, Casandra, Neo4J &others http://j.mp/l32SqMhttp://j.mp/l32SqM via@DZone
@__NeverGiveup Oh YAY, oui tume redis ! *-* Hm, on s'rejoint à14h au bahut ? :o
JE L REDIS JE FOLLOW BACKSUR @Fuckement_TL
une question : "How to useServiceStack Redis in a webapplication to take advantage ofpub / sub paradigm"http://t.co/EOgyLU1http://t.co/EOgyLU1 #redis #web
Nice - Cassandra vs MongoDB vsCouchDB vs Redis vs Riak vsHBase vs Membase vs Neo4jcomparison http://bit.ly/l32SqMhttp://bit.ly/l32SqMfrom @kkovacs
This website is open source software developed by Citrusbyte. The Redis logo was designed by Carlos Prioglio.
Sponsored by
Commands Clients Documentation Community Download Issues
Lanyrd.com
simonw-follows:{144,21345,12328...}europython-attendees:{344,21345,787...}
redis.sinter( 'simonw-follows', 'europython-attendees')
Our redis sets...
Lanyrd.comadd a conferenceadd a conference you are signed in as simonw, do you want to sign out?
calendarcalendar conferencesconferences coveragecoverage profileprofile
searchsearch
JUNE2011
Florencein Italy
EuroPython 2011EuroPython 2011The European Python Conference
You'respeakingAT THIS EVENT
(short URL)
119 speakers
9780
PEOPLE
attending
PEOPLE
tracking
TELL YOUR FRIENDS!
Tweet about thisevent
Topics
Django
Plone
Pyramid
Python
Twisted
19–26http://ep2011.europython.eu/
View the schedule on Lanyrd
Save to iCal / iPhone / Outlook /GCal
@europython
#europython
lanyrd.com/ccdpc
AndreasSchreiber@onyame
AndrewGodwin@andrewgodwin
AndriiMishkovskyi@mishok13
ArminRonacher
AlanFranzoni@franzeur
AlessandroDentella
Alex Martelli
Ali Afshar@aliafshar
AnnaRavenscroft
Anselm Kruis
Antonio Cuni@antocuni
Armin RigoEdit topics
Lanyrd.com
Main Wiki
apache > lucene > solr
Search the site with Solr Search
Powered by Lucid ImaginationLast Published: Sat, 04 Jun 2011 12:23:42 GMT
Welcome to Solr
What Is Solr?Get StartedNews
May 2011 - Solr 3.2 ReleasedMarch 2011 - Solr 3.1 Released25 June 2010 - Solr 1.4.1 Released7 May 2010 - Apache Lucene Eurocon 2010 Coming to Prague May 18-2110 November 2009 - Solr 1.4 Released20 August 2009 - Solr's first book is published!18 August 2009 - Lucene at US ApacheCon09 February 2009 - Lucene at ApacheCon Europe 2009 in Amsterdam19 December 2008 - Solr Logo Contest Results03 October 2008 - Solr Logo Contest15 September 2008 - Solr 1.3.0 Available28 August 2008 - Lucene/Solr at ApacheCon New Orleans03 September 2007 - Lucene at ApacheCon Atlanta06 June 2007: Release 1.2 available17 January 2007: Solr graduates from Incubator22 December 2006: Release 1.1.0 available15 August 2006: Solr at ApacheCon US21 April 2006: Solr at ApacheCon21 February 2006: nightly builds17 January 2006: Solr Joins Apache Incubator
What Is Solr?
About
WelcomeWho We Are
Documentation
Resources
Related Projects
Lanyrd.com
add a conferenceadd a conference you are signed in as simonw, do you want to sign out?
calendarcalendar conferencesconferences coveragecoverage profileprofile
searchsearch
ON NOWFUTUREPAST
PAST
PUBLISHED
Topic: Django1 conference
5 conferences
51 conferences
Topic: GeoDjango1 conference
Practical Django ProjectsBy James Bennett
June 2009
Django Geek@djangeek
Learning Website Development with Django
FILTER BYtype
FILTER BYpast or futurePast events 55
Future events 5
FILTER BYtopicDjango 344
SearchSearchWe found 467 results for “django”
django SearchSearch
SLIDESDjango Con High Performance Django (slideshare.net)
from High Performance Django at DjangoCon 2008
Sessions 209
Coverage 190
Conferences 60
Books 5
Topics 2
People 1
Lanyrd.com
add a conferenceadd a conference you are signed in as simonw, do you want to sign out?
calendarcalendar conferencesconferences coveragecoverage profileprofile
searchsearch
TODAY
We've found 182 conferences your Twitter contacts are
interested in.
From our blogWelcoming SophieBarrett to teamLanyrd
Today we have a very special
announcement (and for once,
it's not a new feature!) We
would like to welcome the
super-wonderful Sophie Barrett
to the Lanyrd team.
Session schedules inyour calendar
You can now subscribe to event
schedules in your calendar of
choice. Stay up to date at the
event with the schedule in the
pocket where you need it.
Venues (and venuemaps)
Your contacts' calendarYour contacts' calendaryours 24 contacts 182
Astronomy Science
Café Scientifique: Exploringthe dark side of starformation with the HerschelSpace Observatory
United Kingdom / Brighton
21st June 2011
4 contacts tracking
21 Attend
Track
Usability User Experience
Usability Professionals'Association – InternationalConference
United States / Atlanta
21st–24th June 2011
1 contact speaking and 3 contacts tracking
21 Attend
Track
Simon
Willison
Your profile
page
Lanyrd.com
add a conferenceadd a conference you are signed in as simonw, do you want to sign out?
calendarcalendar conferencesconferences coveragecoverage profileprofile
searchsearch
Natalie DowneNatalie DowneUnfollowON TWITTER
1614
You currently follow @Natbat onTwitter
on Twitter
CONFERENCESspoken at
CONFERENCESinvolved with
Browse by year
2011 ! 2010 ! 2009 !2008 ! 2007 ! 2006 !2005
Appears with
CSS Maintainable CSS Maintainability
Design Process Internet Explorer Debugging
Show and tell Personal Projects
Web Standards Crowdsourcing
WildlifeNearYou RSI Community
Event Planning Geek Night User Experience
Usability Usability Testing JavaScript
Progressive Enhancement
Unobtrusive JavaScript Flexible design
Planning Artifacts Social Software
Motivation CSS3 CSS Layout
Project Management Dev Fort
Rapid Application Development
PAST CONFERENCES
spoken at
Five Pound App #22 Brighton, United Kingdom
30th March 2010
CSS3 WizardryWorkshop
Brighton, United Kingdom29th January 2010
Clearleft PechaKucha: Volume One
Brighton, United Kingdom22nd January 2010
BarCamp London 7 London, United Kingdom
24th–25th October 2009
Clearleft brown baglunch: Natalie Downe
Brighton, United Kingdom9th June 2009
BarCamp London 6 London, United Kingdom
28th–29th March 2009
Oxford Geek Night 11 Oxford, United Kingdom
25th March 2009
Flash Brighton - March24th, 2009
Brighton, United Kingdom24th March 2009
CSS systems withNatalie Downe
Brighton, United Kingdom4th March 2009
Namics InternationalTechnologyConference
Basel-City, Switzerland10th October 2008
BarCamp London 5 Richmond upon Thames,
United Kingdom27th–28th September 2008
BarCamp London 4 London, United Kingdom
31st May to 1st June 2008
Kiwi Foo 2008 Warkworth, New Zealand
1st–3rd February 2008
BarCamp London 3 London, United Kingdom
24th–25th November 2007
BarCamp London 2 London, United Kingdom
17th–18th February 2007
BarCamp London London, United Kingdom
2nd–3rd September 2006
PAST CONFERENCES
involved with
@media 2010 London, United Kingdom
8th–11th June 2010
Stage manager for the mainstage
Full Frontal 2009 Brighton, United Kingdom
20th November 2009
Volunteer
dConstruct 2009 Brighton, United Kingdom
4th September 2009
Volunteer
UX London 2009 London, United Kingdom
15th–17th June 2009
Volunteer
Skillswap on Speed Brighton, United Kingdom
29th October 2008
Organiser
<head> WebConference
24th–26th October 2008
Volunteer
dConstruct 2008 Brighton, United Kingdom
5th September 2008
Volunteer
Skillswap Goes Pretty Brighton, United Kingdom
20th August 2008
Organiser
Skillswap goesCerebral
Brighton, United Kingdom9th July 2008
Organiser
Skillswap goesJavascript
Brighton, United Kingdom11th June 2008
organiser
Oxford Geek Night 3 Oxford, United Kingdom
25th July 2007
Organiser
Oxford Geek Night 2 Oxford, United Kingdom
11th April 2007
Organiser
BarCamp London 2 London, United Kingdom
17th–18th February 2007
Organiser
Oxford Geek Night 1 Oxford, United Kingdom
7th February 2007
Organiser
PAST CONFERENCES
attended
Frontiers ofInteraction 2011
Florence, Italy20th–21st June 2011
EuroPython 2011 Florence, Italy
19th–26th June 2011
Foo Camp 2011 Sebastopol, United States
10th–12th June 2011
DiscoveringMadagascar
London, United Kingdom25th May 2011
OpenTech 2011 London, United Kingdom
21st May 2011
Devnest - TwitterTuesday
London, United Kingdom3rd May 2011
Pub Standards LXV London, United Kingdom
14th April 2011
DevTank April London, United Kingdom
13th April 2011
London WebStandards: Fun andGames
Holborn, United Kingdom11th April 2011
Etsy Code as Craft Austin, United States
12th March 2011
SXSW Interactive2011
Austin, United States11th–15th March 2011
The LaunchConference
San Francisco, UnitedStates23rd–24th February 2011
Ignite Silicon Valley #3 Mountain View, United
States11th February 2011
Strata 2011 Santa Clara, United States
1st–3rd February 2011
Thirsty Bristol Bristol, United Kingdom
27th January 2011
San Francisco DjangoMeetup GroupNovember:Deployment Bonanza
San Francisco, UnitedStates17th November 2010
Think VitaminJavaScript
13th September 2010
State of the Map 2010 Girona, Spain
9th–11th July 2010
Explore 2009 London, United Kingdom
13th–15th November 2009
Future of Web AppsLondon 2009
London, United Kingdom30th September to 2ndOctober 2009
BarCamp Brighton 4 Brighton, United Kingdom
5th–6th September 2009
@media 2009 London, United Kingdom
25th–26th June 2009
London Open HackDay 2009
London, United Kingdom9th–10th May 2009
Barcamp Austin 4 Austin, United States
14th April 2009
SXSW Interactive2009
Austin, United States13th–17th March 2009
Future of Web AppsExpo 2008
London, United Kingdom8th–10th October 2008
BarCamp Brighton 3 University of Sussex, United
Kingdom6th–7th September 2008
reboot10 Copenhagen, Denmark
25th–26th June 2008
@media 2008 London, United Kingdom
29th–30th May 2008
Accessibility 2.0 London, United Kingdom
25th April 2008
Future of Web Design- FOWD
London, United Kingdom17th–18th April 2008
Webstock 2008 Wellington, New Zealand
11th–15th February 2008
Web 2.0 Expo Berlin2007
Berlin, Germany5th–8th November 2007
Future of Web AppsExpo 2007
London, United Kingdom3rd–5th October 2007
BarCamp Brighton Brighton, United Kingdom
8th–9th September 2007
dConstruct 2007 Brighton, United Kingdom
5th–7th September 2007
Hack Day London2007
London, United Kingdom16th–17th June 2007
@media 2007 London, United Kingdom
7th–8th June 2007
Future of Web DesignLondon 2007
London, United Kingdom18th April 2007
Future of Web Apps2007
London, United Kingdom20th–22nd February 2007
dConstruct 2006 Brighton, United Kingdom
8th September 2006
LUGRadio Live 2006 Wolverhampton, United
Kingdom22nd–23rd July 2006
@media 2006 London, United Kingdom
15th–16th June 2006
SXSW Interactive2006
Austin, United States10th–14th March 2006
The Future of WebApps
London, United Kingdom8th February 2006
Geek Dinner withDave Shea
London, United Kingdom23rd January 2006
dConstruct 2005 Brighton, United Kingdom
11th November 2005
Google open house London, United Kingdom
10th November 2005
OpenTech 2005 Hammersmith, United
Kingdom23rd July 2005
FUTURE CONFERENCES
speaking at
London Insites TourEvening
London, United Kingdom19th July 2011
FUTURE CONFERENCES
attending
Activate - Summer2011
Kent, United Kingdom5th August 2011
dConstruct 2011 Brighton, United Kingdom
2nd September 2011
Brooklyn Beta 2011 Brooklyn, United States
12th–14th October 2011
Full Frontal 2011 Brighton, United Kingdom
10th–11th November 2011
Explore 2011 London, United Kingdom
18th–20th November 2011
FUTURE CONFERENCES
tracking
The Guardian'sActivate Summit 2011
London, United Kingdom22nd June 2011
UX Tuesday, June2011
London, United Kingdom28th June 2011
Croydon Creatives Croydon, United Kingdom
29th June 2011
WDCNZ Wellington, New Zealand
14th July 2011
Science OnlineLondon Conference
London, United Kingdom2nd–3rd September 2011
Flash on the Beach2011
Brighton, United Kingdom11th–14th September 2011
PICNIC '11 Amsterdam, Netherlands
14th–16th September 2011
The Do Lectures 2011 Cardigan, United Kingdom
15th–19th September 2011
Fronteers 2011 Amsterdam, Netherlands
6th–7th October 2011
Playful 2011 London, United Kingdom
21st October 2011
Build 2011 Belfast, United Kingdom
7th–11th November 2011
“Co-founder of @lanyrd - Enthusiastic worldexplorer and digital creative.”
Session topics
Coverage
SEE ALL COVERAGE
11 items in total
FAVOURITE COVERAGE
See 6 items thatNatalie Downe hasfavourited
Book credits
TECHNICAL REVIEWER
The Ultimate CSSReferenceTECHNICAL EDITOR
CSS Mastery
@Natbat
9 times
Simon Willison
5 times
Nigel Crawley
4 times
Reinier Zwitserloot
4 timesNeil Crosby
4 times
Cristiano Betta
4 times
Matt Westcott
4 times
Gareth Rushgrove
bio from Twitter
Full speaking history for Natalie Downe
Online conference
Online conference
Online conference
3 videosMost recent added 7months ago
5 slide decksMost recent added 7months ago
1 write-upAdded 7 months ago
2 linksMost recent added 7months ago
faq ! blog ! privacy ! services ! colophonFollow @lanyrd on twitter. add a conferenceadd a conference
Trick #4: Hashed static asset filenames in CloudFront
Lanyrd.com
cdn.lanyrd.net/js/global.ed81d119.js
global.js
global.ed81d119.js
Lanyrd.com
BenefitsFar futures expiry headersCache-Control: max-age=315360000Expires: Fri, 18 Jun 2021 06:45:00 -0000 GMT
Guaranteed updated CSS in IEDeploy new assets in advance of applicationOld versions stick around for rollbacks
Lanyrd.com
./manage.py push_staticMinifies JavaScript and CSSRenames files to include sha1(contents)[:8]Pushes all assets to S3
Part 2: Challenges
Challenge #1: HTTP requests
Lanyrd.com
Talking to an API?What if it fails to load?What if it takes 30 seconds to load?
Lanyrd.com
A user gives you a URL...urllib.urlopen(url).read() ?
what if it's a huge file?what if it's a slow loading tarpit?what if it's the URL to your private, firewalled Solr instance?http://10.0.1.1/solr/select/?q=secrets
Lanyrd.com
Safe URL consumptionConnection timeoutsLogging and ProfilingHost validationHTTP caching / if-none-match / etcdjango.httpclient ?
Challenge #2: Profiling and debugging production systems
Lanyrd.com
Debugging in development rocks!
django_debug_toolbar = awesomeassert False = instant useful 500 pageimport pdb; pdb.set_trace() adds a command line debugger to your running application!
Lanyrd.com
DEBUG = FalseDEBUG = "BLIND"
Lanyrd.com
from django.views.debug import technical_500_responseimport sys
class UserBasedExceptionMiddleware(object): def process_exception(self, request, exception): if request.user.is_superuser: return technical_500_response(request, *sys.exc_info())
Lanyrd.com
mysql-proxyVery handy lua-customisable proxy for all of your MySQL trafficWorst documented software everlog.lua - logs out ALL querieshttps://gist.github.com/1039751
Lanyrd.com
django_instrumented(Unreleased) code I wrote for LanyrdCollects various runtime stats about the current request, stashes a profile JSON in memcachedWrites out the profile UUID as part of the HTMLA bookmarklet to view the profile
Lanyrd.com
Lanyrd.com
Django improvementsDEBUG as a global setting is an anti-patternMore low-level hooks for measuring, well, pretty much everythingLive profiling tools (as a third-party product)
MVC Mini Profiler
http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html
Challenge #3: Zero downtime deploys
Lanyrd.com
Challenge: deploy upgrades, including database upgrades,without downtime
Lanyrd.com
Deploy DB changes separately from code changes
Make backwards compatible schema changesWe deploy code to a none-web server just so we can run "./manage.py migrate"(you are using South, right?)Use symlinks for instant rollback to previous code
Lanyrd.com
Read-only modeIf you can flip your site in to read-only mode, upgrades become a lot easier
Flip to read only modeReplicate DB to a new machineRun migrations, then testSwitch app servers to new database(On EC2, fire up an entire new web cluster)
Feature flags: finely grained version of this
Lanyrd.com
The lesson wekeep on learning
Lanyrd.com
Global settings are BADDatabase settings => multi-db (1.2)Cache settings => multi cache backends (1.3)Haystack backends => multi backends (v2.0)
Lanyrd.com
Global settings are BADDatabase settings => multi-db (1.2)Cache settings => multi cache backends (1.3)Haystack backends => multi backends (v2.0)
DEBUGTIME_ZONEMiddleware?
Lanyrd.com
Settings should be modifiable at run-time
We're hiring a developer/web operations person in London, come and talk to us!
Thank you!http://lanyrd.com/sfwcq
[ [