pick n mix – djugl sep 09

8/8/2019 Pick n Mix DJUGL Sep 09

1/56

Ben Firshman


2/56


3/56

Performance:* Incredibly important, especially now web applications doing increasingly complex things.

* Amazon found that 100ms of latency caused 1% loss in sales* Extra half a second on Google search results caused 20% drop in trafc

* Especially when altering data, it still needs to be responsive:* Posting a tweet to followers* Uploading images* Importing API data

* Running stuasynchronously


4/56

* Running TASKS asynchronously


5/56

Tasks

Independent components

Loosely coupled

Easily scalable

* Independent components that can run concurrently* Loosely coupled: only depends on initial input* Thanks to which, theyre easily scalable.

* Essentially:* Stu you can send o to run in the background.* Stu the user doesnt immediately care about.


6/56

Application

* Say we have our application* and a bunch of tasks that need running* What runs these tasks?


7/56

Application



8/56

Application

??

?

?



9/56

Application

Worker Worker

* Workers process the tasks.* Unlike ghetto queues done with cron jobs and databases, these are processes that havetasks pushed to them* These could either be on the same server as the application, or, thanks to independence,spread out onto many* Some way for application to publish tasks that need to be completed and distributing toworkers


10/56

Application

Message Queue

* Message Queues are useful for this. (explain)* The messages can just be pickled tasks


11/56

Application

Message Queue

* Message Queues are useful for this. (explain)* The messages can just be pickled tasks


12/56

Application

Message Queue


13/56

Application

Message Queue

Application

* Nice thing with message queues: we can have multiple applications push things to thequeue.* Means the frontend servers can be scaled horizontally without any of the taskinfrastructure caring


14/56

Application

Message Queue

Application

* Nice thing with message queues: we can have multiple applications push things to thequeue.* Means the frontend servers can be scaled horizontally without any of the taskinfrastructure caring


15/56

Application

Message Queue

Application

Worker Worker


16/56

Application

Message Queue

Application

Worker Worker


17/56

Application

Message Queue

Application

Worker Worker


18/56

Application

Message Queue

Application

Worker Worker


19/56

Application

Message Queue

Application

Worker Worker


20/56

Application

Message Queue

Application

Worker Worker


21/56

* This is where celery comes in* Ask Solem Hoel has written an excellent piece of software that provides everything youneed for a task queue


22/56

Celery

Worker daemon


23/56

Celery

Worker daemon API for defining and executing tasks


24/56

Celery

Worker daemon API for defining and executing tasks

Handles communication with message queue (AMPQ or

STOMP)


25/56


26/56

Code!

>>> from myapp.tasks import ImportTweetsTask

>>> ImportTweetsTask.delay(username=londonpython)

* This is the way a simple task is run.* If youre just writing something that writes the result to the database, this is fine


27/56

Code!

>>> result = ImportTweetsTask.delay(username=londonpython)

>>> result.ready()False

>>> result.ready()True

>>> [t.text for t in result.result][u'I will also be looking into paying for pizza forDJUGL attendees next Thursday at Global Radio officesin Leicester Square.', ...]


28/56

map

>>> import operator

>>> from celery.task import dmap

>>> dmap(operator.add, [[1, 2], [4, 2], [3, 8]])[3, 6, 11]

* Callable is pickled, so it must be available on the workers path


29/56

More goodies

Periodic tasks as a distributed replacement for cron

View for Ajax to check status

Statistics

Task sets for progress bars

Retry tasks on failure

* Stats: how many tasks have been executed by type, time taken etc


30/56

http://www.celeryq.org/
http://www.celeryq.org/http://www.celeryq.org/


31/56


32/56

py.test

No boilerplate tests

Useful test failure information

Extensibility

* No boilerplate: Automatic detection of tests and use of built in Python features like assert


33/56

Code!

def test_equality():x = 5assert x == 5

def test_not_equal():y = 9assert y != 2

def test_articles_exist(articles):assert articles.objects.count() > 0

* funcargs provide fixtures


34/56


35/56

Funcargs

def test_something(client):assert 'Success!' in client.get('/path/').content

def test_article_view(rf):request = rf.get('/article/4/')response = article_view(request, 4)assert 'Test title' in response.content


36/56

Funcargs cont.

def test_middleware(settings, client):settings.MIDDLEWARE_CLASSES = (

'app.middleware.SomeMiddleware',)assert 'Middleware works!' in client.get('/')


37/56

Decorators

@py.test.params([dict(a=1, b=2), dict(a=3, b=3)])

def test_equals(a, b):assert a == b

@py.test.urls('myapp.test_urls')def test_something(client):

assert 'Success!' in client.get('/some_path/')


38/56


39/56


40/56

syndication-view

An attempt to clean up django.contrib.syndication


41/56

syndication-view


Feeds as class based views

d


42/56

syndication-view



Backwards compatible

d


43/56

syndication-view



Backwards compatible

Eventually, refactoring generation of feeds including full

Atom support

C d !


44/56

Code!

from django.contrib.syndication.feeds import Feedfrom articles.models import Article

class ArticleFeed(Feed):title = "example.com news"link = "/sitenews/"description = "Latest news from example.com."

def items(self):return Article.objects.all()[:15]

Y k


45/56

Yuck.

from articles.feeds import ArticleFeed

feeds = {'articles': ArticleFeed,

}

urlpatterns = patterns('',# ...(r'^feeds/(?P.*)/$',

'django.contrib.syndication.views.feed',

{'feed_dict': feeds}),# ...

)

C d !


46/56

Code!

from syndication.views import Feedfrom articles.models import Article

class ArticleFeed(Feed):title = "example.com news"link = "/sitenews/"description = "Latest news from example.com."

def items(self):return Article.objects.all()[:15]

M h b


47/56

Much better.

from articles.feeds import ArticleFeed

urlpatterns = patterns('',# ...(r'^articles/feed/$', ArticleFeed()),# ...

)


48/56

http://github.com/bfirsh/syndication-view/

If you have any pet hates of contrib.syndication, let me know!
http://github.com/bfirsh/syndication-view/http://github.com/bfirsh/syndication-view/


49/56

django mptt


50/56

django-mptt

Making trees out of Django models.

Efficient retrieval.

Methods added to models for traversing trees(get_children(), get_root(), get_decendants()etc)

* Efcient retrieval as opposed to a simple parent field and retrieving children with relatedname

Code!


51/56

Code!

from django.db import models

import mptt

class Category(models.Model): name = models.CharField(max_length=50) parent = models.ForeignKey('self', null=True,blank=True, related_name='children')

mptt.register(Category)


52/56

Code!


53/56

Code!

from django.db import modelsimport mptt

class Category(mptt.Model): name = models.CharField(max_length=50) parent = models.ForeignKey('self', null=True,blank=True, related_name='children')


54/56

http://github.com/bfirsh/django-mptt/
http://github.com/bfirsh/django-mptt/http://github.com/bfirsh/django-mptt/


55/56

http://benfirshman.com/@bfirsh
http://twitter.com/bfirshhttp://twitter.com/bfirshhttp://benfirshman.com/http://benfirshman.com/


56/56

Credits

http://www.flickr.com/photos/ilovetechnology/3048585444/http://www.flickr.com/photos/tomhakase/312785554/http://www.flickr.com/photos/pensive-reflections/3829587788/http://www.flickr.com/photos/purpleslog/183842413/

http://www.flickr.com/photos/chrissie64/1425138918/http://www.flickr.com/photos/patlejch/2951799028/http://www.flickr.com/photos/mckenzieo/1601233059/http://www.flickr.com/photos/webel/2429902979/
http://www.flickr.com/photos/webel/2429902979/http://www.flickr.com/photos/webel/2429902979/http://www.flickr.com/photos/mckenzieo/1601233059/http://www.flickr.com/photos/mckenzieo/1601233059/http://www.flickr.com/photos/patlejch/2951799028/http://www.flickr.com/photos/patlejch/2951799028/http://www.flickr.com/photos/chrissie64/1425138918/http://www.flickr.com/photos/chrissie64/1425138918/http://www.flickr.com/photos/purpleslog/183842413/http://www.flickr.com/photos/purpleslog/183842413/http://www.flickr.com/photos/pensive-reflections/3829587788/http://www.flickr.com/photos/pensive-reflections/3829587788/http://www.flickr.com/photos/tomhakase/312785554/http://www.flickr.com/photos/tomhakase/312785554/http://www.flickr.com/photos/ilovetechnology/3048585444/http://www.flickr.com/photos/ilovetechnology/3048585444/

pick n mix – djugl sep 09

Documents