Transcript
Page 1: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

A Related Matter:Optimizing your webapp by using django-debug-toolbar,

select_related(), and prefetch_related()

Christopher Adams DjangoCon 2014

https://github.com/adamsc64/a-­‐related-­‐matter

Page 2: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Christopher Adams

• Software Engineer at Venmo

• Twitter/Github: @adamsc64

• I’m not Chris Adams (@acdha), who works at Library of Congress

• Neither of us are “The Gentleman” Chris Adams (90’s-era Professional Wrestler)

Page 3: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 4: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Django is greatBut Django is really a set of tools

Page 5: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Tools are greatBut tools can be used in good or bad ways

Page 6: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

The Django ORM: A set of tools

Page 7: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Manage your own expectations for tools

• Many people approach a new tool with broad set of expectations as to what the think it will do for them.

• This may have little correlation with what the project actually has implemented.

Page 8: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

As amazing as it would be if they did…

Page 9: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Unicorns don’t exist

Page 10: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

The Django ORM: An abstraction layer

Page 11: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Abstraction layers

• Great because they take us away from the messy details

• Risky because they take us away from the messy details

Page 12: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Don’t forgetYou’re far from the ground

Page 13: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

The QuerySet API

Page 14: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

QuerySets are Lazy

Page 15: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

QuerySets are Immutable

Page 16: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Lazy: Does not evaluate until it needs to

Page 17: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Immutable: Never itself changes

Page 18: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Each a new QuerySet, none hit the database

• queryset  =  Model.objects.all()  

• queryset  =  queryset.filter(...)  

• queryset  =  queryset.values(...)

Page 19: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Hits the database (QuerySet is “evaluated”):

• queryset  =  list(queryset)  

• queryset  =  queryset[:]  

• for  model_object  in  queryset:          do_something(...)

Page 20: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Our app: a blog

Page 21: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Modelsclass Blog(models.Model):! submitter = models.ForeignKey('auth.User')!!class Post(models.Model):! blog = models.ForeignKey('blog.Blog', related_name="posts")! likers = models.ManyToManyField('auth.User')!!class PostComment(models.Model):! submitter = models.ForeignKey('auth.User')! post = models.ForeignKey('blog.Post', related_name="comments")!

Page 22: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

List View

def blog_list(request):! blogs = Blog.objects.all()! return render(request, "blog/blog_list.html", {! "blogs": blogs,! })!

Page 23: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

List Template

Page 24: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 25: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Detail Viewdef blog_detail(request, blog_id):! blog = get_object_or_404(Blog, id=blog_id)! posts = Post.objects.filter(blog=blog)! return render(request, "blog/blog_detail.html", {! "blog": blog,! "posts": posts,! })!

Page 26: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Detail Template

Page 27: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 28: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

SQL Queries?

Page 29: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

If you can’t measure it…

Page 30: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

…you’d never know if there are problems.

Page 31: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 32: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

First view:The blog list page

Page 33: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 34: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 35: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 36: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 37: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 38: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

select_related()• select_related works by creating an SQL join and

including the fields of the related object in the SELECT statement.

• For this reason, select_related gets the related objects in the same database query.

• However, to avoid the much larger result set that would result from joining across a ‘many’ relationship, select_related is limited to single-valued relationships - foreign key and one-to-one.

Page 39: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

List View

def blog_list(request):! blogs = Blog.objects.all()! blogs = blogs.select_related("submitter")! return render(request, "blog/blog_list.html", {! "blogs": blogs,! })!

Page 40: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 41: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 42: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 43: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Second view:The blog detail page

Page 44: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 45: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 46: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 47: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 48: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 49: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 50: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 51: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

prefetch_related()• prefetch_related does a separate lookup for

each relationship, and does the ’joining’ in Python.

• This allows it to prefetch many-to-many and many-to-one objects … in addition to the foreign key and one-to-one relationships.

• It also supports prefetching of GenericRelation and GenericForeignKey.

Page 52: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Detail Viewdef blog_detail(request, blog_id):! blog = get_object_or_404(Blog, id=blog_id)! posts = Post.objects.filter(blog=blog)! posts = posts.prefetch_related(! “comments__submitter", "likers",! )! return render(request, "blog/blog_detail.html", {! "blog": blog,! "posts": posts,! })!

Page 53: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()
Page 54: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Summary

• The QuerySet API methods select_related() and prefetch_related() automate some best practices to avoid extra queries in views/templates.

• Use select_related() for one-to-many or one-to-one relations.

• Use prefetch_related() for many-to-many or many-to-one (e.g. reverse foreign key) relations.

Page 55: A Related Matter: Optimizing your webapp by using django-debug-toolbar, select_related() and prefetch_related()

Thanks!Christopher Adams (@adamsc64)

https://github.com/adamsc64/a-related-matter


Top Related