Download - ScaleFail
![Page 1: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/1.jpg)
ScaleFail(and MPs’ expenses)
![Page 2: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/2.jpg)
Copyright © Steve Bell 2009
![Page 3: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/3.jpg)
![Page 4: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/4.jpg)
![Page 5: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/5.jpg)
![Page 6: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/6.jpg)
![Page 7: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/7.jpg)
![Page 8: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/8.jpg)
![Page 9: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/9.jpg)
How we built it
![Page 10: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/10.jpg)
![Page 11: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/11.jpg)
Crash #1: more Apache children than MySQL connections
![Page 12: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/12.jpg)
![Page 13: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/13.jpg)
![Page 14: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/14.jpg)
unreviewed_count = Page.objects.filter( votes__isnull = True).distinct().count()
![Page 15: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/15.jpg)
SELECT COUNT(DISTINCT `expenses_page`.`id`)FROM `expenses_page` LEFT OUTER JOIN `expenses_vote` ON ( `expenses_page`.`id` = `expenses_vote`.`page_id` ) WHERE `expenses_vote`.`id` IS NULL
![Page 16: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/16.jpg)
unreviewed_count = cache.get('homepage:unreviewed_count')if unreviewed_count is None: unreviewed_count = Page.objects.filter( votes__isnull = True ).distinct().count() cache.set('homepage: unreviewed_count', unreviewed_count, 60)
![Page 17: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/17.jpg)
• With 70,000 pages and a LOT of votes...
• DB takes up 135% of CPU
• Cache the count in memcached...
• DB drops to %35 of CPU
![Page 18: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/18.jpg)
unreviewed_count = Page.objects.filter( votes__isnull = True ).distinct().count()
reviewed_count = Page.objects.filter( votes__isnull = False ).distinct().count()
![Page 19: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/19.jpg)
unreviewed_count = Page.objects.filter( is_reviewed = False ).count()
![Page 20: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/20.jpg)
Migrating to InnoDB on a separate server
![Page 21: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/21.jpg)
ssh mps-live "mysqldump mp_expenses" |sed 's/ENGINE=MyISAM/ENGINE=InnoDB/g' |
sed 's/CHARSET=latin1/CHARSET=utf8/g' |ssh mysql-big "mysql -u root mp_expenses"
![Page 22: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/22.jpg)
“next” button
![Page 23: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/23.jpg)
def next_global(request): # Next unreviewed page from the whole site all_unreviewed_pages = Page.objects.filter( is_reviewed = False ).order_by('?') if all_unreviewed_pages: return Redirect( all_unreviewed_pages[0].get_absolute_url() ) else: return HttpResponse( 'All pages have been reviewed!' )
![Page 24: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/24.jpg)
def next_global(request): # Next unreviewed page from the whole site all_unreviewed_pages = Page.objects.filter( is_reviewed = False ).order_by('?') if all_unreviewed_pages: return Redirect( all_unreviewed_pages[0].get_absolute_url() ) else: return HttpResponse( 'All pages have been reviewed!' )
![Page 25: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/25.jpg)
import random
def next_global_from_cache(request): page_ids = cache.get('unreviewed_page_ids') if page_ids: return Redirect( '/page/%s/' % random.choice(page_ids) ) else: return next_global(request)
![Page 26: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/26.jpg)
Next time, I’ll use redis
![Page 27: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/27.jpg)
Read-only mode saved our bacon
![Page 28: ScaleFail](https://reader034.vdocuments.us/reader034/viewer/2022051609/545ca623af7959be098b48df/html5/thumbnails/28.jpg)
Copyright © Martin Rowson 2008