web performance culture and tools at etsy
DESCRIPTION
This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you don't have a formalized performance team in your company.Originally presented at the Boston Web Performance Meetup on Aug 24, 2011.TRANSCRIPT
![Page 1: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/1.jpg)
Web Performance Culture and Tools at EtsyMike BrittainDir. of Engineering, InfrastructureEtsy
Boston Web Performance MeetupAug 24, 2011
![Page 2: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/2.jpg)
OverviewEtsy and EngineeringMake PerformanceMatterTools and Process
![Page 3: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/3.jpg)
![Page 5: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/5.jpg)
MonthlyWeather Report$38 MM in sales1.9 MM items sold
http://etsy.me/weather-report-june-2011
![Page 6: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/6.jpg)
MonthlyWeather Report990 MM page views
http://etsy.me/weather-report-june-2011
![Page 7: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/7.jpg)
Engineering
TechnologiesLinux, Apache, MySQL, PHP, MemcacheSolr, Squid, Hadoop, Amazon S3, EC2, EMRGanglia, Cacti, Nagios, Graphite, Splunk, and some of our own...
![Page 8: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/8.jpg)
Engineering
Teams90 Engineers3-6 engineers+ product developer and/or designer
![Page 9: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/9.jpg)
Engineering
Continuous Deployment~40 releases per day inc. app code and config changes
![Page 10: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/10.jpg)
1. Write code2. Code review3. Automated tests
![Page 11: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/11.jpg)
1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod
![Page 12: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/12.jpg)
1. Write code2. Code review3. Automated tests4. Dev ⇾ QA ⇾ Pre-Prod ⇾ Prod5. Monitor!6. Monitor!7. Monitor!
![Page 13: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/13.jpg)
Engineering
Data-Driven Development45,000+ metrics50+ dashboards
![Page 14: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/14.jpg)
MakePerformanceMatterHave a story
![Page 15: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/15.jpg)
Business Impact
Measure“Our own years of testing have conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.”
Abundance of research from Google, Bing, AOL, Amazon, Shopzilla, etc.http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html
![Page 16: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/16.jpg)
Business Impact
MeasureBounce rateSearch conversion
![Page 17: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/17.jpg)
Business Impact
MeasurePurchase funnelAd impressions, page views and tracking discrepanciesSocial, engagement
![Page 18: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/18.jpg)
Operations
Site Stability
Contention for shared resources like database, memcache, solr, or even web server processes
Measure perf for discrete piecesof your infrastructure
![Page 19: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/19.jpg)
Morale
Happy Engineers
![Page 20: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/20.jpg)
Getting to work
Where toStart?Focus your efforts where it makes sense for your business
![Page 21: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/21.jpg)
Getting to work
Where toOptimize?Tiers of serviceSLAs
![Page 22: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/22.jpg)
Getting to work
FocusBeware of CTS:Constant Tweaking Syndrome
![Page 23: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/23.jpg)
Getting to work
FocusFriends don’t let friendstweak without graphing
![Page 24: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/24.jpg)
MakePerformanceMatterHave your story
![Page 25: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/25.jpg)
Tools andProcess atEtsy
![Page 26: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/26.jpg)
Process
Etsy’s Perf TeamStandardize patternsCreate tools and reportsCoordinate efforts
![Page 27: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/27.jpg)
Process
Server-sidePerformanceThere is nothing worse than a blank page or a spinningthing-a-ma-jigger
![Page 28: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/28.jpg)
Process
Server-sidePerformance95th Percentile> 800 ms is a P2 bug
![Page 29: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/29.jpg)
Process
Server-sidePerformance
![Page 30: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/30.jpg)
Tools
LoggingPerf-related data belongs in your server logsapache_note()
![Page 31: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/31.jpg)
Tools
Logging$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;
register_shutdown_function()
apache_note('php_microsec', $timer_diff);
![Page 32: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/32.jpg)
LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined
Tools
Logging
![Page 33: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/33.jpg)
LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined
Tools
Logging
![Page 34: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/34.jpg)
LogFormat "%{True-Client-IP}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{etsy_user_id}n %{php_bytes}n %{php_microsec}n %D" combined
Tools
Logging
![Page 35: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/35.jpg)
web0060 66.249.71.110 - - [24/Aug/2011:04:16:52 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - 13399576 505780 554876
Tools
Logging
![Page 36: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/36.jpg)
Tools
Analyzegrep ... access.log | awk ...grep "/listing/" access.log | \awk '{sum=sum+$(NF-1)} END {print sum/NR}'
grep "/listing/" access.log | \awk 'BEGIN {max=0} {if ($(NF-1)>max) max=$(NF-1)} END {print max}'
![Page 37: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/37.jpg)
Tools
Analyze
![Page 38: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/38.jpg)
Tools
Analyze1. Capture perf data in logs page gen., Boomerang, Gomez2. Aggregate Splunk, Logster, StatsD3. Record in Graphite
http://graphite.wikidot.com
![Page 39: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/39.jpg)
Tools
Analyze
![Page 40: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/40.jpg)
Tools
Logster1. Read new log entries every minute2. Parse and aggregate useful numbers3. Send to Graphite http://github.com/etsy
![Page 41: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/41.jpg)
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppppp!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web1101 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.web0002 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!web0201 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0077 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0355 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0052 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0089 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.web1101 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!web0055 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!web0001 [04:28:54 2011] [warning] [client 10.101.x.x] Oh noooooooooooweb0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!web0034 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
![Page 42: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/42.jpg)
Fatals Errors Warnings
![Page 43: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/43.jpg)
:)
![Page 44: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/44.jpg)
Tools
StatsDCollects metrics from your app code and sticks them in GraphiteStatsD::increment("logins.success");StatsD::timing("gearman.time", $msec);
![Page 45: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/45.jpg)
Tools
StatsD90th pct
average
lower
![Page 46: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/46.jpg)
:) :)
![Page 47: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/47.jpg)
Tools
Perf Dashboard
![Page 48: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/48.jpg)
Tools
I/O ProfilerLightweight, inline profilingStart and end times wrapped around service calls databases, memcache, apc, etc.
![Page 49: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/49.jpg)
Tools
I/O Profiler
![Page 50: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/50.jpg)
Tools
I/O Profiler
![Page 51: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/51.jpg)
Tools
Client-sideTestingGomez (API), Boomerang, WebPagetest
![Page 52: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/52.jpg)
Tracks front-end best practices
Tools
ShowSlow
![Page 53: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/53.jpg)
Tools
ShowSlowInternal instanceAutomated testingEnd-point for I/O ProfilerTrending on individual rules http://showslow.com
![Page 54: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/54.jpg)
Other thoughts
YSlow &Page SpeedMind your development process3rd-party content and ga.js“Use a CDN”
![Page 55: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/55.jpg)
Other thoughts
Device-SpecificDesignThe mobile web is very muchabout designing for performanceScreen size, pixel density, connection speed, wi-fi vs. cellular, browser cache size, local storage, connections per host, metered pricing, etc.
![Page 56: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/56.jpg)
MakePerformanceMatterHave your storyUse tools and processto focus your efforts
![Page 57: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/57.jpg)
We are Hiringhttp://etsy.com/careersSoftware Engineering positions available in a number of teams, including Analytics, Operations, Web Performance, Payments, Core Platform, Front-End, Internal Apps, Search, Security, and more...
![Page 58: Web Performance Culture and Tools at Etsy](https://reader033.vdocuments.us/reader033/viewer/2022060107/554bb017b4c905b8618b593d/html5/thumbnails/58.jpg)
ThankYouMike [email protected]@mikebrittainCodeAsCraft.etsy.comgithub.com/etsygraphite.wikidot.comshowslow.com