shopzilla - performance by design

Performance By DesignA look at Shopzilla's path to high performance

Tim Morrow, Senior Architect

11/3/2009

Performance By Design | 11/3/2009

Agenda

• Our Company and what we do

• Design approach

• Detailed site architecture

• Architecture evolution

• Front end performance techniques

2


3

Shopzilla, Inc. - Online Shopping Network

100M impressions/day

20-29M UV’s per Month

8,000+searches per second

100M+Products


4

Background

• In 2000, the business was simple - bizrate.com in the US

• Co-brands, API and shopzilla.com

• Multivariate testing, Europe and multiple data centers


Growth

5


Original Architecture

6


Performance and Scalability Issues

• High latency due to sequential resource access

• Data fetching sometimes O(n)

• Lengthy Time To First Byte

• Memory constrained

• Lacked visibility into performance issues

7


Development Issues

• Serving 9+ web site experiences

• Different look-and-feel; localization

• New features carry high risk

• Shared code-base limited team autonomy

8


9

Our Build Approach

• Start over, stay simple

• Build 2 weeks at a time, deliver every page ASAP

• Manage risk by managing exposure


New Design Principals

• Simplify layers

• Decompose architecture

• Define SLAs

• Continuous performance testing

• Utilize caching

• Apply best-practice UI performance techniques

10


New Site Architecture

11


Site Technologies

• Java 1.6, Tomcat 6

• Spring MVC

• TAL-like templating engine

• Services are JAX-RS implemented with Apache CXF

• Hibernate 3 with Ehcache

• Oracle 10g Database

• Oracle Coherence Data Grid

12


13

Performance SLAs


Web Application Tier14

Service Calls


Web Application Tier

15


16

www.bizrate.com/digital-cameras


Decomposed Services

http://…/service/v3/reviews?pid=419943686&show=30&sort=newestFirst

17

<ProdRevResponse> <ProductReviews> <ProductReview> <UserRating>8.0</UserRating> <Title>advanced and begining photographer alike will benefit</Title> ... </ProductReview> </ProductReviews></ProdRevResponse>


Service Invocation

18

• Connection pooling

• Stale connection checking

• Hardware load balancers

• Connection and Socket Timeouts

• O(1) invocations on a single page

• JAXB XML->Java unmarshaling


How Do We Performance Test?

• Highly Concurrent requests

• Dozens of services

• Database access

• Search Engine invocations

• Meta-data lookups

19


Performance Testing Environment20


Analyzing Performance

• Emit server-side performance information

10.61.35.25 198.133.178.17 - - [15/Jun/2009:18:48:13 -0700] "GET /digital-cameras/ HTTP/1.1" 200 195063 - - www.bizrate.com unique_id=c52efc5b-740b-44d4-8693-587b6b756564!rst=22

start=1245116893247;elapsed=19;requestId=c52efc5b-740b-44d4-8693-587b6b756564;startDate=2009-06-15 18:48:13.247-0700;url=http:///services/content/v7/topsearchService/BR/US/402/0/21;

start=1245116893248;elapsed=24;requestId=c52efc5b-740b-44d4-8693-587b6b756564;startDate=2009-06-15 18:48:13.248-0700;url=http:///search/v5/US/12/product_search?keyword=digital+cameras

• Build server-side call graphs

21


Profiling

• YourKit Java Profiler

• Identify Syncronization Bottlenecks

22


Production Performance Monitoring

23

• JMX Mbeans

• Graphite graphing


Caching

• Started with a simple replicated local cache

• Cached data stored in the service process

• Cannot scale to large data sets

• Read-through caching not always suitable

• Moved to Oracle Coherence

24


Oracle Coherence

• Distributed data grid

• Dynamic cluster membership

• Automatic data partitioning

• Continuous availability

• Computation in the grid

25


Case Study #1 – URL Builder

• Rule set describing most optimal URL structures

• High frequency access from site

• Backoffice system to compute rules

26


Coherence Solution

27


Results

• Continue to meet performance SLAs

• In production for over 6 months

• Very successful project

28


Case Study #2 - Keyword Metadata

• Map of long IDs to object model describing landing page

• Over 600 million entries

• Entry point for the majority of our traffic

• Home-grown partitioned, distributed cache

29


Challenges

30

Time Consuming and intricate

Restarts Required


Solution

31

Automatic read-through

Automatic Expiration

Scalable


Results

• Faster acquisition of new paid placements

• No restarts

• Less software to maintain

• Great performance

32


Cache Architecture

• 6 physical instances

• 8-way, 32 Gb RAM

• 16 JVMs with 1.5 Gb heap

• Distributed Cache

• Database read-through

• LRU expriy based on object count

33


Performance Testing Coherence

• Performance test application against the grid

• Test scenarios such as server loss and data population

34


UI Performance Techniques

• 80% of time spent rendering the page

• Yahoo currently list 34 best practices

35


Minimize HTTP Requests

• Combined files

• CSS Sprites http://spriteme.org/

36


Use a CDN

• Move your content closer to end users

• Reduce latency

• Every resource except for dynamic HTML

• Offloads 100s of gigabytes per day

37


Expiry, Compression and Minification

• Expiry headers instruct the Browser to use a cached copy

• > 2 days considered “Far Futures”

• Use versioning techniques to allow forced upgrades

• Compressing reduces page weight

• Minifying may still reduce size by 5% even with compression

38


Reduce DNS lookups

• Yahoo recommends 3 – 4 DNS lookups per page

• Base page: www.bizrate.com

• Javascript & CSS: file01.bizrate-images.com

• Static images: img01.bizrate-images.com

• Dynamic images: image01.bizrate-images.com

• 3rd party Ads are a different story

39


Avoid Redirects

• Redirects delay your ability to server content

• We strive for zero redirects

• Exceptions:

– Redirect after POST

– Handling legacy-styled URLs

– Links off-site for tracking purposes

40


Use a Cookie-free domain

• Don’t send cookies when requesting static resources

• Buy a separate domain name

– bizrate-images.com

• Saves many Kb of upload bandwidth

• Revenue increased by 0.8%

41


Do Not Scale Images in HTML

• Don’t request larger images only to shrink them

• We utilize a dynamic image scaling server

• CDN caches and delivers exact image size

42


Make favicon.ico small and cacheable

• Can interfere with download sequence

• 2Kb+ multi-layered version:

• 318 byte version:

• We save 100s of megabytes per day

43


Flush the Buffer Early

• Delivers content to users sooner

• By default Tomcat flushes every 8Kb of uncompressed content

• Investigate more proactive flushing

44


Web Performance Measurement

• Continuous monitoring of full page load performance

45


46

Can you spot our release dates?

shopzilla.com 10/13/08

bizrate.com 10/17/08


47

Yeah, yeah… but did we make money?


48

Site conversion rates increased 7-12%

Revenue = sessions x conversion % x CPC


49

Performance Impacts Abandonment


50

Performance Penalties & bizrate.co.uk

~120% SEMSessions


51

Performance Summary

• Conversion Rate +7% - 12%

• Page View’s +25%

• US SEM Sessions +8%

• Bizrate.co.uk SEM Sessions +120%

• Infrastructure Required (US) -50% (200 vs 402 nodes)

• Availability 99.71% 99.94%

• Product Velocity +225%

• Release Cost $1,000’s $80


5252

Is Performance Worth The Expense?

YES!


5353

Simplicity, quality, performance are design decisions


5454

Questions?


Thank You!

Blog: tech.shopzilla.com

Email: [email protected]

Jobs: jobs.shopzilla.com


References

• http://developer.yahoo.com/performance/

• http://www.oracle.com/technology/products/coherence/index.html

• http://www.evidentsoftware.com/products/clearstone.aspx

• http://www.yourkit.com/

• http://spriteme.org/

• http://www.keynote.com/

• http://jakarta.apache.org/jmeter/

• http://cxf.apache.org/

• http://graphite.wikidot.com/

56

shopzilla - performance by design

Technology