rackspace whitepaper: testing magento performance… · rackspac t y 7 ww.rackspace.co.uk 2...

RACKSPACE WHITEPAPER:

TESTING MAGENTO PERFORMANCE

Rackspace Ltd. 5 Millington Road | Hyde Park Hayes | Middlesex | UB3 4AZPHONE: 0800-988-0100 | INTL: +44 (0)20 8734 2600 | www.rackspace.co.uk

https://twitter.com/rackspaceUK

https://www.facebook.com/RackspaceUK

https://plus.google.com/+RackspaceHosting/posts

https://www.linkedin.com/company/5234?trk=tyah&trkInfo=clickedVertical%3Acompany%2Cidx%3A3-1-7%2CtarId%3A1435327015728%2Ctas%3Arackspa

http://www.rackspace.co.uk


2 RACKSPACE WHITEPAPER:TESTING MAGENTO PERFORMANCE

ContentsAbstract 3

Introduction 4

Testing Methodology 4

Introduction 4

UserJourneys 5

TheOptimisedApplicationStack 6

TestingInfrastructure 6

Presentation of Results 7

Test1:InfrastructurePerformance 7

Test2:CustomerExperience 10

Conclusions 13

Appendices 15

Appendix1:FullResultsTables 16

Appendix2:AboutRackspace 17

Appendix3:AboutNeoLoadfromNeotys 17

Appendix4:AboutIllamasqua 17

Appendix5:AboutMagento 18

Appendix6:AbouttheAuthor 18








ABSTRACT

IfyousearchtheInternetforstatisticsaboutwebsitepageloadtimesandtherelationshipwithconversion,youwillfindplentyofrelevantexamples(e.g.Walmart,AmazonandFirefox).Theclearmessageisthatoptimisingperformancewilldirectlyimprovecustomerexperience,whichcanleadtomoresales.Onethingthatisoftenoverlookedbyonlineretailersisperformancemeasurement.Carrying out regular testing not only enables a benchmark of user experience, it also enablestheidentificationandremediationofoperationalbottlenecks.Understandinghowasiteperformsunderloadwillsupplytransactionmetricswhichhelppredictcapacityandensurethatmaximumreturncanbeachievedfromtargetedmarketingactivity.

Magento is the fastest growing e-commerce application for production websites in use today(InternetRetailertop500http://www.internetretailer.com/top500/).Thousandsofretailerstrusttheplatformworldwidebuteffectivedeploymentontoahostinginfrastructurecanbeachallenge.Thiswhitepaperpresentsthebenefitsofoptimisingtheapplicationstacktoincreaseperformance.

Byanalysingdifferentapplicationconfigurationsitispossibletovalidatethebenefitsofatuneddeployment.Thisdocumentwillalsodemonstrateaclearmethodologyforperformancetestingandmeasuringtheresults.








INTRODUCTION

In order to demonstrate that optimisations deliver asignificant performance improvement for Magento weput together a comprehensive testing strategy. TherearemanytoolsavailablebutweuseNeoload.Itisanall-embracing performance tool and highly configurable,enablingdetailedanalysis.

READ MORE ABOUT NEOLOAD IN THE APPENDIX.

Theprocessofbuildinguser journeys,whichaccuratelymimickedhuman browsing, took a significant effort. This reflected thecompletionofpurchasesfromanecommercewebsite,andprovidedmorerealisticresults.Manytestsdonotdothistoasufficientlevel,meaning the outcomes can be unreliable. If you just siege test awebsitethenyouaren’tabletotestthedifferencebetweenbrowseandtransactionalcustomers.

Thefirstplacetostartwastobuildanumberofjourneyswhicheachvirtualuserwouldbeassociatedwith.Twogroups reflecteduserswhowerejustbrowsingandtwowerefortransactionalusers.

In order to test both infrastructure performance and realisticshopping simulations, two tests were performed. The first used adefault Magento installation, effectively a blank store with onlycore features. The second used configuration and data from areal customer store www.illamasqua.com. This included functionalplugins,aproductcatalogueandregisteredusergroup.

READ MORE ABOUT OUR CUSTOMER ILLAMASQUA IN THE APPENDIX.

Wewantedtoreflectawebsiteunderpeaktradingconditionssoaconversionpercentage(browsetobuyratio)ofover5%wasused.Theaverageitemsineachpurchasewasrandomisedfrom1–4.Thedetailsfornumbersareoutlinedineachtest.

Testing Methodology








User JourneysUSER

JOURNEY 1: FILTER

VISITORSVirtualuserswhodidnotlogintothestore

usinganaccount.Filterswereusedtofindaproduct,theproductpagewas

visitedbeforeleavingthesite.

USER JOURNEY 2:

SEARCH VISITORS

Virtualuserswhodidnotlogintothestoreusinganaccount.The

searchfeaturewasusedtofindaproduct,theproductpagewasvisitedbeforeleaving

thesite.

USER JOURNEY 3:

BUYERS

Virtualuserswhosearchedthesitefor

aproduct,addedittotheshopping

basket,thenaddedupto3moreproducts,thenproceededtothecheckout.Halfcompletedtheir

purchaseandtheremainderabandoned

thetransaction.

USER JOURNEY 4:

WISHLIST SHOPPERS

Virtualuserswhosearchedthesiteforarandomproduct,

addedittoawishlist,thenaddedupto3

moreproductsbeforeleavingthesite.

(Homepage > random category > random

sub-category > random product >

end)

(Homepage > random search (from 15

popular) > select a random product >

end)

(Homepage > login > random category >

random product > add to basket > review

basket > checkout or abandon > end)

(Homepage > login > random category > random product

> add up to 4 items to wishlist > review

wishlist > remove an item > end)








The application infrastructure deployed to run Magento was splitinto“standard”and“optimised”builds(orstacks)andwasmadeupofthefollowingconfigurations:

STANDARD:Using the default application versions and settings as provided byCentOS6.

Apache,PHP5.3(withmod_php),Magento1.14(includingFPC)withFileCaching.MySQL5.1.

OPTIMISED:Apache(worker),PHP5.4(FPM),Opcache,Magento1.14(includingFPC), Redis object and session caching, Percona 5.6 (with tablesconvertedtoInnoDB).

In addition to deploying a more capable set of applicationsthe optimisation skill comes in carrying out tuning to meet theinfrastructure.On theweb tier this involvedconfiguringApache tohaveprocessesrunningandpreparedformaxusersandtheavailableserverresources.Thiseffectivelysetsupthesolutionona“readyfortraffic”mannerandreducestimespenttobeginaprocess.Theuseofcachingservicesenablesobjectstobestoredinmemoryandcanberead immediatelyasopposedtothewaitexperiencedwithreadingfromadiskorstoragedevice.

TESTING INFRASTRUCTUREThesamesetofhardwarewasusedforeachtesttohelpensurethatthere was no variation in performance due to server hardware ornetworkingdevices.TorepresentahighlyavailableEnterpriseEditionapplicationwedeployedthefollowingphysicalserverplatform:

Perimeter Firewall: CiscoASA5515x

Load Balancer: F5BigIP1600sLTM

Web Server: Dellr720(dual8coreintelprocessor2.7GHz,64GBRAM,2x300GBSASHDD(RAID1))

Database Server: Dellr720(dual6coreintelprocessor2.5GHz,64GBRAM,2x300GBSASHDD(RAID1),HBAcard)

SAN Deployment: 100GB “gold level performance shared SAN” presented forMySQL database storage. 250GB “silver level performance

sharedSAN”presentedforNFSshare.

The Optimised Application Stack








TEST 1: INFRASTRUCTURE PERFORMANCETo test theeffectofperformanceon infrastructure, two iterationswererunagainstadefaultMagentoEnterpriseinstallation.ThefirstwasastandardLinuxstackdeployment(apache,PHPandMySQL),thesecondconsistedofaRackspaceoptimisedconfiguration.Thefollowingpopulationof500virtualuserswasdeployed:

TOTAL100%

QTY500

FIG1.0PopulationofvirtualuseresdeployedinTest1.

Filtervisitors 38.0% 190

Searchvisitors 25.4% 127

Buyers 6.4% 32

Wishlistshoppers 30.2% 151

TOTAL 100% 500

Thetestswererunoverafixedtimeperiodof6minutes,simulatingathroughputof32transactions.

RESOURCE USAGE: WEB SERVER

Comparingtheuseofresourcesallowsustomeasurethestrainontheservers.Inbothcasesalowervalueisbetter.Aseriousimpacton performance leading to considerable delay and even failure todisplaypageswilltakeplacewhen100%utilisationoccurs.

74

CPU % Memory %

32

47

20

standardoptimised

Presentation of Results








RESOURCE USAGE: DATABASE SERVER

Resources on the database server come under pressure at hightransactionlevel. Intermsofadatabasetransactionthisrelatestoactivityofloggedinuserswhomightbecheckingorderhistorytocompletingapurchase.

10

CPU % Memory %

3

40

53

standardoptimised

AVERAGE PAGE RESPONSE TIME – 92.2% DECREASE

4.309

Average Page Response Time (s)

0.335

standardoptimised

Pageresponsetimeisameasureofthetimetakenforallelementsonthepagetobegathered.Thisdoesnottakeintoaccountthetime

92.2%decreaseinpageresponse

time.

Lowerpageresponsetimemeansaquickerloadtime– thisisvitaltoimprovecustomerexperienceandsiteconversions.

Lowerpageresponsetimemeansaquickerloadtime– thisisvitaltoimprovecustomerexperienceandsiteconversions.








37.4%increaseinpagesservedper

second.

abrowsertakestorenderthepage.Forthismetricalowervalueisbetter.Theaveragevalueisacrossallthepageswhichwereserved.

PAGES SERVED PER SECOND – 37.4% INCREASE

Pagesservedpersecondisameasureoffullpagedelivery.Forthismetricahighervalueisbetter.Ontwoidenticalserverinfrastructuresonecanseethetruevalueofapplicationperformance.

19.8

Pages per Second

27.2

standardoptimised

Theimpactofvirtualuserloadontheserverresourceswassignificant.Theoptimiseddeploymentnotonlyperformedbetter,but itdiditwithalowerimpactontheservers.ThewebserversusedlessthanhalftheresourcesavailableforbothCPUandmemory,toserve500userscompleting32transactions.Interestinglythepeakmemoryuseontheoptimiseddatabaseserverwashigherduetotheincreasedeffectiveness of the web nodes and the optimised configurationmakingbetteruseoftheresourcesavailable.

Based on this test, using an optimised application infrastructuredeployment would deliver a significantly improved efficiency ofresources. This would allow you to host a solution using a lowerspecificationofserverandstilldeliverbetterperformance.

Higherpagespersecondmeansyoursitecanhandlepeaksoftrafficmoreefficiently.








TEST 2: CUSTOMER EXPERIENCEBytestingperformanceagainstcodefromanactualMagentostore,weareabletogaugetheimpactonacustomerinamorerealisticfashion. Again we used a standard Linux stack and a Rackspaceoptimisedconfiguration,butthistimewehadcodeanddatafromourcustomer,Illamasqua.

TOTAL100%

QTY1000

FIG2.0PopulationofvirtualuseresdeployedinTest2.

Filtervisitors 36.0% 360

Searchvisitors 24.0% 240

Buyers 12.0% 120

Wishlistshoppers 28.0% 280

TOTAL 100% 1000

This test was used to measure the speed of transactions. 60transactionswereprocessedandthetimetocompletethisfor1000virtualuserswasmeasured.Inthiscase,ratherthanusingafixedtimeperiod,as soonas theuserswereprocessedand the transactionswerefinishedthetestwascompleted.

AVERAGE PAGES PER SECOND – 122% INCREASE

47

Pages per Second

104.5standardoptimised

122%increaseinaveragepages

persecond.

Anoptimisedsolutionisabletoservemorepagestocustomers,thatmeansaquickersite.








As before there is clear increase in performance based on thequantityofpagesservedbytime.Thesefiguresarehigherthantheprevioustestduetothescheduleofvirtualuserdeployment.

AVERAGE PAGE RESPONSE TIME (S) – 92.9% DECREASE

7.19

Average Page Response Time (s)

0.355

standardoptimised

Bycomparingthesefigureswiththosefromtheprevioustestwecanseetheimpactofadditionalpageelements.Thedifferencewiththeoptimisedsolutionclearlyshowsthebenefitsofthecachingstrategy.

TOTAL TIME TO PROCESS TRANSACTIONS (S) – 48.9% DECREASE

Thismetricmeasuresthetimetocompletethetransactions.Alowervalueisbetter.Fromthisresultwecancalculateanumberofstatisticswhichrelatetocommercialperformanceofthesite.

48.9%decreaseintotaltime

toprocess

transactions.

Gettingmorecustomerstotransactionquicklymeansyoursitewillbemoreeffectiveatpeaktrading.








standard

optimised

11:58

06:07

Time to process all users (m:s)

Again the optimised servers showed a significant performanceincreasewithaverynoticeabledropintimeforpagerequests.Thisresulted in theuser journeysand transactionsbeingcompleted inalmosthalfthetime.

Speedoftransactionsandcontentdeliveryaredirectlyproportionaltocustomerexperienceandtheconversionrateofanecommercesite.Asyoucanseethereisanalmostdoublingofperformance.Whileitisimpossibletoinferthatthesitewouldhavedoublethenumberofpayingcustomers,asignificantincreasecouldbeexpected.

Inthistest,becauseweareusingrealdata,thenumberandvalueof orders processed was recorded. 60 transactions contained anaveragebasketof2.8itemsatavalueof£120.00.Intheoptimisedinstancethiswascompleted in6minutes7seconds.Thisequatestoastoreturnoverof£70,632perhourfortheoptimisedstore.Thestandard deployment took 11 minutes 58 seconds, with the sameitemvaluesequatesto£36,072perhour.

This is an overall increase in commercial productivity of 95.8% ornearlydoublethetotalordervalue(perhour).

95.8%increasein

transactions

perhour

fortheoptimised

store.

589ordersperhour(optimised)vs301ordersperhour(standard)








The results clearly demonstrate an increase in performance when you deploy a Rackspace optimised application infrastructure stack.

Before running the tests these are the results we expected to see,forseveralreasons.Theprocessofoptimisationmakesthebestuseofresourcesavailable,forexamplePHPprocessesareconfiguredtobewaitingtouseavailableCPUcores.Thisstateofreadinessmeansthatthereisnodelayinspawningaprocess.Thisleadstoareductioninresponseandhenceaquickerpageload.Usingthemostefficientresources available is also important, for example caching usingmemorywillproduceamuchbetterlevelofperformancethanwritingtoaphysicaldisk(wherethedataneedstobewrittenandread).

Eachof the rewardsmade frommakinga change to configurationmake very little difference to the overall system performance.Carryingoutalargenumberofsmallincreaseshowever,candeliverameasurableincrease.Thisaggregationofmargingains,leadingtomuchwiderimprovement,waspopularisedbyBritishCyclingCoachSirDavidBrailsford.Itisthismethodologyofaddressingeveryaspectofapplicationdeploymentandtuningwhichdeliverssuchimpressiveresults.

Being able to carry out performance tuning on an infrastructuredeployment in this way takes experience and knowledge. TheRackspaceoptimisationshavebeentriedandtestedwithhundredsof customer websites, across varying server configurations withdifferentinstallationsofplugins.Beingabletoassessthebestplacetostartwhenimplementingsignificantchangesrequiresthoughtandplanning.

Performance testing should be part of any software developmentcyclenotonlydoesthishelpidentifysolutionbottlenecksbutdeliversthecapacitymetricthataplatformcanhandle.Notallcustomersareabletodeployacompletetestingenvironmenttoallowsuchaclearcomparisonwhich iswhy it is so important tobenchmarkasiteatregularintervals,particularlywhenmakingamajorcodechange.Theother advantage with knowing the capacity of the site is that youcanpredictareturnoninvestmentoninfrastructure.Itisveryeasytoworkoutwhenyouneedtodeploymoreresourcesandtheassociatedbenefittheywillbringcanbeclearlymeasuredtomakeabusinesscase.

Conclusion








Thiswhitepaperisforinformationalpurposesonlyandisprovided“ASIS.”RACKSPACEMAKESNOWARRANTIES,EXPRESSORIMPLIED,ASTOTHEACCURACYORCOMPLETENESSOFTHECONTENTS IN THIS WHITEPAPER. All test results and customer examples illustrated here arebaseduponthetestedenvironments,experienceswiththereferencedRackspaceservices,andarenotnecessarily indicativeof the futureperformanceofRackspace services.Actual costofspecifichostedenvironmentandperformancecharacteristicswillvarydependingonindividualcustomerconfigurationsandusecase.

RACKSPACERESERVESTHERIGHTTODISCONTINUEORMAKECHANGESTOITSSERVICESOFFERINGSATANYTIMEWITHOUTNOTICE.EXCEPTASSETFORTHINRACKSPACEGENERALTERMSANDCONDITIONS,CLOUDTERMSOFSERVICEAND/OROTHERAGREEMENTYOUSIGNWITHRACKSPACE,RACKSPACEASSUMESNOLIABILITYWHATSOEVER,ANDDISCLAIMSANYEXPRESSORIMPLIEDWARRANTY,RELATINGTOITSSERVICESINCLUDING,BUTNOTLIMITEDTO,THEIMPLIEDWARRANTYOFMERCHANTABILITY,FITNESSFORAPARTICULARPURPOSE,ANDNONINFRINGEMENT.

ALTHOUGHPARTOFTHEWHITEPAPEREXPLAINSHOWRACKSPACESERVICESMAYWORKWITH THIRD PARTY PRODUCTS, THE INFORMATION CONTAINED IN THE WHITEPAPER ISNOTDESIGNEDTOWORKWITHALLSCENARIOS.ANYUSEORCHANGESTOTHIRDPARTYPRODUCTS AND/OR CONFIGURATIONS SHOULD BE MADE AT THE DISCRETION OF YOURADMINISTRATORS AND SUBJECT TO THE APPLICABLE TERMS AND CONDITIONS OF SUCHTHIRD PARTY. RACKSPACE DOES NOT PROVIDE TECHNICAL SUPPORT FOR THIRD PARTYPRODUCTS,OTHERTHANSPECIFIED INYOURHOSTINGSERVICESOROTHERAGREEMENTYOUHAVEWITHRACKSPACEANDRACKSPACEACCEPTSNORESPONSIBILITYFORTHIRD-PARTYPRODUCTS.

Rackspace and Fanatical Support are either registered service marks or service marks ofRackspace US, Inc. in the United States and other countries. Third-party trademarks andtradenamesappearinginthisdocumentarethepropertyoftheirrespectiveowners.Suchthird-partytrademarkshavebeenprintedincapsorinitialcapsandareusedforreferentialpurposesonly.Wedonotintendouruseordisplayofothercompanies’tradenames,trademarks,orservicemarkstoimplyarelationshipwith,orendorsementorsponsorshipofusby,theseothercompanies.

The use of the word ‘partner’ or ‘channel partner’ does not imply a partnership relationshipbetweenRackspaceandanyothercompany.

©2015RackspaceUS,Inc.Allrightsreserved.








Appendices








TEST 1:

STANDARD OPTIMISED %

Averagepages/s 19.8 27.2 +37.4%

Averagehits/s 464.6 194.7 -58.1%

Totalpages 8632 8604 -0.3%

Totalhits 202566 61518 -69.6%

AverageRequestresponsetime 0.187s 0.056s -70.1%

Totalhiterrors 0 0 +0%

Errorrate 0 0 +0%

AveragePageresponsetime 4.31s 0.335s -92.2%

Totalthroughput 751.88MB 753.27MB +0.2%

Averagethroughput 13.80Mb/s 19.07Mb/s +38.2%

Totaluserslaunched 500 500 +0%

Totaliterationscompleted 500 500 +0%

Totalactionerrors 0 0 +0%

Totaldurationalerts 0% 0% +0%

TEST 2:

STANDARD OPTIMISED %

Averagepages/s 47.0 104.5 +122%

Averagehits/s 633.7 1,219.3 +92.4%

Totalpages 33823 38473 +13.7%

Totalhits 455637 448711 -1.5%

AverageRequestresponsetime 0.548s 0.039s -92.9%

Totalhiterrors 68 31 -54.4%

Errorrate 0 0 +0%

AveragePageresponsetime 7.19s 0.355s -95.1%

Totalthroughput 2640.89MB 1578.93MB -40.2%

Averagethroughput 29.38Mb/s 34.32Mb/s +16.8%

Totaluserslaunched 1000 1000 +0%

Totaliterationscompleted 1000 1000 +0%

Totalactionerrors 0 0 +0%

Totaldurationalerts 38.3% 16.3% -57.4%

APPENDIX 1: FULL RESULTS TABLESThetestingplatformdeliveredahugeamountofresults,presentingandreviewingthemallwasnotpossibleinthisdocument.Thesetablescovermoremetricswerecorded:








Rackspace(NYSE:RAX)isthe#1managedcloudcompany.Itstechnicalexpertise and Fanatical Support® allow companies to tap the powerof the cloud without the pain of hiring experts in dozens of complextechnologies.Rackspaceisalsotheleaderinhybridcloud,givingeachcustomerthebestfitforitsuniqueneeds—whetheronsingle-ormulti-tenant servers, or a combination of those platforms. Rackspace is thefounderofOpenStack®,theopen-sourceoperatingsystemforthecloud.Headquartered in San Antonio, Rackspace serves more than 300,000businesscustomersfromdatacentersonfourcontinents.

NeoLoad (testing software from Neotys) is a load and performancetesting software solutiondesigned forwebandmobileapplications tomore realistically simulate user activity and monitor server behaviour.NeoLoad’s technology allows you to perform load and performancetesting more quickly, efficiently and frequently, which means you canconfidentlydeployyour internet, intranetornativemobileapplicationsnomatterwhicharchitecture theyuse, even thenewest suchasGWT,Silverlight, Flex and Ajax Push. With NeoLoad, the simple tasks areautomated,andthedifficulttasksaresimple.

Neotys’loadandperformancetestingsolutioncanhelpyouprepareyourapplicationtoperformwellunderload.Theyprovideaneffectiveloadtestingsolutionthatcanhelpyoubuildyourwebormobileapplicationtomeetusers’expectationsinproduction.

Thismake-upbrand,thatempowerspeopletoexpresstheirindividuality,hasbeenmakingasceneinLondonsinceitslaunchinNovember2008.Theproductofaheadymixofinfluences;fromitsrootsinthedarkandillicit1920sBerlinclubscene,toarichheritageinthemanufactureofmake-upforfilmandtheatre,Illamasquatakesitsinspirationfrommembersofthe‘alternativescenes’forwhomself-expressionisparamount.Illamasqua’shighly-pigmented, long lastingproductsprovideconsumersandmake-upartistsalikewiththeproductsandinspirationtoexpressthemselves,whoevertheyareandwhateverlooktheywanttoachieve.

APPENDIX 2: ABOUT RACKSPACE

APPENDIX 3: ABOUT NEOLOAD FROM NEOTYS

APPENDIX 4: ABOUT ILLAMASQUA








Magentooffersflexible,scalableecommercesolutionsdesignedtohelptheircustomersgrowandsucceedonline.Thecost-effectivetechnologyplatformmakesitpossibletocontrolthecontent,functionality,andlookandfeelofyouronlinestore.

Theyalsooffersupport, services,andtrainingtohelpyouensureyoursuccess. A comprehensive network of solution partners and certifieddevelopersofferexpertiseandresourcestohelpdesign,build,andhostanonlinestore.Thepartnersalsoofferthousandsofappsandextensionstohelpaddcustomfeaturesandfunctionality.Ultimately,theecommerceplatformprovidesanonlinebusinesswiththeessentialtoolstotransactwithaglobalcustomerbase.

MikeBainbridgeisaRackspacesolutionsarchitectwithaspecialisationindesigninginfrastructureforhighperformance,transactionalwebsites.Withover15yearsintheITindustryandabackgroundintechnicalpre-sales,Mikehasbeenworkingspecificallywithe-commercecustomersforthelast4years.Hehasdesignedsolutionsfromsmall,high-growthstart-upstolargeestablishedenterpriseorganisationswithcomplexdisasterrecovery requirements - covering all market sectors. Understandingtrafficpeaksandhelpingcustomerstogetpeaceofmindbyperformancetesting their environments have been a particular focus in the last 18months.WorkingwithourcustomerstosolvebusinesscriticalproblemsiswhatgetsMikeoutofbedinthemorning.

APPENDIX 5: ABOUT MAGENTO

APPENDIX 6: ABOUT THE AUTHOR






rackspace whitepaper: testing magento performance… · rackspac t y 7 ww.rackspace.co.uk 2...

Documents