World®’16
Next-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagementSudhrityMondal- Sr.PrincipalConsultant- CATechnologies
DO5X56S
DEVOPS
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
ApplicationPerformancehasalwaysbeenanimportantcomponentofapplicationquality,butthegrowthinonlinechannelsofinteractionmeansthatformanyorganizations,performanceisnowcritical.Poorperformancecanimpactcustomerexperience,revenues,profitability,brand,andinsomecases,theviabilityoforganizations.
Currentapproachesforensuringtheperformanceofanapplicationpriortoimplementationintoproductioncanbeantiquated,expensiveandtimeconsuming.ThissessionoutlinesnextgenerationperformancetestapproachthatcanbeadoptedbyorganizationsusingCABlazeMeter,ServiceVirtualizationandTestDataManagementsolutions.Adoptingthisapproachcanofferawiderangeofbenefitsincludingefficiencygainsinapplicationdelivery,costsavings,improvedagility,betterperformingapplicationsandimprovecustomerexperience.
SudhrityMondalCATechnologiesSr.PrincipalConsultant
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Agenda
APPLICATIONPERFORMANCETEST
THEHURDLESTOPERFORMANCETESTING
NEXT-GENPERFORMANCETESTING
Q&A
1
2
3
4
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
YourAppsAretheFACE ofYourBusiness
DoestheirperformancedriveLOYALTY orFRUSTRATION?
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
WhyPerformanceMeansEverythingOverview
68% 25%
Ofcustomerinstallbaseatriskofbeinglostifbusinessesfailtodeliverapositive
applicationexperience*
Ofuserswilldemand anapploadtimeof6secondsorless*
*Software:TheNewBattleGround forBrandLoyalty,2015,Zogby Analytics
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
WhyPerformanceTest
§ Identifiesproblemsearlyonbeforetheybecomecostlytoresolve§ Producesbetterquality,morescalablecode§ PreventsrevenueandcredibilitylossduetopoorWebsite
performance§ Capacityplanningforfutureexpansionorevents§ Toensurethatthesystemmeetsperformanceexpectationsunder
givenlevelsofload§ Exposebugs thatdonotsurfaceincursorytesting
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
ApplicationPerformanceTest
§ Processofdeterminingthespeedoreffectivenessofanapplicationandtheresourcesusedtoruntheapplication.
§ Evaluatestheresponsetime,throughput andutilizationofsystemtoexecuteitsrequiredfunctions– incomparisonwithdifferentversionsofthesameapplication,or– competitiveproduct
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Metrics/KPI
• CustomerExperience• Applicationresponsiveness• Serverutilization• Network
TypesofTest
• StressTesting• LoadTesting• PerformanceTesting• SoakTesting• BenchmarkTesting• BaselineTesting
Factors
• Throughput• ResponseTime• ArrivalRate• Benchmarking• ErrorRate/Availability
ApplicationPerformanceTest
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
PerformanceTest– ApplicationScenarios
§ Whataspectofapplicationgoingtotest?– Breakdownintoseparatescenariosortests?– Actionstobeperformed?
§ Example– Bookaflight:– Logintoapplication– Searchairlineflights– Bookflight– Payforflightwithcreditcard– Logoutofapplication
§ Scenariomix
Script %ofusers
Total
Search&Bookflight 20 200
Search forflightswithoutbooking 60 600
Changeanexistingreservation 10 100
TicketAgentrunningareport 10 100
Total 100 1000
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
PerformanceTest– ResultsSummary
§ Top5SlowestPages– Slowestpagesencounteredduringloadtest
§ Top5SlowestTransactions– Donotmeettheperformancecriteriadefinedattestinception
§ Top5SlowestQueryStatements– SlowestQuerywhichistypically80%ofperformanceproblems
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
PerformanceTest– Coreactivities
§ Plananddesigntestsbasedonacceptancecriteria§ Prepareand/orRecordtestplan(s)§ Identifyandpreparetestenvironment(s)(prod/pre-prod)§ ProvisionTestData§ Executetests§ Analyzeresults§ Performtuning/Changeconfiguration,retest
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
PerformanceTest– Environments
§ Production§ Pre-Production/Staging§ ComponentlevelperformancetestsduringCI/CDinSDLC§ Application/EnvironmentDependenciesandConstraints
DataConstraints
SystemConstraints
CostConstraints Scalability
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
PerformanceTestData
§ Testdataandwhereisitused§ Creationoftestdata§ Unique/non-uniquedata§ Applicationownerrole§ TestDataprivileges§ PIIdatamasking§ Parameterizationofvalues
TestScenario
Id LoginId Name Password Group Pay-Period
Approver 1 45678329 Smith,John
123456 8532 03/01/06-03/31/06
Approver 2 74563422 Hansen,Joe
123456 2345 03/01/06-03/31/06
Approver 3 74569422 Strong,Rodney
123456 5345 03/01/06-03/31/06
DataSecurity
TestDataCoverage Automation Data
Volumes
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Limitedtestresourcesand
lackofAutomation
ofalltestingisstillmanual70%
Source:BloorResearch,2014
TestDataforsetup&staging
oftimespentlookingfor50%
Source:BloorResearch,2014
TheHurdlestoPerformanceTesting
Buildofvariousdev&testing
Environments
ofcriticaldependenciesareunavailable whendevandtestteamsneedthem
56%
Source:vokeMarketSnapshotTM Report:ServiceVirtualization– January2015
PerformanceTestingStilltheBottleneck
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AgileAlignedPerformanceTestingSupportedwithAdvancedTechnologyNext-GenPerformanceTesting
TESTENVIRONMENTSIMULATION
Simulatemissingsystems,APIstosaveupto640Hrs
PERFORMANCETESTAUTOMATION
Shift-Left,IntegratewithCI100%TestCoverage
TESTDATAMANAGEMENT
Reducetimespentwaitingondataby95%
ORCHESTRATEDPERFORMANCETESTINGStreamlinedset-upofcompleteandfullyautomatedperformancetestexecution
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DEMOCRATIZINGPERFORMANCETESTINGINTHECONTINUOUSDELIVERYERA
CABlazeMeterisanext-generation,SaaS andOn-Premise,continuousapplicationperformancetestingsolutiondesignedtohelpDevOpsandAgileteamsbeginweb,mobile,microservices,and
APItestingearlyintheSDLCandinproduction.
PureSaaS,NoSetup
100%Open-Source
Compatibility(“LoadandGo”)
PublicCloud,VirtualPrivateCloudand/orBehindTheFirewall
MassivelyScalable
– 2Million+virtualusers
PerformanceTest
OrchestrationusingTaurus
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
WhyWorryAboutTestData?
Compliance/security
Customersatisfaction/quality
Productivity/agility/efficiency
Skyrocketingcosts
ServerErrorin‘/contractor’Application.
RuntimeError
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TestDataManagementApproaches
COPYPRODUCTIONDATA SUBSET MASK
Formostcompanies,thisisTDM:§ Slowanderrorprone§ 20%–30%coverage§ Nonegativeorfuturetestscenarios§ Nodatafornewdevelopment
Dataismaskedandinsertedintodifferentdatastoresfordev/test
§ Refreshtimingvariesduetoworkrequired§ Datanotavailableinparallelorondemand§ Timewastedwaitingfororlookingfordata§ Highinfrastructurecosts
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Substitution Variables
Combinable Functions
CATestDataManager
Data Model
Generation
Bulking Scripts
ProductionData/Files
TestData
Warehouse
PerfTestEnvironments
1 2
4 5
SecureDataSubsets
XMLFiles
XLS
SQLFiles
CSVFiles
APIHTMLFiles
FD
TXTFiles
NoSQL
3
6
TDMApproaches– SyntheticTestDataGeneration
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DataBulking PerformanceDataDelivered
Format Bulking
Requirements
Functions
Structure
1millionrecords
DataGeneration
Generateasmallernumberofrecordsfirst(example:10,000)
UseBulking/Multiplierfunctiontomultiplytherecordstoamuchlargernumber
(example:1,000,000)
SyntheticDataGenerationforPerformanceTesting
CATestDataManager
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TheHurdletoPerformance?Constraints/BottlenecksforPerformanceTesting
IncompleteDevelopmen
t
ofteamsexperiencedelays indevelopmentandQAduetounavailabledependencies
56%
Over
80%
79%
Source: voke Market Snapshot TM Report: Service Virtualization – January 2015
ofteamsfaceprohibitiverestrictions,timelimitsoraccessfeesonneeded3rdpartyservices
ofcriticaldependenciesareunavailablewhendevandtestteamsneedthem
DataConstraints
SystemConstraints
CostConstraints
ScalabilityConstraints
http://bit.ly/SV-ROI
23 ©2016CA.ALLRIGHTSRESERVED.
Un-ConstrainedPerformanceTesting
ESB
Mainframe SystemofRecord ERP SaaS/API
Availability Sync Capacity Cost
CAServiceVirtualization
OrderManagement
CABlazeMeter
CATestDataManager
ServiceVirtualization
theWindtunnelofSoftware
BlazeMeterRunhundredsofLoadTestsinParallelupto2millionVirtualUsers
from25+Geolocations
TestDataManager
Createhigh-quality,compliant,‘fitforpurpose’testdata
CABlazeMeter• DrivePerformanceTestOn-
PremiseorintheCloudandorchestrate
CATestDataManagemer• GeneratePerformanceTestData
CAServiceVirtualization• SimulateMissingEnvironments
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
OneChallenge:DataSynchronization
Application
iSeries/AS400 z/OSDB2,IMS,VSAM
Oracle,MSSQL,DB2Informix,Teradata,..
FlatFiles–CSV,Excel,XML,..
TestExecution
ProductionData
SYSTEMUNDERTEST
SYSTEMSOFRECORDWhataboutthedata
inyourVirtualServices?
CATestDataManagerSubsets/Masks/Generates
TestData
CASV
CAServiceVirtualizationsimulatesmissingSystems/Interfaces
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DataSynchronizationbetweenVirtualandRealsystems
Application
iSeries/AS400 z/OSDB2,IMS,VSAM
Oracle,MSSQL,DB2Informix,Teradata,..
FlatFiles–CSV,Excel,XML,..
TestExecution
ProductionData
SYSTEMUNDERTEST
TESTFAILURE+ =DATANOTSYNCED
CATDMInterface
SYSTEMSOFRECORD IfTestDataisn’taligned,testcaseswillFAILbecauseofdatasyncissues
CASV
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TheSolution
Application
iSeries/AS400 z/OSDB2,IMS,VSAM
Oracle,MSSQL,DB2Informix,Teradata,..
FlatFiles–CSV,Excel,XML,..
TestExecution
ProductionData
SYSTEMUNDERTEST
TESTSUCCESS!+ =DATASYNCED
SYSTEMSOFRECORDCATDMIntegrationbridgestheData
alignmentgapandallowsforcreation/updateofvirtualserviceswith
synchronizeddata
CASV
CATDMInterface
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
SolutionBenefits
BETTERBUSINESSOUTCOMES
READYFORSEASONALPEAKS
INCREASESECURITY
IMPROVEDQUALITY REDUCEDCOSTS
§ Applicationsperformasexpected
§ ImprovedCustomerExperience
§ PreventRevenueLoss
§ Applicationstestedforpeakloads
§ Identify&resolvebottlenecksbeforecriticalbusinessevents
§ CapacityPlanning
§ Matchtherightdatatotherighttest
§ ContinuousPerformanceTestingaspartofCI/CD
§ Achieve100%performancetestcoverage
§ Eliminatetheriskofinternal/externalbreachesandfines
§ EnsurePIIisprotected
§ Simulatesystemsthatareoutsidescopewithrealisticlatency
§ EliminateWaittimesforenvironmentavailability
29 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
MustSeeDemos
ModernizeAppDeliveryIntegratedCDTheater5- DOV501P
DeliverTestDataFasterTestDataManagerTheater5- DOV511P
DeliverBetterAppsServiceVirtualizationTheater5- DOV507P
OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P