tech vision: next-generation performance testing with blazemeter, service virtualization and test...

World®’16

Next-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagementSudhrityMondal- Sr.PrincipalConsultant- CATechnologies

DO5X56S

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

ApplicationPerformancehasalwaysbeenanimportantcomponentofapplicationquality,butthegrowthinonlinechannelsofinteractionmeansthatformanyorganizations,performanceisnowcritical.Poorperformancecanimpactcustomerexperience,revenues,profitability,brand,andinsomecases,theviabilityoforganizations.

Currentapproachesforensuringtheperformanceofanapplicationpriortoimplementationintoproductioncanbeantiquated,expensiveandtimeconsuming.ThissessionoutlinesnextgenerationperformancetestapproachthatcanbeadoptedbyorganizationsusingCABlazeMeter,ServiceVirtualizationandTestDataManagementsolutions.Adoptingthisapproachcanofferawiderangeofbenefitsincludingefficiencygainsinapplicationdelivery,costsavings,improvedagility,betterperformingapplicationsandimprovecustomerexperience.

SudhrityMondalCATechnologiesSr.PrincipalConsultant


Agenda

APPLICATIONPERFORMANCETEST

THEHURDLESTOPERFORMANCETESTING

NEXT-GENPERFORMANCETESTING

Q&A

1

2

3

4


YourAppsAretheFACE ofYourBusiness

DoestheirperformancedriveLOYALTY orFRUSTRATION?



WhyPerformanceMeansEverythingOverview

68% 25%

Ofcustomerinstallbaseatriskofbeinglostifbusinessesfailtodeliverapositive

applicationexperience*

Ofuserswilldemand anapploadtimeof6secondsorless*

*Software:TheNewBattleGround forBrandLoyalty,2015,Zogby Analytics


WhyPerformanceTest

§ Identifiesproblemsearlyonbeforetheybecomecostlytoresolve§ Producesbetterquality,morescalablecode§ PreventsrevenueandcredibilitylossduetopoorWebsite

performance§ Capacityplanningforfutureexpansionorevents§ Toensurethatthesystemmeetsperformanceexpectationsunder

givenlevelsofload§ Exposebugs thatdonotsurfaceincursorytesting


ApplicationPerformanceTest

§ Processofdeterminingthespeedoreffectivenessofanapplicationandtheresourcesusedtoruntheapplication.

§ Evaluatestheresponsetime,throughput andutilizationofsystemtoexecuteitsrequiredfunctions– incomparisonwithdifferentversionsofthesameapplication,or– competitiveproduct


Metrics/KPI

• CustomerExperience• Applicationresponsiveness• Serverutilization• Network

TypesofTest

• StressTesting• LoadTesting• PerformanceTesting• SoakTesting• BenchmarkTesting• BaselineTesting

Factors

• Throughput• ResponseTime• ArrivalRate• Benchmarking• ErrorRate/Availability

ApplicationPerformanceTest


PerformanceTest– ApplicationScenarios

§ Whataspectofapplicationgoingtotest?– Breakdownintoseparatescenariosortests?– Actionstobeperformed?

§ Example– Bookaflight:– Logintoapplication– Searchairlineflights– Bookflight– Payforflightwithcreditcard– Logoutofapplication

§ Scenariomix

Script %ofusers

Total

Search&Bookflight 20 200

Search forflightswithoutbooking 60 600

Changeanexistingreservation 10 100

TicketAgentrunningareport 10 100

Total 100 1000


PerformanceTest– ResultsSummary

§ Top5SlowestPages– Slowestpagesencounteredduringloadtest

§ Top5SlowestTransactions– Donotmeettheperformancecriteriadefinedattestinception

§ Top5SlowestQueryStatements– SlowestQuerywhichistypically80%ofperformanceproblems


PerformanceTest– Coreactivities

§ Plananddesigntestsbasedonacceptancecriteria§ Prepareand/orRecordtestplan(s)§ Identifyandpreparetestenvironment(s)(prod/pre-prod)§ ProvisionTestData§ Executetests§ Analyzeresults§ Performtuning/Changeconfiguration,retest


PerformanceTest– Environments

§ Production§ Pre-Production/Staging§ ComponentlevelperformancetestsduringCI/CDinSDLC§ Application/EnvironmentDependenciesandConstraints

DataConstraints

SystemConstraints

CostConstraints Scalability


PerformanceTestData

§ Testdataandwhereisitused§ Creationoftestdata§ Unique/non-uniquedata§ Applicationownerrole§ TestDataprivileges§ PIIdatamasking§ Parameterizationofvalues

TestScenario

Id LoginId Name Password Group Pay-Period

Approver 1 45678329 Smith,John

123456 8532 03/01/06-03/31/06

Approver 2 74563422 Hansen,Joe

123456 2345 03/01/06-03/31/06

Approver 3 74569422 Strong,Rodney

123456 5345 03/01/06-03/31/06

DataSecurity

TestDataCoverage Automation Data

Volumes


Limitedtestresourcesand

lackofAutomation

ofalltestingisstillmanual70%

Source:BloorResearch,2014

TestDataforsetup&staging

oftimespentlookingfor50%

Source:BloorResearch,2014

TheHurdlestoPerformanceTesting

Buildofvariousdev&testing

Environments

ofcriticaldependenciesareunavailable whendevandtestteamsneedthem

56%

Source:vokeMarketSnapshotTM Report:ServiceVirtualization– January2015

PerformanceTestingStilltheBottleneck



AgileAlignedPerformanceTestingSupportedwithAdvancedTechnologyNext-GenPerformanceTesting

TESTENVIRONMENTSIMULATION

Simulatemissingsystems,APIstosaveupto640Hrs

PERFORMANCETESTAUTOMATION

Shift-Left,IntegratewithCI100%TestCoverage

TESTDATAMANAGEMENT

Reducetimespentwaitingondataby95%

ORCHESTRATEDPERFORMANCETESTINGStreamlinedset-upofcompleteandfullyautomatedperformancetestexecution


DEMOCRATIZINGPERFORMANCETESTINGINTHECONTINUOUSDELIVERYERA

CABlazeMeterisanext-generation,SaaS andOn-Premise,continuousapplicationperformancetestingsolutiondesignedtohelpDevOpsandAgileteamsbeginweb,mobile,microservices,and

APItestingearlyintheSDLCandinproduction.

PureSaaS,NoSetup

100%Open-Source

Compatibility(“LoadandGo”)

PublicCloud,VirtualPrivateCloudand/orBehindTheFirewall

MassivelyScalable

– 2Million+virtualusers

PerformanceTest

OrchestrationusingTaurus



WhyWorryAboutTestData?

Compliance/security

Customersatisfaction/quality

Productivity/agility/efficiency

Skyrocketingcosts

ServerErrorin‘/contractor’Application.

RuntimeError


TestDataManagementApproaches

COPYPRODUCTIONDATA SUBSET MASK

Formostcompanies,thisisTDM:§ Slowanderrorprone§ 20%–30%coverage§ Nonegativeorfuturetestscenarios§ Nodatafornewdevelopment

Dataismaskedandinsertedintodifferentdatastoresfordev/test

§ Refreshtimingvariesduetoworkrequired§ Datanotavailableinparallelorondemand§ Timewastedwaitingfororlookingfordata§ Highinfrastructurecosts


Substitution Variables

Combinable Functions

CATestDataManager

Data Model

Generation

Bulking Scripts

ProductionData/Files

TestData

Warehouse

PerfTestEnvironments

1 2

4 5

SecureDataSubsets

XMLFiles

XLS

SQLFiles

CSVFiles

APIHTMLFiles

FD

TXTFiles

NoSQL

3

6

TDMApproaches– SyntheticTestDataGeneration


DataBulking PerformanceDataDelivered

Format Bulking

Requirements

Functions

Structure

1millionrecords

DataGeneration

Generateasmallernumberofrecordsfirst(example:10,000)

UseBulking/Multiplierfunctiontomultiplytherecordstoamuchlargernumber

(example:1,000,000)

SyntheticDataGenerationforPerformanceTesting

CATestDataManager


TheHurdletoPerformance?Constraints/BottlenecksforPerformanceTesting

IncompleteDevelopmen

t

ofteamsexperiencedelays indevelopmentandQAduetounavailabledependencies

56%

Over

80%

79%

Source: voke Market Snapshot TM Report: Service Virtualization – January 2015

ofteamsfaceprohibitiverestrictions,timelimitsoraccessfeesonneeded3rdpartyservices

ofcriticaldependenciesareunavailablewhendevandtestteamsneedthem

DataConstraints

SystemConstraints

CostConstraints

ScalabilityConstraints

http://bit.ly/SV-ROI

23 ©2016CA.ALLRIGHTSRESERVED.

Un-ConstrainedPerformanceTesting

ESB

Mainframe SystemofRecord ERP SaaS/API

Availability Sync Capacity Cost

CAServiceVirtualization

OrderManagement

CABlazeMeter

CATestDataManager

ServiceVirtualization

theWindtunnelofSoftware

BlazeMeterRunhundredsofLoadTestsinParallelupto2millionVirtualUsers

from25+Geolocations

TestDataManager

Createhigh-quality,compliant,‘fitforpurpose’testdata

CABlazeMeter• DrivePerformanceTestOn-

PremiseorintheCloudandorchestrate

CATestDataManagemer• GeneratePerformanceTestData

CAServiceVirtualization• SimulateMissingEnvironments


OneChallenge:DataSynchronization

Application

iSeries/AS400 z/OSDB2,IMS,VSAM

Oracle,MSSQL,DB2Informix,Teradata,..

FlatFiles–CSV,Excel,XML,..

TestExecution

ProductionData

SYSTEMUNDERTEST

SYSTEMSOFRECORDWhataboutthedata

inyourVirtualServices?

CATestDataManagerSubsets/Masks/Generates

TestData

CASV

CAServiceVirtualizationsimulatesmissingSystems/Interfaces


DataSynchronizationbetweenVirtualandRealsystems

Application




TestExecution

ProductionData

SYSTEMUNDERTEST

TESTFAILURE+ =DATANOTSYNCED

CATDMInterface

SYSTEMSOFRECORD IfTestDataisn’taligned,testcaseswillFAILbecauseofdatasyncissues

CASV


TheSolution

Application




TestExecution

ProductionData

SYSTEMUNDERTEST

TESTSUCCESS!+ =DATASYNCED

SYSTEMSOFRECORDCATDMIntegrationbridgestheData

alignmentgapandallowsforcreation/updateofvirtualserviceswith

synchronizeddata

CASV

CATDMInterface


SolutionBenefits

BETTERBUSINESSOUTCOMES

READYFORSEASONALPEAKS

INCREASESECURITY

IMPROVEDQUALITY REDUCEDCOSTS

§ Applicationsperformasexpected

§ ImprovedCustomerExperience

§ PreventRevenueLoss

§ Applicationstestedforpeakloads

§ Identify&resolvebottlenecksbeforecriticalbusinessevents

§ CapacityPlanning

§ Matchtherightdatatotherighttest

§ ContinuousPerformanceTestingaspartofCI/CD

§ Achieve100%performancetestcoverage

§ Eliminatetheriskofinternal/externalbreachesandfines

§ EnsurePIIisprotected

§ Simulatesystemsthatareoutsidescopewithrealisticlatency

§ EliminateWaittimesforenvironmentavailability


Questions?


MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P


Thankyou.

Stayconnectedatcommunities.ca.com