decrease test build time, not test quality with ca blazemeter

World®’16

DecreaseTestBuildTimeNOTTestQualityRefaelBotbol- ServicesClientManager- CATechnologies

DO5X53E

DEVOPS

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation

Abstract

Today,yourbuildtimeconsistsofmuchmorethanjustthetimeittakesJenkinstobuildyourapplication.Italsoincludesregressiontests,deploymenttodifferentenvironments,andmaybeotherapplicationspecifictasksyouwanttoautomateaspartofyour“build”.

Yourbuildtimecansignificantlyinfluencehowoftenandwhenyoudecidetorunyourbuilds(i.e.whetheryourunyourbuildsweekly,nightly,twiceaday,orjustpercommit).Allofthishasadirectimpactonyourproductivityandefficiency.

Buthowcanyoudecreaseyourtestbuildtimewithouttriggeringadropinproductquality- oreventhenumberoftestsyourun?TakealookatanewapproachtotestautomationwithintheCIprocess,andlearnhowto:• Increasethequantityandtypeoftestsrunwhilereducingyourbuildtime• IncludeyourperformancetestsaspartofyourCIprocesstoidentifyperformance

degradationissuesearlyintheprocess• Shiftleftallthewaytothedeveloper.SeehowdeveloperscancreatetestswithDSLusing

theirexistingJSON/YAMLknowledge

Thissessionwillstartoffwithaninteractivediscussionbeforeturningintoademonstration/hands-onworkshop.Attendeeswantingtoparticipateinhands-onwillrequiretobringtheirownlaptopstoexecutetheexercises.

RefaelBotbol

CATechnologiesServicesClientManager

Agenda

SAVETIME:BYPARALLELING YOURTESTS

DELEGATE:ENABLEEACH PERSONACONTRIBUTETOTHEEFFORT

FOCUS:CONCENTRATEONBUILDING NOTONMAINTAININGTESTS

SCALE:EXECUTEBUILDSASMANYASYOURBUSINESSREQUIRES

REDUCETHEBUILDTESTINGPHASEWHILEINCREASINGTESTCOVERAGE

DEMONSTRATION/WORKSHOP5

Whichtypeoftestshouldwerun?

§ Functional(API/UI)

§ Regressions

§ CriticalBusinessProcesses

§ Performance

§ Smoketests

OpenSourceToolsyouprobablyuseforTesting

§ Functional(API/UI)– Selenium/JMeter/PhantomJS

§ Regressions – Selenium/JMeter/PhantomJS

§ CriticalBusinessProcesses– Gatling/JMeter/Selenium

§ Performance – Gatling/JMeter/Locust/Tsung

§ Smoketests– Selenium/JMeter/Gatling

Agenda

SAVETIME:BYPARALLELING YOURTESTS

DELEGATE:ENABLEEACHPERSONACONTRIBUTETOTHEEFFORT

FOCUS:CONCENTRATEONBUILDINGNOTONMAINTAININGTESTS

TestingLayers• Monitoring• Smoketests• FullscaleperformancetestsProd

• Criticalbusiness processes• PerformancetestsStaging

• FunctionalUI• Functionalbackend• Regressiontests• Criticalbusiness processes

Integration

Divide,Parallel&Conquer

§ Planyourtestsperlayerandparallelasmuchaspossible

§ Stopthebuildonceadesignatedcriticaltesthasfailedand…– Notifythedeveloperregardingtheissue– Freeresourcesforotherbuilds

§ Buildonce,runmany(evenwhenaddingnewtests)

DSLbasedTestCreationusingYAMLinTaurus

DescribeTestExecutionScenario

SetPass/FailCriteria

SAVETIME:BYPARALLELINGYOURTESTS

DELEGATE:ENABLEEACH PERSONACONTRIBUTETOTHEEFFORT

Whocancontributetothebuild’stests?

§ Coders – ByprovidinglowleveltestsforAPI/UIandregressions

§ Testers – Byprovidingsystemwidetestsandregressions

§ Operations – Byprovidingcriticalkeypointsandcreatingtestswithpass/failcriteriaforthem

Whocancontributetothebuild’stests?

§ Product– Byprovidingpass/failcriteriaforbusinessprocesses

§ Marketers– Bycompletingtheproductviewandprovidingmarketingscenarios

§ Performance – Byimplementingperformancetests

Thisishowwe(can)doit

§ Coders– UsingcodebasedenginesorevenYML/JSONDSL

§ Testers – TheywouldprobablytellyouhowJ

§ Operations– Automatingthescriptsandprovidingpass/failcriteriatomakesurenothingbreaks

TheHow– SurprisingPart

§ Product – Providingpass/failcriteriaforstagingandproductionenvironments(andafterseveralruns,evenfortheintegration..)

§ Marketers – Launchinganewevent?- WhynottestisbyusingaDSLwiththesepages?

§ Performance – Theseguysalreadyhavethescriptsandtheproductionvalues.Theonlythingthatisleft,istoincludeit.

Coders/TestersExamples

§ AsimpleAPIexample

§ Acomplexscenario

§ Java/Pythonseleniumtests

§ JMeter/Gatlingtests

Product/OperationsExamples

§ Createtestsandsetcriteria

§ Setpass/failcriteriaperenvironment

§ Easilyincludeitaspartofyourtestexecution

MarketingExample

AddPerformanceTeststoyourbuild

Summary

§ First,takeadeepbreath…youhavemadeitsofar,youdeserveitJ

§ WecanuseDSLbasedonYML/JSONtocreatetests

§ Wecanchoosetheflavoroftoolswhichbestsuitsus,andreuseourexistingscripts

§ WecanalsousetheDSLtosetpass/failcriteriaanddescribethetestexecution

FOCUS:CONCENTRATEONBUILDING NOTONMAINTAININGTESTS

BuildOnce– UseMany

§ Option1:Separateeverything– Haveyourscriptsrepresentonetestatatime– HaveoneormoreDSLdescribingtheexecutionperbuildper

environment– HaveoneormoreDSLdescribingthepass/failcriteriaperenvironment

Thisgivesyoumaximumagilityoncontrollingyourbuildtests– Addingtestsisfast– Criteriaareappliedtotestsautomaticallyperenvironment

BuildOnce– UseMany

§ Option2:Haveclustersoftests– Groupyourteststogetherbaseontype,functionalityorany

characteristicsthatfitsyou– Attachtherightexecutionandcriteriapergroup

Thisisthebestoptionforspecialprojectssuchas– Anewcampaign– Satelliteprojectswhicharen’tpartofeverybuild– Performancetests

Option1ExecutionExample

Option2ClusteredTestExample

UsetheForce CloudtoScale

§ SometesttypescanconsumeallofyourJenkinsWorker’sresources

§ Sometimes,theamountoftestsissimplytoohighforoneWorker

§ Youshortenedthebuildtimeandnoweveryonewantstobuild…

OneWorkertoTestThemAll

§ TheJenkinsWorkerholdsthefilesandconfigurations

§ TheWorkerstartthetestsusingcloudresourcestoscaleit

§ ResultsaretransmittedtobacktotheWorker

§ BasedontheresultstheWorkerdecides:– Lunchingthenextpackoftests– Stoppingthebuildduetobadresults– Markthebuildassuccessfulandcontinuetothenextstep

UsetheForce CloudtoScale

Seleniumgeo-locationscale

OneSmallThingBeforeWeGo

GettingTaurus

§ Overall,weinstallTaurusbyusingpip.

§ InformationperOScanbefoundhere:http://gettaurus.org/install/Installation/

GetyourTaurusupandrunning

§ Plugyourflashdrive

§ InstallVirtualBox

§ Installvagrant(.dmg file)

§ Openterminalandcreateafolder

§ Gotothatfolderanddovagrantboxaddmy-Taurus-boxfile:///d:/path/to/taurus/box.box

GetyourTaurusupandrunning

§ Onthatfolderdovagrantinit my-Taurus-box

§ Vagrantup

§ Vagrantssh

§ Onceyou’reinsidethevagrant,docdTaurusScript

§ Do:bzt TaurusScenario.yml

TheChallenge

1. RunaJMeterscriptwithTaurus

2. Createyourown(simple)YMLDSLfileandrunitwithTaurus

3. RunaGatlingtestwithTaurus

4. ParallelthemJ

5. RunyourYMLDSLwith3differentexecutors(JMeter,GatlingandSelenium)andseethedifferencebetweenthem

6. Use–reporttoseetheBlazeMeter’sreport

Questions?DO5X53E

RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T03PLeadershipPanel:ContinuousDeliveryintheFinancialServicesIndustry

11/16/2016at04:30pm

DO5T14SAnalystKeynote:ContinuousDelivery:MakingDevOpsAwesome

11/17/2016at10:30am

DO5X56SNext-GenerationPerformanceTestingwithBlazeMeter,ServiceVirtualizationandTestDataManagement

11/17/2016at04:30pm

MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P

DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu

decrease test build time, not test quality with ca blazemeter

Technology

code.org blazemeter case study

make it fast - nicj.net · new relic) profiling...

tech vision: next-generation performance testing with...

democratized performance test platform -blazemeter ·...

codefresh + blazemeter webinar: continuous testing for...

unemployment and jobs in poland. unemployment reasons deep...

decrease amyloid plaque decrease neurofibrillary tangles...

load testing with jmeter, blazemeter, new relic

gleim commercial pilot faa knowledge test · gleim...

· ≥4 µg/m3. increase/decrease > 4 days ....

ph decrease

the total economic impact™ of ca blazemeter

opioid epidemic –task force decrease supply, decrease

an automated testing strategy targeted for efficient use...

ticketfly blazemeter case study

chuinti13 decrease networks

decrease fetal movement.prof.salah

doesneuropmal®improve...

· • value will decrease if: • costs increase and...

decrease asthma attack