test driving an organisation - anztb driving an organisation v2_0.pdf · test execution • need a...
TRANSCRIPT
Test Driving an Organisation
Paul O’Keeffe
Shaun Loh
We Have a Problem • Coreapplications• Businessideasslowtogettomarket
• Manualtestingtimesdominatedthereleasecycle
• Longtailtestcyclesdelayingrelease
• Defectdetectionlateintheprocess
• Somedefectsgettingtoproduction
• Multipleindependenttestteamssharinglittle
• Lackoftrustintestsduetoinconsistentcoverageandexecution
Quality / Speed Delivery • Qualityinitiative• Morphedinto“SpeedDelivery”
• Cycletimeisperhapsthemostimportantmetricinsoftwaredevelopment
• Reducethetimebetweenchangerequestandproductiondelivery
GUI Test Automation • Webapplications
• Desktopapplication
• Tightcouplingbetweenapplications• Initiallybuildasafetynet
• Thenbuildoutapplicationcoverage
• BusinesslogicintheGUIswasrifesowecouldn’tsensiblytestattheAPIlevelorevenattheHTTPlevel
What Test Pyramid? • Thetestpyramidminusthebaseandmiddle;◦ andthetop
• Webuildthetop,becausethebaseandmiddlecouldn’tbe
• Codewasnotdesignedtobetestable◦ Frameworks!
• Swisscheesepyramid
• Normally,buildingafewhappydaysscenariosatthetopofthepyramidistherightapproach,butwithoutagoodbaseormiddle,fragilityisworse
• Somanythingswouldbreaktests• Who/whattoblame?
What Could Possibly Go Wrong? • Infrastructure• Environment
• Data• Applications
• Everything…
• StabilisingaGUItestautomationsuiteisincrediblyhardwork
Infrastructure • Cloud(AWS)
• Butproductionstillonpremise,sodifferent
• Longlivedsnowflakes• Provisioningwasgraduallybeingautomated
• Bothfortestingandapplicationinfrastructure
Controllable Environment • Naildownenvironmentonbuildagentsprecisely
• Spreadoutintonailingdowneverythingexternalwhichischangingand/orcouldcauseproblems• Hardwarespecs• Manualtesterssharingenvironments• Batchjobs• Softwareupdates
• Limitedduetoownershipbyotherteams• Separationofacceptancetestingfromdevelopmentteams
Data • Don’trelyonacopyofproductiondata• Becauseyoucan’trelyonit
• Alwaysgeneratethedatayouneedforatest• Productiondataandbatchjobstakingtoolongtoruninunderspecifiedenvironments
Test Isolation • Creatinganewmemberforeachtest
• Naturalisolation• Datawascompletelyscopedbymember
• Ensureseachtestiscompletelyisolatedfromanyother
Applications (with Real Users) • Usersareapologistsforpoorapplicationbehavior• Robotsaren’t
• Realusersseemedtohavebeentrainedtohaveahightoleranceforslowapplicationresponses
• Synchronousbusinessprocessesbuiltontopofasynchronoussystems
• Usersjustrefreshuntilthingshavefinished• Realusersworkedaroundapplicationerrorsliketheevery10thindexproblem
• Doesn’tmaketestautomationanyeasier,though
Application Driver Layer • Applicationmodels
• Pagemodels
• Growthisasyouneeditforthetestsyou’vewritten• Don’tputahardcaponittoosoon◦ Cucumber/Gherkin◦ Commercialtools◦ Otheropensourcetools(whoshallremainnameless)
• Domain,notUI
Application Driver Layer • Mosttoolsputahardcaponapplicationmodelling,usuallyatthepagelevel
• Needmoreflexibility.Noteverythingisshapedlikeacucumber
• Renewalsmodelisagoodexample
• Coderetainstheflexibility,sinceithasthenecessarymechanismsforabstraction
• Askyourselfwhetheryourtoolcancompetewithcodeonthisfront?
Page Models • Disappointinglackoffreetoolstodobasicpagemodellingascode
• LearnedalotabouthowtodogoodpagemodellinginJavaandC#
• SomeadvancesonthecurrentlyacceptedstateoftheartIMHO
• DesignofpagemodelDSLpreventedrun-timeerrorsandenabledautocompletion
• Projectcomingsoonwithsomeofourdesigns
• DSLsforAPIsandGUIs.Sometimesthesameonesowecouldswitchthetwoeasily
Asynchronous Behaviour • Nofixeddelays(sleeps!)• Pollingloops
• Events
Weapon of Mass Destruction
Effect on Systems Under Test • Weaponofmassdestruction
• Unexpectedloads/greaterscale• Runningourtestscausedahigherloadthanproduction• Duetohaving30buildagentsallrunningtestsfasterthannormalusers
• Buttestenvironmentsaremuchscaleddownversionsofproductiontocutcosts
• Singleserverratherthanclusters(multicastproblem)
• Datasizes?• Insignificantcomparedtoproduction• Butgrowthovertimewithoutresetcausedissues
Impact • Manualtestingisusuallyprettylowimpact
• Automatedtestingishighimpact
• ContinuousIntegrationisevenmoreimpact
• Existingsystemsmayonlybedesignedtohandlemanualtests
• Testingperformance/scaledemandsmaybehigherthanproduction!!!• Especiallyifyouwantfastdevelopmentcycles
• Forcerestoftheorganisationtochange/improve
Test Execution • NeedatoolwiththerightgranularityforGUItestautomation.
• Runeachtestandcollectresultswouldbeaboutright.
• CIservertocoarse.Runalltestsinonebuild.• Testgridtoofine.RuneachUIactionremotelyfromacentralcontroller.
• BrokethingsupbyhavingabuildpertestonCIserver,butnotwellsupported.
Continuous Integration Server • Asopposedtothepracticeofcontinuousintegration• Supportforlongrunningacceptancetests
• Scalabilityofconfiguration• Newgeneration:pipelineascode(oratleastbetterconfiguration)◦ GUIshamperthis
MSHTA • TheUncannyValleyofwebapplications• MSHTAapplicationsareapaintotest
• Nearlyawebappbutnotquitesoyouneedspecialtools• Frameworkdidn’tallowapplicationtoberuninabrowserjustfortesting
• WouldbegoodtohaveafreetoolwhichcoulddothisaswellasSeleniumdoesforweb
• MSHTAhavesomedifferences• Oftenmoredialogwindowwizards• Lockeddownbrowser• Fewercontrols
• Testautomationforcesorganisationawayfromthisstyleofapplication
Recording Evidence • Videorecording>• Sequenceofscreenshots>
• Finalscreenshot>• Logsonly>
• Nothing
• Someproblemsarehardtoreproducejustbyrunningthetestagain
• Observability
• Convincingevidenceforotherswithintheorganization• Otherbenefits
Licensing • Permachinelicensingschemesarepainfulfordesktoptestdevelopment,butmoresoforagridofbuildagentsinthecloud
• Donotbuytoolswithlicenceswhichdiscouragethenumberofpotentialenvironments
• Sitelicencesarefine
Results • Shortenedreleasecyclesbymanyweeks
• Increasedconfidencetoperformreleases,wheretheymightnototherwisehavebeenattempted
• Butotherchangesareslow
New State
Test Driving the Organisation • TDDawholeorganisation• Conway'sLaw,sortof• Canmakemovesandreorganisemoreeasilywiththesafetynet
• Removalofredundantdelivery/testingphases
• Smallenvironments
• Fastercycletimes• Eachchangemakesthisalmostmorepossible• Loweroverhead• Costsaving• Earliertesting• ShiftLeft
Opportunity: Data Load • Builttoolsformanualtesterstosetupdataforexploratorytesting
• Alsoforothermanualtestswehadn’tyetautomated
• Removeddependenceonproductiondataandfindingappropriateexamples
• Wherepossible,usedAPIsfordataloadinourtestsetup
• Chickenandegg• Nolongeranyneedforproductiondata,sowechangedtheorganisationinawaywhichsolvedourownproblemandseveralothers
• LearningandDevelopment(trainingdata)
• PusheddevelopmentteamstoopenupAPIs
Automate (Almost) Everything • AnalysistoolslikeDashyandHelpy.• Don’tjustautomateyourtests,automateyouranalysisandanythingelserepetitiveyouhavetodo.
Test Execution • Runyourtestsallthetime?• Tradeoffs
• Atleastwhenanythingchanges,whichinourcasewasprettymuchallthetime
Deployment and Other Automation • Startedpushingthingstowardsautomateddeployments
• Wewouldhelpoutwiththis,sincewehadautomationskills
• Softwaredevelopmentskillsofteamflowedoverintosmallproductdevelopment
• RobotProcessAutomation(RPA)
Think Different • Developerviewoftestautomation
• Fresheyes
• Embracecode
• Improvedqualityoftestcode
• Bringtestinganddevelopmentclosertogether
Way of Working • Agile,butonlywithinoursmallpartoftheSDLC
• Collaborative
• Pairingondemand◦ Moarpairing!!!◦ Particularlywhereincreaseskillsorshareknowledge,wheretheyareunevenlydistributedwithintheteam
• Honestretrospectives• Continuousimprovementactionsactuallyactedupon
• Attemptedtoinfluenceotherteams
• Asfaraswecouldgowithintheconstraintsoftheorganization
Invest in your People • Super-interns• Andotherswithcomplementaryskillsfromelsewhereintheorganisation• Limitedbudget• Negotiatedtogetreallygoodinternsandthentrainthemwell
• Peopledon'tneedtobeanexactmatchforrequiredskills• Someprogrammingabilityiswelcome,butnoneedforlots• CancomefromotherITbackground• Goodtohavealternativeperspectives(e.g.support)
• Needtobekeentolearn• Cangetfantasticleverageatlowcost.Wedid.
Invest in your People • Axesoflearning/growth• Maturesoftwaredevelopment• Automatedtesting• Enterprisetraining/constraints
• Highlycollaborativeenvironment/team• Workwithotherswithdevelopment/testing/domain/applicationexperience• Balanceofperhaps50%ofteamalreadyexperienced
Invest in Your People • Freedomtofailandlearn• Psychologicalsafety• Givespeopleconfidencetotrynewthingsearlier
• Handsonimmediately
• Internsbecamethementors
• Growexistingpeoplefromtheorganisation• Graftthemontoanewplanttoreignitetheirmotivationandgrownewskills
• Chocolate