Download - selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests
![Page 1: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/1.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.Copyright©DeNACo.,Ltd.AllRightsReserved.
MethodstoSustainLongTermOpera5onsofE2EAutomatedTests
SeleniumCommi<erDay2017July1st2017
SWETGroupToshiyaKomoda(@toshiya_komoda)
![Page 2: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/2.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Bio
■ ToshiyaKomoda
⁃ Serversideengineer⁃ TestEngineer
■ developedgamepla?orms,usingautomatedtests
■ currentlyworkinginSWETgroup
⁃ SWET-SoFwareEngineerinTest
⁃ dedicatedteamfortestengineering
![Page 3: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/3.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
mobagegameplaMorm
■ Mainlyabrowsergamepla?orms⁃ tensofmillionsusersplaygamesonthepla?orm
■ Avarietyofservicesfor3rdpartydevelopers⁃ OpenIDConnect⁃ paymentservice⁃ Communityservice
• friend,chat⁃ andmore
![Page 4: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/4.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
HistoryofmobagegameplaMorms
■ BusinesstrendshiFsto⁃ browsertonaNve⁃ pla?ormtogamepublisher
■ SNll⁃ alotofusersplaygamesonourpla?orms⁃ theprofitishuge
■ Ourgoal⁃ maintainhighavailabilityandstrictbackwardcompaNbility⁃ withminimummaintenancecost
automatedE2Etestsuiteshasbeenkeytoachievethisgoal
![Page 5: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/5.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Somesta5cs
■ 5⁃ yearsoperaNons
■ 22⁃ testsuites⁃ groupedbasedonthefuncNonality
■ 700⁃ browsertestcases
■ 60000⁃ webAPItestcases
■ enabledeveloperstoreleaseservercodeswithoutmanualQA
⁃ reducelead5me
⁃ reduceQACost
![Page 6: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/6.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Lifecycleoftestsuites
■ DevelopmentPhase
⁃ buildE2EtestsasQAforservicedevelopment
■ HealthyPhase⁃ alltestsuitesarestableenough
■ BrokenPhase⁃ environmentalchanges,testdatacorrupNon
⁃ increaseoftheraNoofunstabletests…
■ DeadPhase⁃ noonecanpasstheE2Etests⁃ QAcostandleadtimeincreasealot
![Page 7: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/7.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Lifecycleoftestsuites
■ DevelopmentPhase
⁃ buildE2EtestsasQAforservicedevelopment
■ HealthyPhase⁃ alltestsuitesarestableenough
■ BrokenPhase⁃ environmentalchanges,testdatacorrupNon
⁃ increaseoftheraNoofunstabletests…
■ DeadPhase⁃ noonecanpasstheE2Etests⁃ QAcostandleadtimeincreasealot
Gain > maintenance cost
![Page 8: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/8.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Methods to keep tests healthy
![Page 9: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/9.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
1.Periodicexecu5onofalltestsuites
■ Goal⁃ improvestabilityoftestsuites⁃ minimizecosttofixbrokentests
■ executealltestsuitesinallenvironments
■ weexecutestestsonceinaday
■ teststabilityisrequired
■ findbrokentestsasmuchaspossible
![Page 10: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/10.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
2.Defineclearprocesstoexcludetestcases
■ Goal⁃ Controlcostandcoveragetrade-off
■ AllowtoexcludeandreNretestcases⁃ highlyflakytestcases⁃ brokentestcasesthatarehardtofix
■ Exampleprocess⁃ idenNfythereasonofthetestfailure
• offcourse,ifthereisabuginproducNonservice,fixit.
⁃ ifthereasonisnotabuginproducNonservice⁃ makeJIRANcketsandexcludethetestcases
![Page 11: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/11.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
3.Improvetestdebuggability
■ Goal⁃ minimizetestmaintenancecost⁃ maximizethenumberofengineerswhocanseetests
■ Testlanguage⁃ shouldbefamiliarlanguagetoservicedevelopers
■ TestInfrastructureshouldprovide⁃ performance⁃ retrymechanism⁃ precisedebuglog
![Page 12: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/12.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
3.Improvetestdebuggability
■ LivepreviewandvideorecordsforUItests■ zalenium
⁃ h]ps://github.com/zalaxndo/zalenium⁃ developedbyzaland⁃ Aseleniumgridextension
![Page 13: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/13.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
4.MaketestslooselycoupledwithSUT(systemundertest)
■ Goal⁃ maketestshardtobreak⁃ maketestsrobusttochangesofSUT
■ WebAPItestsarerelaNvelyeasytomaintain.Why?⁃ clearinterfacebetweentestsandSUT
■ UItestsarenot.Why?⁃ hard-codedpagelinkinformaNon⁃ hard-codedDOMidorothera]ributesvalues
■ CanwemakeUItestslooselycoupledwithSUTasAPItests?⁃ 2ongoingprojectswillbeshown
![Page 14: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/14.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
4.aVisualregressiontes5ngwithsitecrawler
■ Goal⁃ removehard-codedlinkinformaNonfromUItestcodes
■ usecrawlertechniquesinregressiontests⁃ detectlinksinwebpagesautomaNcally⁃ takescreenshotsofallpagesinthesite
2-up swipe
![Page 15: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/15.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
4.bInputtopiciden5fica5onwithmachinelearning
■ Goal⁃ removehard-codedDOMa]ributesvaluesfromUItestcodes
■ usenaturallanguageprocessingtechniquetohandleinputforms⁃ basedonaresearchpaper「UsingSemanNcSimilarityforInputTopicIdenNficaNoninCrawling-basedWebApplicaNonTesNng」,Jun-WeiLin,FarnWang
• h]ps://arxiv.org/abs/1608.06549
■ ourprototypeshowspromisingresults
⁃ canesNmate80%ofinputforms
⁃ experimentsforloginanduserregisterforms
■ canbeusedwiththeprevioussitecrawler
![Page 16: selenimu commiter day 2017: Methods to Sustain Long Term Operations of E2E Automated Tests](https://reader034.vdocuments.us/reader034/viewer/2022050613/5a6623627f8b9a214f8b5a7d/html5/thumbnails/16.jpg)
Copyright©DeNACo.,Ltd.AllRightsReserved.
Summary:MethodstoSustainLongTermOpera5onsofE2EAutomatedTests
1. PeriodicexecuNonofalltestsuites
2. Defineclearprocesstoexcludetestcases
3. Improvetestdebuggability
4. maketestslooselycoupledwithSUT(systemundertest)