HowILearnedtoStopWorrying&LovetheBug
Picture Courtesy of: Dr. Sarah Ford (a.k.a. Mrs. Matthew Hertz)
¨ Cannotprovetherearenobugs¤ Canonlyshownobugsexistonthosetests
HowToWriteTests
Testingshowsthepresence,nottheabsence
ofbugs
¨ 20%ofcodehas80%ofbugs¤ Modulesthataremostcomplex,intricate,ordetailed¤ Locationswhereexpectationsofdatamightdiffer¤ Codeintransition:frequentlychangedmodules¤ Anyplacewhereprogramrequiresuserinput
¨ Focustestingeffortstoconcentrateonthesebugs¤ Tests(&testing)expensive&simplertoolsforeasycode¤ Automationmatters;errorsoftenoccuratjoinpoints
WhatToTest&Why
Assumetheworst:Focustestingon
unlikelysituations
TestsKeyConcept
InputTests…
…andFinally
¨ Inrealworld,somecasesmaynotbeworthtesting¤ Mustassumebugsexistsoideallytesteverything¤ Savetime,donotchangeinputtochecksameidea¤ Simplegetters&setterseasy,butcheckbeforecommit¤ Focusonpossibilities,donotcheckimpossiblecases
WheretheBugsAren’t
GoodTests
public class Stock { private double cost; //Constructor&gettersimple&skippedforspace
//Decreasescostofastock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }
//Evenmorecodewouldbehere,werethisnotanexampleforclass
public class StockTest { @Test public void t1() { Stock ibm = new Stock(141.31); assertEquals(141.31, ibm.reduceCost(0), 0.001); assertEquals(141.31, ibm.getCost(),0.001); }
MoreGoodTests
public class StockTest { @Test public void t2() { Stock siri = new Stock(7.10); assertEquals(0.0, siri.reduceCost(8), 0.001); assertEquals(0.0, siri.getCost(),0.001); }
public class Stock { private double cost; //Constructor&gettersimple&skippedforspace
//Decreasescostofastock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }
//Evenmorecodewouldbehere,werethisnotanexampleforclass
NotaGoodTest
public class StockTest { @Test public void t3() { Stock htz = new Stock(15.09); assertEquals(?????, htz.reduceCost(-100), 0.001); assertEquals(?????, htz.getCost(),0.001); }
public class Stock { private double cost; //Constructor&gettersimple&skippedforspace
//Decreasescostofastock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }
//Evenmorecodewouldbehere,werethisnotanexampleforclass
NotaGoodTest
public class Stock { private double cost; //Constructor&gettersimple&skippedforspace
//Differencefromcostatwhichpeoplesoldstock;deltaismax.dropincost //Returnsupdatedvalueofcost public double reduceCost(double delta) { }
//Evenmorecodewouldbehere,werethisnotanexampleforclass
public class StockTest { @Test public void t3() { Stock htz = new Stock(15.09); assertEquals(?????, htz.reduceCost(-100), 0.001); assertEquals(?????, htz.getCost(),0.001); }
TestsKeyConcept
Assumetheworst:Focustestingon
unlikely(butNOTimpossible)situations
¨ Smallbugsinloopscancreatehugeerrors¤ Lotsoftimeexecutingincreasesoddsofhittingrarecase¤ Oftenerroronlyappearswhenresultsused,notinloop¤ Debuggingoftentricky,sincemanyscenariostotestout
¨ Runoften+hard-to-debug==criticaltotestwell¤ Findingbugsimportant,sincequalitydependsonthis¤ Knowingbugsexistsuseless;mustalsosimplifyfixes¤ Sonarrowingbug'scausejustasneededasfindingbug
LoopTestingImportant
TypesofLoops
¨ Forallsimpleloops,tryinputsthat:¤ Skiploopentirely¤ Make1passthroughtheloop¤ Make2passesthroughtheloop¤ Makempassesthroughtheloop,where(m>2)
¨ Ifloopexecutedatmostntimes,tryinputsthat:¤ Maken-1&npassesthroughtheloop
SimpleLoop
¨ Firsttestsetrunsallouterloopsexactlyonce¤ Innerloopruns(min+1),average,(max-1)&maxtimes
¨ Thenrunallbuttwoinnermostloopsexactlyonce¤ Innerloopsrun(min+1),average,(max-1)&maxtimes
¨ Testsshouldcontinuegrowingloop-by-loop
NestedLoops
TypesofLoops
ConcatenatedLoops
¨ Ifloopsareentirelyindependent¤ Noconditions,variables,orvaluesincommon¤ Woo-hoo!Justperformsinglelooptestsoneach
¨ Otherwisetreatasnestedloops&makelifeeasier¤ Workasifthefirstloopistheoutermostloops
UnstructuredLoops
¨ Figureouttheprocessleadingtothisdecision¤ Burnartifactsandcoderesultinginthisabomination
UnstructuredLoops
¨ Figureouttheprocessleadingtothisdecision¤ Burnartifactsandcoderesultinginthisabomination¤ Anyoneinvolvedshouldterminatedimmediately
UnstructuredLoops
¨ Figureouttheprocessleadingtothisdecision¤ Burnartifactsandcoderesultinginthisabomination¤ Anyoneinvolvedshouldterminatedimmediately
¨ ReWrite“missing”documents,startingfromscratch
¨ Unittestsgoodforsometasksworkingonback-end¤ Butwhatabouttasksimplementingfront-endcode?¤ Userwantsresultsandonlyknowswhattheycansee¤ Correctresultsimpossibleifback-endfailsunittests
¨ Back-endcodeveryimportantsocannotskiptests¤ Butinvisibletouserandclientdoesnotcareaboutcode
Back-EndTesting
¨ Needtotestfront-endtasksthatdisplayinformation¤ GUIclassescanbecheckedagainstuserstories¤ JUnittestcaseslessusefulperformingthesetests¤ Automationlackshumantouch;cannotcheckaesthetics
Front-EndTesting
¨ Worstapproach:Clickingaround&seewhatbreaks¤ Simple&fast,butmaynotdiscoveractualcauseofbugs¤ Unrepeatable&slowwhencheckingentiresystem¤ Donebydevelopers,tendstofollowexpecteduses
Front-End"Testing"
¨ Betterapproach:Step-by-stepscripttestsforerrors¤ Lowoverhead&simple,butalsoeasytoforgettorun¤ Discoverunexpectedbugsbyhavingtestersrunscripts¤ Goodrules-of-thumbexisttofindmanycommonerrors¤ ListintaskinZenHub;manywantfilestoholdscripts
Front-EndTesting
¨ Betterapproach:Step-by-stepscripttestsforerrors¤ Lowoverhead&simple,butalsoeasytoforgettorun¤ Discoverunexpectedbugsbyhavingtestersrunscripts¤ Goodrules-of-thumbexisttofindmanycommonerrors¤ ListintaskinZenHub;manywantfilestoholdscripts
Front-EndTesting
¨ Bestapproach:AutomatetestingwithUItool/code¤ Createssetupcosts,butguaranteespredictableresults¤ Cancompensateforloadtimes&otherrealissues¤ Oftenincludebothprogramming&scriptingsetups
ValidationTesting
¨ Allowsautomatedtestingofweb-basedapplications¨ Testsuitereportsresultsofrunning1ormoretests¨ Oftencreatemanytestcases;eachexposes1bug
¨ AddtestsinJava/C#/PythonwithWebDrivermodule¤ ManylanguageshaveSeleniumlibrariestodrivetests¤ Loadspage&definesAPIusedtoevaluateitscontents
¨ IfusingIDE,abletocreate&runsinbrowser¤ IDEeasiertouse:canrecordactionsinbrowserastest¤ WillalsoallowupdatingorrewritingSelenesescript
Selenium
¨ Easiestmoduletouse,butneedsChrome*towork¤ DownloadviaChromeWebStoretobereadytouse¤ Scriptsmostlyrecordedbyclickingonelementstotest
¨ Startprocessusingthecommandopentoloadpage¤ click[AndWait]"clicks"onitemthatyouidentify¤ Scriptcanalsoentertextintoelementusingtype
¨ LikexUnittests,reliesonassertionstodefinechecks¤ assertTitlecheckstitleofpage(textshownontab)¤ CheckiftextonpageusingverifyTextPresent ¤ verifyElementPresentchecksifelementonpage
SeleniumIDE
public static void main(String[] args) { WebDriver driver = new EdgeDriver(); driver.get("http://www.google.com"); WebElement el = driver.findElement(By.name("q")); element.sendKeys("Hawaiian-Print Computer"); element.submit(); WebDriverWait stall = new WebDriverWait(driver, 10); boolean result = stall.until(
new ExpectedCondition<Boolean>() { public Boolean apply(WebDriver d) { return d.getTitle().startsWith("Hawaiian"); }}); System.out.println("Met expectations: " + result); driver.quit(); }
SeleniumWebDriver+Java
driver = webdriver.Firefox() driver.get("https://cse.buffalo.edu/~mhertz") assert "Matthew Hertz" == driver.title crselnk = driver.find_element_by_xpath( "/html/body/table[2]/tbody/tr/td[1]/p/a") crselnk.click() result = WebDriverWait(driver, 10).until( lambda x : "CSE442" in x.title) assert result
SeleniumWebDriver+Python
¨ Inputoverflows:Typestringlongerthannormal/fits¤ Checkthattextisaccepted(orprovidesGOODerror)¤ Iftextisaccepted,areresultsreadableorusable?¤ Specifytext(trymanysizes)inscriptscheckingthis
CommonFront-EndErrors(1)
¨ Structureoverflow:Makepanelslargerthanpage¤ Doesthiscreateerrorsorissystemabletohandledata¤ Doitemsresize,scroll,orprovidewaytoseeeverything?¤ Similartolooptests;detailinginputstousecritical
CommonFront-EndErrors(2)
¨ ViolateAssumptions:Assumeusersjerks(ordumb)
CommonFront-EndErrors(3)
¨ ViolateAssumptions:Assumeusersjerks(ordumb)¤ Theywillmakeworstchoice.Howdoessystemreact?¤ Whatifneededfilesdeleted,networklost,orsimilar?¤ Scriptexplainhowtostart&whaterrorshouldbeshown
CommonFront-EndErrors(3)
¨ DuplicationIssues:Repeatedlyentersameinput¤ IfWhentheyadd/removemultipletimes,whatiserror?¤ Doesapphandle(orprovideclues)forimpatientusers?
CommonFront-EndErrors(4)
¨ DuplicationIssues:Repeatedlyentersameinput¤ IfWhentheyadd/removemultipletimes,whatiserror?¤ Doesapphandle(orprovideclues)forimpatientusers?¤ “Back”buttontempting,whatdoesitdotowebapp?
CommonFront-EndErrors(4)
¨ InvalidData:Intentionallyforceinvalidresults¤ Scriptactionscreatingillegalstateinwidgetattheend¤ Feb29remaininnon-leapyears?Movestartafterend?¤ Giveintothedarkside&tricksystemintobadstates
CommonFront-EndErrors(5)
¨ Resizingissues:Canithandledifferentwindowsizes¤ Scriptresizingwindow&makesureprogramstillusable¤ Setmonitortosmallerscreen&seeiflayoutworks¤ Tryforcingscrollbaruse&seehowuserswillreact
CommonFront-EndErrors(6)
¨ JavaScriptisahorrible,horriblelanguage
Non-UIJavaScriptTesting
¨ JavaScriptisahorrible,horriblelanguage¤ BrendanEichcreated&implementedJSin10days¤ Namewasbuzzword;neverrelatedtoJava¤ Notintendedasstandard;NetscapelookingtobeatIE¤ "Standards"exist,butimplementationsvarygreatly
¨ Languagecombinesmanyfeaturestoprovideitall¤ OO,functional,ordeclarativecodesupportexists¤ JSoftenimplementsmiddle-tier&front-endlayers¤ Testdifferentlevelsseparatelywouldbeideal,buthow?
Non-UIJavaScriptTesting
¨ RealweaknessofJSisdifficultyintestingcode¤ Notgreatlyused,butMochabestunittestinglibrary¤ Middle-tiertestedviafront-endinmany/mostsituations¤ Nothelpfultounderstand&fix,butshouldfindbugs
¨ GoodlanguagecompilingtoJSisalternateapproach¤ Oncecomplete,usetestsandtoolsfororiginallanguage¤ ScalaJS,TypeScript,&Dartdevelopedforthispurpose¤ Allofthisalsoassumesthatbugnotcreatedbycompiler
Non-UIJavaScriptTesting
Utility:Isituseable
OtherTestingIssues
Utility:IsituseableReliability:Willyouendupleadstoryonnightlynews?
OtherTestingIssues
Utility:IsituseableReliability:Willyouendupleadstoryonnightlynews?Robustness:Howlongofdisclaimerwillitneed?
OtherTestingIssues
Utility:IsituseableReliability:Willyouendupleadstoryonnightlynews?Robustness:Howlongofdisclaimerwillitneed?Performance:WillitfinishbeforeBuffalowinsatitle?
OtherTestingIssues
KeyPoint
=
¨ "Acceptancetests"checkuserstorycomplete¤ Ensuresfeatureworksandreadyfordeployment¤ Runbyclientsohavetobescriptedtests¤ Ensurefullunderstandingofallaspectsoffeature
¨ "Tasktests"checkthatataskiscomplete¤ Ensurestaskcompleteandreadyforinclusion¤ Runbydeveloperssocanbescriptedtestsorunittests¤ Findsbugsduringcoding&throughlaterchanges¤ Alsodefineswhatsuccessliketoenableparallelwork
Terminology
¨ WorkonSprint1¤ Rememberthetests!Testsvalidateyourunderstanding
ForNextLecture