software testing video...• test plans are not a static documents but evolve during the development...

SoftwareTesting

Hans-PetterHalvorsen,M.Sc.

RequirementsAnalysis

Design

Implementation

Testing

Maintenance

PlanningDeployment

SRS

SDD

STD

Code

InstallationGuides

UserGuides

GanttChart

withERDiagram,UMLDiagrams,CADDrawings

TestDocumentation

SoftwareRequirementsSpecifications

SoftwareDesignDocumentsSystemDocumentation

SoftwareTestPlan(STP)

ProjectPlanning

End-UserDocumentation

SystemDocumentation

SoftwareTestDocumentation

SDPSoftwareDevelopment

Plan

GanttChart

TheSoftwareDevelopment

Lifecycle(SDLC)

TypicalSoftwareDocumentation

High-LevelRequirementsandDesignDocuments

UserManuals

SystemDocumentation

InstallationGuides

TestPlans

TestDocumentation

DetailedRequirementsandDesignDocuments

ERDiagram(Database)UMLDiagrams(Code)

Time

Start

Finish

HowtoTest/WhattoTest

CADDrawings,etc.

1.Planning

2.Testing

3.End-userDocumentation(Thepeoplethatshallactuallyusethesoftware)

TechnicalStuff

HowtouseitHowtoinstallit

Proofthatyouhavetestedandthatthesoftwareworksasexpected

(Thestakeholders,thesoftwareteam;architects,UXdesigners,developers)

(QApeople)

(SuperUser/ITdep.)

WHATHOW

(EndUser)

ProjectM

anagem

ent(Ga

nttC

hart,etc.)

(SRS)(SDD)

(STP)(STD)

SoftwareDevelopmentPlan

(SDP)

2.Requierements/Design

MainpurposeofTesting:FindBugs!!

• RequirementsErrors:13%• DesignErrors:24%• CodeErrors:38%• DocumentationErrors:13%• Bad-fixErrors:12%

http://proquest.safaribooksonline.com/book/software-engineering-and-development/9781449691998/chapter-3-engineering-of-software/42?uicode=telemark

WhyFindBugsearly?

SoftwareDevelopmentLifeCycle(SDLC)

TheFirstBugever

Theyfoundabug(actuallyamoth)insideacomputerin1947thatmadetheprogramnotbehavingasexpected.Thiswasthe“first”realbug.

WhatisBugs?• Asoftwarebugisanerror,flaw,failure,orfaultinacomputerprogramorsystemthatproducesanincorrectorunexpectedresult,orcausesittobehaveinunintendedways

• Theyfoundabug(actuallyamoth)insideacomputerin1947thatmadetheprogramnotbehavingasexpected.Thiswasthe“first”realbug.

• Debugging:FindandRemove/FixBugs

“Ifyoudon’tknowhowyourcodeworks,itdoesnotwork– youjustdon’tknowityet”

SoftwareTesting

“50%ofthesoftwaredevelopmentisabouttestingyoursoftware”

DifferentSystemsNeedsDifferentTesting1 2

3 4

7PrinciplesofTesting1. TestingshowsthepresenceofBugs:SoftwareTestingreducestheprobabilityof

undiscovereddefectsremaininginthesoftwarebutevenifnodefectsarefound,itisnotaproofofcorrectness.

2. ExhaustiveTestingisimpossible:Testingeverythingisimpossible!Insteadweneedoptimalamountoftestingbasedontheriskassessmentoftheapplication.

3. EarlyTesting:TestingshouldstartasearlyaspossibleintheSoftwareDevelopmentLifeCycle(SDLC)

4. DefectClustering:Asmallnumberofmodulescontainmostofthedefects/bugsdetected.

5. ThePesticideParadox:Ifthesametestsarerepeatedoverandoveragain,eventuallythesametestcaseswillnolongerfindnewbugs

6. TestingisContextdependent:Thismeansthatthewayyoutestae-commercesitewillbedifferentfromthewayyoutestacommercialofftheshelfapplication

7. AbsenceofErrorisaFallacy:Findingandfixingdefectsdoesnothelpifthesystembuildisunusableanddoesnotfulfilltheusersneeds&requirements

http://www.guru99.com/software-testing-seven-principles.htmlhttp://www.testingexcellence.com/seven-principles-of-software-testing

DifferentTypesofTesting


TypesofTesting

StressTesting

UsabilityTesting

PerformanceTesting

UserTesting

RegressionTesting

Setyp&DeploymentTesting

...

......

RequirementsTesting

GUITesting

FunctionalTesting

NonFunctionalTesting ...

...

...LoadTesting

UsabilityTesting

SecurityTesting

WhodoestheTesting?• Programmers/Developers

– Programmersusuallycreatetestcasesandrunthemastheywritethecodetoconvincethemselvesthattheprogramworks.Thisprogrammeractivityrelatedtotestingisusuallyconsideredtobeunittesting.

• Testers– Atesterisatechnicalpersonwhoserolefortheparticularitembeingtestedisjusttowritetest

casesandensuretheirexecution.Althoughprogrammingknowledgeisextremelyusefulfortesters,testingisadifferentactivitywithdifferentintellectualrequirements.Notallgoodprogrammerswillbegoodtesters.

• EndUsers/Customers– Itisagoodideatoinvolveusersintesting,inordertodetectusabilityproblemsand

toexposethesoftwaretoabroadrangeofinputsinreal-worldscenarios.

TestCategoriesBlack-boxvs.White-boxTesting

White-boxTesting:Youneedtohaveknowledgeofhow(DesignandImplementation)thesystemisbuilt

Black-boxTesting:Youneednoknowledgeofhowthesystemiscreated.

TypicallydonebyDevelopers,etc

LevelsofTesting

UnitTesting

IntegrationTesting

SystemTesting

AcceptanceTesting

Anymodule,program,objectseparatelytestable

Interfacebetweencomponents;interactionswithothersystems(OS,HW,etc)

Thebehaviorofthewholeproduct(system)asdefinedbythescopeoftheproject

Istheresponsibilityofthecustomer– ingeneral.Thegoalistogainconfidenceinthesystem;especiallyinitsnon-functionalcharacteristics

LevelsofTestingUnitTesting:Testeachpartsindependentlyandisolated

IntegrationTesting:Makesurethatdifferentpiecesworktogether.TesttheInterfacesbetweenthedifferentpieces.Interactionwithothersystems(Hardware,OS,etc.)

SystemTesting:Testthewholesystem

RegressionTesting:Testthatitstillworksafterachangeinthecode

LevelsofTesting

UnitTesting

RegressionTesting

IntegrationTesting

System/ValidationTesting

AcceptanceTesting

Start

Finish

Requirements&DesignStartDevelopment

UnitTestsarewrittenbytheDevelopersaspartoftheProgramming.EachpartisdevelopedandUnittestedseparately(EveryClassandMethodinthecode)

TheCustomerneedstotestandapprovethesoftwarebeforehecantakeitintouse.FAT/SAT.

SystemtestingistypicallyBlack-boxTeststhatvalidatetheentiresystemagainstitsrequirements,i.eCheckingthatasoftwaresystemmeetsthespecifications

Integrationtestingmeansthesystemisputtogetherandtestedtomakesureeverythingworkstogether.

Regressiontestingistestingthesystemtocheckthatchangeshavenot“broken”previouslyworkingcode.BothManually&Automatically(Re-runUnitTests)

TestingOverviewTestCategories: TestLevels: TestMethods:

UnitTesting

RegressionTesting

IntegrationTesting

SystemTesting

AcceptanceTesting

Black-boxTesting

White-boxTesting

StressTesting

PerformanceTesting

GUITesting

FunctionalTesting

NonFunctionalTesting

LoadTesting

UsabilityTesting

SecurityTesting

etc.

TestPlanning• Tomaximizetheeffectivenessofresourcesspentontesting,asystematicapproachisrequired

• ASoftwareTestPlan(STP)shouldbecreated

TestDocumentation

21

PlanningTests PerformTests DocumentTestResults


SoftwareRequirementsSpecifications(SRS)SoftwareDesignDocument(SDD)

SoftwareTestDocumentation

(STD)

TestLogs

ThesedocumentswillbethefoundationforallTesting

- Functional&Non-FunctionalRequirements- User&SystemRequirements

TypicalSoftwareDocumentation

High-LevelRequirementsandDesignDocuments

UserManuals

SystemDocumentation

InstallationGuides

TestPlans

TestDocumentation

DetailedRequirementsandDesignDocuments

ERDiagram(Database)UMLDiagrams(Code)

Time

Start

Finish

HowtoTest/WhattoTest

CADDrawings,etc.

1.Planning

2.Testing

3.End-userDocumentation(Thepeoplethatshallactuallyusethesoftware)

TechnicalStuff

Howtouseit

Howtoinstallit

Proofthatyouhavetestedandthatthesoftwareworksasexpected

(Thestakeholders,thesoftwareteam;architects,UXdesigners,developers)

(QApeople)

(SuperUser/ITdep.)

WHATHOW

(EndUser)ProjectM

anagem

ent(Ga

nttC

hart,etc.)

(SRS)(SDD)

(STP)(STD)

SoftwareDevelopmentPlan (SDP)

2.Requierements/Design

WhatisaSoftwareTestPlan(STP)?ADocumentthatanswersthefollowing:• TestingshouldbebasedonRequirements&DesignDocuments• Whatshallwetest?• Howshallwetest?• Hardware/SoftwareRequirements• Whereshallwetest?• Whoshalltest?• Howoftenshallwetest(TestSchedule)?• Howshalltestsbedocumented?

§ Itisnotenoughsimplytoruntests;theresultsofthetestsmustbesystematicallyrecorded.Itmustbepossibletoauditthetestingprocesstocheckthatithasbeencarriedoutcorrectly

§ Systemtests:Thissection,whichmaybecompletelyseparatefromthetestplan,definesthetestcasesthatshouldbeappliedtothesystem.Thesetestsarederivedfromthesystemrequirementsspecification. http://www.softwareengineering-9.com/Web/Testing/Planning.html

ThesethingsneedtobespecifiedintheSTP

TestPlanExampleA. GoalsandExitCriteria(Quality,Robustness,Schedule,PerformanceGoals

oftheProduct,...)B. ItemstobeTested/Inspected(Executablessuchasmodulesand

components,NonexecutablessuchasRequirmentsandDesignspecifications,...)

C. TestProcess/Methodologies(Unit,Functional,Acceptance,RegressionTests,Black-box,White-box,Testmetrics,Bugreportprocess,...)

D. Resources(People,Tools,TestEnvironment,...)E. Schedule(Test-casedevelopment,Testexecution,Problemreportingand

fixing,...)F. Risks(...)G. MajorTestScenariosandTestCases(...)

EssentialsofSoftwareEngineering,FrankTsui;OrlandoKaram;BarbaraBernal,3ed.,Jones&BartlettLearning

AppendixDinEssentialsofSoftwareEngineering

HowtomakeaTestPlan

http://www.guru99.com/what-everybody-ought-to-know-about-test-planing.html

TestCasesListExampleTester:_______________________,Date:________

TestCase OK Failed Description

TheLoginProcedureworks

UserDataSavedintheDatabase

etc

TheTestersfillintheseListselectronically.ShouldbeincludedinSoftwareTestDocumentation

IfTestCasesFails,reportBugsinVSO

TestPlanningSummary• Testplanninginvolvesschedulingandestimatingthesystemtestingprocess,

establishingprocessstandardsanddescribingtheteststhatshouldbecarriedout.• Aswellashelpingmanagersallocateresourcesandestimatetestingschedules,test

plansareintendedforsoftwareengineersinvolvedindesigningandcarryingoutsystemtests.

• Theyhelptechnicalstaffgetanoverallpictureofthesystemtestsandplacetheirownworkinthiscontext.

• Aswellassettingoutthetestingscheduleandprocedures,thetestplandefinesthehardwareandsoftwareresourcesthatarerequired.

• Testplansarenotastaticdocumentsbutevolveduringthedevelopmentprocess.Testplanschangebecauseofdelaysatotherstagesinthedevelopmentprocess.

• Testplanningisparticularlyimportantinlargesoftwaresystemdevelopment.• Forsmallandmedium-sizedsystems,alessformaltestplanmaybeused,butthereis

stillaneedforaformaldocumenttosupporttheplanningofthetestingprocess.http://www.softwareengineering-9.com/Web/Testing/Planning.html

TestEnvironment


WhyDoWeNeedaTestEnvironment?

WhycantwejustuseourownPC?

WhyTestEnvironment?• “ItworksonmyPC”saystheDeveloper• CleanEnvironment• OntheDevelopersPCswehaveallkindofSoftwareinstalledthat

theCustomerdonthave,e.g.DevelopmentToolslikeVisualStudio,etc.

• WeneedtotestondifferentPlatformsandOperatingSystems• CustomersmayusedifferentWebBrowsers• Deployment:TestofInstallationpackages• MakethesoftwareavailableforTesters• etc.

“ItworksonmyComputer”MakesuretotestyoursoftwareonotherComputersandEnvironmentsthanyourDevelopmentComputer!• EverythingworksontheDeveloperComputer• TheCustomersDatabaseisnotthesameasyours• TheCustomermaynotusethesameOS• TheCustomermaynotusethesameWebBrowser• TheCustomerdonothaveVisualStudio,SQLServer,etc.ontheirPersonalComputer

• Etc.=>TestEnvironmentisneeded!

Development Testing Production

DevelopmentEnvironment TestEnvironment

ProductionEnvironment

TypicallytheDevelopersPersonalComputerwithDatabase,WebServerandProgrammingSoftware

ACleanPC/Server(oranetworkwithPCsandServers)whereyouinstallandtestyourSoftware.Todaywetypicallyset-upaVirtualTestEnvironment

TheCustomersenvironmentwhereyouunstallthefinalsoftware(ServersandClients)

ProgrammingenvironmentssuchasVisualStudio,etc.shouldnotbeinstalledinthisenvironment.Youneedtocreate.exefilesetc.inordertomakeyoursoftwarerun.

Developers Developers&Testers Customersuntilfinished

Virtualization

OperationSystem

VirtualizationSoftware

Hypervisor

VM VM VM

Hardware(Computer)

VM VM VM

Guests

Host

VM=VirtualMachines

AHypervisorcanrundirectlyonthecomputerwithoutaHostOS

Windows,Linux,...

Windows,Linux,...

VirtualizationSoftwareAlotofVirtualizationSoftwareexists.Herearesomeexamples:• VMwareWorkstation• VMwareWorkstationPlayer (Freeofchargeandsimpleto

use)• VMwarevSphereandvSphereHypervisor• VMwareFusion(Mac)• ParallelsDesktop(Mac)• MicrosoftHyper-V• VirtualBox• etc.

VMwareWorkstationPlayerVMwareWorkstationPlayerisforpersonaluseonyourownPC.VMwarePlayerisfreeofchargeforpersonalnoncommercialuse.

36

VMwareisacompanythathasbeenspecializingwithinvirtualizationsoftware.http://www.vmware.com

WhenareyoufinishedTesting?


SoftwareTesting

“50%ofthesoftwaredevelopmentisabouttestingyoursoftware”

WhenarewefinishedwithTesting?

Start

Finished

Requirements&Design

Development&Coding

ContinuousTestinginthewholeSDLC!

Alph

a

RCBeta

RTM

FinalDelivery

Testing

Testing Testing Testing Testing

IncreasedFocus

IncreasedFocus

IncreasedFocus

IncreasedFocus

Agile/Scrum:PeriodicallyIterations/Sprintevery14-30days

...... ...

CodeFreeze

SoftwarewithoutCritical Bugs

Requirements&Functionality

FunctionalityRequirements

YoucanneverfindallBugs!ReleasedSoftwaredo haveBugs!

WhentoStopTesting?

• Asimpleansweristostoptestingwhenalltheplannedtestcasesareexecutedandalltheproblemsfoundarefixed.

• Inreality,itmaynotbethatsimple.Weareoftenpressuredbyscheduletoreleasesoftwareproduct.

SoftwareFinished

Time

WhentoStopDevelopment?

“90%”“100%”

Details,smalladjustments,etc.Thelast10%takesalotoftime!!!

Soonerorlateryouhavetosayenoughisenoughandreleaseversion1.0.

Onemustdefinewithinthedevelopmentcompany,developmentteamorindialoguewiththecustomerwhatisdefinedas"goodenough".Softwarewillneverbe100%completeorerror-free!

Time

WhentoStopTesting?

CriticalPoint

Inthebeginningititeasytofindbugswithfewresources

WhenshouldyoustopTesting?(dependsonTime,Budget,etc.)

WhentoStopTesting?• Whenthetesterhasnotbeenabletofindanotherdefectin5(10?30?100?)minutesoftesting

• Allcodereviewsandwalkthroughshavecertifiedthecodeasok

• Whenagivenchecklistoftesttypeshasbeencompleted

• Thecodehaspassedallunittests• Whentestingrunsoutofitsscheduledtime• ...

E.J.BraudeandM.E.Bernstein,SoftwareEngineering:ModernApproaches,2ed.:Wiley,2011.+++

BugTrackingSystems


BugTrackingSystems• A“bugtrackingsystem”or“defecttrackingsystem”isasoftwareapplicationthatkeepstrackofreportedsoftwarebugsinsoftwaredevelopmentprojects.

• Itmayberegardedasatypeof“issuetrackingsystem”.

• Typicallybugtrackingsystemsareintegratedwithothersoftware“projectmanagementapplications”– e.g.,VisualstudioTeamServices,Jira,etc.

https://en.wikipedia.org/wiki/Bug_tracking_system

BugTrackingSoftware• TeamFoundationServer/VisualStudioTeamServices

• Jira• Bugzilla• Clearquest• …(hundreds)

BugReportingandTracking

BugTrackingSystem

Developer

Tester

ReportBugs

FixBugs

CheckifBugsareFixed

CheckifBugFixesareApproved

TestManager,ProjectManager,QADepartment

AssignResponsible

Person

GothroughListsofNotFixedBugs,FixedBugs,etc.

VisualStudioTeamServices


WorkItemsExample

YoucancreateQueries(bothPersonalandTeamQueries)

ListofWorkItems

WorkItemDetails

WorkItems– NewBug

Queries• UsedtofindexistingWorkItems• YoumaycreatedifferentQueriestomakeiteasytofindtheWorkItemsyouneed

• Queriesmaybepersonalorvisibleforeverybodyintheproject(TeamQueries)

51

CreatingaQuery- Example

CodeReview&Refactoring


WhatisRefactoring?• Evenwhenusingbestpracticesandmakingaconsciousefforttoproducehigh-qualitysoftware,itishighlyunlikelythatyouwillconsistentlyproduceprogramsthatcannotbeimproved.

• Refactoringis– theactivityofimprovingyourcodestylewithoutalteringitsbehavior

– achangemadetotheinternalstructureofsoftwaretomakeiteasiertounderstandandcheapertomodifywithoutchangingitsobservablebehavior

Refactoring- Symptoms• CodingStyleandNameConventionsnotfollowed• ProperCommentingnotfollowed• Duplicatedcode(clearlyawaste).• Longmethod(excessivelylargeorlongmethodsperhapsshouldbesubdivided

intomorecohesiveones).• Largeclass(sameproblemaslongmethod).• Switchstatements(inobject-orientedcode,switchstatementscaninmost

casesbereplacedwithpolymorphism,makingthecodeclearer).• Featureenvy,inwhichamethodtendstousemoreofanobjectfromaclass

differenttotheoneitbelongs.• Inappropriateintimacy,inwhichaclassreferstoomuchtoprivatepartsof

otherclasses.=> Anyofthesesymptoms(andmore)willindicatethatyourcodecanbeimproved.Youcanuserefactoringtohelpyoudealwiththeseproblems.


UniversityCollegeofSoutheastNorwaywww.usn.no

E-mail:[email protected]:http://home.hit.no/~hansha/