Download - Software Testing Fundamentals part I
HELSINKI UNIVERSITY OF TECHNOLOGY
T-76.613 Software Testing and Quality Assurance
Lecture 1, 13.9.2005
Software Testing Fundamentalspart I
Juha ItkonenSoberIT
2Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Contents
Why do we need testing?What is software testing?Quality and quality assurance
3Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Why do we need testing?
4Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Why do we need testing?
We need testing to reduce and mitigate riskAll software has faultsWe prefer to invest in testing in order to find the most important faults before the software is released to live operation
Failures that occur during live operation are usually more expensive to deal with than failures than occur during testing
Higher the risks, more we need to testNo risk – no test
5Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Someone will always test your software
Customer or UserCompetitorA government inspector, an industrial analyst, an insurance inspectorAn editor of a wide-spread magazineChannel resellerSystems integratorPotential partnerPotential subcontractorPotential employeeSoftware tester
…and find the nastiest bugs…
6Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Software testing case studies (1/3)
Disney’s Lion King CD-ROM, 1994-199526, December customer support phones began to ring...Multimedia CD was tested only on a few systems – did not work on most common home pc:sA lot of bad publicity, newspaper stories and TV news
Intel Pentium floating-point division bug, 1994(4195835 / 3145727) * 3145727 - 4195835 = 0Problem was detected in Intel’s own tests before the chip was released but the management decided that the problem was not severe or likely enough to fix or even publish itIn the end $400 million costs of replacing faulty chips
7Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Software testing case studies (2/3)
NASA Mars Polar Lander, 1999 Multiple testing teams – one team tested contact switch that turn the landing thrusters off after landing, and another team tested that the leg opening mechanisms worked.When the legs snapped open the vibration caused the contact switch to turn off the thrusters 1800 feet above the Mars surface -> the Polar Lander smashed to the ground.
Patriot missile defense system, 1991System failed to defend against several Iraqi Scud missiles, one of which killed 28 U.S. soldiers.Analysis found a software bug – a small timing error accumulated to the point that after 14 hours of operation the tracking system was no longer accurate.
8Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Software testing case studies (3/3)
The Y2K Bug, early 1970sA programmer was programming a payroll system for his companyThe computer had very little memory for storage and the programmer decided to use only 2 digits to store years, because this would save him a lot of memory in the system that heavily relies on date processingHe considered briefly the problems that will occur when year hits 2000, but decided that this system will surely be replaced by then
It’s estimated that several hundred billion dollars were spent to replace or update this kind of systems to fix potential year 2000 failures
9Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Why do defects occur in software?
Software is written by human beingsWho know something, but not everythingWho have skills, but aren’t perfectWho don’t usually use rigorous methodsWho do make mistakes (errors)
Under increasing pressure to deliver to strict deadlines
No time to check, assumptions may be wrongSystems may be incomplete
Software is complex, abstract and invisibleHard to understandHard to see if it is complete or working correctlyNo one person can fully understand large systemsNumerous external interfaces and dependencies
10Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Sources of defects
EducationDevelopers does not understand well enough what he or she is doingLack of proper education leads to errors in specification, design, coding, and testing
CommunicationDevelopers do not know enoughInformation does not reach all stakeholdersInformation is lost
OversightOmitting to do necessary things
TranscriptionDeveloper knows what to do but simply makes a mistake
ProcessProcess is not applicable for the actual situationProcess places restrictions that cause errors
11Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
What do software faults cost?
Huge sumsAriane 5 ($7 billion)Mariner space probe to Venus ($250 million)American Airlines seat booking system ($50 million)Intel pentium floating point division bug ($400 million)Y2K (estimated several hundred billion dollars worldwide)
Very little or nothing at allminor inconvenienceno visible or physical detrimental impact
Software is not ”linear”small input may have very large effect
”The national annual costs of an inadequate infrastructure for software testing is estimated to range from $22.2 to $59.5 billion.” NIST Final Report: The Economic Impacts of Inadequate Infrastructure for Software Testing. 2002
12Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Safety-critical systems
Software faults can cause death or injuryRadiation treatment kills patients (Therac-25)Aircraft crashes Industrial process control softwareHardware controllers…
13Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Example from Finland
”Helsingin terveysviraston tietojärjestelmä pysähtyi taas
Odotusajat venyivät, lääkärit pelkäävät potilasturvallisuudenvaarantuvan.Helsingin terveysviraston potilastietojärjestelmä saatiintoimimaan keskiviikkoaamuna, mutta jo iltapäivällä se pysähtyitaas. Uuden Pegasos-tietojärjestelmän toimintavaikeudetalkoivat viime torstaina ja tämän viikon alussa se jähmettyikokonaan. Tietojärjestelmän ja tietokannan toimittajan asiantuntijatkorjasivat vikaa koko tiistai-illan, ja keskiviikkoaamuna alkoijärjestelmän vähittäinen käyttöönotto. ”
Helsingin Sanomat 6.2.2003
14Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
What is Software Testing?
15Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
What is software testing?
Finding defectsTrying to break the systemFinding and reporting defectsDemonstrating correct functionalityDemonstrating incorrect functionalityDemonstrating robustness, reliability, security, maintainability, …Measuring performance, reliability, …Evaluating and measuring qualityProving the software correctExecuting pre-defined test casesAutomatic error detection…
16Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
The growth of software testing
- 1956 Debugging oriented periodTesting was not distinguished from debugging
1957 - 1978 Demonstration oriented periodGoals to demonstrate that software satisfies its specification
1979 - 1982 Destruction oriented periodGoal to detect implementation faults
1983 - 1987 Evaluation oriented periodGoal to detect requirements, design, and implementation faults
1988 - Prevention oriented periodGoal to prevent requirements, design, and implementation faults
(Gelperin and Hetzel. 1988. The Growth of Software Testing. Communications of the ACM)
17Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Definition of Software Testing(Glenford Myers 1979)
Testing is the execution of programs with the intent of finding defects.
18Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Definition of Software Testing(Ilene Burnstein 2002)
Testing is a the process of exercising a software componentusing a selected set of test cases,with the intent of revealing defects and evaluating quality.
(Testing can be described as a process used for revealing defects in software, and for establishing that the software has attained a specified degree of quality with respect to selected attributes.)
19Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Definition of Software Testing(Pol & Veenendal 2002)
Testing is a process of planning, preparation, execution and analysing, aimed at establishing the characteristics of an information systemand demonstrating the difference between actual and required status.
20Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Definition of Software Testing(Cem Kaner, 2004)
Software testing is a technical investigation of a product, i.e.an empirical searchfor quality-related informationof value to a project’s stakeholders
21Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Context-driven school of testing
1. The value of any practice depends on its context. 2. There are good practices in context, but there are no best
practices. 3. People, working together, are the most important part of
any project's context. 4. Projects unfold over time in ways that are often not
predictable. 5. The product is a solution. If the problem isn't solved, the
product doesn't work. 6. Good software testing is a challenging intellectual
process. 7. Only through judgment and skill, exercised cooperatively
throughout the entire project, are we able to do the right things at the right times to effectively test our products.
http://www.context-driven-testing.com/
22Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Testing is an integral part of development
Often, testing is seen as some separate, last phase of software development process
That can be outsourced to separate testing team That only deeds to be done just before the release – if there is any time
Testing can not be separated from rest of the software development
Testing is much more than the final acceptance testing phase
Testing has to be involved from the beginningTesters can, and should, contribute in each phase of the software development life-cycle
Testing – is not a phase
23Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Reputation of testing and testers
Traditionally testers and their job have not been respected
Bad programmers and poor designers are put into testing teamThese people do bad job in testing -> their job is not respected
Testing is very creative work that requires as skilled and experienced professionals as any job, in order to be done wellActually, good testing requires probably more skill and experience than programming – the skills, however, are different
24Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
We need professional testers
Developers can’t find their own defectsThey have no motivation for testingThey have no skills for testingThey have no time for testing
Testers have the tester’s mindsetGoal is to find defects – break the softwareGet the defects fixed together with the developersTesters are independent verifiers, they can test the software from the customer’s or user’s viewpointTesters are professional – they have the skills, tools, and experience
25Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Some skills of a good tester
Tester’s (destructive) attitude and mindsetExcellent communication skillsAbility to manage many detailsKnow the different testing techniques and strategies and know when and how to apply themBe familiar with the domain the software is targetedComprehensive knowledge of the software engineering discipline
Knowledge and experience of how software is specified, designed, and developedKnowledge of fault and failure types
26Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Are testers destructive?
Finding defects is a destructive taskRequires destructive attitude and mindset
Testers’ goal is to help the development team develop high quality software
The ultimate goal is to construct software with required level of qualityTesters job is not to be destructive
Destructive attitude towards software during testing tasks – constructive, when reporting, towards people and the development project
27Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Software Quality and Quality Assurance
28Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Describe a high quality software system
Defect freeHigh sales figuresFast to learnNice graphicsEasy to install and upgradeRobustAdaptableCheapSimpleSmall and compactExtensibleOpen-sourceFit-for-purposeFast upgrade cycleOpen accessMaintainableAppeals majority
Feature richAward winning productFast to useLow system requirementsReliable customer supportCoolCompatibleExpensiveBig and complicatedLarge and scalableStandardizedReliable vendorMultipurposeNever crashesSecureFast time-to-marketPerfect for target segment
29Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Definition of software quality
Quality: (1) The degree to which a system, component or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations.
(IEEE Standard Glossary of Software Engineering Terminology [IEEE610.12])
Quality is value to some person(s)(Gerald M. Weinberg, 1992, “Quality Software Management”)
30Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Users’ viewpoint differs from yours
All stakeholders have different viewpoints to qualityCustomerEnd userProgrammerProject managerTester
Note: bug free product can be unacceptable to the user
The goal of a software system is to solve customers problemRequirements should be validated Usability needs to be consideredDocumentation should be testedCustomer feedback process needs to be working
31Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Process quality and product quality
Quality in process -> quality in productNumerous models, but has not been proven to be always true
Project: instantiated process
Quality according to ISO 9126Process quality contributes to improving product quality, which in turn contributes to improving quality in use
Process
Project
32Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
The McCall quality model
Product TransitionProduct Revision
Product Operations
PortabilityReusability
Interoperability
CorrectnessReliabilityEfficiencyIntegrityUsability
MaintainabilityFlexibilityTestability
33Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
ISO 9126 Quality characteristics
ISO/IEC9126
Functionality
ReliabilityPortability
Maintainability
Efficiency
UsabilityThe capability tobe understood,
learned, used andattractive to user.
The capability to provide appropriateperformance, relative to amount of
resources used, under stated conditions.
The capability tobe modified.
The capability to betransferred from one
environment toanother.
The capability to provide functionsthat meet stated or implied needs.
The capabilityto maintain a
specified levelof performance.
34Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
ISO 9126 Quality attributes
FunctionalitySuitabilityAccuratenessInteroperabilitySecurity
ReliabilityMaturityFault toleranceRecoverability
UsabilityUnderstandabilityLearnabilityOperabilityAttractiveness
EfficiencyTime behaviorResource behavior
MaintainabilityAnalyzabilityChangeabilityStabilityTestability
PortabilityAdaptabilityInstallabilityConformanceReplaceability
35Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Focus on different quality attributes in testing
Traditionally emphasis on functionalityGrown emphasis on
UsabilityEfficiencyScalabilitySecurity
Everything has to be accessible from public webInternal quality attributes important for products with long life cycle
MaintainabilityPortabilityTestability…
36Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Good Enough Quality
Defined by James Bach IEEE Computer, 1997, vol. 30, no. 8, pp. 96-98.
To claim that any given thing is good enough is to agree with all of the following propositions:
It has sufficient benefitsIt has no critical problemsThe benefits sufficiently outweigh the problemsIn the present situation, and all things considered, further improvement would be more harmful than helpful
37Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Quality Assurance
Application of sound technical methods and toolsFormal technical reviews and inspectionsSoftware testingEnforcement of standardsDocumentationControl of changeExtensive measurementRecord keeping and reporting of the process
QA vs. TestingDifferent attitudeQA is focused to build in quality and prevent defects to ever happen
constructiveTesting is focused to find defects and show the level of quality
destructive
(1) A planned and systematic pattern of all actions necessary toprovide adequate confidence that an item or product conforms to established technical requirements.(2) A set of activities designed to evaluate the process by which products are developed or manufactured.
38Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Quality Assurance 1. A planned and systematic pattern of all actions
necessary to provide adequate confidence that an item or product conforms to established technical requirements.
2. A set of activities designed to evaluate the process by which products are developed or manufactured.
Application of sound technical methods and toolsFormal technical reviews and inspectionsSoftware testingEnforcement of standardsDocumentationControl of changeExtensive measurementRecord keeping and reporting of the process
39Juha Itkonen, 2005SoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
QA vs. Testing
Different attitudeQA is focused to build in quality and prevent defects to ever happen
Constructive
Testing is focused to find defects and show the level of quality
Destructive
Testing can be seen as a part of QAQA includes
Measuring and trackingProcess improvementAssessing quality and reacting to variances