t-76.5613 software testing and quality assurance plann… · testing activities to identify the...
TRANSCRIPT
HELSINKI UNIVERSITY OF TECHNOLOGY
T-76.5613 Software Testing and Quality Assurance
10.10.2007
Test Planning
Juha ItkonenSoberIT
2Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Outline
Test planning, purpose and usage of a test plan
Topics of test planning
Exercise
References:
IEEE Std 829-1998, IEEE Standard for Software Test Documentation, IEEE, 1998.
C. Kaner, J. Bach, and B. Pettichord, Lessons Learned in Software Testing: A Context- Driven Approach, John Wiley & Sons, 2002.
3Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Great testing challenges
Agency: All testers act on behalf of some client. How do testers faithfully represent them?
Evaluation: Not all failures are obvious. When a bug occurs, how do testers recognize it?
Coverage: There's too much to test. How do testers decide what to test and what to ignore?
Logistics: Testing must pace and respond to the project. How do testers apply their efforts at the right time and place to beuseful?
Reporting: Testing is about information. How do testers report their work in such a way as the information will be received andused?
Stopping: Even though there's too much to test, not all testing is worthwhile to perform. How do testers know when enough's enough?
Bach. 2002Test planning decisions
4Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test Planning – different levels
Test Policy
Test Strategy
High Level Test Plan
Detailed Test PlanDetailed Test PlanDetailed Test PlanDetailed Test Plan
Master Test Plan
Company or product level
Project
Test levels (one for each within a project, e.g. unit, system, ...)
Master test plan
Acceptance test plan
System test plan
Unit and integration test plan
Evaluation plan
Number of needed levels depends on context (product, project, company, domain, development process, regulations, …)
5Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test Documentation by IEEE Std 829-1998
The topic of this lecture
6Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Purpose of test planning
To prescribe the scope, approach, resources, and schedule of testing activities
To identify the items being tested, the features to be tested, the testing tasks to be performed
To assign the responsible for each task
To identify the risks associated with the plan
To make conscious decisions of all these things
Communicating, not just recording
Test planning is part of project planningFocuses on similar issuesIs applied on similar level of detailProject planning vs. Technical design <-> Test planning vs. Test (case) design
7Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Some functions of a test plan
Support quality assessment that enables wise and timely product decisions.
Support the initiation and organization of the test project, including preparations, staffing, delegation of responsibilities, facility acquisition, task planning and scheduling.
Support daily management and evaluation of the test project and test strategy.
Describe and justify the test strategy in relation to product requirements and product risks. Promote awareness of the benefits and limitations of the test strategy.
Support effective coordination, collaboration, and other relations among members of the test team and between the test team and the rest of the project.
Identify and manage any risks or issues that may impact the project.
Specify the deliverables of the test project and the delivery process.
Record historical information in support of process audits, process improvement, and future test projects.
8Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Topics of test planning
what needs to be planned
9Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Checklist for test planning
1. Overall test objectives (why)
2. What will and won’t be tested (what)
3. Test approach (how)Test phasesTest strategy, methods, techniques, …Test case organization and trackingDefect reportingMetrics and statistics
4. Resource requirements (who)Tester assignments and responsibilitiesTest environments
5. Test tasks and schedule (when)
6. Risks and issues (what if)
10Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
1. Overall test objectives
Why are we testing?
The quality goals of the project/product?
What are the most important quality attributes?
What is to be achieved by the testing?
What do we know after the testing?
What are the main risks that testing focuses?
11Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Clarify your mission Examples of testing goals
Find important problems fast
Assure customer that the quality is good enough
Achieve certain quality attributes
Perform a comprehensive quality assessment
Conform to a specific (quality) standard
Maximize testing effectiveness
Minimize testing time or cost
Rigorously conform to certain processes
Satisfy particular stakeholders
12Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
2. What will and won’t be tested (scope)
Identify components and features of the software under test
High-enough abstraction level
Prioritize
Both functional and non-functional aspects
Consider time, resources and risks
Everything can’t be tested and everything that is tested can’t be tested thoroughly
Identify separately components and features that are not tested
Provide reasoning
Makes the prioritization decisions explicit and visible
Steers towards the most important things
13Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Analyze the product to know what to test
Get to know the product and the underlying technology
Learn how the product will be usedFind out how the product is built
Your testing will become better as the project proceeds, because you will be more of a product expert
Ask yourselfDo designers think you understand the product?Can you visualize the product and predict behavior?Are you able to produce the test data?Can you configure and operate the product?Do you understand how the product will be used?Are you aware of gaps or inconsistencies in the design?Have you found implicit specifications as well as explicit?
14Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Analyze the product What and how
What
Users
Use cases
Structure
Functions
Data
Platforms
…
How
Review product and project documentation
Interview designers and users
Compare with similar products
Perform exploratory testing
…
15Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Splitting features into sub-features
Feature to betested 1
Importantsubfeature 1
Importantsubfeature 2 Subfeature 3 Subfeature 4
Find the most important features to test
Not necessarily the same prioritization as for implementation
Requirement importance and risks
16Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test environments
Identification of test environments
Hardware, software, os, network, ...
Prioritization and focusing test suites on each
Number of combinations can be hugeRegression testing in different environments
Scheduling implications
Test lab
Different hardware and software platformsCleaning the machinesSetting up the test dataMoving from platform to anotherPeople vs. hardware needs
17Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
3. Test approach
How testing is performedHow quality goals are achieved and what is the role of testing
Test phasesTest phases are part of development phasesPlanning is also a phaseDefine entry and exit criteria
Test levels
Types of testing, methodsBlack-box vs. White-box Testing techniques
Test completion criteria and pass/fail criteriaBy feature/module if necessary
Reporting and defect management procedures
Communicating and collaborating with development
Tools and automation
Outsourcing
18Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Analyze product risk
How might this product fail in a way that matters?
At first you’ll have a general idea, at best
Your test strategy and your testing will become better as the project proceeds
because you’ll learn more about the failure dynamics of the product.
Ask yourself
Will you be able to detect all significant kinds of problems, should they occur during testing?
Do you know where to focus testing effort for maximum effectiveness?
Can the designers do anything to make important problems easier to detect, or less likely to occur?
19Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Design the test strategy
What can you do to test rapidly and effectively based on the best information you have about the product?
Make the best decisions you can, up front
but let your strategy improve throughout the project
Ask yourself
Is everything in the test strategy necessary?
Can you actually carry out this strategy?
Is the test strategy too generic; could it just as easily apply to any product?
Is there any category of important problem that you know you are not testing for?
Has the strategy made use of available resources?
20Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test cases
How test cases are created and documented
Storing, using, maintaining
Testware architecture
What is needed in addition to the test cases?Stubs, drivers, test data, expected result sets and outputs, test automation scripts, …
Plan how test cases are organized
Prioritization, regression testing
Test suites
Requirements traceability
21Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Organizing test cases
Prioritizing tests
Positive / negative
The most severe failures
The most likely faults
End-user prioritizing the requirements
Most faults in the past
Most complex or critical
…
Create test suites
Test-to-Pass (Positive testing)
Test-to-Fail (Negative testing)
Smoke test suite
Regression test suite
Functional suites
Different platforms
Priorities, different passes
…
22Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
How to apply prioritization
Run first priority tests first
… but think prioritization more like distribution of efforts thanexecution order
Especially if the prioritization is based on requirement prioritiesassigned by functionality or module
Use most of your time for high priority tests
Use some of your time/resources for low priority testsAvoid omitting low priority tests altogetherHigh priority tests might take more time than plannedRisk of missing critical problems in low priority functionality
Prioritization can be wrong!
H HL
t
L H L H = high priority testsL = low priority tests
23Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Deliverables
What documents are produced
Why?
When?
How test plans, designs, and cases are documented?
How test results are logged and reported?
How defects are managed?
24Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Defect reporting
Define process to manage defectsHow defects are managed?By whom?
Bug lifecycleDefect statesPrioritiesSeveritiesWho decides? How? When?Classification
Defect-tracking systemreportingdata query and retrieval (statistics)supports the defect resolution process
What data and metrics are collected?And for what purpose?
25Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test completion criteria
Always risks involved
Risk of leaving major defects in the systemRisk of spending too much time and moneyRisk of losing market position and sales if extend testing
Weakest criteria
Deadline – When all the time and resources are used
Setting the test completion criteria (stop-test criteria) is part of test planning
When testing (or testing phase) is completed?Based, e.g., on coverage of testing, defect rates, defect severities, schedule, …
All the planned tests are executed (and passed)All specified coverage criteria have been metThe goals for number of detected defects have been reachedDefect detection rates have fallen below a specified level
26Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Release Criteria
Testing should be visible in release criteria
Release criteria is not the same as test completion criteria
Release criteria is business decision and it is not test team’s or testers’ responsibility to make release decisions
Testing provides useful information on the quality of the product
Testing should be visible in release criteriaTest resultsDefect metrics and reports
27Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
4. Resource requirements
PeopleHow many, what experience, what expertise? Full-time, part-time, contracts, students? Responsibilities
EquipmentComputers, test hardware, printers, tools.
Office and lab spaceWhere will they be located? How big will they be? How will they be arranged?
Tools and documentsWord processors, databases, custom tools. What will be purchased, what needs to be written?
Outsource companiesWill they be used? What criteria will be used for choosing them? How much will they cost?
Miscellaneous suppliesDisks, phones, reference books, training material. Whatever elsemight be needed over the course of the project.
Define responsibilities
Identify limited/critical resources
Location and availability
28Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Plan also high-level responsibilities
Not just allocating individual tasks for people
Example:
Mark Formatting functionality
Sarah Layout functionality
Dennis Editing commands
Louise Configuration and compatibility
Dave GUI: Usability, appearance, accessibility
Paula Test data and regression testing
Jack Performance and load tests
29Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
5. Testing tasks and schedule
Work Breakdown Structure (WBS)Testing tasks can be divided by e.g.
Areas of the softwareTestable featuresPhases, levels, and types of testing
Assigning task responsibilitiesMapping testing to overall project schedule
Both duration and effortBuild schedule
Number of test cyclesEnvironmentsRegression tests
ReleasesExternal links, i.e. Beta testing
Consider using relative dates
30Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
The idea of relative dates
Task Start Duration
Test planning When initial requirement specification available
2 weeks
Test case design Test plan complete 12 weeks
Test round #1 Code complete build 6 weeks
Test round #2 Beta build 6 weeks
Test round #3 Release build 4 weeks
31Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Estimating testing is not different
“Estimate” can be
Estimate – an approximate calculation or judgment, based on professional understanding of an expertQuote / quotation – fixed price by a contractorTarget – A desired time or cost (for political or commercial reasons)
Estimating any job involves the following
Identify tasksWho should perform the taskWhat resources, what skillsHow much effort for each taskWhen should the task start and finishPredictable dependencies
Task precedence (design test before running it)Technical precedence (add & display feature before edit feature)
32Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Estimating testing is different
Testing is not an independent activity
Delivery schedules for testable items can changeTest environments are critical
Test iterations (rounds)
Testing should find faultsHow long does it take to find a fault?How many faults will be found?
Faults need to be fixedHow long does it take to fix a fault?After fixed, how much testing needs to be repeated?
How many times does this happen? - Two? Twelve?
It takes time to report defects
The more fault reports you write, the less testing you are able to do
33Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Ways to estimate test effort
Guessing
Based on the size of the project / product
COCOMO and heuristics
Number of requirements or modules
Consensus of experts (Delphi)
Previous test effort
In-project experience through incremental planning
Detailed work breakdown of testing tasks
Authorized deadlines
Historical data helps estimating regardless of the method
34Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
6. Risks and issues
Risky areas of the test project and plan – not the product
Critical resources
Dependencies
For each risk
Contingency plans – what if the risk realizesConsequences
Mitigation – how to make the risk smallerProbability and consequences
Prevention – how to prevent the risk altogether
35Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
7. Share the plan
You are not alone
The test process must serve the project
So, involve the project in your test planning process
At least chat with the key members of the team to get their perspective and implicit acceptance to pursue your plan
Is the project team paying attention to the test plan?
Does the project team, especially first-line management, understand the role of the test team?
Does the project team feel that the test team has the best interests of the project at heart?
36Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Share the plan Ways to share
Actively solicit opinions about test plan
Help the developers understand how their actions impacts testing
Get designers and developers to review and approve testing materials
Get people to review the plan in pieces
Improve review ability by minimizing unnecessary text in test plan documents
Record and track agreements
37Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test plan heuristics – is this a good test plan
Important problems fast
Focus on risk
Maximize diversity
Avoid over-scripting
Test to the intent
We are not alone
Promote testability
Test plans aren’t generic
Content or nothing
Don’t program people
The test schedule depends
Avoid the bottleneck
Rapid feedback
Testers aren’t the only testers
Review documentationHeuristic is a rule of thumb, an educated guess, a rule for action.
Kaner, Bach, Pettichord. Lessons Learned in Software Testing. 2002
38Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test plan quality criteria
Usefulness: Will the test plan effectively serve its intended functions?
Clarity: Is the test plan self-consistent and sufficiently unambiguous?
Accuracy: Is the test plan document accurate with respect to any statements of fact?
Adaptability: Will it tolerate reasonable change and unpredictability in the project?
Efficiency: Does it make efficient use of available resources?
Usability: Is the test plan document concise, maintainable, and helpfully organized?
Compliance: Does the test plan meet externally imposed requirements?
Foundation: Is the test plan the product of an effective test planning process?
Feasibility: Is the test plan within the capability of the organization that must use it?
Source: Kaner, Bach, Pettichord. Lessons Learned in Software Testing. 2002
39Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Exercise: Contents of a test plan
Compare the two sample test plans
Burnstein’s Sample test planCollege Course Scheduler
IEEE Std 829 Sample test planCorporate Payroll System
40Juha Itkonen - 2007SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Exercise: Contents of a test plan
Focus to the following sectionsIntroductionFeatures to be testedFeatures not to be testedApproachItem pass/fail criteriaTasks and schedule
What are the good parts?why?
What are the weaknesses and problems?
How would you improve the plan?
Which one of the plans is better?documents better the planning decisionsgives guidance and supports the actual execution and management of the system testing project?