Software TestingSoftware Testing
Paratus Systems, Aundh
AgendaAgenda
• Principles of Software Testing
• Testing Strategies
• Advanced Techniques and Tools
• Assignments
Paratus Systems, Aundh
2
Principles of Software TestingPrinciples of Software Testing
Testing DefinitionTesting Definition• Testing
• It is the process of exercising or evaluating a system or system
component by manual or automated means to verify that it
satisfies specified requirements – [IEEE 83a]
• Process
• Sequence of steps performed for a given purpose [IEEE]
• Software Process
• A set of activities, methods, practices and transformations that
people use to develop and maintain software and associated
products [SEI-CMM]
Paratus Systems, Aundh
3
Principles of Software TestingPrinciples of Software Testing
Testing ObjectivesTesting Objectives• Primary:
• Execute a program with a intent of finding errors to
• Determine whether system meets specification
• Determine whether system meets user’s needs
• Secondary:
• Continuously improve the testing process
Paratus Systems, Aundh
4
Principles of Software TestingPrinciples of Software Testing
Misunderstanding about TestingMisunderstanding about Testing• Testing is debugging
• Testing is not the job of a programmer
• If programmers are more careful testing would be unnecessary
• Testing never ends
• Testing activities start only after the coding is complete
• Testing is not a creative task
Paratus Systems, Aundh
5
Myths and Facts about testingMyths and Facts about testing
Paratus Systems, Aundh
Testing is a single phase in SDLC.
Testing is easy.
Development is worth more than testing.
Complete testing is POSSIBLE.
Anyone can be a Tester.
It’s a continuous process.
Test cases are designed manually.
Testing works throughout SDLC.
All inputs & lines cannot be tested.
Training & skills are required.
6
Principles of Software TestingPrinciples of Software Testing
Theory vs RealityTheory vs Reality
• Theory:• Need Sufficient Time• Base-lined and Frozen Requirements• Automation Testing• Have trained testers: test process, business flow,…
• Reality:• Time Pressures to keep delivery date• Continuously updated requirements• Don’t know when to stop: Update Requirements
continuously (CRs) -> Testers update TC and re-tests• Manual Testing• Lack of dedicated software test environment: share, steal• Inexperienced testers without appropriate training
Paratus Systems, Aundh
7
Principles of Software TestingPrinciples of Software Testing
• The goal of testing is to find defects before customers find them out• Exhaustive testing is not possible; program testing can only show the presence
of defects, never their absence• Testing applies all through the software life cycle and is not an end-of-cycle
activity• Understand the reason behind the test• Test the tests first• Tests develop immunity and have to be revised constantly• Defects occur in convoys or clusters and testing should focus on these
convoys• Testing encompasses defect prevention• Testing is a fine balance of defect prevention and defect detection• Intelligent and well-planned automation is key to realizing the benefits of testing• Testing requires talented, committed people who believe in themselves and
work in teams
Paratus Systems, Aundh
8
Principles of Software TestingPrinciples of Software Testing
Verification and ValidationVerification and Validation
Paratus Systems, Aundh
9
Criteria Verification Validation
Definition The process of evaluating work-products (not the actual final product) of a development phase to determine whether they meet the specified requirements for that phase.
The process of evaluating software during or at the end of the development process to determine whether it satisfies specified business requirements.
Objective To ensure that the product is being built according to the requirements and design specifications. In other words, to ensure that work products meet their specified requirements.
To ensure that the product actually meets the user’s needs, and that the specifications were correct in the first place. In other words, to demonstrate that the product fulfills its intended use when placed in its intended environment.
Question Are we building the product right?
Are we building the right product?
Evaluation Items
Plans, Requirement Specs, Design Specs, Code, Test Cases
The actual product/software.
Activities •Reviews•Walkthroughs•Inspections
•Testing
10
1. Verification is a static practice of verifying documents, design, code and program.
1. Validation is a dynamic mechanism of validating and testing the actual product.
2. It does not involve executing the code. 2. It always involves executing the code.3. It is human based checking of documents and files.
3. It is computer based execution of program.
4. Verification uses methods like inspections, reviews, walkthroughs, and Desk-checking etc.
4. Validation uses methods like black box (functional) testing, gray box testing, and white box (structural) testing etc.
5. Verification is to check whether the software conforms to specifications.
5. Validation is to check whether software meets the customer expectations and requirements.
6. It can catch errors that validation cannot catch. It is low level exercise.
6. It can catch errors that verification cannot catch. It is High Level Exercise.
7. Target is requirements specification, application and software architecture, high level, complete design, and database design etc.
7. Target is actual product-a unit, a module, a bent of integrated modules, and effective final product.
8. Verification is done by QA team to ensure that the software is as per the specifications in the SRS document.
8. Validation is carried out with the involvement of testing team.
9. It generally comes first-done before validation.
9. It generally follows after verification.
Audit-Quality check Inspection-Author, moderator, reader,
tester Walkthrough Review
11
Principles of Software TestingPrinciples of Software Testing
Vtest ModelVtest Model
Paratus Systems, Aundh
12
Principles of Software TestingPrinciples of Software Testing
Defect ManagementDefect Management
Paratus Systems, Aundh
13
Testing StrategiesTesting Strategies
• It includes identifying
• What type of testing would you use for testing the functionality?
• What are the configurations or scenarios for testing the features?
• What integration testing would you do to ensure these features work
together?
• What localization validations would be needed?
• What “non-functional” tests would you need to do?
Paratus Systems, Aundh
14
Testing StrategiesTesting Strategies
Unit TestingUnit Testing
• It is a level of the software testing process where individual units/components
of a software/system are tested
• The purpose is to validate that each unit of the software performs as designed
• It is a method by which individual units of source code are tested to determine if
they are fit for use
• It is concerned with functional correctness and completeness of individual
program units
• It is typically written and run by software developers to ensure that code meets
its design and behaves as intended
• Its goal is to isolate each part of the program and show that the individual parts
are correct
Paratus Systems, Aundh
15
Testing StrategiesTesting Strategies
Integration and System TestingIntegration and System Testing
• Testing the interaction between the modules and interaction with other systems
externally is called integration testing
• Integration testing starts when two of the product components are available and
ends when all component interfaces have been tested
• Integration testing is both a type of testing and a phase of testing
• The testing conducted on the complete integrated products and solutions to
evaluate system compliance with specified requirements on functional and
nonfunctional aspects is called system testing
• It is the first level of software testing where the application is tested as a whole
• Non-Functional aspects include Performance/Load testing, Scalability testing,
Reliability testing, Stress testing, Interoperability testing, Localization testing
Paratus Systems, Aundh
16
Testing StrategiesTesting Strategies
Acceptance TestingAcceptance Testing
• Acceptance testing is done by the customer or by the representative of the
customer to check whether the product is ready for use in the real-life
environment
• Following test cases can be included for acceptance testing• End-to-end functionality verification• Domain tests• User scenario tests• Basic sanity tests• New functionality• A few non-functional tests• Tests pertaining to legal obligations and service level agreements• Acceptance test data
Paratus Systems, Aundh
17
Testing StrategiesTesting Strategies
Alpha and Beta TestingAlpha and Beta Testing
• Alpha testing performed at developers site but not by developing team
• Testing a software product which is not the final version.
• This software does not have to necessarily contain the full functionality
required for an application, however core functionality to accept input an
generate output is required.
• Here virtual environment will be created to simulate real time environment.
• Beta or field testing is done by customers at their locations
• Beta Testing is last stage of testing where a product is sent outside the
company or offer the product for free trial download
Paratus Systems, Aundh
18
Testing StrategiesTesting Strategies
Performance TestingPerformance Testing
• The testing performed to evaluate the response time, throughput and utilization
of the system, to execute its required functions in comparison with different
versions of the same product(s) or different competitive product(s) is called
performance testing
• It is done to ensure that a product• Processes the required number of transactions in any given interval
(throughput)• Is available and running under different load conditions (availability)• Responds fast enough for different load conditions (response time)• Delivers worthwhile return on investment for the resources – hardware
and software – and deciding what kind of resources are needed for the
product for different load conditions (capacity planning)• Is comparable to and better than that of the competitors for different
parameters (competitive analysis and bench marking)
Paratus Systems, Aundh
19
Testing StrategiesTesting Strategies
Security TestingSecurity Testing
• Security testing is a process to determine that an information system protects
data and maintains functionality
• To check whether there is any information leakage
• To test the application whether it has unauthorized access and having the
encoded security code
• To find out all the potential loopholes and weakness of the system that can
cause loss of important information
• To identify the vulnerabilities and subsequently repair them
• It helps in improving the current system and ensuring that the system will work
for longer time
Paratus Systems, Aundh
20
Testing StrategiesTesting Strategies
White Box and Black Box TestingWhite Box and Black Box Testing
• White box testing is a way of testing the external functionality of the code by
examining and testing the program code that realizes the external functionality
• It is also known as clear box, or glass box or open box testing
• It takes into account the program code, code structure and internal design flow
• Black box testing is done without the knowledge of the internals of the system
under test
• It is done based on requirements
• It addresses the stated requirements as well as implied requirements
• It encompasses the end user perspectives
• It handles valid and invalid inputs
Paratus Systems, Aundh
21
Testing StrategiesTesting Strategies
Basis path testingBasis path testing• Basis path testing, a structured testing or white box testing technique used for designing
test cases intended to examine all possible paths of execution at least once. • Creating and executing tests for all possible paths results in 100% statement coverage
and 100% branch coverage.• Example: Function fn_delete_element (int value, int array_size, int array[]){
1 int i;location = array_size + 1;
2 for i = 1 to array_size3 if ( array[i] == value )4 location = i; end if; end for;
5 for i = location to array_size6 array[i] = array[i+1];end for;7 array_size --;
}
Paratus Systems, Aundh
22
Testing StrategiesTesting Strategies
Basis path testingBasis path testing
Paratus Systems, Aundh
• Steps to Calculate the independent paths• Step 1 : Draw the Flow Graph of the Function/Program under consideration as
shown below:
• Step 2 : Determine the independent paths.• Path 1: 1 - 2 - 5 - 7 • Path 2: 1 - 2 - 5 - 6 - 5 - 7 • Path 3: 1 - 2 - 3 - 2 - 5 - 6 - 5 - 7• Path 4: 1 - 2 - 3 - 4 - 2 - 5 - 6 - 5 - 7 23
Testing StrategiesTesting Strategies
Equivalence testingEquivalence testing
• Equivalence partitioning is a software testing technique that involves identifying a small
set of representative input values that produce as many different output conditions as
possible
• It is useful to minimize the number of test cases when the input data can be divided into
distinct sets, where the behavior or outcome of the product within each member of the set
is the same
• Example: Identify equivalence classes for this requirement “If a pupil has total score
>=75, he will pass the exam, otherwise will fail (total score is an integer)”
• Search Function Requirement: User fills in search field then clicks search. The system will
find all search strings in the document and highlight them. If none is found, message on
form to indicate no results found (search like Notepad)
• Savings Account: 3% rate of interest is given if the balance in the account is in the range
of $0 to $99.99, 5% rate of interest is given if the balance in the account is in the range of
$100 to $999.99, and 7% rate of interest is given if the balance in the account is $1000 and
above, identify three valid equivalence partitions and one invalid partition.
Paratus Systems, Aundh
24
Testing StrategiesTesting Strategies
Graph Based testingGraph Based testing
Paratus Systems, Aundh
25
• Graph based testing methods are applicable to generate test cases for state machines
such as language translators, workflows, transaction flows and data flows
Testing StrategiesTesting Strategies
Test MetricsTest Metrics• Metrics derive information from raw data with a view to help in decision making.• Metrics can be classified as product metrics and process metrics
• Product metrics can be further classified as • Project metrics
• Effort variance, Schedule variance, Effort distribution• Progress metrics
• Testing Defect Metrics• Defect find rate, Defect fix rate, Outstanding defects rate, Priority
outstanding rate, Defects trend, Defect classification trend, Weighted
defects trend, Defect cause distribution• Development Defect Metrics
• Component-wise defect distribution, Defect density and defect removal
rate, Age analysis of outstanding defect, Introduced and reopened defects
rate• Productivity metrics
• Defects per 100 hrs of testing, Test cases executed per 100 hrs of testing, Test
cases developed per 100 hrs, Defects per 100 test cases, Defects per 100
failed test cases, Test phase effectiveness, Closed defect distribution
Paratus Systems, Aundh
26
Testing StrategiesTesting Strategies
Test ReportingTest Reporting• Test incident report
It is a communication that happens through the testing cycle as and when defects
are encountered
• Test cycle report
It gives• A summary of activities carried out during that cycle• Defects that were uncovered during that cycle, based on their severity and
impact• Progress from the previous cycle to the current cycle in terms of defects fixed• Outstanding defects that are yet to be fixed in this cycle• Any variations observed in effort or schedule
• Test summary report• Phase-wise test summary, which is produced at the end of every phase• Final test summary reports, which has all the details of all testing done by all
phases and teams, also called as “release test report”
Paratus Systems, Aundh
27
Advanced Techniques and ToolsAdvanced Techniques and Tools
GUI testingGUI testing
• Graphical User Interface [GUI] Testing – methods used to identify and conduct GUI tests,
including the use of automation tools
• Elements of GUI Testing –
• A process• A GUI Test Plan• A set of supporting tools
• Running the Test Cases –
• Mouse position capture• Event capture
Paratus Systems, Aundh
28
Advanced Techniques and ToolsAdvanced Techniques and Tools
Functional testingFunctional testing
• It involves product features and functionality
• It tests product behavior
• Result conclusions are simple steps written to check expected results
• Results vary due to product implementation
• Testing focus is on defect detection
• Product and domain knowledge are required for testing
• Failures are normally due to code
• Testing phases include unit/component, integration and system testing
• One-time setup is required for a set of test cases
• Test cases are executed many times
Paratus Systems, Aundh
29
Advanced Techniques and ToolsAdvanced Techniques and Tools
Automated Testing Tools featuresAutomated Testing Tools features• Reliable : Tests perform precisely the same operations each time they are run, there by
eliminating human error.
• Repeatable: We can test how the application reacts after repeated execution of the same
operation
• Comprehensive : We can build a suite of tests that covers every feature in our application
• Reusable : We can reuse tests on different versions of an application, even if the user
interface changes.
• Better Quality Software: Because you can run more tests in less time with fewer resources
• Fast: Automated Tools run tests significantly faster than human users.
• Cost Reduction: As the number of resources for regression test are reduced
Paratus Systems, Aundh
30
Advanced Techniques and ToolsAdvanced Techniques and Tools
Automated Testing Tools selectionAutomated Testing Tools selection
Step 1: Define Tool requirements based on Project’s testing strategy and:Business requirementsTool audienceBudget constraints
Step 2: List possible Tool Candidates in the required category & collect Tool Data
Step 3: Compare each tool with its basic features against the Tool requirements defined in Step 1 and the cost
Step 4: Evaluate/Rank the Tools
Step 5: Select or recommend Tool with highest ranking / best pricing
Step 6: Arrange a Demo of Tool in Test Environment
Step 7: Finalize
Paratus Systems, Aundh
31
Advanced Techniques and ToolsAdvanced Techniques and Tools
Mobile TestingMobile Testing
• Mobile Application Testing is a process by which a developed application of a mobile
device is tested for its functionality, consistency and usability
• Challenges involved include:• Wide varieties of mobile devices• Different mobile operating system• Different mobile network operators• Input method• Hardware compatibility
• Type of testing include:• Functional testing• Laboratory testing• Performance and Load testing• Security testing• Localization testing• Usability testing
Paratus Systems, Aundh
32
Advanced Techniques and ToolsAdvanced Techniques and Tools
Testing Tools SeleniumTesting Tools Selenium
• Selenium is a Functional Automation Tool for Web Applications
• It is an open source tool (no cost involved with it)
• It supports the languages like HTML, Java, PHP, Perl, Python, Ruby and C#
• It supports the browsers like IE, Mozilla Firefox, Safari, Google Chrome and Opera
• It supports the operating systems like Windows, Linux and Mac
• It is very flexible when compared to QTP and other functional tools, because it supports
multiple languages
• It is mainly built on 3 components:• Selenium IDE• Selenium RC• Selenium Grid
Paratus Systems, Aundh
33
Advanced Techniques and ToolsAdvanced Techniques and Tools
Testing Tools JUnitTesting Tools JUnit
• JUnit has become the standard tool for Test-Driven Development in Java
• JUnit test framework is a package of classes that lets you write tests for each method,
then easily run those tests
• TestRunner runs tests and reports TestResults
• You test your class by extending abstract class TestCase
• To write test cases, you need to know and understand the Assert class
• To start using Junit, create a subclass of TestCase, to which you add test methods• Name of the class is important – should be of the form TestMyClass or MyClassTest• This naming convention lets TestRunner automatically find your test classes
• Each assert method has parameters like: message, expected-value, actual-value
Paratus Systems, Aundh
34
Advanced Techniques and ToolsAdvanced Techniques and Tools
Testing Tools – monkey talkTesting Tools – monkey talk
• Previously named “FoneMonkey”
• Tool for automated testing of iOS, Android, HTML5 and Adobe Flex applications
• It is an integrated environment for recording, customizing, running and managing test
suites
• Free and Open Source
• Powerful and robust record and playback functions for mobile applications
• Powerful IDE
• Fully cross platform
• No jailbreaking needed
Paratus Systems, Aundh
35