prioritizing user-session-based test cases for web applications testing sreedevi sampath, renne c....
TRANSCRIPT
Prioritizing User-session-based Test Cases for Web Applications Testing
Prioritizing User-session-based Test Cases for Web Applications Testing
Sreedevi Sampath, Renne C. Bryce,
Gokulanand Viswanath, Vani Kandimalla,
A. Gunes Koru
University of Maryland
University of Nevada
Sreedevi Sampath, Renne C. Bryce,
Gokulanand Viswanath, Vani Kandimalla,
A. Gunes Koru
University of Maryland
University of Nevada
Demand in Secure & Reliable Web Applications
Demand in Secure & Reliable Web Applications
A web application is a set of static or dynamic web pages that are accessible through a browser over a network
Critical part of business for many organizations Must be available 24/7 Failures in this domain result in losses of millions
of dollars
A web application is a set of static or dynamic web pages that are accessible through a browser over a network
Critical part of business for many organizations Must be available 24/7 Failures in this domain result in losses of millions
of dollars
Complications for Automated TestingComplications for Automated Testing
Fix bugs and deploy a new version within a short time frame
Number of integration technologies
- third party reusable modules
- a well-defined, layered architecture
- dynamically generated pages with
dynamic context
- extensions to an application framework Changes in database information
Fix bugs and deploy a new version within a short time frame
Number of integration technologies
- third party reusable modules
- a well-defined, layered architecture
- dynamically generated pages with
dynamic context
- extensions to an application framework Changes in database information
Previous Work: Testing with User-Session-based Test
Previous Work: Testing with User-Session-based Test
Logs of actual usage data are converted into test cases to run regression tests (Spenkle et al, 2005)
Efficient at detecting faults but unscalable with larger numbers of user sessions (Elbaum et al. 2005)
Reduced user-session-based test suites on “covering all base requests” criterion (Sampath 2007)
Logs of actual usage data are converted into test cases to run regression tests (Spenkle et al, 2005)
Efficient at detecting faults but unscalable with larger numbers of user sessions (Elbaum et al. 2005)
Reduced user-session-based test suites on “covering all base requests” criterion (Sampath 2007)
Previous Work: Prioritizing Test CasesPrevious Work: Prioritizing Test Cases
Schedule the test cases according to some criterion to satisfy a performance goalCriteria: semantic differences between 2 programs (Binkley 1992)
coverage of requirements (Offut 1995) fault exposure potential (Rothermel 2001)fault likelihood (Elbaum 2002)length of tests for GUI-based programs (Bryce & Memon
2007)
All strategies prioritize test cases for C and Java programs.
Schedule the test cases according to some criterion to satisfy a performance goalCriteria: semantic differences between 2 programs (Binkley 1992)
coverage of requirements (Offut 1995) fault exposure potential (Rothermel 2001)fault likelihood (Elbaum 2002)length of tests for GUI-based programs (Bryce & Memon
2007)
All strategies prioritize test cases for C and Java programs.
Prioritization for Web Application Testing
Prioritization for Web Application Testing
This work Expands upon previous prioritization work to develop
strategy specifically for web application testing Empirically evaluates strategies using user-session-based
test suites Gives guidance to testers based on the results of their
empirical evaluation
This work Expands upon previous prioritization work to develop
strategy specifically for web application testing Empirically evaluates strategies using user-session-based
test suites Gives guidance to testers based on the results of their
empirical evaluation
Motivation and Approach OutlineMotivation and Approach Outline
Criterion: Length Based
Base Request Long to Short (Req-LtoS)
Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)
Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)
Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)
Criterion: Length Based
Base Request Long to Short (Req-LtoS)
Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)
Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)
Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)
Test suite prioritization uses the entire test suite for execution but the test cases are ordered based on criteria that attempts to detect faults as quickly as possible in test execution cycle
Approach: Test Case Prioritization Criteria:Test Length
Approach: Test Case Prioritization Criteria:Test Length
Order by the number of HTTP base requests in a test case
Descending order of length (Req-LtoS) Ascending order of length (Req-StoL),
where length of a test case is defined as the number of base requests the test case contains counting duplicates.
Order by the number of HTTP base requests in a test case
Descending order of length (Req-LtoS) Ascending order of length (Req-StoL),
where length of a test case is defined as the number of base requests the test case contains counting duplicates.
Approach: Test Case Prioritization Criteria:Frequency-based Prioritization
Approach: Test Case Prioritization Criteria:Frequency-based Prioritization
Test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages in the entire test suiteSequences are considered: in terms of base requests that involve interactions between JSP and JSP servlet
pages of size 2
Test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages in the entire test suiteSequences are considered: in terms of base requests that involve interactions between JSP and JSP servlet
pages of size 2
Two Ways to Prioritize Frequently Accessed Sequences
Two Ways to Prioritize Frequently Accessed Sequences
All Accessed Sequences (AAS)For each sequence in the application, beginning with the most frequently accessed sequence, test cases that have max. occurrences of these sequences are selected for execution before other test cases in the test suite
All Accessed Sequences (AAS)For each sequence in the application, beginning with the most frequently accessed sequence, test cases that have max. occurrences of these sequences are selected for execution before other test cases in the test suite
Most Frequently Accessed Sequence (MFAS)Orders test cases in decreasing
order of the number of times that
sequence appears in the test case
Systematic Prioritization by Parameter-ValuesSystematic Prioritization by Parameter-Values
Pages that contain parameters for which users may specify valuesPages that contain parameters for which users may specify values
Criterion:Length Based
Base Request Long to Short (Req-LtoS)
Base Request Short to Long (Req-StoL)Parameter Value Long to Short (PV-LtoS)Parameter Value Short to Long (PV-StoL)
Frequency BasedMost Frequently Accessed Sequence (MFAS)All Accessed Sequences (AAS)
Parameter Value CoverageUnique Coverage of Parameter Values (1-way)2-way Parameter-Value Interaction Coverage (2-way)
Systematic Prioritization by Parameter-ValuesUnique Parameter-Value Coverage
Systematic Prioritization by Parameter-ValuesUnique Parameter-Value Coverage
The 1-way criterion selects a next test that maximizes the number of parameter-values that have not appeared in previously selected tests
The 1-way criterion selects a next test that maximizes the number of parameter-values that have not appeared in previously selected tests
Systematic Prioritization by Parameter-ValuesParameter-Value Interaction Coverage
Systematic Prioritization by Parameter-ValuesParameter-Value Interaction Coverage
The 2-way criterion selects a next test that
maximizes the number of 2-way parameter-value
interactions between pages that occur in a test
The 2-way criterion selects a next test that
maximizes the number of 2-way parameter-value
interactions between pages that occur in a test
Systematic Prioritization by Parameter-ValuesLength by Parameter-Value Counts
Systematic Prioritization by Parameter-ValuesLength by Parameter-Value Counts
Prioritize test by the number of parameter-values in a test case
Prioritize test by the number of parameter-values in a test case
Approach OutlineApproach Outline
Criterion: Length Based
Base Request Long to Short (Req-LtoS) Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)
Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)
Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)
Criterion: Length Based
Base Request Long to Short (Req-LtoS) Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)
Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)
Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)
Experimental EvaluationExperimental EvaluationEvaluation is based on: Fault detection rate Average percent of faults detected Execution time of the test suite
Subject Applications (test suites): Book - e-commerce bookstore CPM - create grader accounts for TA Masplas – web app for regional workshop
Evaluation is based on: Fault detection rate Average percent of faults detected Execution time of the test suite
Subject Applications (test suites): Book - e-commerce bookstore CPM - create grader accounts for TA Masplas – web app for regional workshop
Evaluation MetricsEvaluation Metrics Rate of fault detection
Average percentage of faults detected
Rate of fault detection
Average percentage of faults detected
Study is based on:
Finding most faults in earliest tests (first 10% of tests executed)
Finding 100% of the faults earliest
Experimental Evaluation (contd…)Experimental Evaluation (contd…)
Categories of fault seeded: Data Store - faults that exercise code
interacting with data Logical - code logic errors in control flow Form based Appearance - change in perceiving user’s
view of the page Links - change of hyperlink locations
Categories of fault seeded: Data Store - faults that exercise code
interacting with data Logical - code logic errors in control flow Form based Appearance - change in perceiving user’s
view of the page Links - change of hyperlink locations
Result Analysis for CPMResult Analysis for CPM
AAS is most effective technique to find most faults in the earliest test
2-way parameter value interaction is most effective technique to find all the faults at the earliest
Result Analysis for MasplasResult Analysis for Masplas
Req-LtoS - Test length based on number of base requests (descending) is most effective technique to find most faults in the earliest test
2-way parameter value interaction is most effective technique to find all the faults at the earliest
Result Analysis for BookResult Analysis for Book
1-way parameter value interaction is most effective technique to find most faults in the earliest test
MFAS and 1-way parameter value interaction are most effective techniques to find all the faults at the earliest
Summary of resultsSummary of results
Choice of the testing technique is based on the tester’s goal and characteristics of web application The 2-way parameter value interaction and PV-LToS technique is to be
used when the primary goal is to find 100% of faults at the earliest Frequency based technique needs to be used to validate the functionality of
frequently used web pages of an application
Choice of the correct prioritization would help in finding and fixing the faults in application quickly, thus saving cost impact
Choice of the testing technique is based on the tester’s goal and characteristics of web application The 2-way parameter value interaction and PV-LToS technique is to be
used when the primary goal is to find 100% of faults at the earliest Frequency based technique needs to be used to validate the functionality of
frequently used web pages of an application
Choice of the correct prioritization would help in finding and fixing the faults in application quickly, thus saving cost impact
Future WorkFuture Work
Consideration of costs associated with prioritization strategies
Evaluate hybrid approaches to prioritization (combining prioritization techniques)
Extension to frequency based metrics to include other metrics
Consideration of costs associated with prioritization strategies
Evaluate hybrid approaches to prioritization (combining prioritization techniques)
Extension to frequency based metrics to include other metrics