a planning-based approach to gui testingatif/presentations/sqw2000.pdf · a planning-based approach...
TRANSCRIPT
![Page 1: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/1.jpg)
1
A Planning-based Approach to GUI Testing
Atif M. MemonMartha E. PollackMary Lou Soffa
Dept. of Computer ScienceUniversity of Pittsburgh
2
Problem
GUI
UnderlyingCode
Testing a GUI
Interactions between theGUI and the Underlying Code
![Page 2: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/2.jpg)
3
High-level Outline
• Design of PATHS (Planning Assisted Tester for grapHical user interface Systems)
• Implementation Status• Overview of Planning & Test Case
Generation
4
GUISpecifications
Planner based Test CaseGenerator
Test CasesCore
The Entire GUI Testing Framework
![Page 3: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/3.jpg)
5
OracleInformation
AugmentationTool
Test Suite(Test Cases +
Oracle Information) Output
6
Test cases and Oracle Information
from Previous Version
CoverageReport
Output
Update forRegression
Testing
Regression Testing
Algorithms
Coverage EvaluationAlgorithms
Coverage Criteria
![Page 4: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/4.jpg)
7
UserAssertions
ProfileInformation
Domain Information
Usage Algorithms
The Entire GUI Testing Framework
8Implementation StatusGUI
SpecificationsGUI
Specifications
Automated Planning-based
Test Case Generator
Automated Planning-based
Test Case Generator
Test CasesTest Cases
AutomatedGeneration
![Page 5: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/5.jpg)
9
Automated OracleCreator
Automated OracleCreator
Test OraclesTest Oracles
10
Test Executor
GUI Under Test
AutomatedExecution
![Page 6: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/6.jpg)
11
Oracle Procedure
Test MonitorACTUAL STATE:(isCurrent ROOT)(Contains ROOT D(Contains ROOT D
12
• Individual User Events– NOT ENOUGH !– Sequences of User Events lead to
Different States• Test Case: Sequence of User Events
What is a GUI Test Case ?
![Page 7: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/7.jpg)
13
• Randomly Choose Sequences• Expert Chooses Sequences• Generate Sequences from a
Description of the GUI
Generating Test Case Sequences
This is the text. This is the text.Sequence
Each Sequence Transforms the GUI State
14
• Automatically Generate Test Cases for EXPECTED TASKS
Novel Idea
Initial State Goal State
MultipleTest Cases
This is the text. This is the text.
![Page 8: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/8.jpg)
15
AI Planning and GUI Testing
• Recent Advances in AI Planning Propositional Planners
• Very Fast• Based on
– Flow-Graphs [Koehler et al. ‘97]
– SAT Solving [Kautz & Selman ‘96]
• AI Planning, Especially Suitable for– Generating Test Cases– Automatically Generating Verification Information– Regression Testing
16
A Plan
InitialState
GoalState
ActionA
ActionB
ActionC
![Page 9: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/9.jpg)
17
InitialState
ActionA
ActionB
ActionC
GoalSTATE
InitialState
ActionB
ActionA
ActionC
GoalState
18
A Plan for a GUI Task
SelectText(“This”)
SelectText(“text”)
SetFontSize(18)
MouseClick(U)
Initial State
This is the text.
Goal State
This is the text.
is theThis text.
is theThis text.
![Page 10: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/10.jpg)
19
Overview of Test GenerationPhase Step Test Designer Automatic
Planning-basedSystem
Setup 1 Derive PlanningOperatorsfrom GUI
2 Code Preconditionsand Effects ofOperators
Test CaseGeneration
3 Specify a Task(Initial and GoalStates)
4 Generate TestCases
20
Straightforward Approach
• Define One Operator for each User Action
Operator :: CUTPreconditions:
isCurrent(Menu2).
Effects:FORALL Obj in Objects
Selected(Obj) ADD inClipboard(Obj)DEL onScreen(Obj)DEL Selected(Obj)
ADD isCurrent(Menu1)DEL isCurrent(Menu2).
Menu2
Menu1
First Order Predicate Logic
File Edit View Ins
Cut
![Page 11: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/11.jpg)
21
Exploit the GUI’s Structure
• Reduce the Number of Operators– System more Efficient– Easier for the Test Designer
22
Opening Modal Windows
English (United States)
OK Cancel Default...
Set Language
![Page 12: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/12.jpg)
23
Opening MenusFile
Send To
Mail Recipient
24
Interacting with the Underlying Software
UnderlyingSoftware
Edit
Copy
![Page 13: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/13.jpg)
25
Create Hierarchical Operators
Two Types of Abstractions– Combine Buttons Create System-
Interaction Operators– Decompose GUI Hierarchically Create
Abstract Operators
26
Create System-Interaction Operators
File_SendTo_MailRecipient= <File + SendTo +
MailRecipient >
Sys-Interaction Operator:
File
Send To
Mail Recipient
![Page 14: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/14.jpg)
27
Create Abstract Operators
SelectFromList()Default
OKCancel
Language Window’s Operator Set
Main GUI’sOperator Set
…Set Language
SelectFromList()Default
OKCancel
...
English (United States)
OK Cancel Default...
Straightforward Approach
Set Language
…Set Language
...
Main GUI’sOperator Set
Using Abstraction
28
Create Abstract Operators
SetLanguage()
SelectFromList(“English(US)”) OK
AbstractOperator
Planner
SelectFromList()Default
OKCancel
Language Window’s Operator Set
SetLanguage()
... ...High Level Plan
Sub PlanDefineAbstraction
![Page 15: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/15.jpg)
29
Effects of Exploiting the GUI’s Structure
• Reduction in Planning Operators– 325 operators 32 operators– Ratio 10:1 for MS WordPad– 20:1 for MS Word
• System Automatically Determines the System-interaction and Abstract Operators
30
Initial State
Goal State
This is the text.
This is the text.
![Page 16: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/16.jpg)
31
Test Case
SelectText(“This”)
FormatFont(“This”, 18pt)
FormatFont(“text”, Underline)
SelectText(“text”)
PrimitiveOperator
PrimitiveOperator
AbstractOperator
AbstractOperator
INITIAL
GOAL
32
FormatFont 18 OK FormatFont Underline OKPlanner Planner
![Page 17: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/17.jpg)
33
Format Font Format FontMapping Mapping
SelectText(“This”) Format Font 18 OK SelectText
(“text”)
Format Font Underline OK
34
Alternative Test Case
SelectText(“This”)
SelectFromList(18)
FormatFont(“text”, Underline)
SelectText(“text”)
PrimitiveOperator
PrimitiveOperator
PrimitiveOperator
AbstractOperator
SelectText(“This”)
SelectFromList(18)
SelectText(“text”)
Format Font Underline OK
SelectText(“This”)
FormatFont(“This”, 18pt)
FormatFont(“text”, Underline)
SelectText(“text”)
PrimitiveOperator
PrimitiveOperator
AbstractOperator
AbstractOperator
INITIAL
GOAL
![Page 18: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/18.jpg)
35
Methods to Generate AlternativeTest Cases
• Different Results from Planner• Abstract Operator Decompositions• Linearizations of the Partial-order
Plan
36
Experimental ResultsGenerating Test Cases and Deriving Verification Info.
00.10.20.30.40.50.60.70.80.9
1 6 11 16 21 26 31 36 41 46 51 56Test-Case Length
Tim
e (s
ec.)
![Page 19: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/19.jpg)
37
(Task) Plan No.
Plan Time (sec.)
Sub Plan Time (sec.)
Total Time (sec.)
1 3.16 0 3.162 3.17 0 3.173 3.2 0.01 3.214 3.38 0.01 3.395 3.44 0.02 3.466 4.09 0.04 4.137 8.88 0.02 8.98 40.47 0.04 40.51
38
Task No. Plan Length Time (sec.) Plan Length Time (sec.)1 18 8.93 3 0.112 20 47.62 4 0.183 24 189.87 5 0.144 26 3312.72 6 7.185 - - 3 0.16 - - 4 13.01
Single Level Hierarchical
![Page 20: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/20.jpg)
39
Experimental Results (contd…)Executing Test Cases and Verifying Results
0
1
2
3
4
5
1 6 11 16 21 26 31 36 41 46 51 56Test-Case Length
Tim
e (s
ec.)
40
Test Oracles & Regression Testing
• Exploiting Planning Algorithms to get Expected Behavior of GUI
• Using Hierarchical GUI Model for Efficient Regression Testing
![Page 21: A Planning-based Approach to GUI Testingatif/presentations/SQW2000.pdf · A Planning-based Approach to GUI Testing Atif M. Memon Martha E. Pollack Mary Lou Soffa Dept. of Computer](https://reader031.vdocuments.us/reader031/viewer/2022011901/5f077f957e708231d41d4804/html5/thumbnails/21.jpg)
41
Concluding Remarks
• Overview of PATHS & Current Status• Planning & Test Case Generation• Automatic Generation of
Preconditions and Effects from GUI Specifications