visual gui testing: the technique, problems and the futuresast.se/q-moten/2014/q1_2014_vgt.pdf ·...
TRANSCRIPT
Visual GUI Testing: The technique, Problems and the Future
Emil Alégroth and Michel Nass
In collaboration with:
Statement 1: Automated testing lowers testing cost!
Requirements Engineering
$
Development $
Testing $
Development process
Total $
TESTING
Maintenance Manual delta-testing
Statement 2: The execution speed is essential for automated tests!
Nightly runs
Statement 3: Regression testing tools find all defects!
Button 1
Button 2
Button 3
Button X Button 1
Button 2
Button X
Click <button 1> Verify <button1, green> Click <button 2> Verify <button 2,green> Click <button 3> Verify <button3, green>
Click <button 1> Verify <button1, green> Click <button 2> Verify <button 2,green> Click <button 3> Verify <button3, green>
Click <button 1> Verify <button1, green> Click <button 2> Verify <button 2,green> Click <button 3> Verify <button3, green>
Click <button 1> Verify <button1, green> Click <button 2> Verify <button 2,green> Click <button 3> Verify <button3, green>
Statement 4: One tool to rule them all!
Unit testing
Integration testing
System testing
Acceptance testing
Customer needs
System reqs.
FR reqs.
Code
XUnit
Visual GUI Testing
Statement 5: Automated testing can replace manual testing
Coverage
Manual Exploratory Testing
Background to GUI-based testing
Code
System components
GUI Components
GUI
Testing of the system through the GUI, not
testing of the GUI!
Change code
Regression testing
GUI-based testing techniques
Manual GUI-based testing Structural GUI-based testing Visual GUI Testing
Manual GUI-based testing
Costly Tedious Error-prone
Test step
Input Expected output
1 Click on button x
Button x changes color
2 Click on button y
Button y changes color
.. … …
N Click on button z
Button z changes color
Structural GUI-based testing
Okey
OK Color: Gray Size: 50x100px Label: “OK”
Pros and Cons
Independent of window size
Finds “invisible”
widgets
Complex tools
Automatic <> Manual Complex component database
Coordinate based for certain Widgets Requires collaboration between
testers and developers Evaluate testability
Environment- and Widget dependent
Visual GUI Testing
Test step
Input Expected output
1 Click on button x
Button x changes color
2 Click on button y
Button y changes color
.. … …
N Click on button z
Button z changes color
Automate test-scenarios…
…using image recognition and AI…
…to emulate end-user behavior for automated System and
Acceptance testing.
Pros and Cons
Implementation independent Tests existing UI without modifications
Tests in the same way as a human Programming skills are not required
Easy to understand the test cases Simpler and more accurate recording
Reliable playback
Uses the cursor
Limited playback speed Difficult to understand how the tool “thinks”
JAutomate Demo
Comparing the Techniques (1/2)
Code
System components
GUI Components
GUI Visual GUI Testing / Manual GUI-based testing
Structural GUI-based testing
Comparing the Techniques (2/2)
WebDriver driver = new FirefoxDriver();
driver.get("http://en.wikipedia.org/wiki/Main_Page");
WebElement about= driver.findElement(By.xpath("//*[@id='n-aboutsite']/a"));
about.click();
Thread.sleep(5000);
String title=driver.getTitle();
assertEquals(title, "Wikipedia:About - Wikipedia, the free encyclopedia");
driver.quit();
Maintenance cost
High
Low
Medium
Manual Structural Visual
Based on Academic literature and empirical research in industrial practice
Execution speed
High
Low
Medium
Manual Structural Visual
Based on Academic literature and empirical research in industrial practice
Flexibility
High
Low
Medium
Manual Structural Visual
Based on Academic literature and empirical research in industrial practice
Required technical knowledge
High
Low
Medium
Manual Structural Visual
Based on Academic literature and empirical research in industrial practice
Research into VGT
Applicability of VGT
New VGT tools:
JAutomate
VGT tools: Initial
support for VGT
VGT in industry: VGT in a real
industrial project VGT:
Challenges, problems and
limitations
Saab in Gothenburg Sikuli and CommercialTool
Saab in Järfälla Real development project
Saab in Gothenburg and Järfälla Two development projects
Inceptive JAutomate, Sikuli and CommercialTool New VGT
approaches: Random VGT
Academic Experiment Sikuli VGT
maintenance: long-term feasibility
Saab in Växjö Sikuli
Maintenance of VGT
VGT suite System X,
v1.0
VGT suite System X,
v2.0
VGT suite System Y,
v2.0
Development: 100 hours
Maintenance: 56-62 hours
Maintenance: 12-18 hours
1 year ~1 month
Is your Test Suite rotting?
?
Future research
“Adopting VGT Automated Testing And Research”, E. Alégroth, M. Nass, H. Olsson, 2014?
?
Conclusion
“Exploratory” Intuitive
Costly Error-prone
Tedious
Fast Record/Repla
y
Maintenance Complexity
!Applicability
Flexibility Maintenance Emulate user
Intuitive
Slower Uses the
cursor How does
the tool think
Choice of technique depends on the
development context!
Statement 4: One tool to rule them all!
Unit testing
Integration testing
System testing
NFR testing
Acceptance testing
Customer needs
NFR reqs.
System reqs.
FR reqs.
Code
XUnit
VGT