smartest: proposal for accelerating the detection of …...ana b. sánchez #drupalcampspain2015...
TRANSCRIPT
![Page 1: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/1.jpg)
SmarTest: Proposal for accelerating the detection of faults in Drupal
Ana Belén Sánchez #DrupalcampSpain2015
![Page 2: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/2.jpg)
Ana B. Sánchez #DrupalcampSpain2015
¿Who am I? Where do I come from?
Ana Belén Sánchez [email protected]
Research Group of Applied Software Engineering
(www.isa.us.es)
Escuela Técnica Superior de Ingeniería Informática
University of Seville
I am a researcher and a Teacher
![Page 3: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/3.jpg)
Ana B. Sánchez #DrupalcampSpain2015
¿Why am I here, in a Drupal Conference?
Ph.D. in automated testing on variability-intensive systems
![Page 4: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/4.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Variability-intensive systems
![Page 5: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/5.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Testing problems
Too many configura-ons!
Time-‐consuming
![Page 6: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/6.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Test case selection techniques reduce the test space by selecting a portion of the configurations to be tested.
![Page 7: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/7.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal.
![Page 8: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/8.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Test case selection techniques reduce the test space by selecting a portion of the configurations to be tested.
Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal.
![Page 9: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/9.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Testing variability-intensive systems
Artificial models to test
How to evaluate these testing techniques?
![Page 10: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/10.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Towards real highly-configurable tools
Studied Tools
![Page 11: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/11.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Testing of real highly-configurable tools
1. Available code
2. Detailed fault reports
3. Automated test cases
4. Extensive documentation
5. > 630,000 users and developers
6. > 30,000 modules
![Page 12: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/12.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our approach
Ana B. Sánchez, Sergio Segura, Jose A. Parejo and Antonio Ruiz Cortés. Variability Testing in the Wild: The Drupal Case Study Published in Software and Systems Modeling Journal, 2015
(http://www.isa.us.es/publications/type/article-journal/9998/variability-testing-wild-drupal-case-study)
![Page 13: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/13.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our approach: The Drupal case study
Module .info files
Drupal documentation
Module JIT
![Page 14: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/14.jpg)
Ana B. Sánchez #DrupalcampSpain2015
The Drupal variability model
48 modules 21 dependencies
![Page 15: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/15.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Drupal information
LoC
Module Size
Changes Commits
Tests 352 test cases 24152 assertions
![Page 16: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/16.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Drupal information
CC
Cyclomatic Complexity
Drupal Installations
Drupal Contributors
![Page 17: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/17.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Drupal faults
Bug tracking System
3.232 Faults in v7.23 160 integration faults:
• 132 caused by 2 modules
• 25 caused by 3 modules
• 3 caused by 4 modules
![Page 18: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/18.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Module Size Faults Module
Changes Faults
Faults v7.22
Faults v7.23
Module CC Faults
![Page 19: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/19.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Module Developers Faults
Tests ? Faults ?
Optional Modules Faults
91 %
Core Mandatory Modules Faults
41 %
![Page 20: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/20.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Module Size
Faults Module Changes
Faults Faults v7.22
Faults v7.23
CC Faults Tests ? Faults ?
Developers Faults
![Page 21: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/21.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Optional Modules Faults
Core Modules Faults
91 % 41 %
![Page 22: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/22.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Statistical study
Optional Modules Faults
Core Modules Faults
91 % 41 %
72 % normal faults
16 % major faults
7 % minor faults
5 % critical faults
No apparent correlation
![Page 23: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/23.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Test case prioritization criteria
Size–driven Criterion
Fault–driven Criterion
CC-driven Criterion
Change–driven Criterion
Priori-za-on criteria based on previous sta-s-cal study
![Page 24: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/24.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Evaluation
94.2 % Test case selection
Fault-driven Prioritization
2.
Could we accelerate the detection of faults in Drupal using previous information?
86.2 % Test case selection
1.
![Page 25: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/25.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Applying our proposal to Drupal
SimpleTest
![Page 26: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/26.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Applying our proposal to Drupal
Analyzing the SimpleTest module
Main view SimpleTest
![Page 27: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/27.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Main view SmarTest
Customizable dashboard
![Page 28: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/28.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
![Page 29: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/29.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
![Page 30: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/30.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Test Prioritization SmarTest
![Page 31: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/31.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Continuos feedback in real time SmartTest
![Page 32: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/32.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
¿What improvements do we add to SimpleTest?
SimpleTest
ü Designed to make testing easier
ü A powerful automated testing
ü Selector to select tests to execute
ü Detailed inform. about test execution
ü Possibility to follow the trace of errors
SmarTest
ü Customizable Dashboard with run-time extracted data to guide the testing
ü Test prioritization fo detect faults faster
ü Automated testing with feedback in real time
ü Time-out option to automatically stop the test execution
![Page 33: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/33.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Our proposal: SmarTest
Things that we are working on…
• Improve the dashboard with more information and new widgets
• Include new test prioritization criteria and multi-prioritization criteria
• Study the integration of SmarTest with other test systems: Behat, phpUnit…
• Improve the SmartTest UI
• Add the SmarTest to Drupal sandbox
• ¿Any suggestions?
![Page 34: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/34.jpg)
Ana B. Sánchez #DrupalcampSpain2015
Acknowledgments
This work was partially supported by the European Commission (FEDER), the Spanish and the Andalusian R&D&I programmes (Grants IPT-2012-0890-390000 (SAAS FIRE- 46 WALL), TIN2012-32273 (TAPAS), TIC-5906 (THEOS), TIC-1867 (COPAS)).
![Page 35: SmarTest: Proposal for accelerating the detection of …...Ana B. Sánchez #DrupalcampSpain2015 Testing variability-intensive systems Test case selection techniques reduce the test](https://reader033.vdocuments.us/reader033/viewer/2022050105/5f4373813711010717037df4/html5/thumbnails/35.jpg)
Ana B. Sánchez #DrupalcampSpain2015