lcg-spi: sw-testing
DESCRIPTION
LCG-SPI: SW-Testing. LCG AppArea internal review (20/10/03) [email protected] LCG/SPI LCG Software Process & Infrastructure. Outline. SPI SW-Testing. SW-Testing Overview LCG-SPI SW-Testing Test Frameworks Test Documents User support. Nightly Building System. SPI SW-Testing. - PowerPoint PPT PresentationTRANSCRIPT
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 1
LCG-SPI: SW-Testing
LCG AppArea internal review (20/10/03) [email protected]
LCG/SPILCG Software Process & Infrastructure
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 2
Outline
SPI SW-Testing
• SW-Testing Overview• LCG-SPI SW-Testing• Test Frameworks• Test Documents• User support
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 3
•Software testing should be an integral part of the software development in the LCG App Area.•All level of software testing should be run as part of an automatic process.
SW-Testing Overview
GOAL (from RTAG2)
Software testing
Integration test
System test
Acceptance test
Unit test
Sw-testing team
IntegrationTests
System Tests
ExamplesLHC experiments
Use in Exp.Test FrameWorks
SW-testing doc
SW- testing support
SPI SW-Testing
SW-testing polices
Code Documentation
Software developer
CODE
Work PackageTest
Automated testing
Nightly Building
System
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 4
LCG-SPI SW-Testing
Aim: to help developers: to help projects:
• to produce test code suitable to be run in automatic way • to document their tests
• to plan their test phase• to run all their test in automatic way • to browse and scan easily the test results and failures
Our inputs: • Contacts within HEP-community.• What is available as free open source code.
Our constraints: • Avoid commercial software and licensing problems. • Avoid “do it yourself solutions”•Try to adopt commonly used open-source software.
Requirements from the users : • Work with different languages (C++, Python, …) and in different environments (platforms & compilers)• Allow two ways of testing: (1) “check the output of the test-code” (2) “check the test results inside the test-code”.• Provide a easy way to integrate existing tests.• Allow the creation of dependencies among tests.• Easily establish a criteria if a test should pass or fail in a given release• Organize the tests by components or packages / integration / system /acceptance .• Regression testing and check respect to the previous sw-release.• Easy integration with the Nightly Building System.• Provide also a graphical interface to run the tests and examine the test results.
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 5
Test Frameworks: Global picture
X-Unit familyJUnit QtUnit
CppUnit PyUnit
Oval
Test Scripts
Old tests
SW Product Examples
Unit testing
Acceptanc
etesting
QA activity
End User Installation
checks
Nightly Building
Project Release
Sw-Testing
QMTest• Uses a GUI for creating and running tests (also in batch).• Can run tests in parallel, supports execution of a single test or many at once (test-cases & test-suites) • Organizes tests hierarchically• Records dependencies among tests
Top layer
Bottom layer
- Adaptable to theprogramming language and developer- Prepared to be run in automatic way
- Integrates different ways to test.
- Common environment to run the tests and to access the test results.
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 6
•Main Scope: Unit–testing
•The same “assertion style” in different languages.
•Provides:- The name of the test case that failed.
-The name of the source file that contains the test.
-The line number where the failure occurred
• Different platforms/compilers: (Linux/Solaris/Windows)
Test Frameworks: X-Unit family
X-Unit family
JUnit
QtUnit
CppUnit
PyUnit
A simple test:
1. Subclass the TestCase CppUnit or PyUnit class
2. Override the method runTest().
3. When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 7
Test Frameworks: Oval
Oval:• validation and regression • developed & used within CMS experiment
•Can be use for Unit-test to Validation tests.
•Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”)
• It is possible to set different run environments.
• Can run external scripts and external binaries.
Ovalfile(configuration)
ExecutableTest
Source Code
Referencefile
Log file
Oval run
Ovaldiff
Oval build
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 8
Test Frameworks: QMTest
QMTest
•Uses a graphical interface for creating and running tests
•The configuration files are in XML and can be created from the GUI. We provide also script to do it
• Runs tests in parallel
• Organizes tests hierarchically
• Supports execution of a single test or many at once
• Records dependencies among tests
• Can be run in batch mode -> easy integration with the Nightly-Building systems
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 9
Test Documents
• For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities.• We use a template•Test Case template collects information about the existing tests and how they cover the expected functionality of the software.
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 10
User support: Web information
http://spi.cern.ch/
Test Frameworks
HowTo
Test doc
Policies
http://spi.cern.ch/testing
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 11
User support: How-To
3 Follow sw-testing policies
Automatic testing requires some standardization
1
2
M Gallas CERN EP-SFT
LCG-SPI: SW-Testing 12
• SW-Testing Overview• LCG-SPI SW-Testing• Test Frameworks• Test Documents• User support
SPI SW-Testing
Thanks to:• LCG-PI team • LCG-POOL team• LCG-SEAL team• LCH-experiments
• Software testing is a needed step in the software development process.
• SPI provides test frameworks, templates and help to have a coherent automatic testing procedure with a common interface to run and browse the test results.
•The proposed testing tools were chosen following the criteria of a simple approach, tools in use in the experiments and freely available.
Final Remarks
• Following standards makes life easier to SPI and the users. •Automatic testing is always needed and is a way to consolidatethe software.• Bug reports and typical use-cases from the users should be converted into tests
• SPI SW-testing has to evolve and consolidate followingthe needs of the LCG projects. •All SPI sw-testing tools, templates, documentation can be used by any project.