testing tools and jenkins - european southern observatory · pdf filetesting tools and jenkins...

ESO Pipeline Workshop, Oct 24 2014

on behalf of

ESO Pipeline Systems Group

Testing Tools and Jenkins

Artur Szostak


Purpose

● Using a number of testing tools to identify problems and bugs in the software.

● Integrated into the Jenkins continuous integration system to run these tools over the code base in a regular and systematic manner.

● Give feedback to the developers as fast as possible when problems/bugs are detected.

● Use the tools to guide development of unit and regression tests.● Help assess the quality of the code.● Provide facilities for developers to extend or add their own

testing jobs quickly and easily.


Jenkins – Continuous Integration

● Standard jobs are maintained in Jenkins for all pipelines, CPL, HDRL and esorex.

● These are run regularly and automatically when changes are detected in the source code repository.

● Users can also create customised build and test jobs which can be triggered either manually or automatically.


Tests Covered in Jenkins

● Testing compilation and linking on a number of platforms.● Check for compiler and linker warnings.

● Running unit tests.● Running static analysis with Cppcheck.

● Coverity Scan is also used for CPL.● Various additional sanity checks for naming conventions etc. are

performed with the staticcheck script.● Running regression tests using esorex on test data sets.● Running unit tests under Valgrind to check for memory leaks and

errors.● Code coverage for unit tests and regression tests.● Creation and testing installation of pipeline kits.


Jenkins Results

● Summary information available per Pipeline.● Lots of additional information available in GUI.


Warnings and Unit Tests Example


Cppcheck Results Example


Staticcheck Script Example


Valgrind Results Example


Coverage Results Example


Conclusion

● Using Jenkins in Pipeline Group for ~ 1.5 years with positive feedback from developers.

● Integrating testing and static analysis tools into Jenkins made the tools:

● easier to use,● and used more often.

● Automated testing with Jenkins (or equivalent system) is the only feasible way to perform systematic large scale testing.

● Consortia are encouraged to run Jenkins like setups for regular testing of the code during development.


Backup slides...


Sample Cppcheck Commands

URL=http://svnhq2.hq.eso.org/p2/trunk/Pipelines/iiinstrument/iiinstrumentp

svn co $URL

cd iiinstrumentp

find . -type f -name "*.h" | xargs -L 1 dirname \

| sort -u > paths.txt

cppcheck --force --enable=all --std=c99 --std=posix \

--includes-file=paths.txt ./


Commands for staticcheck

./bootstrap

./configure --enable-maintainer-mode \

--prefix=`pwd`/testOutput

make clean

make install

make check TESTS=

URL=http://svnhq2.hq.eso.org/p2/trunk/Pipelines/ci-jenkins/scripts/staticcheck

svn export $URL

./staticcheck -namespace=iiinstrument -strict \

-extlibs=cpl,hdrl -excludes='\./hdrl/\.libs/.*,\./irplib/\.libs/.*'


Sample Valgrind Commands

./configure --enable-maintainer-mode \

--prefix=`pwd`/testOutput

make clean

make install

make check TESTS=

cd iiinstrument/tests

valgrind --leak-check=full --track-origins=yes \

.libs/iiinstrument_dfs-test


Commands for Code Coverage

./configure --prefix=`pwd`/testOutput \

CFLAGS='-g -O0 --coverage' \

LDFLAGS='-g -O0 --coverage'

make clean

make install

find . -name "*.gcda" -delete

make check

lcov --compat-libtool --directory ./ -c -o coverage.info

genhtml -o coverage_html coverage.info

firefox coverage_html/index.html


Jenkins – Setup

● Master running on 24 core machine with 64GB RAM and RAID storage.

● Only ~ 1% CPU used by Jenkins master when idle. Up to 10-15% during peak load.

● About 16GB used by Jenkins JavaVM instance.● 2.5TB partition dedicated for Jenkins build logs. Currently using ~

45% of this.● 16 physical and 36 virtual build slaves running various

versions of Apple OS X, Scientific Linux, Fedora, Debian, Ubuntu, OpenSUSE, CentOS.

● Running ~ 750 standard jobs and ~ 80 customised user jobs for 25 pipelines, CPL, HDRL, esorex and dependencies.

testing tools and jenkins - european southern observatory · pdf filetesting tools and jenkins...

Documents