testing tools and jenkins - european southern observatory · pdf filetesting tools and jenkins...

ESO Pipeline Workshop, Oct 24 2014

on behalf of

ESO Pipeline Systems Group

Testing Tools and Jenkins

Artur Szostak

ESO Pipeline Workshop, Oct 24 2014

Purpose

● Using a number of testing tools to identify problems and bugs in the software.

● Integrated into the Jenkins continuous integration system to run these tools over the code base in a regular and systematic manner.

● Give feedback to the developers as fast as possible when problems/bugs are detected.

● Use the tools to guide development of unit and regression tests.● Help assess the quality of the code.● Provide facilities for developers to extend or add their own

testing jobs quickly and easily.

ESO Pipeline Workshop, Oct 24 2014

Jenkins – Continuous Integration

● Standard jobs are maintained in Jenkins for all pipelines, CPL, HDRL and esorex.

● These are run regularly and automatically when changes are detected in the source code repository.

● Users can also create customised build and test jobs which can be triggered either manually or automatically.

ESO Pipeline Workshop, Oct 24 2014

Tests Covered in Jenkins

● Testing compilation and linking on a number of platforms.● Check for compiler and linker warnings.

● Running unit tests.● Running static analysis with Cppcheck.

● Coverity Scan is also used for CPL.● Various additional sanity checks for naming conventions etc. are

performed with the staticcheck script.● Running regression tests using esorex on test data sets.● Running unit tests under Valgrind to check for memory leaks and

errors.● Code coverage for unit tests and regression tests.● Creation and testing installation of pipeline kits.

ESO Pipeline Workshop, Oct 24 2014

Jenkins Results

● Summary information available per Pipeline.● Lots of additional information available in GUI.

ESO Pipeline Workshop, Oct 24 2014

Warnings and Unit Tests Example

ESO Pipeline Workshop, Oct 24 2014

Cppcheck Results Example

ESO Pipeline Workshop, Oct 24 2014

Staticcheck Script Example

ESO Pipeline Workshop, Oct 24 2014

Valgrind Results Example

ESO Pipeline Workshop, Oct 24 2014

Coverage Results Example

ESO Pipeline Workshop, Oct 24 2014

Coverage Results Example

ESO Pipeline Workshop, Oct 24 2014

Conclusion

● Using Jenkins in Pipeline Group for ~ 1.5 years with positive feedback from developers.

● Integrating testing and static analysis tools into Jenkins made the tools:

● easier to use,● and used more often.

● Automated testing with Jenkins (or equivalent system) is the only feasible way to perform systematic large scale testing.

● Consortia are encouraged to run Jenkins like setups for regular testing of the code during development.

ESO Pipeline Workshop, Oct 24 2014

Backup slides...

ESO Pipeline Workshop, Oct 24 2014

Sample Cppcheck Commands

URL=http://svnhq2.hq.eso.org/p2/trunk/Pipelines/iiinstrument/iiinstrumentp

svn co $URL

cd iiinstrumentp

find . -type f -name "*.h" | xargs -L 1 dirname \

| sort -u > paths.txt

cppcheck --force --enable=all --std=c99 --std=posix \

--includes-file=paths.txt ./

ESO Pipeline Workshop, Oct 24 2014

Commands for staticcheck

./bootstrap

./configure --enable-maintainer-mode \

--prefix=`pwd`/testOutput

make clean

make install

make check TESTS=

URL=http://svnhq2.hq.eso.org/p2/trunk/Pipelines/ci-jenkins/scripts/staticcheck

svn export $URL

./staticcheck -namespace=iiinstrument -strict \

-extlibs=cpl,hdrl -excludes='\./hdrl/\.libs/.*,\./irplib/\.libs/.*'

ESO Pipeline Workshop, Oct 24 2014

Sample Valgrind Commands

./configure --enable-maintainer-mode \

--prefix=`pwd`/testOutput

make clean

make install

make check TESTS=

cd iiinstrument/tests

valgrind --leak-check=full --track-origins=yes \

.libs/iiinstrument_dfs-test

ESO Pipeline Workshop, Oct 24 2014

Commands for Code Coverage

./configure --prefix=`pwd`/testOutput \

CFLAGS='-g -O0 --coverage' \

LDFLAGS='-g -O0 --coverage'

make clean

make install

find . -name "*.gcda" -delete

make check

lcov --compat-libtool --directory ./ -c -o coverage.info

genhtml -o coverage_html coverage.info

firefox coverage_html/index.html

ESO Pipeline Workshop, Oct 24 2014

Jenkins – Setup

● Master running on 24 core machine with 64GB RAM and RAID storage.

● Only ~ 1% CPU used by Jenkins master when idle. Up to 10-15% during peak load.

● About 16GB used by Jenkins JavaVM instance.● 2.5TB partition dedicated for Jenkins build logs. Currently using ~

45% of this.● 16 physical and 36 virtual build slaves running various

versions of Apple OS X, Scientific Linux, Fedora, Debian, Ubuntu, OpenSUSE, CentOS.

● Running ~ 750 standard jobs and ~ 80 customised user jobs for 25 pipelines, CPL, HDRL, esorex and dependencies.