System testing for PDE frameworks - tools and experiences

Dominic Kempf1, Peter Bastian1, Timo Koch2, Bernd Flemisch2

[1] IWR, Heidelberg University, Germany, [2] LH2, University of Stuttgart, Germany

Dune: A flexible framework for the solution of PDEsThe Distributed and Unified Numerics Environment is a flexible opensource framework for the numerical solution of partial differentialequations with grid-based methods.

Dune is currently developed at roughly 10 universities in Europe andhas an even wider user base. The underlying idea of Dune is to cre-ate slim interfaces allowing an efficient use of legacy and/or new li-braries. Modern C++ programming techniques enable very differentimplementations of the same concept (i.e. grids, solvers, ...) usinga common interface at a very low overhead. Thus Dune ensuresefficiency in scientific computations and supports high-performancecomputing applications.

Dune is also used as a basic building block of domain specificapplication frameworks like DuMuX. This work is carried out incollaboration with the DuMuX development team.

System testing for PDE modelsTesting is a proven key concept for the development of sustainable,high quality research software. In contrast to unit testing, where soft-ware components are tested individually, system testing is about thecomplex interplay of many such components. In the context of nu-merical frameworks for the solution of PDEs, system testing refersto the combination of framework components into a full simulationexecutable. Some problems in defining these tests systematicallyarise:• The combinatoric product of framework features is unfeasibly

large.• The mathematical input to a framework is by definition infinite-

dimensional.• Definition of test samples needs expert knowledge.

We therefore aim for an approach, where one systemtest is onehand-written simulation executable with a description of the reason-able variability for this model.

Simulation

Dimen-sion

1

2 3

4+

Grids(non-)over-

lapping

(non-)conforming

(un)structured

globally/locallyrefined

simplices/quadri-laterals (non-)

conforming

(non-)manifold

Discreti-zation

CG/DG

cell/vertex-centered

FVM

time dis-cretization

polynomialdegree

Solverpre-

conditioner

dofordering

blocking

iterative/ direct

grid (in-)dependent

parallel /sequential

Model

(non-)linear

(in)stationary

scalar/system

coefficientfunctions

boundaryconditions

Another problem in system testing arises from actually determiningtest failure for a PDE problem. We employ a flexible strategy to wrapthe actual run of the simulation executable by a python script. Thatallows us to do postprocessing, that C++ is not suitable for:

Executable

Pythonwrapper

Exit code CTest

This includes testing tools as complicated as regression testingwith VTK data and testing for expected convergence rates (time orspace) on a series of runs.

As many developers do not feel proficient with build systems, westrongly believe that the addition of any test should involve only oneline of CMake code which follows a well-defined interface. We doc-ument our CMake build system with Sphinx for that purpose.

Describing variability in configuration filesOur goal is to describe a set of system tests taking into accountvariability of some variation points. To keep the testing code basemaintainable, we want to avoid code duplication and have the vari-ability description as compact as possible. We extend the syntaxof the configuration file, that usually accompanies a simulationexecutable, to describe variability:

timestep = 0.5, 1.0 | expand

refinementlevel = 3

Meta Ini Fileunder version control, describes

a subset of the variability model

Python scriptfrom dune-testtools, interacting with CMake

Set of ini filesgenerated in the build tree at config-

ure time, one file refers to one test

From this enhanced metaini file, the test suitegenerates a set of reg-ular ini files at config-ure time. This expansionprocess is controlled byDunes CMake build sys-tem. The relevant codeis provided in the moduledune-testtools.

Having implemented a syntax that applies commands (precededby a pipe symbol) to key/value pairs in configuration files, someextension come natural:

timestep = 0.5, 1.0 | expand refine

refinementlevel = 2, 3 | expand refine

Any expansion processes with the same parameter are expandedtogether instead of generating the combinatoric product of variants.

refinementlevel = 2, 3 | expand

output = level{refinementlevel}.output

Curly brackets may be used to reference the actual value of akey after expansion. These curly brackets are resolved iteratively,allowing even nested usage of curly brackets.

Other implemented commands target string manipulation, simplearithmetic and exclusion constraints on the combinatoric product ofvariants etc.

Dune is a C++ framework making excessive use of templates,hence many variation points will be of compile-time nature. Wesupport this by adding a special group, that is parsed by CMake andturned into a set of executables with the key/value pairs becomingpreprocessor variables.

[__static]

SOLVERTYPE = Solver1, Solver2 | expand

Compile-time and run-time variations can be mixed in an ar-bitrary fashion, leading to a two stage expansion procedure:

Meta Ini File+ Source File

Executable

Executable

Executable

CTest

CTest

CTest

CTest

CTest

CTest

CTest

CTest

CTest

Automated testing of multi-module applications (WIP)The Dune ecosystem reliesheavily on modularization. Userapplications typically build ona stack of approximately fiveto twelve Dune modules. Allthese modules are separate gitrepositories. These intermodular dependencies create additionalproblems in continuous integration, such as:

• Choice of remote repositories for upstream modules• Choice of a compatible set of upstream branches/revisions• Distribution of build triggers towards downstream modules• Error feedback from selected downstream modules• Interoperability with CI systems of downstream frameworks

Dune has moved all of its development to a central GitLab instance,which is also used for CI. GitLab CI uses configuration files underversion control. We add a Dune-specific CI config file, that allowsadditional customization, such as upstream remote repositories.

featu

re/A

master

release

s/2.5

featu

re/A

master

release

s/2.5

featu

re/B

featu

re/C

Upstream

module

Testedmodule

Find a suitable upstreambranch by applying the fol-lowing strategies:• Look for manually con-

figured branch in CIconfiguration

• Look for manually con-figured branch in CIconfiguration of otherupstream modules

• Find the most suitablebranch:– Make a list of all heads in the tested repository– Order them by distance of the fork-point (with heuristics as tie

breakers)– Look for first match of a branch of the same name in upstream

repository

Distributing build triggers and results between modules managed onthe same GitLab instance, requires a server application that• runs within the GitLab docker container• implements any additional logic• uses the GitLab API for build triggers etc.

Additional logic is specified in the Dune-specific CI configuration file:

dune-common

dune-geometry

dune-grid

...

Tested module

CI configuration

Trigger from

upstream

Accept failures

f. downstream

dune-pdelab

dune-alugrid

User module

...

This approach can be extended to bridge multiple GitLab instancesto satisfy the needs of downstream frameworks like DuMuX.

[1] P. Bastian, M. Blatt, A. Dedner, C. Engwer, R. Klöfkorn, M. Ohlberger, O. Sander. A generic grid interface for parallel and adaptive scientific computing. Part I: abstract framework. Computing, 2008.[2] B. Flemisch, R. Helmig et al. DuMux: DUNE for Multi-Phase, Component, Scale, Physics, ... Flow and Transport in Porous Media. Advances in Water Resources, 2011[3] D. Kempf, T. Koch. System testing in scientific numerical software frameworks using the example of Dune. Proceedings of the Dune User Meeting 2015, accepted.