WHITE PAPER:

DESIGNING THE OPTIMAL OPEN SOURCE DEVELOPMENT ENVIRONMENT

PART 1 BY THOMAS ARNBJERG / TECHPEOPLE / 2018

1/40

Introduction 2 The process flow and the toolbox 2 Preparation 3 Installation of tools 3 Creating a ‘HelloWorld’ project and introduction to Eclipse 3 Eclipse concepts 6 Debugging 7 Starting a new feature 10 Implementing a new feature 10 New class 10 Editing live parsing 12 Code Completion 15 Unit test 15 Unit test results 16 Static code analysis 17 CppCheck 17 CppLint 18 Code formatting 19 clang-format integration 19 Making code uniform 19 Profiling 20 Memory profiling – leaks 20 Code coverage 21 Code review 21 AgileReview 21 Pull requests (GitHub, VSTS etc.) 22 Gerrit 22 Completing the feature 26 Embedded development 26 Remote file system 26 Shell commands 27 Cross-compilation 28 Remote debugging 30 Jenkins 33 Tasks 33 Visualizing the result 34 Advanced Jenkins – ultimate Jenkins 38 Multibranch Pipeline job 38 Rounding off 39

2/40

Introduction Software development has proved to be a very unpredictable discipline with regard to evaluation of quality, resource usage and duration of projects. There are many reasons for this and they can be found in different development phases. In this article we will be focusing on those close to coding. Writing programs may be compared to a team of different writers who are tasked with writing a very thick bestseller of thousands of pages. A medium-sized computer program may easily consist of 100,000 lines of text and is often developed in parallel by several software developers – and, in the course of time, developers are frequently replaced by others. At the same time, there must be 'coherence' to make a good storyline. So, the challenges include among other things:

● Ensuring that code written by different developers is uniform ● Ensuring that many developers may work on the same program at the same time ● Ensuring that each developer is able to QA his or her code before it’s integrated into

the system ● Ensuring that the overall system will function after changes are implemented

In the course of time, a large variety of tools and disciplines have been developed to assist in managing all these details – including: – integrated development environments which, among other things, provide assistance with formatting, troubleshooting and code completion. – test framework for unit tests, memory profiling, code coverage, etc. – build/continuous integration servers – tools for statistical code analysis – versioning systems

All these tools have existed for a long time in the open source world, but they have often been standalones or difficult to integrate. Commercial solutions are available, but they are often very costly which may bar purchases and ongoing updates. Below follows a description of a well-functioning, fully integrated and free set of tools which provide state-of-the-art support of C++ development on Linux. Many of these tools may also be used in other situations.

3/40

The process flow and the toolbox The description below details a process in which the individual developer creates a branch in the source code to develop a new feature without compatibility issues in relation to the work done by the other developers.

1. A developer starts a new feature. For this purpose, he or she starts a new feature branch

which means that it gets its own sandbox for the development work 2. The developer implements the feature. The development environment provides help by

way of code completion, checking for compliance of code standards and statistical code analysis. Help is also provided for developing unit tests and debugging, locally as well as remotely.

3. The developer finishes development of features and sends it all off for code review. Once this is done, the features branch is merged with the rest of the code base and the feature branch is discontinued.

The following tools are used to support the process – all installed under Linux: 1. Jenkins as Continuous Integration Server with a wide variety of installed plugins (see

https://jenkins.io/) 2. Eclipse IDE for C/C++ development with an extensive range of installed plugins 3. Git versioning systems

A general description of the workflow follows below. The description is supplemented by

screen dumps from strategic spots in the process. The process and screenshots are created on a machine with 'Opensuse Tumbleweed' installed (Gnome UI) and Eclipse Oxygen.3.

Preparation

Installation of tools Jenkins and Git are assumed to be already installed on a server.

Creating a ‘HelloWorld’ project and introduction to Eclipse First, we create a new C++ project, ‘HelloWorld’, as a ‘C++ Managed Build’. Thus, we leave it to Eclipse to generate and maintain the make file which builds the project.

4/40

Creating C++ Managed Build Project x86 (local) build.

We add a source code file with 'main' and save everything in the 'develop' branch.

● Creating the 'main.cpp' with 'main()' We are working in the ‘develop’ branch

Eclipse supports Git ‘out-of-the box’, but during installation of plugins you may also use other versioning systems – e.g. Subversion, Mercurial, VSTS, TFS, cvs and more. We use the ‘Default C++ source template’ for creating ‘main.cpp’. Naturally, this template (and all others) can be customized with the project preferences. In ‘main.cpp’ we add a ‘main’ function that simply writes ‘Hello World’ in the console.

5/40

The content of ‘main.cpp’ with standard formatting and colours.

Changes are saved in Git (see later) and the project is built. Now we have created a complete HelloWorld program in our editor.

The Eclipse editor after completion of ‘HelloWorld’. The red rectangles mark selected conceptual areas in Eclipse.

6/40

Eclipse concepts In the figure above, a number of conceptually important areas are marked. ‘Perspectives’: Eclipse is a highlymodular platform (OSGI based) which may be extended with new functions through plugins. There are thousands of add-ons – some of which add support for entire programming languages and others just add a few functions. To keep complexity in the user interface manageable, all functionality is mapped to one or several perspectives. Only functions relevant to the active perspective are visible. The chosen perspective is displayed at the top, for example ‘C/C++’. Later, we’ll take a closer look at what happens when we switch to the ‘Debug’ perspective. ‘Project Explorer’: All projects in the active workspace are displayed to the left. At the moment we only have ‘HellowWorld’, but we will add more later on. When double-clicking ‘main.cpp’, the file opens in the central editor as illustrated. Moreover, the project explorer shows that we are working in the ‘demo_workspace’ workspace and ‘develop’ branch in our versioning system. The tiny yellow box in the icon for ‘HelloWorld’ and ‘main.cpp’ shows that no changes have been made to our local workspace compared to the versioning system. ‘Outline’: A ‘map’ (‘outline’) of the selected file is displayed to the right. When navigating the file, you simply click an element (e.g. ‘´Main’), and then the editor scrolls to the selected section. ‘Views’: A row of ‘views’ with various information is displayed at the bottom. We have just built ‘HelloWorld’ and the ‘Console Output’ tab is selected.

7/40

Debugging Now, we are going to test ‘HelloWorld and start a debug session.

Starting a debug session

Eclipse starts a debug session and switches to the debugging perspective as indicated in the top right corner.

8/40

A debugging session against ‘HelloWorld’. Eclipse has switched to the ‘Debug’ perspective and this is indicated in the top right corner

The Debug perspective displays the thread’s status and the call stack in the top left corner. By default, a break is made at the first line in ‘main’, indicated by the blue arrow in ‘main.cpp’ in the lower left corner. The usual functions for a debugging session – ‘Continue’, ‘Pause’, ‘Stop’, ‘Step Into’, ‘Step Over’, and ‘Step out of’ are displayed at the top center. Various ‘debugging views’ are displayed to the right with ‘Variables’ selected, and we can see that the value of ‘argc’ is 1. We enable ‘HelloWorld’ by clicking ‘Continue’ (the green arrow), and the program ends after ‘HelloWorld’ is written in the console as indicated. Notice that the console output for the latest debugging seance is always available when you go to the ‘Debug’ perspective and choose the completed process in ‘Debug’. If several applications have been debugged, there will be an equivalent section for each application.

9/40

Completed debugging session of ‘HelloWorld’

Eclipse supports a wide array of simultaneous debug sessions – also of different types (e.g. 2 local – a Java and a c++ at the same time with two remote debugging seances against an ARM target). Return to the C++ perspective by clicking the ‘C/C++’ perspective button.

Selecting the C/C++ perspective.

10/40

Starting a new feature We will now create a new feature; instead of writing ‘Hello World’ in the console, we will write the message found in an instance of the ‘Greeting’ class. Therefore, we’ll start by branching out to a new ‘feature_greeting’ branch

Starting a new feature Workspace now switches to the feature branch.

Eclipse switches to the new feature branch as illustrated in ‘Project Explorer’.

Implementing a new feature

New class We add a new class, ‘Greeting’, and choose to create a unit test for it. The unit test is placed in the ‘tests’ sub folder.

11/40

Creating the new ‘Greeting’ class with appertaining test.

The new class is now displayed in the ‘Project Explorer’. Since the unit test is not part of the ‘HelloWorld’ application itself, we have created an exclusion filter for the ‘tests’ folder to exclude the source code found here from the ‘HelloWorld’ build. This is indicated by striking through the name. New files are marked with a blue question mark which indicate that there is, as yet, no versioning. Finally, a brown ‘dirty’ marking of the ‘HelloWorld’ root itself is seen as an aggregated status indicating that changes have been made to the tree below.

12/40

After creating the ‘Greeting’ class with appertaining unit test.

Two tabs have been opened in Eclipse for editing the header and the body file. The contents of both files are determined by default formatting of file templates which are of course configurable.

Editing live parsing We now add an argument for the constructor for ‘Greeting’. As we are typing, Eclipse is live parsing changes and updating status as illustrated.

13/40

Live parsing of ‘Greeting.h’.

The figures mentioned above illustrate the following: ● The small star in the ‘Greeting.h’ title indicates that the files has not yet been saved. ● In the editor for ‘Greeting.h’, a red wavy line indicates that there is a problem with the

‘std::string’. ● Mousing over the red bug to the left of the editor displays the cause – ‘Type

std::string could not be resolved’ (because we have not included the header). ● A small red marking is displayed in the right-hand side of the editor. When clicking it,

the editor scrolls to the marking. Naturally, this is not relevant for a file as small as this one, but the feature is useful for large files.

● A red X in both ‘Project Explorer’ and ‘Outline’ in the left and right-hand side indicates that there is an error.

● Last but not least the ‘Problems’ view points out that there is an error in the ‘Greeting.h’ file on line 13 and that this is a semantic error.

Eclipse is now requested to add the missing header and save. The red marking disappears.

14/40

Automatic addition of missing headers.

We initiate a new build and another series of errors are displayed because we did not add the new constructor parameter to the cpp file which can be seen in the build console.

Console output of build with missing argument in constructor.

Double-clicking the line in the console makes Eclipse open ‘Greeting.cpp’ and highlight the error.

15/40

We correct the error and add the ‘greeting’ attribute to the class as illustrated. After that, the build moves ahead without any problems.

New ‘greeting’ attribute in the header file

Edit of the cpp file

Code Completion ‘main.cpp’ now has to be modified to use the new class. Consequently, we are adding a new line at the beginning of ‘main’ and declare a ‘greeting’ variable of the ‘Greeting’ type as illustrated (<Ctrl>+<Space> for code completion).

The greeting variable is introduced. Code completion is activated with <Ctrl>+<Space>

When run, the program will now write ‘Hi there’ in the console.

Unit test We still need to add a unit test to our ‘HelloWorld’. Therefore, we create a unit test project which shares its source code with ‘HelloWorld’. If we make any changes in ‘HelloWorld’, this is then mirrored all through the unit test project. The figure below illustrates the setup.

16/40

Unit test program ‘HelloWorldUnitTest’ for HelloWorld. The code in ‘HelloWorld’ has been imported as a link and is common for the two projects.

The new project ‘HelloWorldUnitTest’ in ‘Project Explorer’ is shown in the figure above. Notice the ‘HelloWorld’ sub folder which has an icon with a small arrow on top. The arrow shows that this is a link, and it is this mechanism which allows the code to be reused in several projects. We have moreover created an exclusion filter for HelloWorld’s main.cpp since the test application must have its own main function in ‘testmain.cpp’. Notice also that the ‘tests’ folder has NOT been filtered out in ‘HelloWorldUnitTest’ as opposed to ‘HelloWorld’, because we are now going to use the unit test source code.

Unit test results Eclipse supports unit test frameworks which will provide reports in Boost, Google test or TAP format. By way of example, we use catch2 with TAP reporter here.

17/40

Unit test run in Eclipse. A file has been made on purpose on the other ‘REQUIRE’ line. The results show one error

Static code analysis

CppCheck Eclipse integrates with CppCheck. We make a minor change in the ‘Greetings’ class thereby changing the constructor, so the argument is not const ref.

Changing ‘Greeting.h’, where the constructor argument is no longer const ref.

Now, we activate CppCheck and get warnings in ‘HelloWorld’ as well as ‘HelloWorldUnitTest’ (since the code is shared). The editor indicates the warning with a yellow wavy line.

18/40

Result of running CppCheck. Several warnings are now displayed in the various views.

Notice that in Project Explorer and Outline, yellow warning triangles are now used as warning indications. The rules which are to be checked and the severity attached to each violation can of course be customized.

CppLint Google has created a Python script, CppLint.py, for checking compliance with their C++ coding standard and others have adapted the tool according to their specific requirements. CppLint also integrates with Eclipse and the result of scanning ‘HelloWorld’ with CppLint is shown below.

19/40

As is the case with CppCheck, the individual rules may be turned on or off and severity may be configured. It is also possible to customize CppLint.py directly or to fetch a derived version.

Code formatting

clang-format integration Some of the things found by CppCheck and CppLint in the previous section are due to simple source code formatting. For example, CppLint complains about tab characters instead of spaces. A number of these may be removed via Eclipse integration with the clang-format tool. clang-format is customizable in a huge variety of ways, but out-of-the-box Google’s style, Webkit, Mozilla, LLVM and Chromium are supported. Eclipse can be configured to auto format code when saving. The result is that all trivial warnings/errors disappear.

Making code uniform Now, we will update HelloWorld to satisfy CppCheck and CppLint. This makes the code uniform. The finished result is shown below.

20/40

The result after making the code uniform with regard to compliance with CppCheck and CppLint

The direct result is that all code now has uniform formatting with regard to brackets and use of spaces. All files now have a Copyright notice, all header guards are uniform, etc.

Profiling Eclipse integrates with default GNU tools like Valgrind and GCOV.

Memory profiling – leaks See below how Valgrind reports an introduced memory leak in which memory is allocated to an array but delete does not match the array allocation.

Reporting a memory leak by the Valgrind view. We allocate memory for an array but only

21/40

release a single item.

Code coverage If ‘HelloWorld’ is run with GCOV profiling turned on, the result is a report like the one shown below:

Result of running with GCOV profiling turned on.

This means that we will be able to see which lines are affected by a given run.

Code review

AgileReview AgileReview provides code review directly in Eclipse. Reviews can be created in a special review project type and the comments may be mapped to the source code directly in the feature branch. Below follow examples of review comments directly in Eclipse for HelloWorld.

22/40

Code review of HelloWorld directly in Eclipse.

The review comments are shared with other developers via the versioning system – which in this case is Git.

Pull requests (GitHub, VSTS etc.) Code review can be done in a number of ways. If the project uses a hosted solution like Github, the service may have built-in code review features in connection with a pull request (by the way, Github may be fully integrated in Jenkins, but this belong to another and separate subject).

Gerrit Gerrit functions as a step between Jenkins and the development environment – (see https://www.gerritcodereview.com/). This drawing is taken from https://www.gerritcodereview.com/ and illustrates the way in which code changes are reviewed before being reforwarded to Jenkins.

23/40

Gerrit Concept (drawing copied from https://www.gerritcodereview.com/)

Configuration of Gerrit is likewise too large a subject to be included in this article – it should be mentioned, however, that Eclipse has several plugins for supporting Gerrit which can be seen by searching the Eclipse Marketplace (‘Help->Eclipse Marketplace’ menu).

24/40

Gerrit integration in Eclipse

Notice, however, that the choice of branching strategy and review tool are typically related.

Completing the feature Consequently, the feature development may be completed:

● When all parts of the feature have been implemented ● When all unit tests are carried out satisfactorily ● When code coverage is adequate ● When memory profiling does not exhibit any problems ● When staticcode analysis with CPP Check and CPP Lint does not trigger any

warnings All checks are carried out in the feature branch and thus BEFORE integration into the common code base. After that, the feature can be completed in Eclipse.

25/40

Completing the feature.

Eclipse now merges the feature branch into ‘develop’ and switches workspace to ‘develop’.

26/40

The result of the completion of a feature.

Merge is only carried out against the local Git copy which is indicated by the upward arrow displayed in Project Explorer. It can now be verified that everything still works against the local ‘develop’, before it is all pushed to the Git server.

Push to Upstream Now, nothing further needs to be pushed to the Git server.

Embedded development Eclipse supports shared code among projects as was demonstrated in connection with the unit test program for ‘HelloWorld’, enabling us to seamlessly create a link to the real ‘HelloWorld’ source code. Eclipse also supports cross-compilation and remote debugging against Linux. A new cross-compiled project is therefore easily created in which the code from ‘HelloWorld’ is reused. All in all, there is no big difference between developing against a x86 application and an ARM application. The simplest way of doing this is if the Linux platform supports SFTP. The barrier between the development machine and the embedded device is practically eliminated here.

Remote file system The screen below shows the Eclipse user interface with the ‘Remote System Explorer’ perspective selected and connection established to a Raspberry Pi.

27/40

Eclipse in the Remote System Explorer perspective (as can be seen at the top).

Two nodes are displayed at the left. ‘Local’ is the development machine and when exploded the file system is displayed. Below a Raspberry Pi with the IP address 192.168.1.42 is displayed. We are in ‘My Home’ which is the pi user’s home folder and ‘Root’ which is ‘/’. Files can be moved between the two using copy-paste.

Shell commands It is also possible to start a remote terminal. Thus, all shell commands may be executed.

28/40

Starting terminal Remote terminal against a Raspberry Pi

In the figures below, two terminal sessions have been started – in one of which we have been running ‘top’.

Two terminal sessions. In one of them we are running ‘top’.

Cross-compilation We are now going to create a cross-compiled project in which we reuse the code from ‘HelloWorld’ and create a debug session. First, the ‘RpHelloWorld’ project is created.

29/40

Selecting cross-compilated project Tool chain settings.

The result is displayed below. Again, we have created links to the source code in ‘HelloWorld’ and then built the project.

30/40

Cross-compiled ‘HelloWorld’.

As indicated, we now have an ‘arm/little endian’ binary which is built with the rp tool chain.

Remote debugging We will now configure a remote debug configuration enabling us to carry out debugging in target.

Configuring remote debugging.

When clicking ‘Debug’, ‘RpDebug’ is downloaded to the target, gdbserver is started, and a break is carried out in the first line of ‘main’ as shown.

31/40

Starting a remote debugging session against Raspberry Pi.

So, this is the exact same experience as when developing against the x86 machine (though with a somewhat longer start time because download has to be made to target). If a ‘ps aux’ is made in the terminal we created a short while ago, the debugging is seen more clearly.

32/40

The debugging seance as seen in a remote terminal with the ps command.

You are able to do exactly the same as in an ordinary debugging session, which means single stepping, evaluating variables, etc. Moreover, several sessions may be started at the same time. They can easily exist side by side – the port on the gdbserver must be different though.

33/40

Jenkins

Tasks The previous sections detailed the way Eclipse may be used to support all relevant phases of the development of a feature. The continuous integration server is placed at the other end, ensuring that everything can be built and that all tests are run. When changes are pushed to the Git server, they are detected by Jenkins which will then do the following:

● Run CPPCheck on the code ● Run CPPLint on the code ● Build HelloWorld ● Build HelloWorlUnitTest ● Run HelloWorldUnitTest and evaluate whether the unit test has succeeded ● Run Valgrind profiling and publish the results ● Run GCOV profiling and publish the results ● Count the number of code lines in the various languages and publish the results

Visualizing the result The following figures illustrate the result:

34/40

CppCheck, Unit Tests (TAP format) and Code Coverage.

35/40

SLOCCount (Source Lines Of Code Count), Valgrind analysis

36/40

Coverage details

37/40

SLOCCount details

CppLint details

38/40

CPPLint error – at the far bottom.

This means that Jenkins carries out QA on the consolidated code base on an ongoing basis.

Advanced Jenkins – ultimate Jenkins The previous sections showed us how Jenkins may be configured to build the Eclipse project and carry out a number of quality related activities before a build is accepted. The configuration was made against a single branch in Git (develop), but we can actually improve on that. The key terms are ‘Multibranch Pipeline’ and ‘Jenkinsfile’.

Multibranch Pipeline job Multibranch Pipeline jobs in Jenkins allow Jenkins to clone the build configuration dynamically when new branches are created in the source control tool (in this case Git). The specific meaning of this is that once the feature branch is created during development, all QA services carried out by Jenkins (unit test, static code analysis, etc.) are cloned to the new feature branch thus facilitating QA of the feature BEFORE merging back to ‘develop’ (in the GitFlow branching pattern). This eases integration quite substantially, and in reality there is no longer any excuses for breaking the build in ‘develop’, since all integration issues in the feature branch ought to have been solved. This automatic cloning of build setup is based on the assumption that the branch contains a file named ‘Jenkinsfile’ which describes all the activities to be carried out in a build. The ‘Jenkinsfile’ is typically placed in the branch in which the feature branch is created – in this article this is the root of ‘develop’.

39/40

Rounding off In the previous sections we have demonstrated how a fully integrated development environment around Jenkins, Eclipse and Git (and GitFlow) can be configured.

Developers will see how, at the start of a new feature, they get their own sandbox for the development. They receive tools support for development with code completion and formatting. On the build server, their codes are controlled using static code analysis and unit test. Code coverage is measured and quality criteria/goals can be set out. Once a feature is completed, ‘develop’ is merged into the feature branch and it is ensured that everything runs in Jenkins. After that, there is a merge back to ‘develop’ and the feature is finished. All things considered, the integration task becomes so much simpler and most of it happens in the feature branches.

Everything is open source-based without costs for licenses and all the projects go several years back and are actively maintained. Following up on development and upgrading on an ongoing basis is thus manageable.

Last but not least, giving something back to those who have made it possible to create a professional development setup that does not require investment of large amounts of money is considered to be good style. You might perhaps consider donating part of the savings to the open source projects or contribute with developer resources to make the experience even better.