institutionen för datavetenskap - diva portal802248/fulltext01.pdf · 1.4. questions chapter 1....

Institutionen för datavetenskap Department of Computer and Information Science

Final thesis

Continuous Integration – A comparison

between theory and practice

by

Martin Sandberg

LIU-IDA/LITH-EX-G--15/005--SE

2015-02-27

Linköpings universitet

SE-581 83 Linköping, Sweden

Linköpings universitet

581 83 Linköping

“masterthesiseng” — 2015/4/7 — 13:10 — page — #1

Final thesis

Continuous Integration - Acomparison between theory and

practice

by

Martin Sandberg

LIU-IDA/LITH-EX-G–15/005–SE

2015-02-27

Supervisor: Johan Glannfjord

Examiner: Kristian Sandahl

“masterthesiseng” — 2015/4/7 — 13:10 — page — #2

“masterthesiseng” — 2015/4/7 — 13:10 — page i — #3

Abstract

To develop software in a larger scale, some kind of software developmentmethod is needed to organize the development. Many different softwaredevelopment methods have emerged since the beginning of software devel-opment, and Continuous Integration (CI) is one of them. Many companieshave applied, or tried to apply CI in their business. Some companies weresuccessful, some were not. This thesis examines what CI is in the theory,and compare it to how it works in the practice in the department ”Prod-uct Development Traffic Control” (PD TC) which is a part of the ”ProductDevelopment Unit LTE Multistandard Radio” (PDU LMR) within Erics-son. The theory of CI is examined mostly through literature studies. CI inpractice was examined through interviews with developers and employeesworking with the CI-machinery at PD TC. The comparison between thetheory and the studied company revealed that it is difficult to adapt CI per-fectly to a large organization. This does not imply that CI is implementedin the wrong way, but the benefits of implementing CI in large projects maynot be as enormous in comparison with smaller projects.

i

“masterthesiseng” — 2015/4/7 — 13:10 — page ii — #4

“masterthesiseng” — 2015/4/7 — 13:10 — page iii — #5

Contents

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.6 Prerequisite . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 What is CI? 32.1 The basics with CI . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Example of a CI-system . . . . . . . . . . . . . . . . . . . . . 52.3 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3.1 Reduce Risk . . . . . . . . . . . . . . . . . . . . . . . 62.3.2 Less work . . . . . . . . . . . . . . . . . . . . . . . . . 62.3.3 Deployable software at any time . . . . . . . . . . . . 6

2.4 Disadvantages . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.4.1 Takes time to start up and to maintain . . . . . . . . 82.4.2 Extra expenses . . . . . . . . . . . . . . . . . . . . . . 8

3 What is needed? 93.1 Version Control System . . . . . . . . . . . . . . . . . . . . . 93.2 Developers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 CI-server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.4 Build script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.4.1 Code-inspection . . . . . . . . . . . . . . . . . . . . . 113.4.2 Compiling . . . . . . . . . . . . . . . . . . . . . . . . . 113.4.3 Database integration . . . . . . . . . . . . . . . . . . . 113.4.4 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 123.4.5 Deployment . . . . . . . . . . . . . . . . . . . . . . . . 12

3.5 Feedback mechanism . . . . . . . . . . . . . . . . . . . . . . . 12

iii

“masterthesiseng” — 2015/4/7 — 13:10 — page iv — #6

CONTENTS CONTENTS

4 PDTC’s CI-system 134.1 VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.1.1 Rebase . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2 Developers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.3 CI-server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.4 Build-script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.4.1 Code-Inspection . . . . . . . . . . . . . . . . . . . . . 164.4.2 Integration . . . . . . . . . . . . . . . . . . . . . . . . 164.4.3 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.5 Feedback mechanism . . . . . . . . . . . . . . . . . . . . . . . 17

5 PDTC’s CI workflow 195.1 Phase 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.1.1 Review . . . . . . . . . . . . . . . . . . . . . . . . . . 205.1.2 Verification . . . . . . . . . . . . . . . . . . . . . . . . 21

5.2 Phase 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.2.1 Mainline Delivery Queue . . . . . . . . . . . . . . . . . 225.2.2 Builds and tests . . . . . . . . . . . . . . . . . . . . . 225.2.3 SDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

5.3 The end? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6 Experiences among the developers 24

7 Analysis 26

8 Improvements 288.1 No daily basis . . . . . . . . . . . . . . . . . . . . . . . . . . . 288.2 More automation . . . . . . . . . . . . . . . . . . . . . . . . . 298.3 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

9 Discussion 309.1 Ethic & environment . . . . . . . . . . . . . . . . . . . . . . . 31

10 Dictionary 32

11 Interviews 3411.1 Interview 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3411.2 Interview 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3511.3 Interview 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3511.4 Interview 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

iv

“masterthesiseng” — 2015/4/7 — 13:10 — page 1 — #7

Chapter 1

Introduction

1.1 Motivation

Continuous Integration (CI) has become a very important concept in devel-opment of the LTE system at Ericsson. Implementation of CI-mechanismshas mainly been carried out in order to improve the quality of the product,but also to increase efficiency of the development and reduce the deliverytime to the market. CI-mechanisms are crucial to the organization, and tobe able to gain full potential of the mechanisms and to meet future demandsit is fundamental to improve the understanding of the mechanisms.

1.2 Purpose

The purpose with this thesis is to study how Continuous Integration is sup-posed to work according to the theory, and investigate how it works in thepractice at PD TC. These differences will be analyzed to evaluate whetherthe current way of working is the proper way, or if there are any possibleimprovements to perform.

1.3 Method

The structure of the work behind this thesis could be described with fol-lowing phases: First, literature studies within CI was performed to get abasic understanding of what CI is. After this phase, the examination of CIin PD TC was performed. It was achieved through workshops, discussionsand interviews with developers as well as employees working with the CI-machinery. I had very good contact with a cross functional team (XFT),that the thesis is mainly based on. I reconnected to the theory continuouslywhile examining the CI in practice to get a deeper understanding of thedifferences between practice and theory.

1


1.4. QUESTIONS CHAPTER 1. INTRODUCTION

1.4 Questions

• What is Continuous Integration according to the theory?

• How does Continuous Integration work at PD TC?

• What are the differences between PD TC’s adaption of CI and thetheory?

• Are there any possible improvements that could be considered at PDTC?

1.5 Limitations

Following limitations have been decided due to the limited scope of thisthesis:

• There will be no investigation of how Continuous Integration is imple-mented in other departments within Ericsson, or companies outsideEricsson.

• When investigating the CI-machinery, focus will be on the part ofLTE’s CI-machinery that PD TC is using.

• There will be no implementation of eventual improvements found.

• I have chosen not to include deeper theories or ideas about CI, but theoverall description of what CI is.

1.6 Prerequisite

This thesis is written with following prerequisites:

• Basic knowledge in computer engineering.

• Basic knowledge of what a version control system is.

2


Chapter 2

What is CI?

CI stands for Continuous Integration, and was initially taken from the Daily-build method by Microsoft in the 90’s [1]. It was later included in the soft-ware development method eXtreme programing (XP), in the end of the 90’sas one of it’s original methods [2]. The name eXtreme programming comesfrom the idea of taking all ”best-practices” to an extreme level [3]. The goalwith CI was to prevent the occurrence of major problems in the course of thesoftware integration, through integrating more frequently [3]. In a softwaredevelopment project where several developers are involved, the integrationprocess really is something that can contribute with major problems. In-tegration is a very unpredictable step in the development process, and caneasily be more time consuming than the actual programming process [3,p. 49].

Due to this, it is significant to test the whole system frequently alreadyfrom an early stage. But often this is not done. Instead, each individualpart is developed until it is finished before testing of the whole system takesplace [4].

Here is where CI comes in. CI is basically a concept to easily integrateall code that is written, and also compile and test it more frequently. Andthis should be done more or less automatically. Through this approach,developers should submit the code to the CI-server on daily basis. The CI-server integrate and test the code automatically. It doesn’t imply that thisprocess solves all the issues, but when integration problems occur, developersfind out this already a short while after a faulty line of code is written. Thus,CI provides much faster feedback than if this should be done manually inthe end of the development process [4].

3


2.1. THE BASICS WITH CI CHAPTER 2. WHAT IS CI?

2.1 The basics with CI

The basics with CI could be described in short as [4]:

• To integrate more often. The faster the code integrates, the fasterthe developer finds out if it works with the rest of the code imple-mented by other developers. With integration means that the differentcomponents of the code shall be put together, think of it like a puzzle.

• To improve efficiency by testing all the code more frequently.Through this, the search area gets smaller once problems arise.

• Reduce the manual work by automating as much as possible interms of compiling, testing, database integration and more.

• Be able to produce functional, executable software at anytime that has already been tested and is ready to be delivered.

4


2.2. EXAMPLE OF A CI-SYSTEM CHAPTER 2. WHAT IS CI?

2.2 What could a CI system look like?

Below follows one example of how a CI system could look like. It is possibleto adapt CI in different ways, depending on the different needs, but this isthe fundamental idea [4].

1. A developer submits his or her code to the repository in the versioncontrol system. It is of great importance to write new or change oldtest cases for the added or changed functionality. These test casesshall be submitted together with the updated code to the repository.

2. After code submit, the CI-server discovers that a change has occurred,and fetches the new code from the version control repository. After thisthe CI-server executes a build script that contains compiling, databaseintegration and the execution of the updated test cases. What is in-cluded in this step could vary, but according to the theory, this is thefundamentals.

3. When the build1 is finished, the CI-server sends direct feedback to theinvolved project members.

Figure 2.1: Illutsration of a CI-system [4, p. 40]

1What the definition of a build is could vary, but in this thesis a build includes notonly compiling and integration, but also testing

5


2.3. ADVANTAGES CHAPTER 2. WHAT IS CI?

2.3 Advantages

2.3.1 Reduce Risk

The fundamental idea with CI is that every developer shall submit theircode at least on daily basis. This leads to many integrations every day. Atevery integration, several tests should be executed to ensure that everythinglooks ok. With the help from the CI-server’s feedback mechanism it enablesthe developers to get rapid feedback on their code, which results in [4]:

• The developer gets to know about the faults that get introduced inthe code already when the code is submitted to the version controlrepository, and not in a later phase of the project [4, p 24].

• It becomes easier to get an overview on the status the software has inreal-time [4, p 31].

• If the code is tested with the same process against the same scriptsall the time, there will be less assumptions about the software qualityand/or the reliability of third party modules [4, p 23-24].

• The feeling about the quality of the software gets better. This isbecause it is possible to at any time see what a certain version ofthe software has been tested against, and if it has passed or failedthese tests. Without CI-mechanisms it becomes more difficult to sayto what extent a new piece of code will affect the rest of the software[4, p 31-32]

2.3.2 Less work

It becomes much easier when having a CI-server that takes care about muchwork that usually has to be done manually. It can be tasks as compiling,database integration, testing, code inspection and feedback on the qualityof the code. In this way the project members get more time to do otherimportant work. It also becomes less risky to miss some step in the processbecause everything is run automatically at every code submit [4].

2.3.3 Deployable software at any time

Due to the early and continuous integration, testing and deployment, itis possible to deliver the software as a ”whole product” to an end-user orcustomer at any time. The product may not contain all functionality in theearly part of the project, but it has at least been tested and deployed. Thismakes it a lot easier to ship the product to the customer. If not using CI, andintegrating occurs for the first time in a later part of the project, it is likelyto encounter unexpected problems. Suddenly, when integrating, intractableissues emerge, that puts the deadline at stake due to unpredictable time

6


2.3. ADVANTAGES CHAPTER 2. WHAT IS CI?

consumption to solve the issues. In worst case, this could lead to loosing acustomer. This is not preferable for any project [4].

7


2.4. DISADVANTAGES CHAPTER 2. WHAT IS CI?

2.4 Disadvantages

2.4.1 Takes time to start up and to maintain

To be able to obtain a fully functional CI-machinery time to set it all up isneeded. If there is a need to change the way of working and a high numberof processes to obtain CI this can be quite time consuming. There is alsosome time needed to maintain CI [4]. In very small projects this might notbe suitable.

2.4.2 Extra expenses

To preform CI in a good way, separate hardware to run the integrationserver on is needed. This can be seen as an expensive investment [4].

8


Chapter 3

What is needed toimplement CI?

3.1 Version Control System

First of all, a version control system (illustrated in figure 3.1) is needed. Aversion control system manages a central or distributed storage space, wherethe developers fetch and submit their code to [4][7]. This space is called arepository [4]. First, the developer fetches the code from the repository.Then the developer implement his or her changes to the code. After this,when the developer is finished, he or she submit the updated code backto the repository which makes it possible to other developers to fetch thenew changes. The version control system record all changes to the files itmanages in the repository [5]. It comes with possibilities such as to [5]:

• Compare a file with an older version of that file to see what changesthat has been made.

• See who made the changes that might introduced a problem to thecode

• Revert one or several files to older versions

There are many different types of version control systems on the market, butPaul M. Duvall, the author of Continuous Integration - Improving SoftwareQuality and Reducing Risk advocates the use of Subversion in his book,because it is freely available, and it has a set of useful features [4, p. 8].

9


3.2. DEVELOPERS CHAPTER 3. WHAT IS NEEDED?

Figure 3.1: Illustration of a centralized Version Control System.

3.2 Developers

As a developer, it is very important to assure that the code is working asintended before submitting it to the repository. This is to prevent to breakthe code in the repository. Therefore, all developers should compile, testand integrate their code on their local workstation before submitting. Thetests should be implemented by the developers themselves. It is enoughto only execute unit tests1 of the implemented code before submit. In thatway, there is less risk that the code in the repository is going to break. Aftereverything works on the local workstation, the developer submits the codeto the repository. It is important to not forget to also submit the newlyimplemented test cases as well. Further on, a rule of thumb is that eachdeveloper should submit at least on daily basis. In this way there will bemuch easier to troubleshoot when something goes wrong, because the searcharea gets smaller [4].

1Unit test = Test of the smallest testable piece in the code, e.g. a class

10


3.3. CI-SERVER CHAPTER 3. WHAT IS NEEDED?

3.3 CI-server

An integration server, preferred dedicated, is needed to host a CI-server.The CI-server’s mission is to with short intervals pull the version controlrepository for changes. If a change has occurred, the CI-server fetch thenew or updated code and tests and executes a build script [4].

3.4 Build script

The build script is a script that normally executes following tasks:

• Code-inspection

• Compiling

• Database integration

• Tests

• Deployment of the software

The whole process executed by the build script is called a build (yes,a build could be more than just a compile) [4, p. 4]. What is included inthe build script can differ, but above tasks is the fundamentals. It is veryimportant that the build script does not take too long time to execute. Arecommendation according to the theory of CI is top 10 minutes. The reasonfor this is that only one developer can build his or her code at a time. Ifthe build script takes too long time, it becomes a bottleneck. The tasksexecuted by the build script is described in the following text below.

3.4.1 Code-inspection

It is important to have some kind of code inspection in the build script.Maybe the project have decided different rules or criteria that the sourcecode should follow? For instance, automated checks that guarantee theserules and criteria are fulfilled could be included[4].

3.4.2 Compiling

Is one of the basic features of CI. It means that the CI-server should automat-ically compile the source code submitted to the version control repository,and make an executable file [4].

3.4.3 Database integration

Some people might think that database integration and code integrationare two different things. It is, but why not automate that step to? Thedatabase integration part can be a simple script where all the data tablesare dropped, and created again (but with new data) [4].

11


3.5. FEEDBACK MECHANISM CHAPTER 3. WHAT IS NEEDED?

3.4.4 Testing

Maybe the most important part of the build script is the testing part [4].The developers should (as mentioned in section 3.2) implement and executeunit tests before submit. But hopefully there are a lot more tests that existstogether with the rest of the code. If all these tests should be executed thiscan easily make the build script take more than the suggested 10 minutes[4, p 141]. Therefore it is very important to have some strategy to somehowcategorize the different test cases. A good idea to have in mind to speed upthe testing is to start small. For instance, in the build script that triggeredby every code submit, execute only unit tests. Then a second build scriptcould be implemented, that executes periodically. For instance, once everynight. This build script could contain the more time consuming tests likecomponent tests, system tests and functional tests [4, p. 138].

3.4.5 Deployment

Finally we have come to the last part in the build script. After the abovesteps have passed, it is time to deploy the software. Deployment is the stepwhen the software is executed and tested in an environment as ”customer-like” as possible. This can be considered as the last step in the testingpart.

3.5 Feedback mechanism

When the build script has been executed, a result shall be sent to the de-veloper that submitted the code. This can be done through RSS, e-mail ortext message to name a few examples. The build script has to pass all thedifferent tasks to pass. It is enough with just one failed task, and the resultis set to ”fail”. When a task in the build script is failed, it is of highestpriority to fix this. It is the developer who submitted the faulty code thathas the responsibility for this [4].

12


Chapter 4

PDTC’s CI-system

4.1 Version control system

PD TC is using the version control system Git. Git was created 2005 tomanage the source code for the Linux core. Git is known as a rapid anddistributed version control system, and probably the best thing with Gitcompared to other version control systems is its branching model [7].

Branching could basically be described as making a copy of the existingcode in mainline, and start a ”new project” with it. This is not what actuallyhappens, but it is easier to describe it that way. The mainline is the ”codefoundation” which should be fully tested and fully working. New code iswritten upon this code (often in form of a branch) and later on merges backinto the mainline to become the new mainline. It is widely used within PDTC, as a way to enable many parallel activities at the same time withoutmessing with the mainline. For example each new feature that is beingdeveloped is normally started as a new branch. Figure 4.1 illustrates ina very brief way how branching works from creation of the branch to theintegration back into mainline. Commits to the mainline shown in figure4.1 could be commits from other branches or commits made directly on themainline.

Figure 4.1: Illustration of a branch.

13


4.1. VCS CHAPTER 4. PDTC’S CI-SYSTEM

4.1.1 Rebase

Before integrating the code back to mainline, a rebase is needed to get thelatest changes from mainline. A rebase is a command in Git and basicallychanges the base1 of a branch, to integrate recent changes performed onmainline into the branch. In this case, eventual integration problems occursoutside the mainline which is very good.

Figure 4.2: Illustration of a branch merged back into mainline (same asfigure 4.1). The branch is based on mainline commit ”MC1”.

Figure 4.3: Instead of basing the branch on mainline commit ”MC1”, itis now rebased and based on the latest mainline commit ”MC3”. Whathappens here is basically that you add all the code implemented in thebranch upon the newer mainline commit ”MC3” instead.

Figure 4.4: The branch now have changed base to ”MC3”, and when in-tegrating back to mainline it becomes much easier since there are no morechanges performed on mainline since last rebase.

1A base is the commit in the mainline where the branch was started or previouslyrebased from

14


4.2. DEVELOPERS CHAPTER 4. PDTC’S CI-SYSTEM

4.2 Developers

As a developer in a cross functional team at PD TC the responsibilities areboth to implement code and test cases (described more in detail in section4.4). Instead of executing these tests locally before submitting the codeto the mainline as described in theory, these tests executes on a dedicatedserver. At PD TC there are no requirements how frequently the code shallbe submitted to the CI-server, this responsibility is on the team. Someteams submit their work more often than others, and usually the integrationproblems increase at less frequent deliveries.

4.3 CI-server

What is meant to be a CI-server in the theory I rather prefer to call ”CI-machinery” in the practice. The reason for this is mainly because the ex-pression “CI-server” is not very correct, since it implies the use of only onemachine, one CI-server. In the parts of LTE’s CI-machinery PD TCis using,there are actually several machines taking care of the functionality of theCI-server. However, when talking about the CI-server or CI-machinery werefer to the actual software used in the server(s).

The first part of the CI-machinery is Gerrit. It is to Gerrit the devel-opers submit their code, and also where the CI-machinery begins. Gerritis basically a web based code review system where the developers revieweach others code [10]. It is closely integrated with Jenkins to easily get anautomated process chain. Jenkins is a tool that at it is website is describedas ”an extendable open source continuous integration server” [11]. It is de-signed to automatically build, integrate and test software [11]. When newcode has been submitted to Gerrit, Gerrit communicates with Jenkins toperform the integration and the testing.

In the theory the code is first put in a version control repository and thenpulled to the CI-server. At PD TC the code first arrives to the ”CI-server”,and if the build passes, it is sent to the repository. Gerrit could be describedas a gate keeper to the repository, protecting it from broken builds.

4.4 Build-script

What is called ”build script” in the theory we can say is composed of thefollowing at PD TC:

• Code-inspection

• Integration

• Tests

15


4.4. BUILD-SCRIPT CHAPTER 4. PDTC’S CI-SYSTEM

4.4.1 Code-Inspection

The code inspection part at PD TC can be done either through pair pro-gramming, or through a code review in Gerrit (explained in section 4.3).When programming in pairs, the developers continuously review each oth-ers code. When using code review in Gerrit, the code first has to be reviewedand approved by the developer who submitted the code, and normally byone of his or her team members too.

4.4.2 Integration

As mentioned in section 2.1, integration is the ”puzzling part” in the CIprocess. First, every piece must work for themselves. Then every piece mustwork together with their neighbours, and finally with all pieces together.The integration process at PD TC occurs in different levels. To be able tounderstand this integration process I have to explain the different parts ofthe code.

First, at the lowest level there are products. A product can either bea software unit, an interface unit or a library unit. An interface unit is aunit where different protocols and interfaces used are defined. A libraryunit can be explained as a regular C++ library that contains a collection ofcommon classes and functions. Software units contains the implementationof functionality.

At the next level, several of these products integrates to a Load Module(LM). This is how deep this thesis will go, but i will briefly explain the nextsteps anyway. After the LM’s are built there are two separate ways in theCI-machinery depending on the technology developed:

• Technology 1: The LM’s gets integreated directly to an Upgrade Pack-age (UP)2

• Technology 2: The LM’s gets integrated to Load Module Container(LMC), and then integrates with other LMC’s to later form a UP.

4.4.3 Testing

There are different tests executed in the CI-machinery. Below I will explainwhich different tests there are, and what these tests are verifying.

4.4.3.1 A-test

A-test is actually a test framework, but when executing the collection of thefollowing tests below, it is mentioned as executing one A-test.

• Basic test: Is a kind of unit test, and is testing only one productindependent of any other product. It is the developers responsibilityto write these test cases.

2Upgrade Package = The final software used in a base station [6]

16


4.5. FEEDBACK MECHANISM CHAPTER 4. PDTC’S CI-SYSTEM

• FS verify: A file structure verification test that checks that the filestructure applies to some given rules.

• Pure Coverage: Checks whether the basic tests are ”visiting” everyline of code. This is a test to verify that new test cases has beenwritten for the submitted code.

• Purify: Check for memory leaks in the code

• Model validation: Checks that the models3 that have been createdor changed follow a specific set of rules.

• Lint: Is a static code analysis test that checks the code for usualmistakes that developers tend to introduce in the code. For instance,if a developer forgets to check if a pointer is null before using it, thistest will fail.

4.4.3.2 Multi component test

A multi component test is as it sounds testing different components to-gether (in this case when talking about components, we are talking aboutproducts). It requires some additional experience and understanding of theenvironment around the product in focus to implement this kind of tests.This is due to the fact that more than one product is involved in the test-ing, and you have to understand the dependencies between them. The multicomponent tests are not necessarily executed in the CI-machinery upon codesubmit for the current products. However, they have to be executed beforethe feature is considered as fully implemented.

4.4.3.3 Node test

Node test means that tests in an environment with real nodes4 are exe-cuted. Tests like setting up a call, attaching cell phones to a base stationare performed in a node test.

4.5 Feedback mechanism

The feedback is a very important key factor in the theory of CI. At PDTC the feedback is sent through an e-mail to the developer submitting thecode. It is also possible to follow the progress of the build in Gerrit. If thedeveloper want more detailed information about the load module build orthe A-test(s) there is a link to the Jenkins build page for the current build.The developers get feedback on following levels, regardless of the result:

• The code review in Gerrit

3Model = A textual or visual description of a part in a software4Node = A real node in the mobile network. For example a base station

17


4.5. FEEDBACK MECHANISM CHAPTER 4. PDTC’S CI-SYSTEM

• The load module build executed by Jenkins

• The A-tests executed by Jenkins.

• If the submitted code reaches the mainline or not

18


Chapter 5

PDTC’s CI workflow

Figure 5.1: Illustration of PD TC’s part of LMR’s CI-machinery.

19


5.1. PHASE 1 CHAPTER 5. PDTC’S CI WORKFLOW

The CI-workflow at PD TC is divided into two phases: ”Pre-integration”and ”Mainline integration”.

5.1 Phase 1: Pre-integration

Figure 5.2: The pre-integration phase.

The pre-integration phase contains following parts:

• Review

• Builds and tests

5.1.1 Review

When it is time for the developers to submit the code, it is submitted toGerrit. Gerrit sends a job request to Jenkins. Also, an e-mail is sent to thestake holders involved with the submitted code that a code review needsto be done. The submitted code is not stored in a visible repository wheredevelopers can fetch the code from, just a temporarily repository. Gerrit hasa grading system where the developers reviewing the code can put a gradeon the submitted code. To pass this stage the submitted code needs at leastto be approved by two reviewers. If a reviewer reject the submitted code forsome reason, the pre-integration phase will fail directly and the submittingdeveloper is notified by e-mail.

20



5.1.2 Builds and tests

In parallel with the code review in Gerrit, Jenkins receives the job requestfrom Gerrit and starts one or several A-tests and Load Module Builds (de-scribed previous in 4.4.3).

Jenkins dynamically chooses a couple of A-tests to run. This technique iscalled Multi-dependency A-test, and selects the different A-tests dependingon what dependencies there are from the submitted code.

In parallel with the Dependency multi A-test, Jenkins start building ofproducts into load modules (LM-builds). If Jenkins fails with either the testsor the LM-builds, the pre-integration phase fails and feedback is sent via e-mail to the developer submitting the code. If both tests and LM-buildspasses in Jenkins together with the code review in Gerrit, the delivery isqualified for phase two.

21



5.2 Phase 2: Mainline integration

Figure 5.3: The mainline integration phase.

When the pre-integration phase is finished, it is time for the secondphase, which contains:

• Mainline Delivery Queue

• A-tests and LM-builds

• Source Delivery Check

5.2.1 Mainline Delivery Queue

Before entering the mainline there is a queue-system. Instead of forcingthe developers to wait until there is a free time slot before submitting, thedevelopers submit directly. The submitted code is then put in the queueuntil the resources needed are free.

5.2.2 Builds and tests

In the mainline integration phase, there is a need to build and test thesubmitted code again. This is because the mainline the pre-integration wasbased on for the different deliveries might have changed. This is performedby doing a rebase (explained in section 4.1.1, and submit a job requestto Jenkins that performs the Dependency multi A-test and the LM-buildagain. If it fails, we go back to first phase again. If the build is approved, itis submitted to the ”Source Delivery Check”.

22


5.3. THE END? CHAPTER 5. PDTC’S CI WORKFLOW

5.2.3 Source Delivery Check

In this step, the system executes multi component tests with real nodesinvolved. If the delivery pass this stage, it is submitted to the mainline. Ifit fails, the build is discarded. In either way feedback is sent to the involveddevelopers.

5.3 The end?

No, the CI-machinery does not end here. After the delivery has made itto the mainline, the load modules are integrated to an Upgrade Package(UP), and additional testing occurs. Here the more time consuming testingis executed several times a day. But this is beyond the scope of this thesis.

23


Chapter 6

Experiences among thedevelopers

To be able to get the employees perspective on what CI is and what ithas done to their work I set up several interviews. There were a total ofseven formal interviews. The interviews were held with developers from anXFT, working mostly with feature development, and also employees workingwith the CI-machinery. In addition to these interviews, there were alsoseveral informal interviews, workshops and coffe-break-chats that also tookplace. All this material was gathered and documented, and in this chapter Iwill summarize the experiences of Continuous Integration presented in thismaterial.

Maybe the most important aspect when implementing a new softwaredevelopment method like CI into a software developing company is how thedevelopers working with the code experience the changes. Based on theinterviews held, I think the knowledge about what CI is, is rather good. Allthe participants in the interviews described CI as a technique to integrateand test more frequently, which is comparable to the definition of CI. Buthow big change has it been since the introduction of CI? Are there anymajor differences experienced by the developers that deliver the actual code?Based on the interviews done with the XFT at PD TC, they all can agreethat they notice some improvement, but not a huge one. Compare this tothe examples in the book, where the introduction of CI has become more orless like a religion for some. Why do the developers not think it have beena huge improvement then? In the interviews I asked how the process lookedlike before CI. The answer I got from the developers that had worked sincethe ”transformation into CI” was that the process almost looked the same.But there was actually one big difference: Before CI, the developers had toexecute the A-tests, MCT’s and other tests manually and locally on theirown computer. The first disadvantage with this approach is the differentuser environments that exist. Secondly, and even more crucial - If this was

24


CHAPTER 6. EXPERIENCES AMONG THE DEVELOPERS

forgotten, unverified code slipped through, and the time taken before anyonenoticed an eventual fault was longer. After CI was introduced, these testsare now executed automatically at code submit. This might seem as a smallchange to the developers, but I think the change is vital to the organization.Every step done manually could be a risk. If it is possible to automate, thiscould save time, reduce assumptions and so on. If it is possible to proceedto next step without testing, this WILL happen - humans make mistakes.Another reason to why the changes made didn’t seem to affect the developersthat much could be that the changes are not that visible to the developerswriting the code. Instead the actual change by implementing CI was themachinery “behind the fence” from the developers perspective.

25


Chapter 7

Analysis

In the theory of CI the most important task is to “fix the broken build”. Thismeans, when integrating with the mainline and the code stops working - thehighest priority is to fix this. Meanwhile, no deliveries shall be done, andno one should fetch code from the repository with the broken build. Thisapproach should not be feasible within PD TC, where hundreds of developersare involved. You can’t stop deliveries from hundreds of developers whensomeone breaks the build. That would be very expensive. Instead, PDTC use branches to separate different feature developing teams from eachother. If one developer submits broken code to the team’s feature branch- only the team (or the people involved in the feature) will get affected.Before even having the possibility to deliver to the mainline the code mustgo through a safety net that consists of various testing to ensure the deliverydoesn’t break the code. This is a very good idea compared to the theorywhere it’s the developers responsibility to test the code before delivery.

The theory of CI also advocates that all code shall be integrated to themainline at least once a day. This works perfectly fine in smaller projects.But when the number of developers increases it becomes a problem. Imaginehow it would look like if these hundreds of developers in PD TC shoulddeliver to the same mainline every day. I would not say it is going to beimpossible, but it is very difficult. You cannot integrate hundreds of codepieces into one piece without any problems. First of all, if implementedperfectly according to the theory, integration should not take more than 10minutes. This means a total of 16 hours integration time for a hundreddevelopers. Then we do not include any broken builds, everything has towork perfectly. We clearly see that the equation does not add up - it is notpossible to integrate every piece of code individually.

What PD TC has done in order to adapt CI to fit their process betteris mainly to use branches. Every feature or task are mainly developed in abranch. The team responsible for the branch decides when they do rebases,when they deliver, and what to deliver to the mainline. They also decide

26


CHAPTER 7. ANALYSIS

when the individual developers should deliver to the branch. Some teamsdeliver often, and solve integration problems continuously. Other teams canstay outside mainline on a branch for months without doing any rebasesbefore integrating to the mainline. There are no doubts about that therewill be more integration problems for the later example. It does not meanthat all teams that are ”on branch” for a long time are not aware of the risks.Sometimes when major changes are to be done it might not be possible todo rebases or integrate with mainline before this change is fully performed.

The branching technique is a good approach, because it solves the prob-lem with impossible high number of deliveries into mainline. Instead ofintegrating hundreds of deliveries from individual developers, we are nowtalking about tens of branches. But still, we have the fact that daily deliv-eries into mainline from each branch are rare.

One consideration regarding the differences between the theory and PDTC’s implementation of CI is the integration time. According to the theoryan integration should not take more than 10 minutes. When deliveringcode, the first integration with the mainline takes significantly more thanthat. If this integration succeeds the delivery is eventually put in a queueto the mainline. If that is the case, a new integration with the mainline hasto take place, which also usually takes more time than the recommended10 minutes. The time taken in this step depends on the number of otherdeliveries in front in the queue that is integrated with the mainline and couldaffect the current delivery. This means in worst case scenario, the integrationprocess to mainline could be quite far away from the 10-minute build. Onthe other hand, this might be acceptable. It is a very large organizationwith far more than ten developers. And as Daniel Stahl, an employee atEricsson mentioned in his research report - ”studies that validate the benefitsof Continuous Integration in large scale development projects is lacking”.However, he also found out that there are proof of some benefits by using CIin large scale development projects, but as the participants in the interviewsthat were held in this thesis said - ”The difference is not huge”.

27


Chapter 8

Possible improvements

When I first started with this thesis, I was sure that there would be a largedifference between the practice and the actual theory, with many parts toimprove. Later on I realized that the CI-machinery at PD TC actually worksvery well, and the number of improvements that I consider could be doneare few.

8.1 No daily basis

The first thing I experienced as a disadvantage was the low number of inte-grations made from each developer. According to the theory, every developershould submit code at least on daily basis. All these code pieces should gothrough the whole CI-machinery and finally the code shall be deployed in areal system1. I mentioned in the analysis chapter that there are hundredsof developers in PD TC, and it would be difficult to integrate hundreds ofpieces of code daily with the mainline. But why not take advantage of thebranching technique that PD TC uses? Instead of hundreds of individualdevelopers, we have tens of branches to integrate with the mainline. Thiswould probably work in the theory. But when talking to the developersthey say it is almost impossible to submit something to the mainline everyday. As one developer said: ”Think if a surgeon has to take a break in themiddle of a heart transplant to test if that part of the transplant worked”.That would probably never work. Exactly like the human body, the systemdeveloped within PD TC is a large and complex system. It is difficult tosubmit what you have done on daily basis. However, one thing that couldbe done in order to increase the number of deliveries to mainline is to di-vide the code into smaller components. If this is possible, it is probably anexpensive improvement since parts of the structure has to be re-designed.

1With a real system it means that the system should be as customer like as possible

28


8.2. MORE AUTOMATION CHAPTER 8. IMPROVEMENTS

8.2 More automation

There are no parts of the CI-machinery I have seen that needs automation,except one part - The code reviews in Gerrit. A lot of code inspectionsexecutes automatically, but the review is done manually. Instead of spendingexpensive time reviewing submitted code, is it possible to automate? If thereview process is changed, everything needed is a piece of code instead ofteaching all the individual developers the new process. If an automation ofthe code review seems impossible now, start small and see how much of thecode review that could be automated.

8.3 Feedback

The feedback in the CI-machinery is good, but it could get better. E-mail isnot the best way of notifying the developers about the feedback. Neither is awebsite that the developers has to visit to get more information. What couldbe an improvement is a monitor that in real-time shows all information thatthe team needs. For instance: Status about the team’s latest build, how bigchange there is on the mainline since last rebase, how big the mainline queueis and so on. Some information could be gathered in radiators to visualizewhether the numbers are considered as good or bad. Another good idea isa sound every time a commit from the team passes all tests and reaches themainline.

29


Chapter 9

Discussion

To enter a large company more or less directly from school and investigate asoftware development method they are using is difficult. I had no experienceof using any particular software development method at all before (exceptwaterfall method1 if that one counts). So I had to confide in the experienceamong the employees. On the other hand it could be good to let an outsideperson examine the methods used within a company to see it from anotherperspective. And I think that it turned out pretty well. I had a very goodbook [4] as a base while studying the theory. It was easy to understandand easy to read, even for me who is not a good reader. In fact it was sogood, that I almost only used that book in the theoretical part. This couldof course be a disadvantage compared to reading several different books inthe subject. But due to the limited scope of this thesis, in combination withmy reading difficulties i decided to mainly go with that book. Through thebook I also found interesting persons (like the author and other CI-gurus)that I looked up. Through these people I found more useful informationthat I used as well. I consider the sources very reliable that I used for thetheoretical part. I also tried to bring more focus in the investigation of PDTC’s CI-system, since I knew that could take more time than I planned for.Even if I tried to focus more on the real world investigation, this was actuallynot enough. I could easily spend all my 10 weeks on this investigation, sowe decided to narrow down the scope of the thesis to make it manageable.As I mentioned in the introduction I reconnected to the theory continuouslyall the time while examining the CI in practice. This was a very instructiveway to work, since it enabled fast feedback regarding if the practice atPD TC were close to the theory or not (actually a little bit as one of theadvantages with CI).

In the first weeks during the thesis I was seated together with the XFTmentioned in the thesis. This was very good since I quickly could ask ques-

1Waterfall method is a software development method where you have strict phasestypically in following order: Pre-study, implementation, testing

30


9.1. ETHIC & ENVIRONMENT CHAPTER 9. DISCUSSION

tions and start discussions. But since a new member entered the team duringthe time of the thesis, i had to move to another location. It was more dif-ficult to have these spontaneously coffee-breaks where a lot of interestingdiscussion was ongoing. However, I do not think this affected the result ofmy thesis since I had got a grip of most parts in the CI-machinery at PDTC by then. Even if I was not seated together with the XFT I had thematerial from the interviews which were very useful since that was the bestsource of information I had in the examination part.

Actually, while examining the CI-machinery I found a disadvantage withthe theoretical part compared to CI at LTE. From a CI perspective if thecode breaks when submitting to the repository, the highest priority is to fixthis. This can happen if one or two statements are true: 1. The developerhas not tested his or her code, 2. The developer has not tested his or her codegood enough. At LTE, instead of submitting broken code to the repositorythe developer is forced to go through this safety net with tests before thecode is automatically sent through in the CI-machinery. I totally agree withLTE’s technique here. If there is any sign of possibility to break the codewhen submitting, this must be prevented in all possible extent. I don’t thinkthe developers should have the responsibility to test the code themselves; itshould be included in the CI-machinery.

9.1 Ethical and environmental aspects

With the use of Continuous Integration, there are a much wider use ofautomation than before. This result in two disadvantages regarding theethical and environmental aspects:

• In the ethical aspect; when automating, the same number of em-ployees might not be needed. This can lead to reductions among thepersonnel working with the development because the number of hoursspent is less than before. It is very difficult to say if this could hap-pen at LTE, but there might be a small risk. On the other hand, thequality of the tasks performed at the workplace increase since it feelsmore useful to write a line of code than compile and test it manuallyfrom time to time if this can be done automatically.

• In the environmental aspect; introducing CI means that morefrequent integration and testing should take place. This creates alarger need for new hardware that should take care of this. All useof hardware affects the environment since a lot of energy is needed.However, one of Ericsson’s three core values is ”perseverance”, andthey work very hard to minimize the environmental footprint.

31


Chapter 10

Dictionary

LMR LTE Mobile Radio (The department responsible for the de-velopment of the LTE technology)

LTE Long Term Evolution (a mobile radio communication tech-nology, also known as 4G)

PDTC Product Development Traffic Control (A departmentwithin LMR)

XFT Cross-functional team (A team that shall be able to workwith several different tasks)

XP eXtreme Programming (A software development methodwhere all good practices from other methods are taken toan extreme level)

32


Bibliography

[1] Steve McConnell’s homepage. (Author of many soft-ware engineering textbooks including Rapid Development).http://www.stevemcconnell.com/ieeesoftware/bp04.htm. Last accessed:2014-12-08.

[2] MartinFowler’s homepage. (Well known author and inter-national spokesperson in the software architecture area)www.martinfowler.com/articles/continuousIntegration.html. Lastaccessed: 2014-11-14.

[3] Beck, Kent Extreme Programming Explained - Embrace Change, Secondedition. Addison-Wesley, 2004

[4] Paul M. Duvall, Steve Matyas, Andrew Glover, Continuous Integration- Improving Software Quality And Reducing Risk. Addison-Wesley, 2007

[5] Git - About Version Control Systems http://git-scm.com/book/en/v2/Getting-Started-About-Version-Control Lastaccessed: 2014-12-01

[6] Interview with Daniel Stahl, senior specialist at CI, LTE 2014-11-12.

[7] Git http://git-scm.com/. Last accessed: 2014-11-20

[8] GitGuru http://gitguru.com/2009/02/18/branching-strategies-overview/. Last accessed: 2014-11-20

[9] Interview with a team member in a cross functional team.

[10] Gerrit Code Review - A quick documentation https://gerrit-documentation.storage.googleapis.com/Documentation/2.9.1/intro-quick.html. Last accessed: 2014-11-18

[11] Jenkins http://jenkins-ci.org/. Last accessed: 2014-11-19

33


Chapter 11

Interviews

In this chapter I’ve gathered notes from interviews that were held.

11.1 Interview 1

What are you working with? What is your role in the team?Designer, working with features within LTE. As a designer I’m involved

in everything from the pre-study phase to the maintenance phase. Also Iam a SCRUM master for the team which can be explained as a link to theoutside of our team.

For how long time have you been working for Ericsson?I’ve been working for Ericsson since 1999 and on LTE since 2009.

What does CI mean to you? What are the differences from ”be-fore”?

It’s just an organizational move. Now we have people that actively workwith CI. There is some kind of positive improvement.

How do you think CI is working for you?It works fine. But it feels like before.

What is good with your ”way of working”?When the team is getting more mature everybody can do everything

which is good. But it takes some time to get there.

What could be improved?Decisions are sometimes taken without taking everything into consider-

ation.

34


11.2. INTERVIEW 2 CHAPTER 11. INTERVIEWS

11.2 Interview 2

What are you working with?Designer

For how long time have you been working with that?6 months

Do you have any experience about CI before this job?Somewhat, yes. I’ve worked for CIRV (Continuous Integration Release

Verification). We maintained the test cases that the cross functional teamswrote.

What is CI to you?You follow up and execute tests more often to find faults faster.

Do you think you are performing CI?Yes. We are writing test cases that should cover the feature being im-

plemented.

11.3 Interview 3

What are you working with? What is your role in the team?Designer and product guardian.

For how long have you been working for Ericsson?Approximately seven years in different positions in LTE.

What is CI to you?CI is something that I send my code to. If everything is working as

intended it reaches the mainline. If something goes wrong I get feedback.The difference I see from before is the automation and the increased testingfrequency. Also you cannot forget to run your tests since it is executed au-tomatically.

Do you think you are performing CI?Yes, in one way or another. But I think it is occurring on different levels.

The thing that might be missing is the connection between the written codeand the multi component tests.

11.4 Interview 4

What are you working with? What is your role in the team?

35


11.4. INTERVIEW 4 CHAPTER 11. INTERVIEWS

I work as a designer with both implementation and testing.

For how long have you been working for Ericsson?Started as a summer worker during 2013, after that I did my thesis at

Ericsson and finally got a job in early 2014.

Do you have any experience within CI before this job?No.

What is CI to you?It feels like continuous testing and automated tests outside ”me”. With-

out CI there should probably be less integration testing.

Do you think you are performing CI?Both yes and no. I think we are performing CI on different levels. The

thing that might be missing to perform CI in a better way is more integrationtesting. What’s good with your way of working? It is a quite new way ofworking. However, the organization has not yet found a way to make use ofGit’s full potential.

36

institutionen för datavetenskap - diva portal802248/fulltext01.pdf · 1.4. questions chapter 1....

Documents