continuous integration - stability, reliability and speed in software development

Whitepaper

/Continuous integration: stability, reliability and speed in software development

Table of Contents

Introduction......................................................................................................................................................................................................3

Why continuous integration?...........................................................................................................................................................................4

Benefits of continuous integration....................................................................................................................................................................6

Challenges and Solutions................................................................................................................................................................................9

Achieving full test automation-gradually.........................................................................................................................................................10

Managing complexity......................................................................................................................................................................................11

Introduction

Stability, reliability and speed are just

some of the words used to describe the

advantages of a software development

practice known as Continuous Integration

(CI). It has transformed systems

development. Martin Fowler, a thought

leader on CI describes1 it as the practice

of integrating code into a shared repository

regularly with automated check-ins

authenticating the build to identify issues.

This procedure allows teams to detect and

resolve problems early and quickly –

before they become major downstream

issues.

CI’s key benefit could be summed in one

word. Quality. Developer teams at

Computaris use CI for one aim - to achieve

excellence. Some projects are more

demanding than others. Larger projects

require change often during the

development lifecycle.

The CI process allows Computaris to

deliver outstanding results – faster.

Computaris has an excellent track record

of delivering successful projects.

Customers come to Computaris for both

new projects and to seek counsel on

existing projects expecting swift

turnaround times combined with excellent

quality. This whitepaper explains best

practice methodology for CI and the

benefits it delivers.

/

Continuous integration: stability,

reliability and speed in software

development

page 3

Why continuous integration?

In today’s dynamic programming environment, software development is getting increasingly complex. For example, systems being built on established languages2 today could easily be outdated3

in a matter of a few years. As such, IT teams need to stay ahead of the curve in software development best practice to create systems and programs that operate effectively not just in the short term, but also continue to deliver results in the future. Teams that adopt CI can ensure continuity.

By using CI, development teams can continuously integrate and modify code within a shared repository several times a day. An automated build verifies each check-in and this allows development teams to detect problems early4. This is an important step to minimize the cost of fixing anomalies and to maximize build quality.

The technique that underpins CI is the integration of code being constantly developed and added to a code base or a source code control system5. In the past, most developer teams relied on a daily build. Now it is at least once a day with each significant change. This enables the developer team to have constant feedback on the status of the software and anomalies are easier to manage.

These are the basic tenets of CI:

> Use a single repository for source code control: This is the developer team’s fail-safe and hub for automation6.

> Self-testing the builds: Have a suite of automated tests to examine the code base for bugs regularly7.

> Validate each commit: Select and stick to a consistent coding standard. Regularly monitor the repository for oversights and

check each commit8.

> Commit often: Team members need to commit regularly within an agreed timeframe to reduce the likelihood of conflict changes9.

> Test accurately: A realistic test environment that reflects the production environment will contribute to obtaining a higher-quality of the final build10.

> Efficient and transparent processes: Having good API documentation with robust processes can ensure that everyone, including customers, are in-sync with the build to avoid needless and costly rebuilds11.

> Deployment strategy: When the business is ready, use a script to deploy the application into production via an automated live test server that's accessible to key stakeholders12.

page 4

2Different Programming Languages

3Fabien Potencier blog

4Mike Jackson, Software Sustainability Institute

5C. Aaron Cois, Software Engineering Institute

6Martin Fowler blog

7Martin Fowler blog

8Paul Jones, Why Coding Standards Matter

9Seth Robertson, Blog

10John Overbaugh, TechTarget

11David Flanders and Malcolm Ramsey, Joint Information Systems Committee

12Eric Reiss, Radar

/



development

Benefits of continuous integration

Faster time-to-market

In the past, development teams were

bogged down with paperwork and had to

write deployment codes for other teams to

follow. This was a silo approach. Such

time-consuming process can mean that

time-to-market for new products and

services could take months – even

years13. CI has transformed and

accelerated time-to-market.

An automated pipeline of continuous

delivery ensures that only good quality

code makes it all the way through14. As

such, automation plays a fundamental role

in CI and helps to build a robust

development work flow where fixing bugs

and improving quality is at the heart of the

system. This level of quality control speeds

up time-to-market and gives businesses a

competitive edge15.

Faster development lifecycles

The usual developer process, without CI,

can often comprise of the following steps:

1. Develop new code OR perform bug

fixing a. This includes Unit tests or

manual smoke tests

2. Commit code in version control

3. Wait for QA teams to raise issues or

verify fixes

4. Receive issues from QA teams and

return to Step 1

Some developers write code, run a few

tests and pass the code to someone else

to verify it16. That’s inefficient, it damages

quality and hampers knowledge sharing

within the team17. Steps 3 and 4 usually

take a long time, measured in hours if not

days. By the time a developer received

feedback, he could have moved on to

different tasks and prioritized something

else. It could take more time for the

developer to familiarize himself with the

code again and retrieve that context. If the

duration for steps 3-4 took even longer

than just a few days, it is very likely that

another developer could have taken over

the fixes and the original developer might

not even be available. All this adds

unnecessary delays and is a setback for a

business hoping to deploy a solution to the

market.

page 5

13Sven Malvik blog

14Florian Motlik, Codeship

15Jason Tee, TheSeverSide

16Scott W Ambler

17Jason Cohen, SmartBear Software

/



development


CI provides a new lease of life for the developer process:

1. Develop new code OR perform bug fixing

a. This includes adjusting the automated tests to verify the new code

b. For large test suites, nominal bugs and glitches should be addressed at this step

2. Commit code in version control

3. CI server (e.g. Jenkins) detects the new commit and runs a new build

4. Team members are notified via e-mail if the CI build found issues Some developers write code, run a few tests and pass the code to someone else to verify it16.

That’s inefficient, it damages quality and hampers knowledge sharing within the team17. Steps 3 and 4 usually take a long

time, measured in hours if not days. By the time a developer received feedback, he could have moved on to different tasks and prioritized something else. It could take more time for the developer to familiarize himself with the code again and retrieve that context. If the duration for steps 3-4 took even longer than just a few days, it is very likely that another developer could have taken over the fixes and the original developer might not even be available. All this adds unnecessary delays and is a setback for a business hoping to deploy a solution to the market. CI provides a new lease of life for the developer process.

The main difference is that steps 3 and 4 are automated and happens in a window of just a few minutes – not days - from the time the developer finished coding. The developer will not have time to lose context, any issue detected by the CI build will be fixed much faster.

Other CI applications referenced by this White Paper aim to have 1 to a few builds per day, a build lasting one to several hours. We try to design our builds to be faster, in the range of 10 to 15 minutes with a test suite having over a thousand full end-to-end automated tests (see “4.4 Parallelization and configuration”).

Quality: from good to great

CI makes the integration process efficient. Without CI, Quality Assurance (QA) engineers could spend more than 50% of total project time running nominal tests for integration. This will shorten the time the QA team spends looking for bugs and other subtle issues18.

page 6

18Gautham Pallapa, blog

/



development


The CI process ensures that the QA team receives code with integration done and the nominal test cases are successful. This frees the QA team to dive deeper and focus on problems, test more scenarios and yield better overall quality and results19 .

Stable codes and demo ready

The fast development lifecycle described in section 3.2 will allow a large part of the code to be stable even when under active development. Agile development methodologies20, which rely on incremental development and getting constant feedback from stakeholders, ensures that the project is on the right track. Regardless of the project management techniques used, by using CI the team can be ready at any time to demo the system. This allows for quick feedback during customer meetings and ensures requirements are well understood and the

project is moving in the right direction. As such, customer-led improvements can be incorporated within a matter of a few hours – and not days21.

Smarter risk mitigation

During a build's lifecycle, there is always the possibility of probable mistakes22. If there are suitable parameters and metrics in place, the status of the build can be constantly monitored accurately. CI enables developer teams to discover and resolve flaws promptly. Furthermore CI provides intelligence for teams to eliminate potential vulnerabilities too. This is especially the case when they have access to historical data which allows QA teams to build test cases to ensure that historic mistakes are not repeated.

CI mitigates risk because check-ups and trials are run frequently - in some cases every few hours - so there's a higher

probably of detecting issues and fixing them before launch. This ensures that businesses have robust solutions they can deploy with confidence. Joe Green, a developer and champion of CI wrote: "Having a good CI system is akin to having a fire alarm in your house. It won't fix your error for you but you can be rest assured that it will flag up the error as soon as it is integrated and that the build containing the error will not make it into your staging or production environments.23

Flexibility

CI underpins flexibility and can foster creativity in the development team. Thanks to the automated nature of CI, development teams have the flexibility to change code quickly and without fear. Fast CI builds give developers the opportunity and freedom to innovate and try new techniques to enhance the system. This is one of the key benefits of CI.

page 7

19Gautham Pallapa, blog

20Agile Alliance

21Duval, Matyas et al. Tech Target

22Francis Adanza

23Joe Green, Smrtr

/



development


Faster onboarding for new team

members

Sometimes onboarding new team

members, especially within a large project,

can take up valuable time. If key members

of the developer team are not available,

bringing someone up to speed on a build

can be challenging. CI makes onboarding

more efficient. It generates API-based

documentation to make the software

architecture easily accessible to new

members on the developer or clients’

teams. You only need to browse the test

cases to have a quick overview of the

functionality of the system.

Having a CI process allows you to have

the tools to automatically build, deploy,

and test the system. These are the same

tools needed for a new developer/QA

engineer to hit the ground fast25.

24Manuel Weiss, Codeship

25Darragh Curran, Inside Intercom

/



development

Challenges and solutions

While there are many benefits to CI,

developer teams and customers can face

a number of challenges from the build

phase to deployment.

Achieving full test automation

gradually

Automating an entire build at the start is

not easy, so start simple26. Manage the

process continually by keeping

deployments consistent and have a well-

defined process that can be constantly

automated and developed over time. This

continuous automation methodology can

help developers automate the entire build.

Even with the 80/20 principle27, achieving

100% automation is obviously more

difficult than 80% automation. Some

developers have found that fixing 20% of

the bottlenecks could give you 80% gain in

speed28. Yet coding automated tests is still

coding that requires careful handling.

Some teams fail to build a reliable CI

process because they consider it is easy

and do not pay enough attention to detail.

Here are some common mistakes:

> Poorly-coded automated tests will cause

team members to spend more time

investigating failures related to the test

suite itself, rather than investigate real

errors of the product that’s being tested.

> After a while, frustration grows, and the

entire test suite is abandoned or rewritten

from scratch.

> Poorly-managed infrastructure will cause

environmental problems and will probably

impact reliable CI builds. These problems

can include: malfunctioning machines,

patchy networks, disks with limited space,

and sluggish server and CPU

performance, which can cause CI builds to

run slowly.

26Eric Reiss, Radar

27Pareto Principle

28Naresh Jain, Key Principles for Reducing

Continuous Integration Build Time

/



development

Managing complexity

Software development best practices must

be carefully applied to test code not only to

production code. Especially for

large/complex test suites with

hundreds/thousands tests, poorly

implementing automated tests will greatly

increase cost of maintenance and may

even lead to completely abandoning the CI

test suite.

It is important for companies to ensure that

their developer/quality assurance teams

adhere to best practices, including Don’t

Repeat Yourself (DRY) and Keep it Simple

Stupid (KISS). In their book, the Pragmatic

Programmer, Andy Hunt and Dave

Thomas coined the phrase DRY29. DRY is

about reducing complexity to manageable

units by dividing a system into pieces30. As

systems are fragmented into components

and subcomponents, complexity could be

reduced to a single responsibility.

Keep It Simple Stupid or Keep it Stupid

Simple (KISS) is another best practice

concept in software development which

means – the simplest explanation is likely

to be the best one31. In KISS, developers

aim to find a resolution as easily as

possible and produce code to solve

complex problems in fewer lines of code –

faster and with remarkably better quality.

Best practice guidelines for complex

coding

> Conduct tests on asynchronous I/O

(Input/Output) for faster and accurate

results instead of relying on “sleep” mode32

> To manage test data efficiently, select an

accessible tool or language so team

members can retrieve data efficiently33

> Test scenarios must produce log files

easy to read and follow, this is mandatory

for easy tracking of issues when the CI

build fails34.

29Andy Hunt and Dave Thomas, The Pragmatic

Programmer

30Chris Peters, Code

31The Apache Software Foundation

32Mixu, blog

33Cohen, Hunter et al., Association of

Computational Linguistics

34Tamir Korem, Cloudify

/



development

Managing complexity

Maintaining build speed

For improved productivity, builds need to be fast. Along with best practice concepts, the developer teams also need to consider the following:

> Ideally a build should be fast enough to be able to run and validate each commit a developer makes in the version control system early and regularly35. Of course this is hard to achieve especially for large projects with teams of many developers where a commit might happen every few minutes

> It is difficult to find an issue with 10 commits and 200 failures out of 1000 tests. Investigating each commit takes time. A fast build exposing problems in each individual commit would minimize costs of debugging/investigating

> Black box testing, where testers are only

aware of what the software is supposed to do and not how it does it, runs much slower than unit tests. Unit tests verify the functionality of a specific section of code. All this can be complex36.

> For black box test, all components of the system under test must be built, deployed and configured fast. Assertions on system behavior, where tests can be run on your assumptions, are black box. They verify the system’s output so they must wait for disk I/O, network I/O, caches to fill up, messages to pass through queues and all other type of events that must happen until a noticeable output from the system can be tested. All these operations take time and making the build fast becomes a challenge.

Parallelization and configuration

The solution to make the test suite faster is to use parallelization, which means

running tests on multiple components or subcomponents simultaneously. One of Computaris’ test suites can run approximately 1000 tests in parallel with corresponding sub-tests concurrently. A rough estimation would be 40 processes each with an average of 100 threads running in parallel. Of course this requires a fair amount of hardware.

Most systems have system-wide configurations which are changed very rarely. A CI build must test various configuration options and this creates a new problem, how can you run parallel tests with different configurations? The solution is yet again parallelization. Computaris has the capability to carry out multiple system tests that can run simultaneously, each instance with its own configuration.

35Wikipedia, Revision Control

36Black Box Testing

/



development

Managing complexity

This approach requires the following:

> The need for robust hardware: multiple

instances of parallel system runs tend to

require a great deal of hardware

resources.

> The need to automate the infrastructure:

Computaris has a regression suite with

over 1000 tests running on 16 parallel

systems being tested and uses tools such

as Chef to manage infrastructure

automatically37. This also helps to solve

the need for many hardware resources.

Having automation allows the team to run

CI builds on commodity hardware that may

fail. Recovery from these failures is

extremely fast using “infrastructure as

code” which eliminates the need for

expensive server hardware38.

Make tracking down real problems easy

and fast

Ultimately, finding and resolving problems

faster is why CI has been used for the

build. For large test suites (e.g. 1000 tests)

if you have 200 tests that fail, the simple

action of checking the failure message of

each of the 200 failures can be time

consuming.

In this instance, a “Root Cause Analysis

Report” which looks at the origin of the

problem and the corrective measures

required can be helpful for the developer

team. The Root Cause Analysis Report

can group tests by cause of failure and

provide an overview of the problems that

caused the build to fail in the first place39.

Decide on your tests

For the CI build to be successful, it is

important for the developer team to settle

on a testing regime. Part of Extreme

Programming (XP), Test-Driven

Development (TDD) uses automated unit

tests to “get something working now and

perfect it later40.” After each test has been

evaluated, similar tests are performed

again until it reaches a satisfactory result.

A collaborative practice, Acceptance Test

Driven Development (ATDD) involves

customer being part of the test design

process to define criteria41. ATDD ensures

that all stakeholders are project members

and they understand precisely what needs

to be done and implemented42.

37 JT Gray, Safari Blog

38Hummer, Rosenberg et al, Distributed Systems

Group

39Alon Linetzki, Root Cause Analysis for Software

Testers

40Narayana Maruvada, Agile Record

41Scott W Ambler 42Gabo Esquivel, blog

42Gabo Esquivel, blog

/



development

Managing complexity

Behavior-Driven Development (BDD) is, as

the name suggests, behavior-based build

which incorporates the practices of TDD

and focuses on language and

interactions43. One benefit of BDD is that it

provides the developer team and customer

a clearer indication as to what the system

should do.

TDD can be more technical and complex

that requires developer teams to become

conversant with the detailed object

model44.

Continuous deployment

Continuous Deployment or continuous

delivery (CD) is described as the practice

of releasing only good build to the final

user45. This concept is based on CI and is

a continuation of this principal to the 'last

mile' for customer delivery46. With CD,

teams can quickly adapt to business

requirements and user needs which in turn

enabled greater co-operation between

operations and delivery, championing best

practice and driving real change47.

Software developers deploy their software

into production each night and this

placement provided a number of benefits:

written software was not wasted,

developers responded quickly to problems

and fast turn-around times led to valuable

partnerships between teams and their

customers48. Similar to CI, CD mitigates

risk and detects bugs fast. Crucially, it also

enables the developer team to progress on

to fully functioning software even faster.

Fine-tuning

As we've seen, CI is one of the most

effective concepts for software

development. However for it to work

perfectly, it requires discipline in the team.

However, developer teams are human

after all. Sometimes, the bigger the team

and the code, the more likely it can get

broken and the lengthier the process will

be to fix it because other business

priorities take precedence49. Team

members can ignore the red flags from

Jenkins and overtime flawed software can

get passed on to the end customer. The

solution? Forbid team members from

including anything into the main repository

and make a script that anyone can access

easily. The script should merge, test, and

commit without exceptions. The developer

of each code should be responsible for

broken tests and the team should raise red

flags before a code gets into the master

repository.

43Agile Alliance

44Gabo Esquivel, blog

45Carl Caum, Puppet Labs

46Martin Fowler, blog

47Florian Motlik, Blog

48Sarah Goff-Dupon, Atlassian Blog

49Yegor Bugayenko, Dev Ops

/



development

/Thankyou!

/[+44]20.7193.9189www.computaris.com

continuous integration - stability, reliability and speed in software development

Technology