enterprise continuous integration presentation

© CMC Media 2008© CMC Media 2008

Challenges and Characteristics of

Enterprise Continuous Integration

Moderator

Patrick EganPublisher – CM Crossroads and Agile Journal

Speakers

Paul Julius – Willowbark Consulting

Anders Wallgren – Chief Technical Officer Electric Cloud

CM Crossroads Webcast Series

www.electric-cloud.com

Challenges and Characteristics of Enterprise

Continuous Integration

October 9, 2008 Slide 3

Introduction

In the past…

Teams spent weeks or longer verifying that applications

worked

Time spent „integrating‟ was tortuous, lead to integration

„storms‟ of broken builds

Developers would have to track down bugs introduced

months earlier

The answer to this painful process is Continuous Integration


CI: The Beginning

Continuous Integration (CI) dictates that, as soon

as changes are made, they are integrated and run

through a rigorous build and test cycle

Build engineers began using various CI tools,

sometimes without management approval

Once these tools became known, they were

embraced by developers who wanted and needed

the instant feedback on the health of their builds

As teams and applications grow over time, many

enterprises find themselves with CI servers

spread across the company


Challenges

With the proliferation of multiple standalone

systems, next step is to investigate the idea of

consolidation

Consolidated approach will help:

Teams to reuse procedures and tools

Better utilize hardware

Free personnel from manual, low value tasks


Challenges in Real World Enterprise CI

No centralized build-test-deploy infrastructure

Difficulty scaling the environment

Testing and resource administration increase in

complexity

Hardware Utilization difficulties

Barriers to sharing and reusing scripts

Difficulty tracking builds and cross-project

reporting


S.T.A.R

The challenges of Enterprise CI require a CI

solution that is:

Scalable

Turnkey

Automatic

Rapid


Scalable

Has many aspects

Must scale by:

Number of projects

Size of an individual project

Number of users of the system

Number of machines in the build cluster

Complexity of workflow

Enterprise CI is fundamentally about many projects and

many users


Turnkey

Enterprise CI has to be turnkey. The number of

builds, number of users, and the reality of 24/7

operations make this imperative

Turnkey implies that from the very beginning,

Enterprise CI must be useable. It must be trivial

to add scripts to the system, without requiring a

rewrite.


Automatic

CI system must be automatic, builds must be

launch-able with a single click, but it must also be

compatible with existing scripts

Three forms of automatic: On demand, On

stimulus and On the clock

On Demand: Builds should be a single click away

On Stimulus: Typically caused by changes in the

SCM system

On the Clock: Replacement for ‘cron’


Rapid

As the system scales to mulitple projects, and the

number of builds increase, builds have to be

rapid.

Hours long builds are a deal breaker

Reducing build times from hours to minutes

typically is done using parallelism

Enterprise CI systems should be able to work in

tandem with build parallelism software


Other Properties

Three other properties are important to

Enterprise CI:

Auditing

Virtualization

Reporting


Customer Example

Problem

Medium sized Product company

10‟s of disparate existing builds

Varied build infrastructure

Nearly impossible to identify a “good” build

Solution

Unified Dashboard for all builds

Consistent build infrastructure

Templated, resuable build infrastructure


Next Steps

It’s easy to get started with an open source CI

system

But as your needs change and your company

grows you have to evaluate the effectiveness of

this solution.


Build vs. Buy

Key question then, should we use an open source

tool, build our own, or buy a commercial tool?


Four Key Areas

As we mentioned four key areas to review

Fast Builds

Software keeps growing = longer builds

On the Clock Builds

Scheduled Builds

The classic „nightly build‟

On-demand Builds

Ad hoc, as needed builds

On-Stimuli Builds

Driven by changes in SCM

October 9, 2008 Slide 17Slide 17

Build Your Own?

Features

Investm

en

t

Trigger builds

after check-ins

Automated build

and test for one build tool, one

machine

Scheduled jobs

Seems pretty straightforward…


Build Your Own?

Features

Investm

en

t

Trigger builds

after check-ins

Database of

build results

Automated build

and test for one build tool, one

machine

Notifications via e-mail/RSS

Scheduled jobs

Web-based

access to results

Just a few finishing touches

CruiseControl

gives you this

for Java/Ant


Build Your Own?

Features

Investm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of

build resultsSingle job can use

multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Multiple servers, concurrency


Features

Build Your Own?In

vestm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of


multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Cancel jobs

View partial

results of

builds in

progress

Monitor system

status

On-demand job

invocation via

Web

Better monitoring and control


Features

Build Your Own?In

vestm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of


multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Cancel jobs

View partial

results of

builds in

progress

Monitor system

status

On-demand job

invocation via

Web

Tools for

extracting

data from logs

Trend reports

Cross-product

reports

Resource

utilization

reports

Annotate builds

after completion

Customizable

reports

Better reporting


Features

Build Your Own?In

vestm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of


multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Cancel jobs

View partial

results of

builds in

progress

Monitor system

status

On-demand job

invocation via

Web

Tools for

extracting

data from logs

Trend reports

Cross-product

reports

Resource

utilization

reports

Annotate builds

after completion

Customizable

reports

Logging and

error reporting

Retry steps after

errors/ failures

Timeouts for

runaway job

steps

Detect resource

failures

Better error handling


Features

Build Your Own?In

vestm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of


multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Cancel jobs

View partial

results of

builds in

progress

Monitor system

status

On-demand job

invocation via

Web

Tools for

extracting

data from logs

Trend reports

Cross-product

reports

Resource

utilization

reports

Annotate builds

after completion

Customizable

reports

Logging and

error reporting

Retry steps after

errors/ failures

Timeouts for

runaway job

steps

Detect resource

failures

Developer access

to production

builds

LDAP

authentication

Role-based

access control

Shared access by

multiple teams

User

impersonation,

password

management

Priorities

Handle multiple

independent groups


Features

Build Your Own?In

vestm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of


multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Cancel jobs

View partial

results of

builds in

progress

Monitor system

status

On-demand job

invocation via

Web

Tools for

extracting

data from logs

Trend reports

Cross-product

reports

Resource

utilization

reports

Annotate builds

after completion

Customizable

reports

Logging and

error reporting

Retry steps after

errors/ failures

Timeouts for

runaway job

steps

Detect resource

failures

Developer access

to production

builds

LDAP

authentication

Role-based

access control

Shared access by

multiple teams

User

impersonation,

password

management

Priorities

Conditional steps

Modular,

composable

process steps

Extensibility

Parameters Control

environment

for jobs

More programming

features


Features

Build Your Own? It Never EndsIn

vestm

en

t

Automated build

and test for one

build tool, one

machine

Scheduled jobs

Trigger builds

after check-ins

Web-based

access to

results


Database of


multiple resources

Run job steps

in parallel

Multiple jobs run

on one resource

simultaneously

Run multiple

builds

simultaneously

Resource

pooling, load

balancing

Multiple servers,

remote invocation

Resource

selection criteria

Cancel jobs

View partial

results of

builds in

progress

Monitor system

status

On-demand job

invocation via

Web

Tools for

extracting

data from logs

Trend reports

Cross-product

reports

Resource

utilization

reports

Annotate builds

after completion

Customizable

reports

Logging and

error reporting

Retry steps after

errors/ failures

Timeouts for

runaway job

steps

Detect resource

failures

Developer access

to production

builds

LDAP

authentication

Role-based

access control

Shared access by

multiple teams

User

impersonation,

password

management

Priorities

Conditional steps

Modular,

composable

process steps

Extensibility

Parameters Control

environment

for jobs

Portable across

hardware/OS

platforms

SCM integration,

bill of materials

Support multiple

languages and

build tools

Web interface for

editing processes

Command-line

interface


Build Your Own

You could try that

Before you do think about…

How much time do you have to write all that code?

How you are going to manage 200+ builds per day?

How are you going to get the build down from 4 hours to 15

minutes?

How are you going to manage hardware failure in your build

system?

Build Managers now control the heartbeat of

software engineering

They need something more than a creaky Perl script


Build Your Own

Fast builds

Buy lots of SMP hardware and try out GNU Make parallelism

or manually parallelize the build.

Scheduled builds

Use cron for that

On demand builds

Build an intranet page, integrate it yourself with the current

build and source code management system

Stimuli builds

Build ad-hoc script attached to source code management

system


Open Source

Fast builds

distcc, ccache, GNU Make -j

Scheduled Builds

cron, CruiseControl

On-demand Builds

Apache Continuum

Stimuli Builds

CruiseControl has some support

SourceControl plug-in


Commercial

Build speed tools

Like ElectricAccelerator

Build visualization and troubleshooting tools

Like ElectricInsight

Build management tools

Like ElectricCommander or CruiseControl

Together these tools enable build managers to

build, package, test, and deploy software

And they enable Agile development


Stepping Stones

Builds up 18 months little by little

Four stepping stones


Stepping Stones (1)

Step 1: Put full builds in the hands of developers

Set up an internal web page that allows a developer to fire off a build

The developer should be able to select:Which branch to build…

… or the location of his personal source tree

Whether to run unit tests or not

The build runs and the engineer gets an email with the result

Start a new rule: developers run on demandbuilds before they commit


Stepping Stones (2)

Step 2: Buy those lava lamps

Set up the full build system to run more than

once per day

Set up a system the feedback to the entire team

whether the full build succeeded

red/green lava lamps or

a flat panel screen mounted high enough for engineering to

see and

a web page showing live build status

Scheduled builds start to be the heartbeat of

engineering


Stepping Stones (3)

Step 3: Speed up the build

Look into tools that can speed up your build

Set an under-60 minute goal for a full build and

smoke test

Announce to the team that multiple integrations

can now happen per day, during the day

These fast builds begin to enable agile

development


Stepping Stones (4)

Step 4: Fully automate

Integrate your fast builds with your source code

management system

When the source code changes run a stimuli

build and test it

Now the lava lamps really matter

Now the build is the heartbeat of engineering

Start referring to yourself as the Build Automator

not the Build Manager


When to buy

Look at $$$ build tools when

Existing builds are too slow; developers complain

Existing builds are frequently broken

Existing builds are blocking implementation of new

techniques (e.g. Agile/XP)

Development teams are spread across the country or the

globe

Other drivers are

Company mergers

Implementation of new target platforms

A rapidly growing code base


Summary

No one builds their own:

Editor

Compiler

Debugger

SCM

Most people don’t build their own:

Issue tracker

Test system

Why DIY the build system?


For More Information

Electric Cloud

408.419.4300

[email protected]

www.electric-cloud.com

Willowbark Consulting

312.560.5351

[email protected]

www.WillowbarkConsulting.com

mailto:[email protected]



http://www.electric-cloud.com/




© CMC Media Inc. 2008© CMC Media Inc. 2008

Questions and Answers

Please post your questions now using

the “Ask a Question” box

on left side of the screen

View other Webcasts in the Series at www.cmcrossroads.com/wc

enterprise continuous integration presentation

Documents