enabling continuous delivery for major retailer using open source jenkins

Masood Jan – KITS CD

Footer

#jenkinsconf

2

KITS Continuous delivery

• The Systems• The VM• The DEV Process• Continuous integration• Build & Release Pipelines• Automated Testing • Branching strategy• Future goals

#jenkinsconf

Footer

The Systems

#jenkinsconf

Footer

The VM• Oracle virtual box – centos VM.• At least 8gb of ram.• All tools like git, eclipse, java, atg, jboss, oracle

, mq manager and fitnesse packed into vm.• Able to run all ATG applications together.• Easy to copy/export/import

Footer

#jenkinsconf

5

The DEV process Hand-out

#jenkinsconf

Footer

The Continuous Integration

• Commit builds• Hourly builds• Daily cut• Release builds

#jenkinsconf

Footer

Commit Builds

• Compiles & runs Junit tests• Records all recent changes• static code analysis PMD, code

coverage, duplicate code• No functional tests run

#jenkinsconf

Footer

Commit Build Reports

#jenkinsconf

Footer

Build status monitor

• This page shows status of critical builds.• This page is visible on the monitor for

everyone to see.• Build Failures will cause the section to

turn RED.• Teams priority will be to make it green

again.

#jenkinsconf

Footer

Build status monitor – Failure view

#jenkinsconf

Footer

SQA (CI) Builds (2 hourly)

• Build is on demand or every 2 hours• Trigger via build pipeline• Creates deployable package and deploys to sqa.• Runs db deploy for DB changes• Triggers fitnesse Automated CI tests.• If succeeded marks the build as success and uploads to nexus.

#jenkinsconf

Footer

DAILY CUT• Agreed to take final build at 4pm everyday.• Successful Build was tag permanently• Deploy release to qam• Inform QA test team to start testing QAM• The build pipeline continues• Release notes could be prepared at this stage.• A draft automated release note is prepared by

jenkins and uploaded to nexus

#jenkinsconf

Footer

NIGHTLY RELEASE

• QAM Should be stable• Decide on promoting the release.• Share release notes• If agreed, Deploy to suat and sanity test• If agreed, Deploy to uatX and sanity test• Release available next morning• Build pipeline continues until release is

promoted for production

#jenkinsconf

Footer

ATG ENVIRONMENT RELEASE

• This View shows which releases numbers on each environment.

• Clicking on the release number hyperlink leads to the release in Nexus repository.

• This page is visible on the monitor for everyone to see.

#jenkinsconf

Footer

Deployments

• KITS Deployment Service

• Non – Prod.

• Production.

#jenkinsconf

Footer16

KITS Deployment Service (KDS)

• Holds deployment, startup and shutdown scripts for jboss & apache• Holds templates for ATG, jboss, apache environment• Holds environment configurations for all environments• Decoupled from development• Is back bone for deployment pipeline• Developed in simple shell scripts.• Provides easy and simple excel based process for creating and updating ATG

environment specific configurations.• Same scripts used from sqa to production• Invoked from jenkins, takes parameters and executes commands• Has its own GIT repository

#jenkinsconf

Footer17

Environment Config Generator

#jenkinsconf

Footer

Non – Prod Deployments in action

• Stop ATG environment• Take DB restore point (optional)• Download release from nexus and

prepare for deployment• Apply DB changes• Deploy ATG applications• Deploy Apache configs and static

content• Update JBOSS configs• Start ATG environment

Footer

#jenkinsconf

19

Non – Prod.Deploy Job

Footer

#jenkinsconf

20

#jenkinsconf

Footer

Production deployment pipelines

• Full outage pipeline• Rolling Data center pipeline• Silo by silo pipeline• Hotfix pipeline

A separate secured Jenkins is used for orchestrating production deployments

#jenkinsconf

Footer

Full outage pipeline

#jenkinsconf

Footer

Rolling Datacenter pipeline

This pipeline is decommissioned

#jenkinsconf

Footer

Silo Deployments – non-outage

#jenkinsconf

Footer

Hotfix pipeline

#jenkinsconf

Footer

High Level View Of Jenkins internals

#jenkinsconf

Footer

Full Build Pipeline

#jenkinsconf

Footer

Automated Functional Testing

• Fitnesse is being used .• Code has been restructured few times to make it easy to understand.• Its still a area that is being worked on.• High priority tests are now being run as part of CI.

#jenkinsconf

Footer

Fitnesse wiki

#jenkinsconf

Footer

Fitnesse Integrated With Jenkins

#jenkinsconf

Footer31

Fitnesse Jobs in action

#jenkinsconf

Footer32

Fitnesse Reports

#jenkinsconf

Footer33

Branching Strategy

#jenkinsconf

Footer34

Areas of Improvement

• Fitnesse testing, need to make it easily understood by all tech and no tech team.• Junit and functional testing need more focus and widely used by development and

testing teams.• Fitnesse on vm’s• Frequent but stable releases to higher environment- this is key for continuous

delivery.• The more automated test coverage we have the more closer to Continuous delivery.

#jenkinsconf

Footer

Future Goals – Cloud

#jenkinsconf

Footer36

TYPICAL DIY.COM LIVE DEPLOYMENT NIGHT

No more stressful releases No more manual deployments When a deployment job has run

30k+ times, it can’t get any better.

#jenkinsconf

Questions?

37