optimizing your ci: lessons learned from a successful jenkins rebuild - juc us west 2015
TRANSCRIPT
Optimizing Your CI: Lessons Learned from a Successful Jenkins Rebuild
Santa Clara, Us West, September 2015Jonathann Zenou
Footer
#jenkinsconf
Greetings fellow humans
• Hi, my name is Jonathann• Living in Tel Aviv, Israel• Originally from Marseille, France• Leading the DevOps effort at Ravello Systems• @Zenoujohn• [email protected]
#jenkinsconf
Footer
What is Ravello Systems ?
#jenkinsconf
Footer
Ravello Systems
• 30 developers • 7 major projects• Kanban like
#jenkinsconf
Footer
A part of our stack
#jenkinsconf
Footer
What are we going to talk about ?
#jenkinsconf
Footer
Old problems
#jenkinsconf
Footer
Jobs Leftovers
• Causes full disk• Workspace Cleanup Plugin• Can pollute your maven repo or npm cache• Solution needed : a new clean environment for every job
Footer
#jenkinsconf
Better resource distribution
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
Backend Team Frontend Team
“waiting for next available executor …”
Footer
#jenkinsconf
Better resource distribution
Backend Team/Frontend Team
Footer
#jenkinsconf
Better resource distribution
Backend Team/Frontend Team
Footer
#jenkinsconf
Better resource distribution
Backend Team/Frontend Team
Footer
#jenkinsconf
Frontend Team leader’s reaction
#jenkinsconf
Footer
Better resource distribution
• Solution needed : a rapid and scalable way to spawn dynamic environments
#jenkinsconf
Footer
Better resource distribution
#jenkinsconf
Footer
Using Docker in Jenkins
• Using Jenkins Docker plugin• Dedicated Docker host for each team• Hosts machine capabilities are a function of team size and
requirements • All images on all hosts• Problems solved : jobs leftovers and resource distribution
#jenkinsconf
Footer
Security
• No active directory or LDAP• Jim uses Dave’s account• Employee leaves the company
#jenkinsconf
Footer
Security
#jenkinsconf
Footer
Security
#jenkinsconf
Footer
Security
• Solution needed: third party dynamic authentication mechanism
#jenkinsconf
Footer
Security
• Using Jenkins Google Login plugin• Coupled with Matrix Authorization Strategy plugin• 2-Factor authentication• Open our Jenkins to the world
#jenkinsconf
Footer
Too much code in jobs
#jenkinsconf
Footer
Too much code in jobs
• Hard to maintain• Can be easily broken• “Who touched this ??”
#jenkinsconf
Footer
Too much code in jobs
• Solution:• Put your scripts in revision control (Github)• Using Jenkins Multiple SCM plugin
#jenkinsconf
Footer
Jobs Versioning
#jenkinsconf
Footer
Jobs Versioning• Solution:
• Using Jenkins JobConfigHistory plugin• Saves old configurations• Diff between jobs configurations• Revert job configuration
• Coupled with Jenkins SCM sync configuration plugin• Saves you whole Jenkins configuration in revision
control (Github)
#jenkinsconf
Footer
Jobs data sharing
Backend Job Frontend Job
Parameters:- Backend version: 1.1- Frontend version: stable
Parameters:- Frontend version: 1.1- Backend version: stable
#jenkinsconf
Footer
Jobs data sharing
• Solution needed: an interface/service that will store data that will be accessible from all the jobs
• Using Jenkins Global Variable String Parameter Plugin• Cannot alter them from a job• But you can using Groovy …
#jenkinsconf
Footer
Jobs data sharing
#jenkinsconf
Footer
Jobs data sharing
• Keep a global parameter for each of product stable version
• Let all the other jobs use these values• Change the global parameter value of a product stable
version automatically when this product passes a green CI• Gist link: https://gist.github.com/johnyzed/2af71090419af2b20c5a
#jenkinsconf
Footer
CI at Ravello Systems
build+
unit tests
environment creation
codedeployment test suites stop
environments
artifactory
log collection
#jenkinsconf
Footer
CI at Ravello Systems
environment creation
#jenkinsconf
Footer
Before digging further
#jenkinsconf
Footer
The pieces are ready• What Do Developers Want ?• Blackbox• No duplication• C.D.R.S• Single form, with a lot of triggers• Using Multijob Plugin
#jenkinsconf
Footer
Jobs sandbox• Improving a job without:
• Breaking it• Stopping other people using it
• Solution needed: a parallel Jenkins environment almost identical to our production Jenkins environment to test the new changes on
#jenkinsconf
Footer
Jobs sandbox• Jenkins environment blueprint on Ravello Systems • Using Ansible:
• git clone• minor changes
• Running regular jobs• All that orchestrated from a Jenkins job
#jenkinsconf
Footer
Jenkins sandbox flow
changes on Jenkins script
branch
create dynamic Jenkins
environmentrun jobs stop dynamic
environment
#jenkinsconf
Footer
Jobs sandbox
#jenkinsconf
Footer
Comparing Jenkins environment
Old Jenkins New Jenkins
Number of jobs 137 56Number of slaves 10 0-50Build and test duration 70 43
#jenkinsconf
Footer
Comparing Jenkins environment
#jenkinsconf
Footer
Take-aways
• Ask your developers how to make their life easier
• Don’t be afraid to break things• Follow @jenkins_release
#jenkinsconf
Footer
One last thing
• Ravello Repo• http://www.ravellosystems.com/repo/• http://www.ravellosystems.com/repo/blueprints/6039143
6
#jenkinsconf
Footer
Questions ?
#jenkinsconf
Footer
Please Share Your Feedback
• Did you find this session valuable?• Please share your thoughts in the
Jenkins User Conference Mobile App.
• Find the session in the app and click on the feedback area.
#jenkinsconf
Thank you