jenkins pipeline @ scale. building automation frameworks for systems integration

Jenkins Pipeline @ ScaleBuilding Automation Frameworks for Systems Integration

CloudBees | Jenkins Automotive and Embedded DayBöblingen, April 05, 2017

Oleg Nenashev, CloudBees

© 2017 CloudBees, Inc. All Rights Reserved. 2

About me. Hardware/Embedded

@oleg_nenashev

oleg-nenashev LibreCoresSt. Petersburg Polytechnic University


About me. Jenkins project

@oleg_nenashev

oleg-nenashev


Oleg’s “Hall of Shame”(c)

• Jenkins Core• Remoting• Plugins


2016

http://bit.ly/nenashev-automotive-day-2016


ØHow/why do we build frameworks for Embedded?ØWhat has changed in Pipeline since 2016?ØHow to effectively use and develop Pipeline?ØLessons learnt

Agenda


• Pipeline 101• Pipeline Generic Features

• Blue Ocean• Declarative Pipeline

• Automation Infrastructure• Agents, Tools, Hardware Peripherals

• Added-value Pipeline features in CloudBees Jenkins Enterprise

What is NOT in the talk?

Ask the Experts,Jenkins Online Meetup

Birds of Feather,Ask the Experts


What is a typical Embedded Product?

`Hardware

External IPFirmware / Embedded

SW

External SW

Development tools / EDAs

Demo software Demo hardware

Developer kits

“Main product”

Documentation


Embedded area. Welcome to the automation hell!

• Almost everything needs to be automated

• Multiple technologies and tools

• Integration testing is important

• Unstable hardware & networks

• Legacy stuff is everywhere

• …


OK, Automotive…

Source: http://www.popsautoelectric.com/auto-electric/

• System of systems• High level of integration• Components by 3rd

parties• Many standards• High quality requirements


1. The most popular CI automation server in the world

2. It’s a framework => flexible• Swiss Army Knife of Automation

3. It’s open source => customizable4. Commercial support is available5. Useful plugins

Why Jenkins?


Система

Docs

Plugins Configs

Infra-structure

SamplesDemos

How do we buildCI Systems?

Infra-structure

Infra-structure

• Agents (FKA ”slaves”)• Tools• Hardware peripherals• Computing grids


Approach: Using “classic” Job types:• Freestyle, Matrix, JobDSL, …

Classic Approaches

Keywords:• Script steps to run the logic• Custom Tools Plugin to manage tools• Throttle Concurrent Builds and Lockable

Resources to dispatch access to hardware• Naginator Plugin to restart builds on-demand• Parameterized Trigger / Copy artifacts to

parallelize tests and to collect results

JUC2015/London

https://jenkins.io/solutions/embedded/


Classic Approaches

Spaghetti Automation

• Barely traceable dependencies

• Complex build flow• Complex

configurations • Duplication

• Expensive maintenance


ü Store build definitions with the projectü Track changes together with the projectü Tests together with the project

Configuration as Code


Configuration as Code in Jenkins

Job ConfigurationJenkins system configuration


Jenkins System Configuration… as Code

External Tools

Jenkins CLI and REST

API

python-jenkins

jenkins-client (java)

Configuration Management

systems

Ansible, Chef

Docker

Solutions within Jenkins

project

Groovy Boot Hooks

Scriptler Plugin

SCM Sync Configuration

UnstableJust examples…


Configuration as Code for jobs

Job DSL

Job Builder Plugin

Jenkins PipelinePipeline is offered by default in Jenkins 2.0

…

Groovy

YAML

Groovy


Pipeline is a…Ø Groovy DSL for automatic flow description *

• Alternatives – Job DSL, Jenkins Job BuilderØ Job type in Jenkins (Pipeline Job)

• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations

Ø Ecosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2

// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"

} http://bit.ly/pipeline-tutorial

19


Why Jenkins Pipeline?


Why Jenkins Pipeline?

• Robustness - master restarts & network

• Parallelization across multiple nodes:• parallel()• node()

• Ability to store the shared logic in standalone Pipeline libraries

• Security

• Two modes: Scripted and Declarative

• Visualization: Stage View, Blue Ocean


Система

Docs

Plugins Configs

Infra-structure

SamplesDemos

Automation Framework Vision

Infra-structure

Infra-structure

SCM

Infrastructure as Code

Configuration as Code

Documentation as Code

Pipeline Libraries

Pipelines


Developer Experience

• Encapsulation• Simple UI• Most of the code is in

GitHub• No need in classic UI

for 90% of users• Notifications to

email/GitHub

Jenkins Pipeline in Embedded2016


2016

http://bit.ly/nenashev-automotive-day-2016


2016 – Summary

• Jenkins can be used to run tests on hardware in embedded and automotive projects

• Jenkins Pipeline is helpful for test runs• Pipeline lacks some critical features


• Management of resources.• Integration with Throttle Concurrent Builds plugin• Use-case: limiting the license utilization in automation

infrastructures (JENKINS-31801)

• Better support of tools.• Common build flow – dozens of tools• Integration with Custom Tools Plugin seems to be the best option,

especially in the case of multiple tool versions (JENKINS-30680)

• Pipeline package manager (JENKINS-34186)• Lack of plugins for EDA tools in Jenkins, code duplications• A common approach is to keep the shared "functions" in libraries• Pipeline Global Library and Pipeline Remote Loader can be used

2016 – Pipeline improvement areas

From my talk at CloudBees |

Jenkins Automotive and Embedded Day

2016


Management of resources• Lockable Resources Plugin – since 2016• Throttle Concurrent Builds – coming soon• Pipeline Milestones


PARTIAL - Better support of tools

• withEnv() for environment variables• tool() step in Pipeline was improved

• Tool reference in the Job• Section in Declarative Pipeline

• Custom tools is not fully integrated• Follow JENKINS-30680• ETA – Q2, 2017


DONE – Pipeline Libs (Pipeline Package Manager)

• Early 2016 • Global Pipeline Library• Pipeline Remote Loader Plugin

• Late 2016• Pipeline Shared Libraries• Support of @Grab – Pipeline libs

can pull common Groovy libs

Library Structure:


• Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸ Jenkins system-wide▸Folder▸ Job


Pipeline Library Definition [1/2]


Pipeline Library Definition [2/2]


Using Pipeline Libraries

• Implicit Loading• Via Jenkins Global or Folder configuration

• Explicit Loading• @Library


Using Pipeline Libraries

com.mycorp.libs.make

Jenkinsfile

makeProject()

makeProject() implementation - global variable

com.mycorp.libs.coverity

com.mycorp.libs.cppunit

Globally-defined library, automatic

update

@Library() definition


Early adoption of any framework

http://devopsreactions.tumblr.com/post/150674592321/petting-your-incomprehensible-piece-of-code

• It is working!• In several pet projects…

Lessons Learnt.What does NOT work well?


• Development Tools• Non-Success Path handling• Declarative Pipeline• Manual release management


Pipeline Developer Experience

• No Debugger• No IDE• No built-in test framework• Limited static analysis features

•Pipeline DSL != Groovy


How do I work?

Intellij IDEAFilesystem SCM Plugin (with patch)

Disclaimer: There may be better ways

• Syntax file• Built-in documentation

Jenkins Test Instance

Source Code (local .git repos)

• Production System Snapshot

• Custom Pipeline Library Settings

• Repositories for libraries• Jenkinsfile repos


Getting the Syntax File


Filesystem SCM Plugin

• Pending pull request:• https://github.com/jenkinsci/filesystem_scm-plugin/pull/2

Library definition in the test instance

!!!


Pipeline Testing

• No standard test framework offered by the Jenkins project• External project: https://github.com/lesfurets/JenkinsPipelineUnit

“mvn verify” has been invoked…

And the execution passed…

For the source Jenkinsfile…


Error Handling

• IMHO: Pipeline is designed for Success Path scenarios

When it “Just works”

When it does not


Error Handling. Top-levelScripted Pipeline Declarative Pipeline

• Global try/catch()

When it does not…


Error Handling

When it “Just works”

When it does not

The interesting part…Failover, repeat


Error Handling. Hardware failover

• Pipeline allows re-running unreliable stages on other nodes.

System failure(script returns 253)

Node #1 Node #2. . .

. . .

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 253")) {// Fatal errorfail("Test run failed")

}}

}

Try all possible boards:


Error handling. Hardware failover

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 255")) {// Fatal errorfail("Test run failed")

} } }

???

• Hard to extract error cause• Special handling is required for

every case• Cannot just use existing parsers

• Build Failure Analyzer• Log Parser• Warnings

• Try-catch blocks pollute the code• Hard to test


Context retention

• No way to propagate any data• Solution - @NonCPS steps within the library steps


Error handling. Hardware failover

Declarative Pipeline

Encapsulation via Libraries

Simple use-case

Other

• Exception try/catch• timeout() wrapper• Proper failover• Logic in @NonCPS


Declarative Pipeline

• It is great for simple use-cases• Publishers & Notifiers

• We do not need a framework for simple use-cases

• Lack of features• “How do I pass option for the Docker Agent”• “How do I declare a resource requirement for Throttling?”• …

• Scripted Pipeline is a recommended fallback, but there is no automatic migration


Declarative Pipeline. Recommendations

• Use Declarative Pipeline with caution• Complex agent definitions – use Scripted Pipeline• Complex throttling – use Scripted Pipeline• Post-build steps – work well• Tool definitions – work well• …


Pipeline vs. Application Lifecycle


Pipeline vs. Application Lifecycle

BUT:• Non-CD process with manual promotion• Dependent products• Snowflake development stages


Solutions

• Input() step + milestone() for synchronization• Multiple Pipelines

• Release() step is coming soon• Release Plugin

Pipeline 1 Pipeline 2Run Selector

Staging Artifact Storage

External ALM tool

CD for the Component 1

CI/CD for the Component 2

Lessons learnt• No area-specific issues so far• High barrier to entry• It worth investment


What I need? 2017

• Development Tools for Pipeline• IDE, Debugger, Test Tools

• Community Pipeline Libraries• Common Embedded cases without plugins

• Promotion engine for Pipeline• Ideally: Merge of standard Promoted Builds and Release plugins

• Plugins for area-specific tools


Takeaways

• Jenkins Pipeline is a powerful engine for highly-customized Jenkins instances

• Jenkins Pipeline can be effectively used in Embedded/Automotive use-cases

• Barrier to entry is still high


More info

• Embedded solutions page:• https://jenkins.io/solutions/embedded/

• Links to the articles and videos• Some categorization

Embedded Solutions Page

…


• Jenkins project:• Website: https://jenkins.io• Embedded Solutions page: https://jenkins.io/solutions/embedded/

• Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples

• CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform

Useful Links


Thank you!

Contacts:

E-mail: [email protected]

GitHub: oleg-nenashev

Twitter: @oleg_nenashev

Software at the speed of ideas

THANK YOU!www.cloudbees.com