the butler is still young – applying modern jenkins features to the embedded area


PhD, Hardware Engineering

Background: R&D in Hardware & Embedded • Intel Labs, Sitronics, Synopsys• Large-scale automation infrastructures

Jenkins Developer, CloudBees• CloudBees Jenkins Platform • Customer support and consulting

Core Team Member, Jenkins Project• Maintainer of plugins • Meetup organizer, GSoC Org Admin

Contributor, LibreCores

About me

@oleg_nenashev

oleg-nenashev

2


The Butler is still young –applying modern Jenkins features to the Embedded areaOleg NenashevCloudBees, Inc.


Do know about Jenkins?

About you


1. The most popular CI automation server in the world

2. It’s a framework• Swiss Army Knife of Automation

3. It’s open-source• Can be customized

4. Commercial support is available

Who is Mr. Jenkins?

© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 6

Computinggrids

DocsHW

peripherals

Plugins Configs

Build agents

JobsJenkins-based Automation System


Do you use Jenkins?

About you


Jenkins is Popular

http://stats.jenkins-ci.org/jenkins-stats/

• Common: ~70% of teams use Jenkins

• Almost 150k installations• ~300 active committers

(over last 6 months)• ~3 new plugins every week• ~50 plugin releases every

week

Hudson Jenkins


Many Embedded companies use Jenkins


LTS release linesBackward compatibility•Jenkins 1.x was compatible for 8+ years•Jenkins 2.x is almost compatible with Jenkins 1.x

It’s also conservative about continuous improvement

10

Jenkins is Conservative


NEW WEBSITE

NOW IT’S A CD AUTOMATION SERVER

FOCUS ON USER EXPERIENCE

UI IMPROVEMENTS

SECURITY IS ENABLED BY DEFAULT

PIPELINE AS CODE

11

Jenkins 2

http://bit.ly/jenkins20Stats:• Released on April 2016• LTS line – July 2016• 50% adoption over six months


Jenkins 2 is a milestone, work continues

USER EXPERIENCE• Better stability of installations• Better UIs (Blue Ocean, etc.)

ARCHITECTURE• (?) Pluggable storage• Remoting changes• Jenkins in the Cloud – ”new” requirements

PROJECT INFRASTRUCTURE• Contributor experience• Better quality of plugins & core finally

12

What’s going on in Jenkins?


GOALS• Share info about new Jenkins

features• Share experience about

Pipeline usage in Embedded• Give stickers & T-shorts for

good questions ;)NO-GOALS

• Continuous Integration 101• Jenkins 101• “Classic” approaches to

Embedded use-cases

About this talk

JUC2015/London

More info:


What is an Embedded Product?

14

`Hardware

External IPFirmware / embedded software

External software

Development tools / EDAs

Demo software Demo hardware

Developer kits

Main Product

Documentation

Integ-ration

Boom!


Almost everything needs to be automated

Multiple technologies and tools

Unstable hardware and networks

Legacy stuff is everywhere

Limited resources

Hardware engineers are not software experts

15

Embedded Area. Welcome to Automation Hell!

TechnicalNon-technical


Approach: Use Free-style and Matrix:https://jenkins.io/solutions/embedded/

Classic Approaches in Jenkins

JUC2015/London

More info:

Keywords:

• Script steps to run the logic

• Custom Tools - manage tools

• Throttle Concurrent Builds and Lockable Resources - dispatch access to the shared hardware

• Naginator - restart builds on-demand

• Parameterized Trigger / Copy Artifact -parallelize tests and to collect results

16


Spaghetti Automation

• Many job dependencies• Difficult build flow

tracking• Complex configurations • Duplication

• Maintenance is expensive

Classic Approaches in Jenkins


ü Storage together with the projectü Tracking together with the projectü Tests together with the project

…. as code?Configuration…Documentation…Pipeline… Everything…


Jenkins Pipeline

19

Configuration as Code for Jenkins jobsNot just a configuration as code


Pipeline is a…

Ø Groovy DSL for automatic flow description *• Alternatives – Job DSL, Jenkins Job Builder

Ø Job type in Jenkins (Pipeline Job)• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations

Ø Ecosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2

// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"

} http://bit.ly/pipeline-tutorial

20


Snippet generator


Stored in SCM • Jenkinsfile

Jenkins can create projects from SCM• Multi-branch Pipeline• Folder with a project• Jobs for branches and pull requests

Works even on the organization level• GitHub, BitBucket, etc.

22

Pipeline…. as code


Multi-Branch Pipeline


Pipeline. Useful Features

Robustness against restarts of Jenkins master

Robustness against network disconnects• sh() steps are based on the Durable Task plugin• Jenkins continues execution once the node reconnects

Parallelization across multiple nodes:• It can be achieved via combination of parallel() and node() steps• No need to use the trigger/copy chain (but you can do it)

Ability to store the shared logic in standalone Pipeline libraries

Visualization: Pipeline Stage View, Blue Ocean

24


Pipeline. Stage View Plugin

25

New UI: Blue Ocean is coming soon!


Example 2: Running tests in Parallel


Embedded Components Testing Approaches

Software Hardware

SimulatorsEDA tools

FPGAsASIC prototypesHardware

Accelerators, etc.

x

27


Example. Jenkins Node with an Attached Board

28

VM or bare-metal

PowerReset

when it hangs

Deploy bit-files,Blasting cables

Debugging cables,firmware upload

We don’t change them,

huh?

Jenkins node

Drivers

Tools

Peripherals

Attached peripherals

Runtime


Issue #1. Hardware is Unstable

29

Nodes @ Site A Nodes @ Site CNodes @ Site B“We borrowed FPGAs here, somebody takes them randomly.”

“This stuff has Windows drivers only.”

“No infra team members here.”

“We run Jenkins nodes on developer laptops.”

Frankenboards“We built them from the garbage we had.”

L

L

L

L

L


Node Failover

Pipeline allows re-running unreliable stages on other nodes.

System failure

Node #1 Node #2. . .

. . .

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 255")) {// Fatal errorfail("Test run failed")

}}

}

Try all possible boards:

30


We want to…• minimize the amount of hardware we use• utilize the hardware as much as possible• share hardware between projects

Nice to have:• reuse the existing hardware▸Developer computers (nightly builds)▸Manual QA labs▸…

31

Issue #2. Hardware is Expensive!

M – x000$

L –x0000$M - http://www.xilinx.com/univ/xupv5-lx110t.htmL - HAPS-64, www.synopsys.com


Access should be dispatched by Jenkins

Builds utilize the hardware during a single phase only

Scheduled availability of nodes

Teams fight for resources

Clean environment (again)

32

Issue #3. Hardware is Shared


Resource Management

33

Dispatching parallel access:• Lockable Resources plugin is

integrated with Pipeline• Throttle Concurrent Builds

plugin is not integrated, license limitations may be a problem

Restricting access:• Job Restrictions Plugin• Node access schedule

echo 'Starting'node('board1') {

stage('Build')git url: 'http://github.com/myorg/myproject.git'withEnv(["PATH+GCC=${tool ’gcc4'}/bin"]) {

sh 'make clean build' }

}

stage(’Run tests')lock(env.NODE_NAME + "-board1") {

// Run something on the board// ...

}}


Tools…

34


1. Local tool installation on a node2. External Tool Managers (CLI)3. Jenkins-based solutions

-Via plugins (if available)-Generic approaches: Custom Tools, EnvInject, etc.-Limited availability in Pipeline

4. Virtualization

Tool usage approaches


Advantages:•Quick setup if you have ready infrastructure•Load balancing on physical hardware: vCenter, Docker Swarm•Works well with configuration as code (e.g. Puppet)•Clean environment•Dozens of plugins in Jenkins

Disadvantages:•Poor scalability when you have many tool combinations•Complicated for BIG tools (common case for EDAs)

Virtualization for tools


Docker PluginDocker Pipeline PluginDocker Custom Build Environment Plugin…Yet Another Docker Plugin

Example. Docker plugins


Example. Docker-based tools and Pipeline

Docker Pipeline Plugin:

• You can share workspaces• External Workspace Manager• https://github.com/jenkinsci/external-

workspace-manager-plugin• You can create build chains


You do NOT need to use a single node


Example

https://github.com/jenkinsci/external-workspace-manager-plugin


Jenkins Pipelines are ComplexHow to reuse them?


Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸Jenkins system-wide▸Folder▸Job


Pipeline Library Definition [1/2]


Pipeline Library Definition [2/2]


Using Pipeline Libraries


More examples:https://github.com/jenkinsci/pipeline-

examples


Pipeline. What’s next?

• Stability / Fault tolerance• UI/UX, Blue Ocean• Plugin integrations• Syntax improvements• (?) Freestyle => Pipeline Converter


Blue Ocean

https://jenkins.io/projects/blueocean/

Ø New UI for Jenkins Ø New UX for common Jenkins usersØ Deep integration with Pipeline


Blue Ocean


Pipeline. Dev tools

Available: •Minimal IDE: Auto-completion•Minimal static analysis•Documentation browser for steps•Library manager

Missing:•Debugger•Real-time code deployment from IDE•Real-time approval of Script Security•Freestyle => Pipeline Converter


Summary. Embedded projects & Pipeline

SO…• We successfully adopted Jenkins Pipeline in several projects ▸E.g. LibreCores CI (http://librecores.org/)

• We got advantage in user experience: Templates, Blue Ocean, Pipeline Model Definitions

LESSONS LEARNT• Pipeline greatly simplifies task parallelization and failover within a single build

• Pipeline saves much time on automation system maintenance• Pipeline is a programming languages, high barrier to entry• Pipeline still lacks Dev Tools and integrations, but it evolves


There are many articles about Jenkins in the Embedded area•Most of them describe the “classic approaches” based on Freestyle jobs

Embedded solutions page:•https://jenkins.io/solutions/embedded/

Blogposts:•https://jenkins.io/node/

54

More Info Embedded Solutions Page

…


Jenkins project:• Website: https://jenkins.io• Embedded Solutions page: https://jenkins.io/solutions/embedded/

Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples

CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform

55

Useful Links


Thank You!

Where to find me:

E-mail: [email protected]

GitHub: oleg-nenashev

Twitter: @oleg_nenashev

Want to get more from Jenkins? go.cloudbees.com