using cloudbees jenkins enterprise to effectively manage the jenkins ecosystem across enterprise

Using Jenkins Enterprise to effectively manage the Jenkins ecosystem across the Enterprise

Sarah Banks, Kostas Gaitanos, Raj Menattamai

Footer

#jenkinsconf About FINRA www.finra.org

2

Financial Industry Regulatory Authority

Ø  Largest  independent  regulator  for  all  securi2es  firms  doing  business  in  the  U.S.  

Ø  >4,000  brokerage  firms  

Ø  ~161,000  branch  offices  

Ø  ~637,700  registered  securi2es  representa2ves  

Our Mission: Ø  Investor  Protec2on    Ø  Market  Integrity  

Our  Technology  is  powerful  enough  to  look  across  markets  and  detect  poten2al  fraud.    

We  oversee  50  billion  market  transac2ons  every  day  

In  2014  ,  we  expelled  18  firms  from  the  industry,  suspended  705  brokers  and  barred  481  brokers  from  doing  business.  

We  also  fined  firms  more  than  $134  million,  and  ordered  res2tu2on  of  $32.3  million  to  investors  who  had  been  harmed.  

Computerized  cer:fica:on  &  con:nued  educa:on.  

Series  7,  63…etc.  

Educating & informing investors

Actively supporting firms’ compliance efforts

Providing independent, vigorous regulation

Inviting active industry involvement

& input

Footer

#jenkinsconf Metrics for Jenkins Enterprise based ecosystem activities

# of Apps # of Folders # of Jobs # of Builds/yr (estimate)

# of Deployments/yr

(estimate) ~110 ~610 ~1,400* ~100K

promotable builds

~75K (across all envs)

3

* # of jobs active over the past quarter; valid jobs requiring maintenance ~ 2.5K

>100 plugins currently utilized

Footer

#jenkinsconf Discussion areas

4

•  Enable deployments to higher environments via the build pipeline –  Development have the ability to deploy to all lower environments

including QC –  Enable Prod Deployment group to execute production deployments

using the exact same approach as for the lower environments. •  Treat all of Jenkins build-deploy infra as software/source

code –  Automatic creation of all infra (i.e folders, jobs, views/pipelines) –  Use the same framework to create all that make up an application

Footer

#jenkinsconf Enable deployments to higher environments via the build pipeline

5

Footer

#jenkinsconf Enable deployments to higher environments via the build pipeline

6

Folder Template when instantiated will create:

v  Deployment jobs for lower environments •  Job template to create standard XL Deploy deployment job *

v  Deployment job for QA with a child job “promote_to_prod” •  Keep the build forever •  Job template to create standard XL Deploy deployment job AND a child job *

v  Deployment job for Prod •  Job template to create a Prod specific XL Deploy deployment job *

v  PROD folder with a Prod Deployment job §  RBAC is set §  promote_to_prod & prod_deploy jobs are created *

* jobs are created based on corresponding job templates

Footer

#jenkinsconf Folder template groovy code handling RBAC

7

•  <com.cloudbees.hudson.plugins.folder.Folder plugin="cloudbees-folder@4.5"> •  <actions/> •  <description></description> •  <properties> •  <com.cloudbees.jenkins.plugins.foldersplus.SecurityGrantsFolderProperty plugin="cloudbees-folders-plus@2.9"> •  <securityGrants/> •  </com.cloudbees.jenkins.plugins.foldersplus.SecurityGrantsFolderProperty> •  <com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider_-FolderCredentialsProperty> •  <domainCredentialsMap class="hudson.util.CopyOnWriteMap\$Hash"> •  <entry> •  <com.cloudbees.plugins.credentials.domains.Domain plugin="credentials@1.16.1"> •  <specifications/> •  </com.cloudbees.plugins.credentials.domains.Domain> •  <java.util.concurrent.CopyOnWriteArrayList/> •  </entry> •  </domainCredentialsMap> •  </com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider_-FolderCredentialsProperty> •  <com.cloudbees.hudson.plugins.folder.properties.EnvVarsFolderProperty plugin="cloudbees-folders-plus@2.9"> •  <properties></properties> •  </com.cloudbees.hudson.plugins.folder.properties.EnvVarsFolderProperty> •  <com.cloudbees.hudson.plugins.folder.properties.SubItemFilterProperty plugin="cloudbees-folders-plus@2.9"/> •  <com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer plugin="nectar-rbac@4.12"> •  <groups> •  <nectar.plugins.rbac.groups.Group> •  <name>PPL_ROLE_TEC_ART_LOCAL</name> •  <member>PPL_ROLE_TEC_ART</member> •  <role>develop_prod</role> •  </nectar.plugins.rbac.groups.Group> •  </groups> •  </com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer> •  </properties>

Footer

#jenkinsconf Treat all of Jenkins build-deploy infra as software/source code

8

The goal is to create all of our build-deploy infrastructure automatically: Ø  Folders Ø  Jobs Ø  Views/Pipelines

v  Create a single framework ( groovy job DSL) for all application-components that

make an application

v  All of the application related info (i.e folders, environments, pipeline views, components, parameters for plugins used etc…) is captured in a JSON file

v  JSON file is the input to the framework, as the framework effectively becomes the “seed” job; when “seed” job runs it will create all of folders, jobs views/pipelines for the application

Footer

#jenkinsconf

9

Properties JSON "AGS": { "scm": "svn", "name": "ENTERPRISEDESKTOP", "rootFolder": "FLDR_ENTERPRISEDESKTOP_DSL_DEMO", "DevQCDeployJobFolder":FLDR_ENTERPRISEDESKTOP_DSL_DEMO/deployment_jobs", "DevQCDeployJobFolderTemplate": "FLDR_DEPLOY_JOBS_use_with_Job_DSL", "ProdDeployJobFolder": "FLDR_ENTERPRISEDESKTOP_DSL_DEMO/deployment_jobs/PROD_JOBS", "ProdDeployJobFolderTemplate": "FLDR_DEPLOY_JOBS_use_with_Job_DSL/PROD_JOBS",

"DevQC": [ { "name": "DEV_deploy" }, : :

JobDSLmethods (class Folders { static void createParentFolders(def asgProperties, def AGS, def AGSFolder,…) { Folders.createAGSFolder(AGSFolder, AGS)

: : }

static void createAGSFolder(def folder, def AGS) { folder.with { :

: : }

JobDSL Framework import jobdslmethods.enterprise.folder.* import groovy.json.JsonSlurper : : Folders.createParentFolders(asgProperties, AGS, AGSFolder, DeploymentJobFolder, DeploymentJobProdFolder) :

Footer

#jenkinsconf Metrics discussion

10

Template Plugins Manual setup of pipeline * 5 mins 30 mins

~4 hrs

Job DSL & Template Plugins

Manual

Jenkins infra as code^

20 mins 4-5 hours

* Times are per job; on an average each app contains 12-13 jobs

~ 1week upfront investment to design/implement

^ metrics captured based on applications with ~20 jobs/pipelines ~3 weeks upfront investment to design/implement