Standardizing and Automating Deployment Configuration

Paychex: David Jozis

Housekeeping This webinar is being recorded

Links to the slides and the recording will

be made available after the presentation

You can post questions via the

GoToWebinar Control Panel

Follow the conversation on Twitter

About me

David Jozis

15+ years experience in coding

Professionally for about 8 years

Enjoying the innovation culture at Paychex for

just over 3 years

1 year as a developer

2 years in in the EBA group at Paychex

(Enterprise Build Automation)

Technology/coding is both my job and my

passion

3

Agenda

Introduction and context

Our build/deploy pipeline

What was wrong with how we configured it

What we did to configure it faster

Standardization

Automation

What we learned

This is not about how we sped up deployments, this is about how we sped up configuring new

deployments

4

Introduction and Context

About Paychex Industry leader in Human Capital Management (HCM)

services with solutions for payroll, human resources,

insurance, and retirement

Established 1971

Headquartered in Rochester, NY; 100+ U.S. locations

600,000+ small- to medium-sized business clients

Technology-enabled service company

24/7/365 days per year support

13,000+ Employees

1,500 IT professionals

Paychex Flex | www.paychex.com

6

The Enterprise Build Automation team (EBA)

A “dev service” team

Develops/maintains/supports the build/deploy/release pipeline for consistent re-use across Paychex

We don’t write business application code

We don’t run the releases

Configure and write plugins for Bitbucket, Jenkins, Gradle, XL Deploy, XL Release

Made up of 17 members, 3 agile teams

Serves ~1,500 IT staff, ~80 agile development teams

~90:1 staff ratio

We interact with lots of different teams

7

Our tools

8

Left to right: earlier to later in build/deploy/release pipeline

Build/Deploy Pipeline ChallengesThe challenges with manually configuring your automated platform

Previous State: Paychex Build/Deploy Pipeline

EnvironmentInfrastructure

Git repositorymanager/store

Runs build,publishes to

XL Deploy

Stores artifacts anddeployment metadata

10

Challenges around Bitbucket phase

Application naming inconsistent with Jenkins job or XL Deploy application

Fragile link between where build artifacts are created and where Jenkins looks for them (maintained in

two places)

Build/test logic in Jenkins is different from running locally

11

Challenges around Jenkins phase

Definition of XL Deploy metadata done through the Jenkins UI

No testing outside of Jenkins (locally building XLD packages?)

No loops or other dynamic configuration

Copy/paste mistakes

Not source-controlled – makes change tracking more difficult

No consistency enforced

Filling up XL Deploy with build artifacts, not just metadata

Dev have access to misconfigure things, such as no tags

12

Challenges around XL Deploy phase

Configuration of “Infrastructure” and “Environments” in XL Deploy done manually

Each team member of our shared services team configured applications slightly differently

Tag usage inconsistent – mainly based around infrastructure

Infrastructure based tags problematic for new datacenter with phased onboarding of applications

Missing information in onboarding requests

Needed to reconfigure every WebLogic deployment for a phased technology upgrade – too slow

13

Challenges around real infrastructure

Cluster naming inconsistent

Incorrect information turned over

WebLogic admin servers down

The infrastructure does not actually exist

Uses slightly different configuration due to manual setup

Deployment failures due to inconsistent/incorrect configuration

14

15

I need my application configured for deployment

Developer EBA member

What sort of application? Does it have properties file, static web content?

It’s a WebLogic application. I think it needs these files.

I only found a cluster “my-exmpl”, is it that? Also, web-server-x doesn’t exist.

Which clusters does it go to? Which webservers do the static content go to?

It’s my-example-cluster and web-server-x.

I don’t know. CC: WebEngineer. (Discussion gets lost between teams.)

Summary of struggles

Lack of consistency

Lack of source of truth

Lack of accountability

Ineffective communication, too much back-and-forth, too many handoffs, etc.

Too slow

16

Standardization and AutomationWhat we did and what we learned

Inspiration from Netflix

Nebula is an opinionated set of plugins for the Gradle build system created by Netflix

Code is built and tested locally using Nebula

Changes are committed to a central Git repository

A Jenkins job executes Nebula, which builds, tests, and packages the application for

deployment

Gradle chosen because:

easy to write testable plugins

reducing the size of a project’s build file

More reading: https://medium.com/netflix-techblog/how-we-build-code-at-netflix-c5d9bd727f15

18

Inspiration from Amazon

Mandate at Amazon, from around 2002

All teams will henceforth expose their data and functionality through service interfaces

Teams must communicate with each other through these interfaces

It doesn’t matter what technology they use

How to apply this to our manual onboarding process?

More reading: https://apievangelist.com/2012/01/12/the-secret-to-amazons-success-internal-apis/

19

Standardization

A standard that isn’t enforced by code is a suggestion

Developed standard with agreement from

Architecture

Development

Infrastructure/Ops teams

Wrote a Java library that does standard validation

Reusable set of classes with common methods for validating different standardized elements

Different classes for different standards – WebLogic, OpenShift, .NET, Java stand-alones,

Pro*C artifacts, etc.

Throws helpful errors that guide the user to adherence

20

Totem

The Onboarding Tool EBA Made

Java/GWT (Google Web Toolkit) Web Application

Exposes scripts that do what we used to do manually

Consistently

Automatically

Instantly

Has both Web UI and REST endpoints for every script

Absolutely minimal inputs

Didn’t use Jenkins Jobs primarily to get finer grained permissions control

Example: Anyone can run a “validate” on a script, but only LDAP group X can run Execute

22

Totem – How it works

Reads information from canonical source of truth in source control

Checks the source of truth against the real world for consistency

Uses the standards library to protect against non-standard configuration and input

Application naming

Tags

Infrastructure configuration

Permissions

Detects current state and only does what is necessary – can rerun

23

Log in

Select a script

Fill out input

Execute!

Wins

Enabled the customers!

No more EBA team involvement with onboarding common technologies

Culture change: Instead of providing a service, develop a platform for self-service

Much more scalable

5x more onboardings today than 10 people could perform in the same time

No deployment failures due to inconsistent XL Deploy configuration

Application specific tags means we control where things go

25

Shift left

Shift validation to run sooner

Created a Gradle plugin for publishing to XL Deploy

Parsed that metadata, applied the same standards library

Get build failures in Jenkins due to standards violations

Shift validation to run sooner again

Run the same Gradle script locally, before check-in

Get failures before committing configuration

26

Package task

Publish task

No duplicate maintenance of artifact paths

Generate/validate application specific tags

Additional improvements

Dynamic/flexible configuration

Can manipulate artifacts during the publish script to publish slightly different versions

Can publish in a loop

Used the above to publish slightly different artifacts for internal/external service deployments

They can be separately deployed, and be tagged separately

Created a Gradle plugin to publish the artifacts to Artifactory

No configuration, apply the plugin and it handles the rest dynamically by reading the XL Deploy

package configuration (and updating it)

Make the Gradle validation plugin check Jenkins job name and Bitbucket repository name in the future?

28

Current State: Paychex Build/Deploy Pipeline

EnvironmentInfrastructure

Git repositorymanager/store

Runs build,runs Gradle

publish script

Publishes buildartifacts toArtifactory,

and package toXL Deploy

Stores binaries

Stores deploymentmetadata

29

30

I’ve configured my Application for deployment. Configure the Infrastructure?

Developer Totem

I checked Web Engineering’s files in source control. Because of your

application name, I can see that it goes to servers X/Y, but X doesn’t

exist in reality, and there’s another server Z that is in that cluster. Please

work with them to resolve it and try again.

I’ve configured my Application for deployment. Configure the Infrastructure?

Done. Here’s all the changes I made.

Efficiency gains

One EBA team’s velocity in 2015 was about 15

points per 2 week sprint

3 onboardings per sprint with nothing else

Totem puts through ~147 points worth of

onboardings per 2 weeks

~10x more throughput

Equivalent of ~40 EBA members

Doesn’t need EBA involvement

EBA team now spends their time on projects that

will advance Paychex CI/CD capabilities

Velocity has increased 3x or more

Much less back-and-forth

0

20

40

60

80

100

120

140

160

One (of the two) EBA teams in2015

Totem in 2017

Story points worth of onboardings per 2 weeks

Story points worth of onboardings per 2 weeks

31

Lessons learned

A standard that isn’t automatically enforced is a suggestion

Groovy DSLs can offer great flexibility for configuration

Self-service apps needs both a UI and REST API

Good automation can result in buy-in and culture change

Demand scales to meet supply

Everyone expects everything Just In Time now

More than 5x output required than 6 months ago

Assumptions will be made about what it does or doesn’t do

Fear of the unknown – training and communication are important!

32

Lessons learned

After you improve, get ready to improve again

Eat your own dogfood

We build and deploy Totem via the same pipeline

Good abstractions give you convenient points to extend from

We’re using the same Gradle scripts now to gather code coverage metrics and associate them with

artifacts

We were able to trivially add publishing binaries to Artifactory instead of XL Deploy because we put

our configuration for deployments in Gradle

Take what you do today, write it in code, then iterate

No need to start with perfection

33

Questions?

Don’t miss our next webinar – Nov. 7th

Guest speaker: Robert E Stroud CGEIT CRISC –

Principal Analyst, Forrester Research

35

Thank you!

▪ Don’t miss our next webinar – Nov. 7th

▪ Guest speaker: Robert E Stroud CGEIT CRISC – Principal Analyst, Forrester Research