rule them all - jax londonliveperson ci / cd: docker / kubernetes dev env build deployment tests /...

RULE THEM ALLCI/CD For Micro-Services Architecture

Nir KorenDevOps CI/CD Team Leader, LivePerson Israel

Nir KorenDevOps CI/CD Team Leader, LivePerson IsraelDoing and implementing DevOps and CI / CD for 10+ years

https://il.linkedin.com/in/nirkoren

https://www.facebook.com/koren.nir

@KorenNir

@nir_koren

AGENDA

▸ About LivePerson▸ About LivePerson Production▸ LivePerson CI/CD in a nutshell▸ Microservices (MSA) in a nutshell▸ Goals for better MSA CI/CD▸ LivePerson CI/CD in a practice

ABOUT LIVEPERSON

“

”

We create a

Digital Contact Center to allow transition

From Voice to Messaging

LIVEPERSON

Founded in 1995, public since 2000 (NASDAQ: LPSN)

More than 20,000 customers worldwide

More than 1000 employees

25B site visits and +300M live engagements Yearly

100% SaaS Cloud Company

Headquartered in NYC with 18 offices in Americas,

EMEA and APAC

SMS

Apps

1990 2000 2010

Phone

Today

People no longer call, they message

Started with SMS in 1992. Now dominant across all age groups and geographies

Messages Facebook Whatsapp

Hangouts Twitter Viber

Snapchat Skype WeChat

LivePerson transforms the world’s largest brands customer care

from voice calls to messaging

Learn more about LivePerson @ www.liveperson.com

ABOUT LIVEPERSON PRODUCTION

LIVEPERSON DATACENTERS MAP11

OaklandVirginia

LondonAmsterdam

Tel Aviv

Sydney

Melbourne

LIVEPERSON PRODUCTION NUMBERS

7Worldwide data centers

3MConcurrent users at peak

1.1MMetrics monitored per minute

10,000Servers (virtual and physical)

7000Production deployments per year

3BVisits per month

ABOUT LIVEPERSON CI/CD

ALIGNMENT: WHAT IS CI?

“A key software development practice where members of a team integrate their work frequently.” Martin Fowler, Continuous Integration: http://martinfowler.com/articles/continuousIntegration.html

LIVEPERSON CI / CD NUMBERS

200Microservices (50% K8S)

7000 Ongoing deployments per year

2500Active GitHub Repositories

15,000Running weekly builds in TeamCity

80TeamCity /Jenkins Build Agents

5CI/CD core Team members

LIVEPERSON CI / CD: PUPPET / RPM SERVICES

BUILD DEPLOYMENT TESTS / SCANS OPERATIONSDEV ENV

REVIEW

CI / CD is orchestrated by TeamCity

LIVEPERSON CI / CD: DOCKER / KUBERNETES

BUILD DEPLOYMENT TESTS / SCANS OPERATIONSDEV ENV

REVIEW

CI / CD is orchestrated by TeamCity

LIVEPERSON MSA RELEASE PROCESS

RELEASE REPO PRODUCTION

All is done by the developer independently

LivePerson developers owns the entire process

from

IDE to PRODUCTION

Continuous Deployment

Continuous Delivery

Auto Auto ManualBuild Deployment Tests Production

Auto Auto AutoBuild Deployment Tests Production

ALIGNMENT: WHAT IS CD?

We at LivePerson runs

CONTINUOUS DELIVERYand on our way to

CONTINUOUS DEPLOYMENT

ALIGNMENT: SUMMARY

END OF SPRINTDEVELOPMENT FIXES DELIVERY

PLAN

FACT

CI

CD

OUR ARCHITECTURE: MICRO-SERVICES

We have a single product but in fact we run

200 products

FROM MONOLITHIC TO MICRO-SERVICES

Monolith Microservices

MONOLITH?

▸ Single Deployment and version.▸ Easy to control the servers in Runtime. ▸ Easy to monitor.▸ Single Codebase / Code repository.▸ Easy for integration.▸ Easy to enforce processes and governance.

MONOLITH?

▸ Spaghetti.▸ Hard to refactor / maintain.▸ Hard to implement Zero-Downtime deployments.▸ Each change ”shakes” the entire system.▸ Complicated and long CI processes.▸ Complicated and long deployments.▸ Hard to implement CD (both deployment and delivery).

MICROSERVICES?

▸ Simplicity.▸ Easy for modeling.▸ Easy to add / remove / refactor services.▸ Increase DevOps Culture & processes.▸ Easy to own a better CI / CD Processes.▸ Both frequent and relevant updates (mainly on cloud).▸ Popular: more and more best practices in the web.

▸ Requires DevOps skills, people and culture. ▸ Hard to control.▸ Hard to perform organizational governance.▸ Complexity of interfaces & integration.▸ Operational overhead.▸ Network and performance overhead.▸ Dependencies management (both build and runtime).

MICROSERVICES?

Microservices or Monolith?

There is no right answer

Microservices or Monolith?

”…don't even consider Microservices unless you have a system that's too complex to manage as a monolith…”

Martin Fowler

Microservices or Monolith?

LIVEPERSON ARCHITECTURE

200 micro-services. Each one has its own:

▸ Code repository (*ies)▸ Code review rules▸ Lifecycle▸ Deployment rules ▸ Server(s) and environment ▸ Expose needed API / Entry points

Doing wrong CI/CD in Microservices

LEADS TO CHAOS

CHAOS IN BUILD

Each team uses whatever makes him feel right

▸ Maven, Gradle, Ant… ▸ NPM? Yarn? Grunt? ▸ JDK / Maven. ▸ Maven dependencies▸ Scripts▸ Releases▸ Artifacts, Versions, Naming conventions... Wild West

▸ Chef, Puppet, Ansible… Each team chooses whatever he likes.

▸ Use docker-compose? Kubernetes? Scripts? Plugins?

CHAOS IN DEPLOYMENT

▸ Different technologies for the same languages.▸ Different code-coverage technologies ▸ Different dependencies versions for the same technology.

CHAOS IN TESTS

CHAOS IN POST BUILD ACTIONS

▸ Team can mark unwanted scans and steps. ▸ Cannot monitor processes and statistics.▸ Unable to enforce security / quality policy

Chaos creates

BAD PRODUCTS

MICROSERVICES CI/CD GOALS

What do we want to achieve in order to

CONTROL THE CI/CD?

What do we want to achieve in order to

SUPPORT OUR DEVELOPERS?

GOAL # 1

All aligned with the right (and relevant) tools

Make all versions (and tools) unified

GOAL # 2

Make (most of the) builds run in the same way

GOAL # 3

Enforce (corporate) governance

GOAL # 4

Transparency to (both) Ops and Dev teams

GOAL # 5

Unified path (and checklist) to Production

GOAL # 6

LIVEPERSON CI/CD IN A PRACTICE

PROJECT ARCHETYPES

IN-HOUSE MAVEN / NPM PLUGINS

IN-HOUSE PARENT POM / NPM

POST-BUILD ACTIONS

DASHBOARD AND REPORTS

UNIFIED PIPELINE TO PRODUCTION

1

2

3

4

5

6

PROJECT ARCHETYPES

PROJECT ARCHETYPES

1-Click self-service that will generate everything you need

Need a new Project? Choose your project technology

Choose your project type

Fill in groupId, artifactId,

Organization…

PROJECT ARCHETYPES

Step 1: Project creation (Maven project example)

Maven directories structure

Deployment Skeleton

Maven POM Skeleton

Dockerfile

TestsSkeleton

PROJECT ARCHETYPES

Step 2: GitHub Repository creation

LivePerson GitHub EnterpriseREST Call

Commit from step 1

PROJECT ARCHETYPES

Step 3: Continuous Integration Job (TeamCity)

LivePerson TeamCity ServerREST Call

PROJECT ARCHETYPES

MASTER Build job (VCS)

BRANCHES Build job

PULL REQUEST Build job

IN-HOUSE MAVEN / NPM PLUGINS

IN-HOUSE PLUGINS – WHY?

▸ You can enforce additional steps by the organizational needs.▸ The plugin does not exists outside. ▸ NPM / Maven Framework will do the job for you. ▸ The user should be able to configure it easily.

IN-HOUSE PLUGINS – WHY?

IN THE SAME WAY TO ALLUnified way to run common actions

IN-HOUSE MAVEN PLUGINS – CODE IT

import org.apache.maven.plugin.AbstractMojo;import org.apache.maven.plugin.MojoExecutionException;import org.apache.maven.plugins.annotations.Mojo;

@Mojo( name = "sayhi")public class GreetingMojo extends AbstractMojo {

public void execute() throws MojoExecutionException {

getLog().info( "Hello, world." );

}}

IMPORT MOJO LIBS

WRITE TO CONSOLE

IN-HOUSE MAVEN PLUGINS – USE IT

<build><plugins>

<plugin><groupId>sample.plugin</groupId><artifactId>hello-maven-plugin</artifactId><version>1.0-SNAPSHOT</version><executions>

<execution><phase>compile</phase><goals>

<goal>sayhi</goal></goals>

</execution></executions>

</plugin></plugins>

</build>

MOJO NAME

WHEN TO RUN IT?

DECLARATION

puppet-deploy-plugin

DEPLOYMENT

ci-provisioning-plugin

OPENSTACK INSTANCE

yum-upload-plugin

YUM REPO UPLOADER

checkmarx-plugin

SECURITY SCAN

k8s-maven-plugin

KUBERNETES DEPLOY

cd-flow-plugin

CONTINUOUS DELIVERY

IN-HOUSE MAVEN PLUGINS @LIVEPERSON

LIVEPERSON PARENTS (POM/NPM)

WHAT IS PARENT POM?

▸ POM (Project Object Model).▸ Parent POM = Super POM. ▸ Specifies parent element and makes your project inherit from it.▸ Enforces projects to use it as a prerequisite for release.

Example of the Maven Parent POM

WHAT IS PARENT POM - EXAMPLE?

Project ReactorLivePerson Parent Module

inherit inherit

Super POM Parent POM Child POM

Main Plugins configurations

maven-compiler-plugin3.7.0

Overwrite / Extend configuration

<configuration><source> 1.8

Declare / use the plugin

<failOnError>false

LIVEPERSON PARENT POM

SERVICE #1

POM

SERVICE #2

POM

SERVICE #3

POM

SERVICE #4

POM

SERVICE #5

POM

LivePersonParent POM

LIVEPERSON PARENT POM

POM

<parent><groupId>com.liveperson.maven</groupId><artifactId>liveperson-parent</artifactId><version><SOME VERSION></version>

</parent>

<plugin><groupId>com.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>${maven.compiler.version}</version>

</plugin>

DECLARATION

USAGE

VARIABLE

LivePersonParent POM

LIVEPERSON PARENT POM

No hardcoded versions but Environment Variables Parameters are being injected on build time by Groovy Hooks

$ mvn clean deploy

$ mvn clean deploy -Dmaven.compiler.version=${compilerVersion}

HOOK

def compilerVersion=‘3.7.0’

POSTBUILD ACTIONS

POSTBUILD ACTIONS – WHY?

Enforce Corporate Governance

POSTBUILD ACTIONS – WHY?

▸ Make sure developers cannot comment / remove important tasks.▸ Make sure we are not violating security standards..▸ Make sure we are using legal and permitted software.▸ Enforce corporate quality standards.

POSTBUILD ACTIONS – WHAT DO WE RUN?

Analyze: what do we run on which build type

Build Types:▸ Normal CI Process▸ Schedules (nightly) CI Process▸ Release Build

Identify build type

NORMALRELEASESCHEDULES

Triggers …

POSTBUILD ACTIONS – WHAT DO WE RUN?

[Commercial] Static code scan for security violations

[Open Source] Static code scan for code quality

[Commercial] Dependency scan for security and licenses

POSTBUILD ACTIONS – WHAT DO WE RUN?

Save build data and info into a local database for reports, dashboard, debugging…

Coverage reports (Istanbul / JaCoCo)

POSTBUILD ACTIONS – HOW?

We use Maven Lifecycle Extension. Extending the Maven Lifecycle Class

@Component( role = AbstractMavenLifecycleParticipant.class, hint = ”delegate")public class MyExtension extends AbstractMavenLifecycleParticipant {

@Overridepublic void afterSessionStart(MavenSession session) throws MavenExecutionException {

// Do something…}

@Overridepublic void afterProjectsRead(MavenSession session) throws MavenExecutionException {

// Do something…}

}

POSTBUILD ACTIONS – HOW?

Identify the Executions type (Session: Ended, Succeeded, Failed….)

ExecutionEvent executionEvent = (ExecutionEvent) event;MavenProject project = executionEvent.getProject();

if (executionEvent.getType() == ExecutionEvent.Type.SessionEnded) {// Do something…

}

Place your Maven extension JAR under your Maven installation:

${maven.home}/lib/ext

POSTBUILD ACTIONS – HOW?

Each jar is being replicated to the build agents to ${maven.home}/lib/ext

Build Agent Build Agent Build Agent Build Agent Build Agent Build Agent

DASHBOARDS AND REPORTS

WE COLLECT ANYTHING WE CAN

We write to the DB from any plugin / process we canWe expose any data we need in a REST API Service

REST API

BUILD PUPPET POST BUILD QA DEPLOY PROMOTION PRODUCTION

DASHBOARD FOR CI SERVICES

UNIFIED PIPELINE TO PRODUCTION

LIVEPERSON CD PIPELINE

Can be automated on green CI or manually on demandUnified process for all services

Check if QA environment

status is green

Run the deployment

(Puppet or K8S)

Save Deployment results and info

Update Slack Channel with the

results

Update ServiceNow with the relevant info

Manage and Orchestrate

MAIN IMPACT

Our CI Processes are

ROBUST AND BULLETPROOF

We know the process and can

SUPPORT ALL PROJECTS CYCLES

Onboarding new services and projects

IT’S EASY

Single point of failure

EASY TO FIX, EASY TO MAINTAIN

87

THANK YOU ALL!https://il.linkedin.com/in/nirkoren

https://www.facebook.com/koren.nir

@KorenNir

@nir_koren