devops: sprinkle dev, sprinkle ops, let's make cake, not mud pies

Sprinkle Dev, Sprinkle Ops, lets make Cake not Mud Pies

Brian Paulsmeyer

Agile Gravy: September 29, 2016

Agile Development Pitfalls

… that warn us of mistakes in our agile implementation

05/03/2023www.centricconsulting.com 2


Waterfall in 2 Week Iterations

•No ordering of stories•Gold plating of features•“Agile Waterfall” Assumptions• Learning from usage of system won’t change features• “All features will come together” in last day of last sprint• “Pause the world for 3+ years for us to finish”• “Value Proposition: All or Nothing”• Architecture can be “known” and not change• Deployments are still hard

•Unfortunate ending for “Agile Waterfall”• “We ran out of money, and have nothing in production.”

Constant Redefining of “Definition of Done”

•Done should simply mean done•Burndown/Burnup no longer reliable•Feature inversion occurs when time spent on defects and maintenance overtakes feature work


1 2 3 4 5 6 7 8 9 10

20 20 2018 17 17 16 16 15

13

0 0 02 3 3 4 4 5

7

Feature InversionFeatures Defects and Hidden Work

Only focusing on Unit Tests and Manual Testing

•Unit tests• Won’t ensure business workflows• Only allows minor refactoring• Failure: Mars Climate Orbiter (1999)

• Team A: Pounds for unit of force• Team B: Newtons for unit of force (1 lb = 4.45 N)• Disintegrated in Mars atmosphere!• Each “Unit”/Component worked in isolation


Continuous Delivery and DevOps

… simple concept, requires diligent implementation


No more “Throw It Over the Wall”


DevOps enables Continuous Delivery

•Team performs activities across disciplines•Apply principles from iteration 0 through delivery


Regression

UATIntegration

Testing

DevOps – Full Lifecycle Delivery Pipeline and Container Management

Local Environments

Builds

Unit TestsMerge

Procedures

Log Interpretation

Dependency Mgmt

Dev Deployments

Traditional DEV activities Traditional OPS activities Traditional QA activities

Prod Deployments

Performance Mgmt

Monitoring

QA Deployment

s

Branch Mgmt

Change Mgmt

Log Management

Configurations

Integration Deployment

s

Performance testing

What It Takes

Whether it be security, regulatory, software limitations, etc., there are many reasons why organizations say CD is impossible


Tooling

Process

Culture

Architecture

Basic Definitions and Levels of Maturity

Continuous…


Automated Deployment to ProductionDeployment

Manual Compile/Build, Manual Testing, Manual DeploymentPain

Automated Deployment to Integration/Staging/Pre-Prod

Deployment Toolset Constant across all Environments

Delivery Automated Integration/Workflow Tests

Automated Component Builds

Automated Component Tests*

Integration Automated Unit Tests

* Many teams mistakenly consider Continuous Integration (CI) complete with only automated unit tests

Pipeline

•Recipe for building, deploying, and testing the system•Grouping of discrete stages for a complete workflow•Coordinates disparate technologies•Provides feedback of system state across environments


Business & Customer

Develop

Customer

Deployto

Production

Deployto

Staging

Deployto QA

Buildand Unit Test

Automated Integration /

Workflow Testing

Detailed Pipeline


Business & Customer

Backlog

Consolidate Features and Hotfixes

Iteration Planning

Build Server“X”

Stories

Development

1 Active Story Per Developer

1Commit

Acceptance Test

Customer

BuildArtifact

Acceptance Test Environment

QA Testing

QA Environment

Auto-Trigger Pipeline:Installs To AIT Environment

Executes Smoke and Acceptance Tests (<10 min)

Manual Trigger Pipeline:Installs To

QA EnvironmentExecutes Smoke Tests

UAT Testing

Manual TriggerPipeline:

UAT EnvironmentUAT Smoke &

UAT Tests

Production

Manual TriggerPipeline:

Prod EnvironmentProd Smoke Tests

UATEnvironment

Product Pipeline


Pipeline Workflow Tools

… to speed implementation


Pipeline Stage Implementation

•Pipeline workflow steps trigger external tools


Business & Customer

Develop

Customer

Deployto

Production

Deployto

Staging

Deployto QA

Buildand Unit Test

Java.Net

TypeScriptPHPetc.

AntMaven

MSBuildMakeetc.

JUnitNunitetc.

BashPowerShellChef/Puppet

VMsDocker/CoreOS

etc.

Automated Integration /

Workflow Testing

SeleniumCucumber JBehaveSpecFlow

Codeceptionetc.

Same Toolsas Deploy to QA

Only ConfigurationDifferences

Same Toolsas Deploy to QA

Only ConfigurationDifferences

Build Tool Jobs (ex. Jenkins)

Key Pipeline Coordination Tool Features

•Pipeline Triggers• Automatic on code check-ins to source control• Manual• Programmatic API

•Workflow• Parallel steps• Sequential steps• Failure-only steps• Graphical view of steps

•Artifact Tracking• Traceability from check-in to final deployment• View of deployed version in each environment• History of deployments• Ability to trigger updates to agile tracking tools (ex. JIRA)

•Environment Support• Same deploy scripts, allows configuration changes per environment


Pipeline Workflow Tool : Jenkins

•On-Site•Version 2.0 Provides Workflow View•Software Language Agnostic


Pipeline Workflow Tool : ThoughtWorks Go

•On-Site•Software Language Agnostic


Pipeline Workflow Tool : Octopus Deploy

• .Net


Automated Testing

… so every day can be deployment day


Automated Component/Integration/Workflow Testing

•Opinions of why automated testing isn’t needed• “Manual testing is sufficient”• “Business only wants to spend money on features”• “Developers only want to code features”• “Our <insert technology> is too hard to test with automation”• “Automated tests run too slow”


Best Case: Missed DeadlineWorst Case: Catastrophic Failure in Production

Result of skipping automated tests

Selenium WebDriver

Direct Test using Selenium WebDriver

WebDriver driver = new FirefoxDriver();// Navigate to webpagedriver.get("http://www.google.com");// Find the Search Field (UI Locator)WebElement element = driver.findElement(By.name(“Search"));// Fill in the dataelement.sendKeys(“something interesting");// Now submit the form. WebDriver will find the form that contains the elementelement.submit();// Assert the test passed


Selenium WebDriver Web Page

Test

Test

Test

Test

Page Object Model


• Separate Test Code and UI Locators

• Supports Don’t Repeat Yourself (DRY) for Element Locators

• Less Technical Test Case

SearchPage page = new SearchPage(selenium);// Enter Search informationpage.search(“something interesting");// Now submit the formpage.submit();// Assert the test passed

Page Object Model(s) Web Page

Test

Test

Test

Test

Selenium WebDriver

ServerAPI

BDD Testing

• Gherkin (English) Text File

• Step Definitions Map Between English Test and System

• Business Readable Test Case

Given user searches for “something interesting”When user performs the SearchThen assert something is true


Page Object Model(s) Web Page

Step Definition

Step Definition

Step Definition

Selenium WebDriver

ServerAPITest

Test

Test

Test

Capabilities

… we need to be successful


Rapid Response to Changing Business

•Need to be able to react to disruptive technology and business models•Stagnant architecture limits new technology adoption• Changes blocked by fear of breaking system• Inability to deploy new systems to production• Unable to adopt new security upgrades• Unable to adopt new software versions


Drive Defects From Most Used Features

“You can’t test quality in”

At a minimum make sure defects aren’t in the standard user workflows



Adjustable/Changeable Architecture

•Magic Quadrant Architecture• “Best” choice today ≠ Best choice tomorrow• Failed technologies and companies in all parts of magic quadrant

•Natural product End-of-Life requires new architecture•Continuous Delivery and DevOps allows fluid architecture changes

OODB(Object-Oriented Database)

EJB 1.0

Cadence

•What if everyday was a normal day•What if those normal days were release days•What if those release days were multiple releases throughout the day

•Low Risk On-Demand Releases


Deployment Styles

… for production deployment


Standard Deployment

•Deployment Steps• Disable user access• Upgrade servers from V1 to V2• Test deployment• Enable user access• Upgrade destabilizes running environment

•Problem Mitigation• Re-Enable backup if possible• Developer troubleshooting requires

customer downtime



Blue Green

Router / Load Balancer

Blue-Green Deployment

•Two Identical Production Environments• Blue• Green

•One Active: Router Controlled•Deployment Steps (Green Active)• Deploy to Blue environment – No users

affected• Perform smoke tests on Blue environment• Optional: Switch subset of users to Blue

(similar to Canary releases)• Switch all users to Blue environment

•Problem Mitigation• Don’t switch to 2nd environment• Developer troubleshooting has no customer

impact

Canary

Router /Load Balancer

Canary Deployment

•One Production Environment•Deployment Steps• Deploy to 1 server (Canary)

Optional: Provision and build new server from scratch

• Router sends portion of users to Canary• Random• Internal users only• User-type based (ex. Test users)

• Gradually upgrade servers to V2• Gradually switch users to V2 servers

•Problem Mitigation• Don’t switch additional users to Canary

environment• Developer troubleshooting has no customer

impact


Case Studies

… that show it can be done

Before Build Automation

•23 Teams• 150 Team Members (Developers, Quality Engineers, Scrum Masters)• 100 Developers

•Service Oriented Architecture• Managers, Engines, Data Access

•Baseline End of PI6 (SAFe)• PI6 demo system setup

• Two weeks to setup• Manual Installations – not repeatable• Individual Teams created “magic instructions” in isolation to install and run solution

• Some teams unable to run full solution• Custom data loaders used to skip workflow steps• Developers unable to test full workflow• Movement of data through workflow steps untested till end of PI

• Update of component versions caused “Day of Downtime” for developers to fix unexpected issues


After Build Automation: Jenkins Easy Button Builds

• Starting Point: a machine with Windows — that’s it!• The rest is automated:

−Provisioning: verify/install tools takes ~50 minutes− Puppet Agent, PowerShell, 7-Zip, VNC− SQL Server, .Net, Visual Studio− Noticia Repono Client

−Install Custom Software takes 5-10 minutes− Clean system from any previous runs− Install/Configure Base Components− Install System

− Install and Run Smoke Tests

At this point the system is ready to be used to do further testing, demos, etc.

• Deploy cycle repeats

Chun

ky M

onke

yVa

nilla

Definition of Done

… that makes our customers happy


Continuous Delivery Definition of Done

•Automated Deployments• Fully Automatic or Push-Button Deploys• Same Executable Scripts for all Environments• No Deployment Hero

•Automated Testing• Unit• Integration• Full System

•Always Shippable• No Temporary Code• No Untested Code• Feature Toggles

•One Backlog• Production Bug Fixes• New Features


About Me


• Our highest priority is building lifelong relationships with clients based on trust, respect, and collaboration.

• We invest in our talented team and support their well being by keeping them challenged and inspired.

• Our localized company structure allows us to play very active roles in the lives of our families and community.

This relationship-centric focus keeps us passionate, committed and motivated in all facets of our lives. That’s why our team is here to stay.

Centric Consulting

• 17 years as a software developer and architect• Full stack developer in Java, .Net, TypeScript (Angular 2), and PHP• DevOps tooling experience including VMs, Docker, Chef, and Puppet• Continuous Delivery and Automated Testing implementations• Experience across industries including FDA and SEC regulated

Continue the Journey • http://centricconsulting.com/centric-st-louis-scaled-agile-workshop/

Thank You for Attending!

devops: sprinkle dev, sprinkle ops, let's make cake, not mud pies

Technology