A DevOps JourneyJon Ar i ld Tørresdal

2

Why DevOps?

• Release planning and management

• Handoffs (queue time)

• Manual labor and coordination to get a application into production

• Frustrating long delivery times (both for features and bug fixes)

• Added risk of releasing too many changes at once

«The gap between Dev and Ops will in most organizations result in a major bottleneck,bringing with it a significant (hidden) cost way outside of IT.»

— Jon Arild TørresdalWhat are these costs?

3

«If it’s painful, do it more often!»

4

Gains of DevOps

• Freeing time for highly educated and paid employees to focus on continuous improvement

• Common competence between Dev and Ops results in better efficiency and innovation

• Faster time to marked

• Fast feedback loops from users, allowing quick adjustments

5

DevOps Reforms in Frende Forsikring

6

Reform - Ops

• External IT operations

• No access to servers in production

• No Load Balancing during deployment

• Everything virtualized except Database Servers

• Changes and deployments was executed by hosting provider

• “No” monitoring

• New (still external) IT operations

• Clear separation between Dev and Ops responsibilities

• Dev has full access to all environments, incl. production

• No changes allowed on servers - all changes are coded as “Infrastructure as Code”

• Everything virtualized, incl. database servers

• Sticky load balancing to ensure optimum scaling deployment

• Monitoring in production by Dev

• Hosting provider delivers bare OS

• Bootstrapping and configuration of servers + deployment is fully automated

Before: Now:

7

Reform - Development

• Team Foundation Server for source control and build

• Some unit testing, but not structured

• Code freeze before each release

• Database schema changes was generated by diffing test and production

• Bug-tracking

• TeamCity for build (build agents runs at Amazon)

• Git for source control

• Unit tests and integration tests when needed

• Never code freeze

• Extreme focus on low complexity and micro services

• Every push to Git can potentially end up in production at any time

• No branching

• Little or no bug tracking

Before: Now:

8

Reform - Deployment

• Deploy packages was handed off to hosting provider

• Release approx. once/mont

• Rollback of code and databases on errors

• Executed early morning or late nigh

• All deployments automated

• Can deploy at any time without any downtime

• Multiple deployments to production every day

• Deployment Pipelines

• Canary releasing (e.g. Customer Center can try out features before the customers)

• Schema changes in databases are forward and backward compatible

• No rollback, forward only - fix error, redeploy

• Continuous Deployment in Test and Pre-Prod, Continuous Delivery in Production

• Full audit of who and when deployments is executed

Before: Now:

9

Reform - Process

• Startup company in rapid growth

• Scrum like

• Business expected release on fixed date

• Long feedback loops

• Manual testing of the "entire" application before release

• Company with high cost focus (combined ratio in insurance)

• Kanban/Lean

• Stakeholders and upper management participate in daily standups

• Aware of the increased risk with infrequent releases

• Actively uses canary releasing

• Continuous testing and verification by stakeholders

Before: Now:

10

9 established principals

11

Principal #1 - Deployment Pipelines

Image from: Continuous Delivery Book

12

Principal #1 - Deployment Pipelines

13

Principal #2 - Deploy vs Release

• You can Deploy to production…

• …without Releasing functionality to users

14

Principal #3 - Feature toggles

Image from: http://labs.qandidate.com

15

Principal #4 - NO branching!• When you branch, you opt out of Continuous Integration!

• No branch, no merge :-)

• The alternative to branching is feature toggles

• If you insist on branching, note the following:

• Reduced delivery capacity

• Reduced quality

• Increased risk of errors

• Added complexity

16

Principal #5 - No manual changes on servers

Image from: http://www.spreadshirt.co.uk

“Admin GUI’s on servers are poison!” - Build 2015

— Jeffrey Snover, Distinguished Engineer and Lead Architect for Windows Server

17

Principal #6 - New project, deploy Hello World

Image from: http://fringe.chrisay.com

18

Principal #7 - Version Control Infrastructure

Image from: http://www.bhmpics.com/

19

Principal #8 - Canary Releasing

20

Principal #9 - Done == In Production!

• No Done Done

• No Done Done Done

• No Done Done Done Done Done….

• Done == In Production!!!

Image from: http://www.definition-of-done.nl

21

The 9 principals1. Deployment Pipeline

2. Deploy vs Release

3. Feature toggles

4. No branching

5. No manual changes on servers

6. New project, deploy Hello World

7. Version Control Infrastructure

8. Canary Releasing

9. Done == In Production Success!

22

Improvement points

• Container adoption

• Smoke testing

• Performance testing/monitoring

• Automating routers, firewalls and other peripheral network equipment

• Smart monitoring and AI agents

• …and continuous improvement in general

23

24

Thanks!

jat@miles.no condep@condep.iohttp://blog.torresdal.net http://condep.io

@torresdal @con_dep

http://github.com/torresdal http://github.com/condep

Questions?