how automated cloud infrastructure setups can help with continuous delivery

Donnerstag, 19. September 13

“How automated cloud infrastructure setups can help with Continuous Delivery”Edmund Haselwanter, [email protected], @ehaselwanter


mailto:[email protected]

mailto:[email protected]

cloudbau

Why Continuous Delivery?

„IT REVOLUTION MANIFESTO“, http://itrevolution.com

For most companies, IT functions as the nervous system and provides an increasing amount of the organizational muscle mass. Most critical business functions are entirely automated within IT, and 95% of all capital projects depend on IT to get done. Today, nearly every business decision will result in at least one IT change.

Gene Kim, Kevin Behr, George Spafford, 2013

E.M. Goldratt, 1984

Theory of Constraints → →


http://itrevolution.com/

http://itrevolution.com/

cloudbau

Wikipedia: http://en.wikipedia.org/wiki/Continuous_delivery

● Version Control

● Continuous Integration

● Automated Testing

● Continuous Deployment

Continuous Delivery


http://en.wikipedia.org/wiki/Continuous_integration




http://en.wikipedia.org/wiki/Automated_testing

http://en.wikipedia.org/wiki/Automated_testing

cloudbau

We got the so!ware development part right!

The core, chronic con!ict that every IT leader faces is the need to simultaneously enable faster time to market (i.e., make as many changes as you can), while providing stable, secure and reliable IT services (i.e., make as few changes as you can).

„Lowering risk of change through tools and culture“John Allspaw, Paul Hammond, 2009 (Flickr)

small changes, o!en

reproducible

standardized

Request Approve

Test Deploy

Develop

Water-

Scrum

- FallGene Kim

expect failure

culture change Dev+Ops

feedback (e.g. tests/metrics/...)


cloudbau

The Last Mile: Dev/Test/Prod Parity

● Hardware con#guration (e.g. amount of RAM, HDD vs. SSD)

● So$ware used (e.g. sqlite in dev, mysql in prod)

● So$ware/library versions

● Deployment (e.g. automated for test, manual for prod)

● System con#guration (debug !ags, logging)

● External dependencies (dev: payment mocked out vs. prod: real gateway)

● Database contents (huge prod DB with millions of entries vs. few entries in dev)

● Operating system (e.g. dev: Mac OS X, prod: Linux)

See also http://12factor.net/


http://12factor.net/


cloudbau

The Last Mile: Dev/Test/Prod Parity

● Log level “normal” to chatty on production

● CI build broken due to minor version change in upstream mocking framework (0.4.1 to 0.4.2)

● Changes in external dependencies (Sun Java to Oracle Java)

● Sessions invalid in production due to time difference on multiple servers. No problem in dev environment (just one app and one db server)

● SQL Query no problem in dev environment but too slow for production data

● Different hardware: 32 vs. 64 bit. libraries missing

● Different hardware: optimized CPU !ags on CI system. build not working in prod

● Staging In-house, Prod AWS Cloud: Latency ok during staging test, problem on production

● Snow!ake back-port: Live Tuning of Kernel/Mysql parameters, got lost a$er re-install

● Development on Mac OSX/Windows, deploy on Linux: File not found (case sensitive)

See also http://12factor.net/




cloudbau

● ] Getting to the solution...


cloudbau

Application

Infrastructure as Code

Infrastructure

Environment


cloudbau

Building Blocks: Tools and Culture

CPU-Virt.

Storage-Virt.

SDN

Con!g-DB

Programmable Infrastructure (Cloud) Executeable System Description

Processes and Culture

Continuos Delivery

DevOpsCross-Functional Teams

Continuous Build

Anti-Fragile Organizations

Binary-Repo System DescriptionOrchestration

Prod Env Stage Env Dev Env


cloudbau

● Chef/Puppet/Cfengine/.... for con#g

● Dependency management (for programming languages, packages etc.)

● Dependency management for con#g scripts as well (e.g. Berkshelf for chef)

● Everything in version control

● Infrastructure can still be different...

First... automate and equalize all environments


cloudbau

● Encode the infrastructure as well

● Describe all the pieces of the infrastructure in code

● Now it’s code: it can be dynamic!

... then use a cloud to equalize infrastructure!


cloudbau

Infrastructure as Code => We need a process

But ... There be dragons

Source Code

Compiler

Artifact

Test X


cloudbau

● OpenStack / Eucalyptus / ...

● Chef/Puppet/Cfengine/ ...

● Jenkins (Building and testing the SW but also infrastructures)

● Net"ixOSS (Asgard + Aminate)

The implementation patterns using OSS


cloudbau

● The So!ware Package itself

● The whole VM

● A prepared container

● ...

New Question in the Cloud Era: What is my deploy Artifact


cloudbau

● The build artifact is a binary.

● Use IaaS API to create the environment

● Deployment is a!er the server creation

● Deploy with automation or any other tool

● The whole VM

● “from source” is just a special case

The So!ware Package is the Artifact


cloudbau

HP Cloud + Jenkins + Puppet

Jenkins

Binary Repo

Puppet Modules

HP Cloud Servers

(OpenStack)


cloudbau

● Build artifact is a image (optimal for stateless service)

● Only install needed, there is no upgrade/deploy

● IaaS must support Loadbalancer as a service (or something similar)

● Orchestration for deployment steps needed

The VM itself is the Artifact


cloudbau

Jenkins + AWS + Asgard + Aminate + Chef

AMIsJenkins Asgard

AWS EC2 (ASG + ELB)

Aminate

+ Chef


cloudbau

● Optimization of the VM idea

● Transportable lightweight OS containers (e.g. LXC/docker)

● Very o$en back to upgrade/deploy

● Needs external orchestration as well

● Very new toolchain emerging (docker/maestro)

A container is the Artifact


cloudbau

https://www.docker.io/the_whole_story/




cloudbau

Jenkins + docker + cfengine + Aminate + Chef

Host VM Jenkins Maestro

AWS EC2 (ASG + ELB)

Docker

+ cfengine


cloudbau

● Have your own environment

● Brings the production setup to the developer

● One description for local VM and Cloud environment (Dev/Prod)

● Needs external orchestration as well

Cloud can help with developer environments as well


cloudbau

OpenStack + Vagrant + Chef

JenkinsBinary Repo

Chef Repo / Server

OpenStack Servers

Vagrant


cloudbau

● Persistent Data: Copy/Clone Production Data to test migrations

● Debugging: Developer can stand up whole environment (even with data)

● Backup/Restore (snapshot whole images/servers for rollback)

One more Thing: Other Areas Cloud can help


cloudbau

● Better Orchestration/ Pipeline support in Jenkins (Commercial CI-Servers are advanced here)

● OpenSource Orchestration Engines (like OpenStack Heat)

● Net"ixOSS working on OpenStack

What’s missing in the OpenSource Ecosystem?


cloudbau

● Cloud/App-Monitoring Solutions (Nagios won’t cut it in dynamic environments)

● Log#les

● Metric driven autoscaling

Don’t forget about Operations


cloudbau

Thank you! Questions?


how automated cloud infrastructure setups can help with continuous delivery

Technology

netcloudbau donnerstag

culturecloudbau donnerstag

aminatecloudbau donnerstag

dev environment

culture change dev

artifact build artifact

prod aws cloud

prod staging