operations as a strategic weapon (part 2)

Operations is a Strategic Weapon

(Part Deux)

Saturday, September 24, 2011

Are you an ...

aaSSaturday, September 24, 2011

-If it's not it should be

-Let's take a look at GW.com.. yep

-they sell GW as a service GreenWidgits_aaS

-Obviously you better have kick “ass” green widgets; Pun intended....

-But today that is not enough....

----------------------------------------

-What does amazon and netflix get that barnes and noble and blockbuster don't ?

_aaS (circle the "S")

-Over the next 45 minutes I am going to try and convince you that the principles behind Devops are the best solution to

address the "S" in your aaS


-Most of you are probably familiar with “Software” TD-I am going to tell you a story about “Infrastructure TD”


-How many ppl know who these guys are?-The dudes who invented Facebook.. rght?-However, this was not there first venture. -Green vs Red Widgits compnay story. -After many arguments the both decide to split up. Cameron to make green widgets and Tyler decides to make red widgets. -They both go to their father each ask for a 1m dollars and the father asks how much money are you going to make. They both say 10 million. -Father calculates the ROR to be 900% and gives em each 1m. 10-1 = 9/1 = 900%-However one of them lied.... The red widgets only return 233%10-1-2 = 7/3 = 233%

Green vs Red Widgits

1 Million2 Million10 Million Profit10 - 3 = 7/3 233% ROR

1 MillionNo TD10 Million Profit10 - 1 = 9/1 900% ROR


-Father is not pleased with Rehaus.

Technical Debt

Vicious Cycle

ToxicOperations Terminal

It Gets Worse...


- It gets even worse. Rehause starts getting what Isreal Ghat of the Cutter Group call the vicious cycle of TD.

- You wind up fixing a lot of things that you didn’t fix in the first place..

- this pulls more resources from delivery good service and compound effect is that you are spending more and more

resources that you should have gotten right the firs time..

- but ever worse the effect of customer satisfaction starts loosing more business and the “V” cycle is out of control.

- TD->VTD->Toxic operations->Terminal

- I call this are you running a business or building a business?

- Toxic operations. Amertrade/etrade story

Tale of Two Startups


- Jesse Robbins my ex boss and CEO of Opscode/Chef did a great post on O’rielly rdar a few years ago called the Tale of

Two Startups.

- The chart looked like this first 4 weeks.

- First chart legacy (I call it the non devops startup/project)

- Second Chart is the (secret sauce startup ... #devops)

- I played around with this using “R” to be cool and I came up with a=140% ROR and 700% ROR

The Meat to Math Ratio

http://radar.oreilly.com/2011/08/meat-to-math-ratio.html


- Alistar Croll has a great post called the meat to math ratio.

- Amazon had $12.95B in Q410 revenues and 33,700 employees, revenue per employee of $384,273.

- For Barnes & Noble: Barnes & Noble: $1.91B in Q410 revenues, and 35,000 employees, meaning a revenue per employee

of $54,571.

- Netflix: $444M in Q409 revenues, and 1,000 employees, meaning a revenue-per-employee of $444,000.

- Blockbuster: $400M in Q409 revenues. The company peaked at 60,000 employees i

- Dropbox: In Q211 Dropbox had $25M in revenues, and 74 employees, for a revenue per employee of $338K.


•I got into a Twttier argument that went like this... • See “Cloud Gone Wrong”

apache/php

memcachedmysql

Rightscale

AWSs3

Businessbusiness

ideas !

s3putssh

Dev

Clouds Gone Wild


-So letʼs start the story in the begining... Biz guy got a great idea for a service.

-They used a classic web2.0 app architecture: apache/php, memcached & mysql. Development done on a single server-Production ran on EC2 and they used rightscale server templates -Release done by pushing code and assets to s3 buckets and then running a parallel SSH scripts to distribute them-This approach seemed to work... They got up to a few hundred nodes pretty fast.. business was cooking

apache/phpmemcached

mysql

Rightscale

AWS s3

Rightscale

AWSs3

Rightscale

AWS s3

yum

puppet

s3put

ssh

3rd party,

&

middleware

apache/phpmemcached

mysql

apache/phpmemcached

mysql

Business

Business3

s3put

ssh

Business2


-First service was such a huge success they decided to launch other sevices-So they ”Copy and pasted” the whole architecture and lifecycle to launch the new businesses-Each new group pushed assets to s3, scripted the distribution, and hacked the rightscripts and templates-Things were obviously getting more complicated, so they did what they were supposed to do and added centralized tooling like puppet and yum-They thought they were doing things the cloud way and that all would be fine

“As-is”

app devs

middlewaredevs

operations

systemeng

EC2

servertemplates

yum repo

RS deploymentRS

deployment

puppet

S3

code,content

code

TARTAR

TARTAR

platform

TARRPM

TARRPM

servertemplates

EXE

AUTO

TOOLS

Provision-time builds

CONTROL

restarts

deploys

reconfigs

PROVISION RELEASE


First we got the team on the whiteboard to map out the “as is” picture. This is a, believe it or not, a simplified version of that.Some of the highlights...-First youʼll notice that different groups had their own path to production... different methods of control, provisioning, and release. -Each group and role seemed to have a different way to editing or storing config-There were differing ways of packaging software... sometimes it might be a .tar.gz other times it might be an RPM.-Shockingly... Some things were even being built directly on production servers.-There was no authoritative source of information is maintained about nodes, application topology, software versions, etc.... -Everything was being stored in S3 buckets... which is great because itʼs so easy to use... but itʼs unversioned and people would just upload whatever was newest for them. But that stuff was never really tested in unison.. so old stuff wasnʼt working with new stuff... and its was unclear what was different or why it was different.-We can go on... but you should get the point that they had the right cloud, the right tools, and lots of smart people, but it all got dangerously out of control very quickly

Other:

-Changes hit all customers at once => Puppet configs in unversioned S3 buckets-Buggy node classification causing provisioning problems = > complex/long node classifier script-”Dead boxes” after provisioning => rightscript/puppet ordering problems-”my box got clobbered!” => puppet, is it supposed to be on or off?-new environment setup was taking longer and longer => from days to weeks because of “fooled by false horizons”-”is the system ready yet?” => nobody knows what “ready” means-Scripts contained a list of role-to-node lists to put things in the right places- scripts crap out on nodes taken out of commission-”software works differently” => rightscale driven compile/installs

Technical


pup

pet

SVN

active

directory

splunk

instance create

resource model

users packages

new node Node a

commands

right

scale

run

deck

hud

son

yum

repo

nag

ios

packages

sys cfg

eventslog data

agent cfg

CONTROL PROVISION RELEASE

Everything starts here

Infrastructure Deveopment Life Cycle


I wonʼt go into too much detail about the tooling that was put into place to support all of this but here are some highlights..-took a loosely coupled toolchain approach... using mostly open source tools -This became their standard stack of “operations middleware”. Of course, we are all used to the notion of application middleware... but to an online service, the management infrastructure is just as much a part of the service as the application itself.-This operations middleware stack is a first class citizen along with application stack and it all goes through the same SDLC... everything is versioned, built, deployed, and packaged via the same process -Once in place, this middleware provides a single path for releasing, provisioning, and controlling anything that goes into an environment.

Other:Management infrastructure based on “swap-able” sets of integrated tools- Organized into three rough categories: Control, Provisioning, Monitoring - Control tools support routine and ad hoc procedures executed as commands/scripts - Provisioning tools support package delivery and post install customization - Monitoring tools actively check health and collect log data

When you think of middleware, you think of where your app code works.. but in the service world, you have an operations middleware that is just as important. All the provisioning and management stuff... itʼs just as important. Itʼs one and the same.

Solve the information problem... where is the system of record? easy in cloud to get basic node data... that comes from the compute service... but what about everything else you need to manage your infrastructure?

Key integrations:- SVN drives everything in the tool chain!- Rundeck synchronizes to RS; must be connected to compute service to know what nodes are provisioned and ready- All packages come through yum

Infrastructure SDLC

Meet your competition


•Betting on better experience

•Attacking / chased by the majors

•Real money at stake


One last quick tale for you all....

So what does it look like when you start with the end in mind...

Let’s look a hot startup called Wealthfront-Its a financial services company... but all you have to know that it manages real people’s retirement funds. It’s not a game or a toy.-They are being chased by the majors so they have to move fast and can’t screw up-To make this work, they subscribe to the theories of these two gentlemen, steven blank and eric reiss, and are using the Lean Startup methodology to run their business... and all you really need to know about that right now is that it all hinges on getting new ideas in front of your customers as fast as possible and getting feedback as fast as possible.

Customer discovery -> Customer Validation -> Customer Creation -> Company Building


So what was immediately identified as a major impediment to that fast cycle?

The traditional development and deployment cycle. Even the 1 - 4 weeks cycles that are so common is too long to wait.


So they rewired it in classic Continuous Deployment fashion...Get rid of the idea of batching up change for a release... They are always deploying as soon as a a piece of code is readyDon’t have staging... that’s a waste of time... production is the real environment anywayDon’t have an explicit QA stage... built extensive automated testing in all across the lifecycle and made quality be everyone’s responsibilityThey are always deploying from trunk... so no need to worry about patching and merging branchesoh and do they whole thing from check-in to running in production in 5 - 10 minutesare they crazy? They don’t think so. They have extreme confidence in this system and are kicking ass with it.

go read: http://eng.wealthfront.comSaturday, September 24, 2011

The results are striking...

They are sometimes averaging 30 deployments to production a day. But they aren’t sacrificing quality!

A release can get from check-in in to production in under 10 minutes but it’s got to pass over 6k tests and then there is a phased rollout with ongoing and extensive system and traffic analysis post deployment.

The result is that their customers are happy. Their partners are happy. And more importantly their team is happy...no burnout and releases are non-events. Want to go on vacation? No problem, any developer can step in and take over your task or fix problem because of extensive test coverage and confidence in their quality systems.

Operations is their strategic weapon

go read: http://codeascraft.etsy.comSaturday, September 24, 2011

The results are striking...

They are sometimes averaging 30 deployments to production a day. But they aren’t sacrificing quality!

A release can get from check-in in to production in under 10 minutes but it’s got to pass over 6k tests and then there is a phased rollout with ongoing and extensive system and traffic analysis post deployment.

The result is that their customers are happy. Their partners are happy. And more importantly their team is happy...no burnout and releases are non-events. Want to go on vacation? No problem, any developer can step in and take over your task or fix problem because of extensive test coverage and confidence in their quality systems.

Operations is their strategic weapon

95% 5%

Necessary Liability Strategic Weapon


subscribe in

iTunes

DevOps Cafe


I also do a podcast with the famous cloud and IT management guru john willis.Interview based series where we talk to all kinds of movers and shakers across the development and operation spectrum.

Between DTO and doing the devops cafe content I get to talk to a lot of companies and see what’s working and what isn’t working.

@botchagalupe

[email protected]

Let’s Talk....


We do this stuff all day long for a lot of large and cutting edge clients... and we love talking about DevOps so drop me a line anytime if you want to talk

@botchagalupe dev2ops.org

[email protected]

Let’s Talk....


We do this stuff all day long for a lot of large and cutting edge clients... and we love talking about DevOps so drop me a line anytime if you want to talk

Extra Slides


CAMS


First get you mind around what you are looking for... Simple framework for categorizing DevOps problems and solutions.

CultureAutomationMeasurementSharing


operations as a strategic weapon (part 2)

Technology