japan aws user group osaka study session #0 lt: devops

DevOpsIntroduction to DevOps movement,

and Some views of the Cloud Computing

Japan AWS User Group Osaka study session #0Lightning Talk @nxhack

2010/11/17

http://www.flickr.com/photos/pmtorrone/23362393/

@nxhack

Background:　25年以上 情報処理業界　17年間 イソターネット業界 Bias:　運用な人　*NIXな人　文系エソジニア

伝えたい事: DevOps 運動と何ぞや そこから見えるクラウドの姿

前説

IT Operations

http://www.flickr.com/photos/cote/54408562/

50年の歴史

The world's first-ever web server

1990年

Web

http://www.flickr.com/photos/cote/54408562/

TraditionalSystems OperationsX

http://www.flickr.com/photos/arthur-caranta/2926332140

Web Operation

CAUTION!WebOps BIAS運用さん視点

DevOpsMovement

http://velocityconf.com/

http://opscamp.org/

http://www.devopsdays.org/

Web AppsWeb Services

StartupScalableGrow fast

Continuous DevelopmentContinuous Deployment

ResearchDevelopment

DeploymentRelease

CustomerFeedback

ContinuallyPivot

(Iterate)

Hypotheses Testing

PDCA cycle

PDCA cycleX

http://www.flickr.com/photos/sdasmarchives/4727993474/

Colonel John (Richard) Boyd

(January 23, 1927 – March 9, 1997) was a United States Air Force fighter pilot and Pentagon consultant of the late 20th century, whose theories have been highly influential in the military, sports, and business.

OODA Loop

Web AppsWeb Services

Shrink WrapXhttp://www.flickr.com/photos/cogdog/116957071/

http://www.flickr.com/photos/arthur-caranta/2926332140

The infrastructure is the application.

The application is the infrastructure.

Devの役割Opsの役割

http://www.flickr.com/photos/aaronjacobs/64368770/

Developer

FeaturesEvolution

http://www.flickr.com/photos/troyholden/4036820768/

Operator

Services Stability

安定性安全性

http://www.flickr.com/photos/greg-palmer/3490903215/

Ops focuson failure

hardware failure

http://www.flickr.com/photos/purplemattfish/3760706750/

hardware failure

http://www.flickr.com/photos/purplemattfish/3760706750/

Avoid SPOF

hardware failure

http://www.flickr.com/photos/purplemattfish/3760706750/

Avoid SPF冗長設備の投資対効果を定量的に示しなさい

bursting

http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes

bursting

http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes

Capacity Planning

http://www.amazon.co.jp/dp/0596518579

bursting

http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes

Capacity Planning

http://www.amazon.co.jp/dp/0596518579

性能設計の根拠と投資対効果を定量的に示しなさい

software failure

http://www.flickr.com/groups/webopsviz/pool/with/4819476138/

software failure

http://www.flickr.com/groups/webopsviz/pool/with/4819476138/

Testing

software failure

http://www.flickr.com/groups/webopsviz/pool/with/4819476138/

Testing????

software failure

http://www.flickr.com/groups/webopsviz/pool/with/4819476138/

Testing????Testing?

operation failure

http://www.flickr.com/photos/proimos/4199675334/

operation failure

http://www.flickr.com/photos/proimos/4199675334/

指示書・手順書

operation failure

指示書・手順書

http://www.amazon.co.jp/dp/4122018331http://www.amazon.co.jp/dp/0691004129

うがぁ失敗!

operation failure

http://www.amazon.co.jp/dp/4122018331http://www.amazon.co.jp/dp/0691004129

orz

Web AppsWeb Services

Continuous DevelopmentContinuous Deployment

http://www.flickr.com/photos/greg-palmer/3490903215/

CHANGE CAUSES OUTAGES

http://www.flickr.com/photos/adjourned/308053363/

http://www.flickr.com/photos/sis/124916325/

http://www.flickr.com/photos/carve/2856365431/

http://www.flickr.com/photos/rocketjim54/2955889085/

“It’s not my code, it’s your machines!”

“It’s not my machines, it’s your code!”

http://www.flickr.com/photos/rocketjim54/2955889085/

本番環境がおかしいじゃないの?俺のテスト環境じゃぁ問題なかったぜ!

サーバ環境はあなたの仕様書通りだぜ。問題はプログラムの方だろう!

Dev Ops

Dev Ops

Dev Ops

Wall o

f Con

fusion

Dev Ops

Wall o

f Con

fusion

Dev vs. Ops

http://www.flickr.com/photos/ammo_hauler/3504563587/

Dev vs. Ops

普通のDev

良いDev

悪いDev

普通のOps良いOps悪いOps

儲かりまへん ややウケ 障害多発Opsは地獄

ぼちぼちでんな バカウケ ドッチラケ

あきまへんリリース遅延Devは激怒 倒産

Dev・Ops 欽ドン! 的マトリックス

普通のDev

良いDev

悪いDev

普通のOps良いOps悪いOps

生ける屍 ◯ 倒産生ける屍 ◎ 倒産倒産 売却 倒産

Dev・Ops 欽ドン! 的マトリックス (辛口版)

ちょっと待てよ

Dev & Ops

Goal:To Enable

Our Business

We make Our Servicestogether

To EnableContinuous DevelopmentContinuous Deployment

じゃぁどうする

Developer

Manifesto for Agile Software Development

We are uncovering better ways of developingsoftware by doing it and helping others do it.Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentation

Customer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items onthe right, we value the items on the left more.

http://agilemanifesto.org/

agile and iterative

ディーン・レフィングウェル (著), 玉川 憲 (監修, 翻訳)

http://www.amazon.co.jp/dp/4798120405/

DevelopersProduct Owners

Testers

DevelopersProduct Owners

Testersたいへんよくできました

OperatorsSystem AdminsDB Admins

Network EngineersDesigners

OperatorsSystem AdminsDB Admins

Network EngineersDesigners

がんばりましょう

どうすんのよ

Operator

Infrastructure:BootstrappingConfigurationMonitoring

agile operations

agile operations?

Infrastructure is HARD

http://www.flickr.com/photos/johnseb/3425464/

Infrastructure is HARD

http://www.flickr.com/photos/johnseb/3425464/

!

agile Infrastructure

IaaSInfrastructure as a Services

Server: EC2Storage: S3, EBSNetwork: ELB, CloudFrontMonitoring: CloudWatchDataBase: RDS, SimpleDB

Pooled IT Resources

On DemandSelf ServicePay as you goFair pricing

API

% ec2-run-instances ami-b232d0db -n 3 --availability-zone us-east-1a! ! ! !RESERVATION r-385c5950 012301230123 defaultINSTANCE i-5bca5a30 ami-b232d0db pending 0 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtualINSTANCE i-59ca5a32 ami-b232d0db pending 1 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtualINSTANCE i-5fca5a34 ami-b232d0db pending 2 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtual

サーバー 3台用意してね☆

Infrastructure as Code

API! API! API! API! API!Infrastructure as Code! Infrastructure as Code! Infrastructure as Code!

http://www.flickr.com/photos/programmerman/2315566040/

Give me an API, or give me DEATH!

http://www.flickr.com/photos/bitchcakes/3895840169/

- 2009/04/05 -「70/30から30/70へ変革」Amazon.comのEコマース・サイトにおいて、エンジニア・リソースの70％がネットワーク、ストレージ・クラスタなどのインフラ障害対応に費やされていて、本業であるEコマースサイトをどう効率よくするかというシステム開発に時間が割けない状態が続いていました。そこで、ハードウェアは壊れる前提で物事を考え、仮想化や分散処理、SOA技術を屈指した共有サービス・モデルに移行したとのこと。最終的には0％にはならないものの許容範囲であるリソースの30％程度に削減することができ、エンジニアはインフラ管理から解放されることとなりました。これらの経験から、拡張性、コスト効率化、信頼性、セキュリティに強い今の姿のAmazon Web Serviceが完成し、真のInfrastructure as a Service(IaaS)になったとのことでした。そして、自社のインフラ資産をCAPEX（資本的支出額）からOPEX（事業運営費用）に転換することができたとのこと。つまり、自社の持つIT資産を武器に利益を生み出すサービスとして事業転換に成功したということですね。

http://blogs.itmedia.co.jp/eno/2009/04/cloud-computing.html榎本 瑞樹さんのBLOG「ベンチャースピリット」 X 「セレンディピティ」より

agile operations を実現するために EC2

は生まれた

かなり本気な余談:IaaSを使う最大の利点は海外を含めた複数の拠点にデプロイできる事。アナタたち日本だけで商売するつもりですか?

Bootstrapping:Infrastructure as a ServicesInfrastructure as Code

agile Infrastructure

を使いこなす

http://www.flickr.com/photos/robenjoyce/4208638969/

Tools!

DevOpsTool Chainshttp://code.google.com/p/devops-toolchain/

Release

Generalized architecture

Control

Orchestration

Dispatcher

Provisioning

Deploy

Config management

OS boot/install

Artifact repository

Build

CI Server Issue tracker

SCM Repository

Model

Asset inventory

Host naming

Identity

CMDBMonitoring

Events

Trending Reporting

Trending

Workflows

Resources

Topology

Configuration

Code

Sources

Scheduler

Wednesday, June 23, 2010

http://code.google.com/p/devops-toolchain/

Release management

Artifact

Build

SCM

Trackerbugzillamantistrac

svncvsgithg

Repository

yum/yastartifactoryarchivaDFS/openEFSHTTTP

eggsgemsperlmod

tgz/ziprpm/deb/pkg/msijar/war/ear

makemavenant/ivy

sbtrakephing

Wednesday, June 23, 2010

http://code.google.com/p/devops-toolchain/

Dev

Monitoring and control

Measurement Instrumentation

Charting, History, SPC

Alarm Mgmt

Op Console, Control

Runbook Automation

nagioszenossopennms

rrdtoolcacti

ControlTierJobschedulerOpenScheduler

Wednesday, June 23, 2010

http://code.google.com/p/devops-toolchain/

Ops

Provisioning

Bcfg2cfengineChefPuppetSmart Frog

Application Service Orchestration

System Configuration

Cloud or VMImage

LaunchOS

Install

KickstartJumpstartCobblerOpenQRMxCAT

XenlxcopenVZEucalyptusKVM

CapistranoControlTierFabricFuncmCollective

Cloud: OS Install:

Command and Control

Configuration

Bootstrapping

Prov

isio

ning

Act

ivity

Wednesday, June 23, 2010

http://code.google.com/p/devops-toolchain/

Dev

Ops

自動化できうるところは全て自動化する

agile Infrastructure を活用しインフラ構築・構成を含めて

自動化する

Configuration:Configuration Management

agile Infrastructure を活用するよう System Architecture を見直す

System Architecture変えないといけない

従来の考え方から脱却新しい考え方が必要

障害対応

http://www.flickr.com/photos/greg-palmer/3490903215/

MTBFMean Time Between FailureX

Failure as a Service

Failure is a failurebut

embrace failure

Your sitewill fail!

http://www.flickr.com/photos/gaelx/2912634902/

MTTDMean Time To Detect

MTTRMean Time To Resolve

余談:今年の Velocity 2010 で、印象に残った事は、Failures/Outages が起こった時の対応と Postmortem (検死報告) の在り方についての議論でした。メキシコ湾原油流出事故の教訓にちなんだんでしょう。

http://www.slideshare.net/lennysan/the-upside-of-downtime-velocity-2010-4564992

Monitoring:Measurement

VisualizeShare

http://blog.b3k.us/ooda.html

WebOps OODA LoopDiagram

ActDecideOrientObserve

ExecutionPlanningResourcing

AnalysisVisualizationCorrelation

MetricsMonitoringAlertingAlarming

Benjamin Black, b@b3k.us

Visible DevVisible Ops

http://www.amazon.co.jp/dp/443407296X/

THE VISIBLE OPS HANDBOOK - 見える運用

Kevin Behr (著), Gene Kim (著), George Spafford (著)

官野 厚 訳出版社: ブイツーソリューション (2006/1/6)

彼らはもう...

http://www.etsy.com/

http://twitter.com/allspaw/status/22742505774

◯

http://twitter.com/allspaw/status/29580411400

◯

Web OperationsKeeping the Data On Time

By

John Allspaw, Jesse Robbins

Publisher:O'Reilly MediaReleased:June 2010

http://www.amazon.co.jp/dp/1449377440/

Opsの役割

http://www.flickr.com/photos/troyholden/4036820768/

Operator

Services Stability

次のステップへ

http://www.flickr.com/photos/alexthomp/3545005410/

Operator

System AvailabilityEfficiency

Dev ♥ Ops

To EnableOur Businesstogether

Culture

http://www.flickr.com/photos/24443965@N08/3285794855/

PeopleProcessTools

http://www.flickr.com/photos/pmtorrone/23362393/

DevOpsCultural and

Professional Movement

Question

http://www.flickr.com/photos/statelibraryqueensland/4903167275/

Appendix:John Allspaw (twitter) (blog)・SlideShare・10+ Deploys Per Day: Dev and Ops Cooperation at Flickr　　(どのプレゼンも素晴らしいですが、このスライドは必ず見てください)

Andrew Clay Shafer (twitter) (blog)・SlideShare・DevOps ‒ You’re Doing IT Wrong « fate = will && choice || circumstance

Adam Jacob (twitter)・SlideShare・Velocity 2010 ‒ Adam Jacob on DevOps・Choose Your Own Adventure OSCON 2010 Edition (PDF)

Appendix:John M Wills (twitter)・SlideShare・What Devops Means to Me

Damon Edwards (twitter) (blog)・DevOps is not a technology problem. DevOps is a business problem.

PodCast:・DevOps Cafe・IT Management Podcast

Conference / UnConference:・DevOps DAYS・OpsCamp・O’REILLY Velocity

Appendix:日本語での情報:・すべての道は DevOps へと : DevOps ‒ mizzy.org ‒ Trac・DevOps: OpenSource Provisioning Toolchain : DevOps ‒ mizzy.org ‒ Trac・DevOpsって何？

・クラウドと「DevOps」を理解する ‒ builder by ZDNet Japan

My Blog:・すでにここにある雲 DevOps

このスライド:・http://www.slideshare.net/nxhack