OpsToolswith Perl

Hokkaido.pm #72012/05/12

riywo

Self Introduction

Ryosuke IWANAGA

a.k.a. riywo(りーお)

DeNA(2009～)

Mobage Server-side

Ops-Engi / DBA / Manager

Acknowledgement

Thanks for JPA and Hokkaido.pm :)

地域PM向け講師派遣支援プログラム

http://japan.perlassociation.org/services/send-a-monger

But...Am I a Perl Monger?

I’m not a Programer

Mobage uses Perl heavily but I have never written the business logic...

But, I read Perl heavily

Because I should cope with all system trouble including application bugs ;P

I wrote some operation tools with Perl

And I also use some Perl tools

But...Am I a Perl Monger?

So...

I’ll try to talk about my knowledge, thought, and tools as much as possible

しょぼくてごめんなさい＞＜

Perl Ops-ToolsIntroduction

kazeburo-ware

http://blog.nomadscafe.jp/@kazeburo

CloudForecast

http://www.slideshare.net/kazeburo/introduction-to-cloudforecast-yapcasia-2010-tokyo

CloudForecastServer Resource Collecting/Monitoring

Easy install

git clone & cpanm --installdeps .

SNMP / RRDs is needed

LightWeight - about 5,000 lines (*.pm)

WAF: Shirahata.pm

See alsoリソースモニタリングツール「CloudForecast」入門 - As a Futurist...( http://blog.riywo.com/2011/02/27/043646 )

GrowthForecast

http://blog.nomadscafe.jp/2011/12/growthforecast.htmlhttp://blog.nomadscafe.jp/2012/02/growthforecast1json.htmlhttp://d.hatena.ne.jp/tagomoris/20120218/1329558305

GrowthForecast

API and Graph Render of RRDTool

Easy install

git clone & cpanm --installdeps .

RRDs is needed

LightWeight - about 1,800 lines (*.pm)

WAF: Kossy.pm

See also

GrowthForecastというグラフ表示ツールで捗る話 - blog.nomadscafe.jp( http://blog.nomadscafe.jp/2011/12/growthforecast.html )

**Alien::RRDtool**RRDs is not CPAN module...

Alien::RRDtool Install rrdtool as CPAN module

extlib/lib/perl5/{archname}/RRDs.pm

extlib/lib/perl5/auto/share/dist/Alien-RRDtool/

Especially useful for perlbrew

* Developed by gfx

See alsoAlien::RRDtool - RRDtoolをCPANから入れる - Perl Advent Calendar Japan 2011 Hacker Track( http://perl-users.jp/articles/advent-calendar/2011/hacker/2 )

chase-tail

chase-tail

$ tail -f error_log | chase-tail -l 10 -t various_error

One sheet perl script

Highlight some keywords

Detect fast flooding of log

See also

垂れ流されるログのおしりを追いかける - (ひ)メモ( http://d.hatena.ne.jp/hirose31/20120120/1327025989 )

@hirose31

App::Ikachan

App::Ikachan$ curl -F channel=\#yappo -F message=oppai http://hoge.com/notice

HTTP API of IRC posting

use it from any languages

Easy install (git clone & cpanm)

See also

YappoLogs: App::Ikachan - 様々なサーバのバッチ処理の結果等を IRC のチャンネルに通知するサーバ( http://blog.yappo.jp/yappo/

archives/000760.html )

@Yappo

Percona Toolkit

Percona Toolkit

Mixed tools - Maartkit and Aspersa

Almost all tools are written in Perl

System utility / MySQL Ops

“pt-query-digest” is quite efficient for performance tuning of MySQL server

See also

Percona Toolkit - Percona Software( http://www.percona.com/software/percona-toolkit/ )

@Percona

Touryo

“Touryo” is configuration management tool

I needed a Configuration Management Tool

Chef is good but difficult to customize

So, I started to think a plan of Touryo

Simple and Lightweight

Testable

able to write DSL in Perl freely

How to write DSL in Perl?

I didn’t know it X(

I asked gfx(@__gfx__) who is a great Perl Monger and works at DeNA

use strict;use warnings;

Hoge->do_test;

#---------------------package Hoge;

my $func;sub Hoge::test(&) { my $block = shift; $func = $block;}

sub do_test { package Hoge; do "dsl.pl"; $func->("DSL");}

[dsl.pl]

test { my $hoge = shift; print "Hello $hoge!\n";};

What can Touryo do?

crontabsysctlrpms

apachefastcgi

app

web01$ touryo test web01

$ touryo run web01FAIL?

- “touryo test host” - check the state of the host - correctly installed? - correctly configured?

- “touryo run host” - setup to the “correct” state

-> Then, what is the “correct” state?

crontabsysctlrpmsmysql

backup

crontabsysctlrpmsmysql

db01 db02

admin:touryo

game1:db:master

game1:web

game1:db:backup

crontabsysctlrpmsmysql

How to define the “correct” state? (1)

crontabsysctlrpms

apachefastcgi

app

backup

crontabsysctlrpmsmysql

web01

db01 db02

role: game1:web blueprint: - apache - fastcgi - app

game1:db:master

role: game1:db:backup blueprint: - backup

game1:web

game1:db:backup

role: game1:db blueprint: - mysql

role: game1 blueprint: - crontab - sysctl - rpms

admin:touryo

Touryo config

- Touryo use “blueprint” config - means what should be installed- The config defined by “role” - should be defined the role of hosts - inherited searching - separated by comma

crontabsysctlrpmsmysql

What is the “blueprint”?

crontabsysctlrpms

apachefastcgi

app

backup

crontabsysctlrpmsmysql

web01

db01 db02

game1:db:master

game1:web

game1:db:backup

blueprint/crontab/blueprint.plbootstrap { my $c = shift; $c->add_attribute({ cron_hour => 5, cron => 'root.tx', });};

test { my $c = shift; is_diff $c->sudo_ssh->crontab, $c->template('cron')->content; done_testing;};

run { my $c = shift; $c->sudo_ssh->crontab($c->template('cron'));};

blueprint/crontab/root.tx0 <: $cron_hour :> * * * /path/to/command

- blueprint locate the directory - named the blueprint name- contain “blueprint.pl” - Perl DSL for Touryo - able to use template engine - Text::Xslate::Syntax::Kolon - using “attribute” valuables - have other useful functions - sudo_ssh, template, crontab - etc...

admin:touryo

crontab blueprint

crontabsysctlrpmsmysql

What is the “attribute”?

crontabsysctlrpms

apachefastcgi

app

backup

crontabsysctlrpmsmysql

web01

db01 db02

game1:db:master

game1:web

game1:db:backup

blueprint/crontab/blueprint.plbootstrap { my $c = shift; $c->add_attribute({ cron_hour => 5, cron => 'root.tx', });};

blueprint/crontab/root.tx0 <: $cron_hour :> * * * /path/to/command

- “attribute” can be used to change the behavior of test/run - changing logic, template, etc..- overwriting of “attribute” - bootstrap - role config - defined for the role - node config - defined for the node(host)

admin:touryo

0 9 * * * /path/to/command

Touryo Configrole: game1:web attribute: cron_hour: 9

0 5 * * * /path/to/commanddefault rendering

If defined attribute onthe Touryo Config ...

How to define the “correct” state? (2)

crontabsysctlrpms

apachefastcgi

app

web01

role: game1:web blueprint: attribute: - apache cron_hour: 9 - fastcgi somaxconn: 10240 - app fcgi_proc: 100

game1:web

role: game1 blueprint: attribute: - crontab somaxconn: 1024 - sysctl - rpms

Touryo config

node: web02 blueprint: attribute: - git fcgi_proc: 10admin:

touryo

crontabsysctlrpms

apachefastcgi

app

web02

game1:web git

cron_hour: 9somaxconn: 10240fcgi_proc: 100

cron_hour: 9somaxconn: 10240fcgi_proc: 10

- You can vary correct “blueprint” and “attribute” by Touryo config - role config and node config

Lightweight/Testable/Freedomfind lib/ -name "*.pm" | xargs cat | wc -l

about 1500 (including html templates)

Like test scripts of Perl Software, you can test the servers’ configuration

you can write anything on blueprint.pl

Conversely you have to write anything

Other Featuresin terms of Programming

use Amon2

easy to develop Web Interface and CLI

use TAP::Parser to get the test result

test function uses Test::More

use IPC::Cmd and “ssh” / “sudo ssh”

easy to configure with .ssh/config or others

blueprint dependance resolving

easy to cascade the config

What language should I use?

There are many languages

script / compile

Perl,Ruby,Python,PHP / C,C++,Java

procedural / OO / functional

Perl,C / Java,Ruby / Scala,Haskell

popular / minority

???(religious problem...)

My ThoughtYou can use any languages you like

Engineer have to treat multi languages

Development speed / Maintenance cost

They are usually trade-off

Popular LLs are often suited for both

Simple / Light / Thin / Useful Modules

Everyone can understand it easily

Perl/Ruby/Python are similar

I ♡ Perl

Fastest to develop for me

I am used to Perl because DeNA use Perl

There are many Perl Mongers around :)

colleague and friends

CPAN has many useful modules

long history and super hackers

Finally...

DeNA ♡ EngineersWe need more powerful engineers

We use Perl and other languages

Ruby/JavaScript/C/C++/Java/Objective-C

SocialGame/Platform/SDK/Data mining

Or let’s make very useful tools with me :)

Server Ops/Database Admin/Network

If you are interested in us, pleeeeeeeeease contact me or DeNA staffs!

hirose31, gfx, hidek, zigorou, nekokak...

Thanks!http://blog.riywo.com

http://twitter.com/riywo

Thanks for JPA, @kazeburo, @hirose31, @Yappo, @Percona,and All Perl Mongers!