coscup
DESCRIPTION
License by http://creativecommons.org/licenses/by-nc-nd/2.5/tw/TRANSCRIPT
Better node.js software
release practice using open
source tool
????????
Software release to production is such a hard practice, especially considering the business
impact and the quality of software . I will demonstrate how to use open source tools to
enhance a node.js web software release process and ensure the better release quality.
The open source tools I will mention will includes Jenkins, Git, Ubuntu dpkg system, node.js
language , ruby , rake, jshint , nodeunit, expresso , YETI, Selenium, and mongodb
?, ??????? 100+ ???? = =
Web Software release best
practice
Agenda
Who am I ?
Release engineering in Y! Shopping
Process
Testing
Deploy Tools
Configuration Management
Who am I ?
thegiive
• A open source lover
• Release manager of Taiwan Yahoo! Shopping site
• Continuous study on software development, release engineering and software quality
• Fall in love with Ruby, VIM, Puppet and Rake
• Trying to do more work rather than write more blog
Release engineering in Y! Shopping
Common image about release
engineer
Yahoo! Shopping release manager
integration hub for more complex software development teams, sitting at the cross between
development, product management, quality assurance and other engineering efforts, also known
as DevOps
DevOps
Release???1 : ???
??????????????
• RD Team ???? , PHP Team , .Net Team , Search Team
• User Team ???? , ?? Team , ?? Team , Search team
Release???1 : ???
?????
Release???2 : bug????
From Book "Beautiful Testing" Chaptor 2
TestROI : ????????
• ACPB : Average Cost of a Production Bug
• ACTB : Average Cost of a Test Bug
• TBN : Total Bug Number
• TDBC : Total Defect Bug Cost
Taiwan Shopping ? ACPB?????? Test ROI ???
Test ROI ??????????? testing ????
Release???2 : bug????
Release???3 : User????
• ??RD?bug?QA????????SE??????RM?????
• User ???? monitor script , ????????????
???? Release Manager , not release engineer
Release???4 : ??????
PHP Team
• Dev process : Scrum
• Source Code Management : Subversion
• Language : PHP and partial C++
• OS : Linux
.Net Team
• Dev process : Waterfall
• Source Code Management : VSS
• Language : ASP.Net and ASP
• OS : Windows
????????????? ... ?????? 100??
release ?release 2000 ??
item
??????
Release quality ??
Y axies stands for ehnancement number / release number
Throughput ??
Y axies stands for
release item / release number
User ?? , ???????
??Release Quality ??????
• 2011 Q4 24? release ??, ? 12??(02:00 ~ 06:00) release
• ???? user ?? manager ?????
??????????????? , ??????? release ???????, ????? 6???
?? release qulaity ???user ???????????
• 2012 Q2 11? release ??, ? 0??(02:00 ~ 06:00) release
• 2012 Q2? 1??07:30 release
????????
Process
Communication : No Surprise !
• ???????????, ?????????????????????? briefing and retrospective , ??????? 18%
• Communication ???? 9???
???????
?????????
• ????
• ???
• double confirm ??????????
??
• ????
• ?? 30min ??, 30???????????? 20000 NTD
??????
• ?????? sharing information ?????
????????
?????????
• Information syncing : 40%
• Discussion : 30%
• decision making : 30%
?????
• Information sync : 5% => ?????????
• Discussion : 15%
• decision making : 10%
Convention over configuration
Release??
• Release ????? ???????????
• ??????? release ??
• ????????? release
• Release ???????? release briefing
• Release ??? Code Freeze
• ???? cost , ?????? retrospective
??????????? sync ??
• ???? Release Plan
• ????????? Plan ??
◦ Item ????,???? , ????
◦ Item ?????? , ???? , ????
◦ ??????? item ????????????
release briefing ?????
• Information sync : 5% => information double confirm
• Discussion : 15% => ???? sync , ??????????
• decision making : 10% => ??sync??????????????
???????
• ???????? ??? release????? ????email
• ????????????????? , ????????????
• ????? , ???? surprise , ???????
Testing
Release Manager ???
• Unit Test : nodeunit , YETI
• Automation : Selenium
• CI ( Continuous Integration ) : Jenkins
????????????????
Unit Test ?????
?? code ???? , 100% RD ???
Automation
• ??????? SLA (Service-level agreement)
• ???? ?? automation coverage rate ??? SLA ? 100%
• ?? release ?????? automation
• Presentation layer ? coverage rate ???? 100% , ???????
Continuous Integration
• ??????? Unit Test and Automation , ???????
• CI ????????????? Unit Test , Automation
• ?????? CI ? auto build ??
Deploy Tool
???SE/Ops ??? Deploy Tool
??: ???? Yinst/Igor
Use os level package system
• Debian/Ubuntu : dpkg
• RedHat/Centos/Fedora : rpm
• FreeBSD : port
• Windows : MSI
Don't use language level package system , ????????? relay on ???????? server
• Webservr : apache, nginx
• Cache : memcached , redis
• SQL : MySQL , PosgreSQL , MongoDB
????? Deploy Script
?????????? script ? deploy code
1 apt-get install npm
2 apt-get install pip
3 apt-get install rubygem
4 apt-get install nginx #web server
5 apt-get install memcached #cache server
6 npm install -g express # Web framework
7 npm install -d
8 pip install pykml
9 gem install amqp # node.js has no rabbitmq client , so use ruby XD
10 apt-get install rabbitmq-server # rabbitmq server
11 wget http://mydev.server/mycode_package.tgz # get my code from dev server
12 tar zxcf mycode_package.tgz # unpack the code package
• ???? hybrid ?? package system , package dependency ?????
• SE/Ops team ?? language package system ??? , ???? RD ????
?? Deploy Script
????? package system , ?????? package version
1 apt-get install nginx=0.1.19-1
2 apt-get install memcached=1.4.13-0ubuntu2
3 apt-get install node-express=2.5.9-1
4 apt-get install pykml=0.0.3
5 apt-get install ruby-amqp=0.9.5-1.
6 apt-get install rabbitmq-server=2.8.4-1
7 apt-get install mycode_package=1.2.3 # ???? code ?? package
??
• SE/Ops team ??????? deploy ???
• CI server ????
???? pkg version
Production System ? ???? pkg system ? depentency
How to pack your package
• ?? code ????? (Code,Config,template)
• ??? code ??? prodcut ????? (??,??????,???,????)
• ????????? (??lib ,??lib ,?? config , ??config )
• ???????????? ( 100? , 10? , 1? , ???? , ???? )
???? Framework ???
Change Management
??? code ???????
• mycode-1.2.3.zip : ?????? change ???? ???? readme
PKG list : ?? SE/Ops ?????
• Upgrade : functionA-config-1.2.3-2012.07.08
• Upgrade : Member-template-2.3.2-2012.07.11
• Donwgrade : functionC-code-3.2.1-2012.03.05
• Upgrade : ShoppingCart-lib-100W-1.2.3-2012.07.20
• Upgrade : WS-stock-lib-200W-3.2.1-2012.07.21
Package pack script ???
Script best practice
• ???
• ? package ???? source code version ???
• Version ??????
• Package ????? package server
• Readme ???????? svn log
Package Version 2.3.1-2012.08.12
[thegiive v123] add some slides change
[RD2 v121] change function a
Configuration Management
debconf
???? config package
• mypkg-config.2.2.3-2012.07.17
• ?????? config ? ????? readme
?????? config ??
1 echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections
2 echo "mypkg mypkg/ws_host string ws1.coscup.org"|debconf-set-selections
3 echo "mypkg mypkg/title string coscup"|debconf-set-selections
4 debian_frontend=noninteractive dpkg-reconfigure
package ??? postinst
1 db_get mypkg/use_memached
2 echo $RET #??? debconf ?????
3 do_config_change($RET)
??????SE/Ops?????
Cache Server ???SE?????? disable cache
1 echo "mypkg mypkg/use_memached boolean false"|debconf-set-selections
2 debian_frontend=noninteractive dpkg-reconfigure
WS1.coscup.org ??????SE???? WS2.coscup.org
1 echo "mypkg mypkg/ws_host string ws2.coscup.org"|debconf-set-selections
2 debian_frontend=noninteractive dpkg-reconfigure
???? title , SE ????? XD
1 echo "mypkg mypkg/title string coscup.org"|debconf-set-selections
2 debian_frontend=noninteractive dpkg-reconfigure
• ????RD ???SE?Daily work?????? RD , SE ??????
• ?? incident ??? , SE ????config ???? , RD??? SE?????
?? mobile login server ??
• ?? mobile loging server ?????????????????
• ?????????? RD ???????RD,SE,??????
• ???? config change , ????? mobile login ?? Web?? login ??
Debconf ????...???
????????deploy script
1 # pkg
2 export debian_frontend=noninteractive
3 apt-get install -y nginx=0.1.19-1
4 apt-get install -y memcached=1.4.13-0ubuntu2
5 apt-get install -y node-express=2.5.9-1
6 apt-get install -y pykml=0.0.3
7 apt-get install -y ruby-amqp=0.9.5-1.
8 apt-get install -y rabbitmq-server=2.8.4-1
9 apt-get install -y mycode_package=1.2.3
10 # config
11 echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections
12 echo "mypkg mypkg/ws_host string ws1.coscup.org"|debconf-set-selections
13 echo "mypkg mypkg/title string coscup"|debconf-set-selections
14 dpkg-reconfigure
• ??????? script ? deploy ?????????? , VM , Cloud instance
• script ???? non-interactive mode ? disable ??????? step
• ?? CI server ?? auto build , auto config
deploy script ?? version
control
?? tag ??? deploy , rollback
7??1? release ? tag ? v2012_07_01 , 7??2? release ? tag ? v2012_07_02
1 # ???1? release
2 git tag v2012_07_01
3 exec_deploy_script
4 # ???2? release
5 git tag v2012_07_02
6 exec_deploy_script
? v2012_07_02 release ??????? rollback ??????(v2012_07_01)
1 git checkout v2012_07_01
2 exec_deploy_script
?? git diff ?????? change
????7??2? release ?????
1 git diff v2012_07_01 v2012_07_02
2 #git diff output
3 .....
4 -apt-get install -y mycode_package=1.2.3
5 +apt-get install -y mycode_package=1.3.1
6 -echo "mypkg mypkg/use_memached boolean true"|debconf-set-selections
7 +echo "mypkg mypkg/use_memached boolean false"|debconf-set-selections
8 .....
9 -echo "mypkg mypkg/title string coscup"|debconf-set-selections
10 +echo "mypkg mypkg/title string coscup.org"|debconf-set-selections
11 .....
• package ????? , config management ????change ???
• ?? RD ??????? diff ???????????
?? git branch ?????
• ?? git ??? branch ???????? deploy script
• ???????? alpha , beta , staging , production ?? branch
??? alpha ???????? beta ??? git merge
1 git checkout beta
2 git merge alpha
3 ...solve confict...
4 git commit
One More Thing
Deploy System : Puppet
• Open source deploy system written by Ruby
• Client Server ????????? pure client
• Config is language, built by Ruby DSL !!!
• ??? template system ???? perfect configuration manage System
• ???OS ? package system , ???????? OS ????
• From Wikipedia ????: Google, Zynga, Sun, Wikipedia, Dell, Twitter...
Puppet Node config
In module
1 class ntpd {
2 package { "ntp": ensure => installed, }
3 file { "/etc/ntp.conf": owner => root, group => root,
4 require => package["ntp"] }
5 @service { "ntpd": ensure => running, enable => true,
6 subscribe => File["/etc/ntp.conf"] }
7 }
8 class ci{ package { "jenkins" : ensure => installed,} }
9 class Webservr{ package { "nginx" : ensure => installed,} }
In main config
1 node basenode { include ntpd }
2 node 'ci.coscup.org' inherits basenode {
3 include ci
4 }
5 node 'web.example.com' inherits basenode {
6 include webserver
7 }
Puppet Package config
?? Package ??
1 package {'vim': ensure => 'installed' }
2 package { "apache2": ensure => '2.2.22-1ubuntu1' }
3 package { 'mysql': ensure => installed,
4 provider => 'msi',
5 source => 'N:/packages/mysql-xxx.msi',
6 install_options => { 'INSTALLDIR' => 'C:\mysql-5.5' }, }
??????? Deploy Script
• Debian / Ubuntu : Apt-get
• RedHat / Fedora / CentOS : yum
• FreeBSD : port
• MacOS : macport
• Windows : msi
• Gentoo : portage
Puppet Template System
Configure Management ??? : ERB
??????? config file
1 $server_name = 'domain.coscup.org'
2 $server_port = 80
3 file {"/usr/conf/nginx.conf": mode => "664",
4 content => template("nginx/config.erb"), }
template
1 server {
2 listen <%= $server_port%>;
3 server_name <%= $server_name %>;
4 .....
5 }
Pure ruby config menas
1 Given I run "ssh localhost -p 2222" interactively
2 And I type "apache2ctl configtest"
3 And the exit status should be 0
• We can write unit test ... so we can TDD
• We can write rspec and cucumber ... so we can BDD
• We can commit our config in Hudson... so we can CI our config
??? Deploy Script ??
1 class web{ package { "nginx": ensure => '0.1.19-1' } }
2 class cache{ package { "memcached" : ensure => '1.4.13-0ubuntu2' }}
3 class kml_lib{ package { "pykml" : ensure =>'0.0.3' }}
4 class message_quene{
5 package { "ruby-amqp" : ensure => '0.9.5-1' }
6 package { "rabbitmq-server": ensure => '2.8.4-1' }
7 }
8 class code_package{
9 include kml_lib
10 include message_quene
11 package { "node-express" : ensure => '2.5.9-1' }
12 package { "mycode" : ensure => '1.2.3' }
13 }
14 node "w1.coscup.com" {
15 include web
16 include cache
17 include code_package
18 }
Q & A
Puppet Config
?? Command
1 exec { 'remove file' : command => 'rm /tmp/pup/etd' , path => '/bin' }
?? Windows Command
1 exec { 'wincommand': command => 'C:\Windows\..\powershell.exe -executionpolicy remotesigned -file C:\test.ps1', }
?? file
1 file{'/tmp/pup': ensure => 'directory'}
2 file{ '/tmp/pup/et': ensure => 'present', content => 'look at me',
3 require => File['/tmp/pup']}
4 file { "/etc/ntp.conf": owner => root, group => root,
5 mode => 0644, source => "puppet:///ntpd/ntp.conf",
6 require => package["ntp"]
7 }