puppet @ seat

45
Puppet @ Seat Seat Internet Factory Tech Talk 5

Upload: alessandro-franceschi

Post on 05-Dec-2014

7.244 views

Category:

Technology


0 download

DESCRIPTION

Presentation on how Puppet has been introduced in Seat Pagine Gialle to automate system administration tasks and easy the cooperation between Ops and Others.

TRANSCRIPT

  • 1. Seat Internet Factory Tech Talk 5
  • 2. Tempus fugitSetup. Congure. Deploy. Troubleshoot. Fix.
  • 3. Panta reiEmails. Tickets. Calls. Meetings.
  • 4. Malatemporacurruntcat /etc/issueDebian 4
  • 5. Errarehumanum est...rm /var/log/messages
  • 6. perseverareautemdiabolicumchmod -R 777 ; su nobody ; rm -rf .
  • 7. Alea iacta estHope driven release management.
  • 8. Devs Versus OpsD: Can you copy /ldap/dritchie/spes.war to prod? O: Open a Ticket.
  • 9. O temporao mores!Big efforts for small outputs
  • 10. Spes ultima dea Have you heard about this Puppet thing?
  • 11. Repetitia iuvantMachines automate better
  • 12. GratisOpenSource from www.puppetlabs.com
  • 13. Cui prodest? Source: www.puppetlabs.com
  • 14. Divide et imperaMany Puppet nodes One Puppet Master
  • 15. package { openssh-server:Status Quo } ensure => present, before => File[/etc/ssh/sshd_config],A declarative language to describe file { /etc/ssh/sshd_config:system status ensure => file, mode => 600, source => /root/learning-manifests/sshd_config, } service { sshd: ensure => running, enable => true, subscribe => File[/etc/ssh/sshd_config], }
  • 16. Veni vidi vici puppet apply
  • 17. DeusExMachina# This File is Managed by Puppet
  • 18. Mater semper certa est, pater nunquam O: Who changed that le?!info: Filebucket[/var/lib/puppet/clientbucket]: Adding /etc/resolv.conf(d7fbc1695489ce896d30b7b04d72887c)info: //test/File[/etc/resolv.conf]: Filebucketed /etc/resolv.conf to main with sumd7fbc1695489ce896d30b7b04d72887cnotice: //test/File[/etc/resolv.conf]/content: content changed {md5}d7fbc1695489ce896d30b7b04d72887c to {md5}958836dd057fdbb33597d688cc6d28a2
  • 19. Ex novoProvision from scratch.
  • 20. DoctumdocesInfrastructure as code
  • 21. Scripta manent,verba volantsvn ci -m Ticket #777
  • 22. FactotumP: I manage. Almost everything.
  • 23. Ignorantia legisnon excusatOnce you Puppettize, you Puppettize
  • 24. Sperimentazione iniziata nel 2010Implementazione sistematica su nuovi server da Ottobre 2010Conversione / Upgrade sistemi esistenti in corso.Produzione a Rozzano - Sistemi gestiti: 167 (growing):- Pagine Bianche- Banners- E-Commerce, Scioppy- Tools PL, Iglu, Spysite, Routing, Fotocontest, Iglu ...- Sistemi infrastrutturali (Dns, Syslog, Deploy, Ldap (soon) )- Sistemi di monitoring (Nagios, Munin)Disaster Recovery - Sistemi gestiti: 34:- Pagine Gialle (Intera liera)- Sistemi infrastrutturali (Dns, Syslog, Deploy, Ldap)- Sistemi di monitoring (Nagios, Munin)
  • 25. # Base nodenode basenode { $dns_servers = ["192.168.39.42","192.168.39.43"] $syslog_server = "syslog-1.pgol.com" $type = "prod" # We assume that most of nodes are of prod(ution) type. $users_auth = "ldap" # By default we want ldap auth[...]}# Rozzano Production sitenode rozzano inherits basenode { $site = "rozzano" $ntp_server = ["ntp1.pgol.com","ntp2.pgol.com"]}node rozzano-dmz inherits rozzano { $zone = "dmz" $users_auth = "local" # No ldap auth for users in DMZ servers}node rozzano-erog inherits rozzano { $zone = "erog"}# Disaster recovery Sitenode dr inherits basenode { $site = "dr" $dns_servers = ["192.168.50.10","192.168.50.11"][...]
  • 26. # FRONTEND BIANCO # CLUSTER CASSANDRA DEVnode fep-pbit-1 inherits rozzano-erog { node dev-cassandra-1b inherits rozzano- include role_fep-pbit erog {} ! $type = "dev" ! include role_cassandranode fep-pbit-2 inherits rozzano-erog { } include role_fep-pbit [...]} # CLUSTER CASSANDRA TEST[...] node test-cassandra-1 inherits rozzano- erog {node test-pbit-1 inherits rozzano-erog { ! $type = "test" $type = "test" ! include role_cassandra include role_fep-pbit }} [...][...] # CLUSTER CASSANDRA PROD node cassandra-1 inherits rozzano-erog { ! include role_cassandra } [...] node cassandra-5 inherits rozzano-erog { ! $has_batch = "true" ! include role_cassandra }
  • 27. class general { include distro include profile include puppet include motd include users include openssh include hosts include resolver include monit include sudo include snmpd include nrpe include munin include rsync include basedirs case $operatingsystem { ubuntu,debian: { # Moduli attualmente funzionanti solo su Ubuntu/Debian include exim include openntpd include apt include rsyslog include unattended-upgrades include hardware } centos,redhat: { include yum } default: { } }}
  • 28. class role_fep-pbit { $role="fep-pbit" include general include nfs::client::fep-pbit include apache include apache::seat::fep-pbit include php::pear include php::dev include php::oci8 include apache::spidertrap apache::module { "rewrite": } apache::module { "proxy": templatefile => "proxy.conf.erb" } php::module { "gd": } php::pear::module { "apc": } php::pear::module { "XML_Serializer": use_package => "no" } php::pecl::config { "http_proxy": value => "$proxy_server" } [...] # Monitoring include monitor::seat::url_fep-pbit #Deploy puppi::project::files { "fep-pbit": source => "http://deploy.${domain}/fep-pbit/deploylist.txt", [...] }}
  • 29. class apache::monitor { # Port monitoring monitor::port { "apache_${apache::params::protocol}_${apache::params::port}": protocol => "${apache::params::protocol}", port => "${apache::params::port}", target => "${apache::params::monitor_target_real}", enable => "${apache::params::monitor_port_enable}", tool => "${monitor_tool}", } # Process monitoring monitor::process { "apache_process": process => "${apache::params::processname}", service => "${apache::params::servicename}", pidfile => "${apache::params::pidfile}", enable => "${apache::params::monitor_process_enable}", tool => "${monitor_tool}", } [...]}
  • 30. $monitor="yes"$monitor_tool=["monit","nagios","puppi",munin]
  • 31. class monitor::seat::url_fep-pbit {# 1240 monitor::url { "Url-1240_PREFFISSI_INTERNAZIONALI": url => "http://1240.paginebianche.it/execute.cgi?ts=9&cb=18&btt=1&nz=CANADA&l=it", port => 80, target => "${fqdn}", pattern => 01139, enable => "true", tool => "${monitor_tool}", } monitor::url { "Url-1240_RICERCA_LOCALITA_E_CAP_PER_PREFISSO": url => "http://1240.paginebianche.it/execute.cgi?ts=19&cb=18&btt=1&prefisso=0744&l=it", port => 80, target => "${fqdn}", pattern => Terni, enable => "true", tool => "${monitor_tool}", } [...]}
  • 32. root@fep-pbit-1:~# puppi checkHost check: 50-Url-1240_PREFFISSI_INTERNAZIONALI [ OK ]HTTP OK: HTTP/1.1 200 OK - 25285 bytes in 0.193 second response time |time=0.192856s;;;0.000000 size=25285B;;;0Host check: 50-Url-1240_RICERCA_LOCALITA_E_CAP_PER_PREFISSO[ OK ]HTTP OK: HTTP/1.1 200 OK - 12417 bytes in 0.071 second response time |time=0.070648s;;;0.000000 size=12417B;;;0Host check: 50-Url-1240_RICERCA_LOCALITA_PER_CAP [ OK ]HTTP OK: HTTP/1.1 200 OK - 12641 bytes in 0.048 second response time |time=0.048381s;;;0.000000 size=12641B;;;0[...]Host check: 50-Mount__store_www.paginebianche.it_doc_root_g[ OK ]/store/www.paginebianche.it/doc_root/gclight is mounted! Type is nfsHost check: 50-apache_process [ OK ]PROCS OK: 540 processes with command name apache2Host check: 50-apache_tcp_80 [ OK ]TCP OK - 0.000 second response time on port 80|time=0.000250s;;;0.000000;10.000000Host check: 50-openssh_process [ OK ]PROCS OK: 3 processes with command name sshdHost check: 50-openssh_tcp_22 [ OK ]TCP OK - 0.000 second response time on port 22|time=0.000151s;;;0.000000;10.000000
  • 33. root@metaportali-mpc:~# puppi deploy configuratorPuppi setup: 00-configurator-RuntimeConfig-Initialization [ OK ]Deploy: 10-configurator-Run_PRE-Checks [ OK ][...]Deploy: 20-configurator-Retrieve_WAR [ OK ]Deploy: 30-configurator-Backup_existing_WAR [ OK ]Deploy: 36-configurator-Disable_extra_services [ OK ][...]Deploy: 37-configurator-Check_undeploy [ OK ]Deploy: 38-configurator-Service_stop [ OK ][...]Deploy: 39-configurator-Run_Custom_PreDeploy_Script [ OK ]Deploy: 40-configurator-Deploy_WAR [ OK ]Deploy: 42-configurator-Service_start [ OK ][...]Deploy: 43-configurator-Check_deploy [ OK ]Deploy: 44-configurator-Enable_extra_services [ OK ][...]Deploy: 80-configurator-Run_POST-Checks [ OK ][...]Reporting: 20-configurator-Mail_Notification [ OK ]REPORT FOR PUPPI - STATUS OKSummary of operations is: /var/log/puppi/configurator/20110303-145104/summaryDetails are in: /var/log/puppi/configurator/20110303-145104/Temporary workdir has been: /tmp/puppi/configurator/ (Will be rewritten at the next puppi run)Runtime config file is: /tmp/puppi/configurator/configFiles have been archived in: /var/lib/puppi/archive/configurator/20110303-145104
  • 34. root@fep-pbit-1:~# puppi rollback fep-pbitPuppi setup: 00-fep-pbit-RuntimeConfig-Initialization [ OK ]Choose deploy to rollback:total 24drwxr-xr-x 2 root root 4096 2011-02-17 17:25 20110217-172418drwxr-xr-x 2 root root 4096 2011-02-24 17:18 20110224-171656drwxr-xr-x 2 root root 4096 2011-02-28 16:54 20110228-165323drwxr-xr-x 2 root root 4096 2011-03-01 17:01 20110301-170009drwxr-xr-x 2 root root 4096 2011-03-01 17:13 20110301-171339drwxr-xr-x 2 root root 4096 2011-03-01 17:16 20110301-171607lrwxrwxrwx 1 root root 47 2011-03-01 17:16 latest -> /var/lib/puppi/archive/fep-pbit/20110301-171607[...]
  • 35. class role_mpc-metaportale { $role="mpc-metaportale"[...] puppi::project::war { "configurator": source => "http://deploy.pgol.com/mpc-metaportale/configurator.war", user => "www-data", init_script => "tomcat-mpc", predeploy_customcommand => "rm -rf /store/tomcat/mpc/webapps/*", predeploy_user => "root", predeploy_priority => "39", deploy_root => "/store/tomcat/mpc/webapps", report_email => "[email protected],[email protected]", enable => "true", disable_services => "monit puppet", } root@metaportali-mpc:~# puppi deploy configurator} Puppi setup: 00-configurator-RuntimeConfig-Initialization [ OK ] Deploy: 10-configurator-Run_PRE-Checks [ OK ] [...] Deploy: 20-configurator-Retrieve_WAR [ OK ] Deploy: 30-configurator-Backup_existing_WAR [ OK ] Deploy: 36-configurator-Disable_extra_services [ OK ] [...] Deploy: 37-configurator-Check_undeploy [ OK ] Deploy: 38-configurator-Service_stop [ OK ] [...] Deploy: 39-configurator-Run_Custom_PreDeploy_Script [ OK ] Deploy: 40-configurator-Deploy_WAR [ OK ] Deploy: 42-configurator-Service_start [ OK ] [...] Deploy: 43-configurator-Check_deploy [ OK ] Deploy: 44-configurator-Enable_extra_services [ OK ] [...] Deploy: 80-configurator-Run_POST-Checks [ OK ] [...] Reporting: 20-configurator-Mail_Notification [ OK ] REPORT FOR PUPPI - STATUS OK Summary of operations is: /var/log/puppi/configurator/20110303-145104/summary [...]
  • 36. Faster Setups Do ut des Tested codeQuick Scalability Url based checksDeployment Agility Site Aware congurationsTesting Environment Standardized deploy requests
  • 37. In medio stat virtus Share needs, constraints, knowledge and skills
  • 38. Hicmanebimus optime Cooperation is fun
  • 39. Full Infrastructure AutomationUnied Infrastructure Reporting Per aspera ad astraSelf Service Release ManagementAutoTesting Release Workow Add here your wildest (IT) dream...
  • 40. ad maiora Graphics: www.tatlin.net