Config Management with

Puppet, GIT & some Ruby magic

Stefan Peer – System Engineer

11.11.2016

Head organization of 369 cooperatives with more than 124.000 single members

Service provider and consulting IT, HR, financial, legal, education and much more

310 employees in total 40% in IT

Raiffeisen Informationssystem (RIS) IT service provider of the Raiffeisen Group

Datacenters in Bolzano and Milano

2

Raiffeisenverband Südtirol

Applications running on different platforms z/OS (Mainframe), Linux, Solaris, Windows

Heavily rely on virtualization and automation VMware, Solaris container

3

IT Systems in RIS

5 5 6 6 7 7 7 8 8 8 8 0

100

200

300

400

500

600

700

800

900

1000

2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

# VMs

# Administrators

Configuration Management is the process of

standardizing resource configurations and enforcing

their state across IT infrastructure in an automated yet

agile manner.

(Puppetlabs)

4

Definition

Growth same effort to make a change on 1 or 1000 servers

Central Governance in a heterogeneous environment with various OS

Traceability / Reporting obliged by certifications such as PCI/DSS or ISAE3402

Rollback revert changes

Durability keep config-state consistent

Consistent Environments hand over changes: Test => QA => Production

5

Why Configuration Management?

6

Admin‘s daily life … before CM – part 1

Hey Linux! We need to change the IP Address

of our secondary DNS server!

Okay, don't worry. Gimme a week.

Don’t have time for that!

Hmm, I could write a script that SSHes into

all our servers and applies the change!

But what about this other

Debian server?

7

Admin‘s daily life … before CM – part 2

Hey Solaris! We need to change the IP Address

of our secondary DNS server!

Okay, lot’s of manual work, but we will have it done by next week!

Hmm, good task for our

intern

Damn! Project delayed

for another week!

8

Admin‘s daily life … with CM

Hey Linux! We need to change the IP Address

of our secondary DNS server!

Ok, hang on, I’ll commit the change into CM. Done, change will be rolled out within half an hour.

Btw. to Solaris servers as well!

Thanks man! Good work!

Where could I go skiing

tomorrow?

9

Let the puppets dance!

Puppet Master

1. facts ex.

I am Frida, a

RHEL 6.8

with 2 cores

Foreman

4. reference config ex. Apache must be running,

listening on Port 443

2. ask ENC ex. who is Frida?

3. classes and params ex. Apache server located

in Bolzano

each server,

every 30 minutes

10

Let the puppets dance!

Puppet Master

6. report ex.

service Apache

failed to start

Foreman

7. forward report ex. service Apache failed

to start on Frida

5. apply reference config ex. service httpd start

ex. for Solaris it would be:

svcadm enable /network/http:apache22

Assign Puppet Classes to hosts (ENC) ex. Icinga Master host

What are your servers doing?

What has changed on server X?

11

Foreman

12

What can I do with Puppet?

Manage files

file {'/etc/httpd/conf/httpd.conf':

ensure => present,

content => template('${module_name}/httpd.conf.erb'),

owner => 'root',

group => 'root',

mode => '0644',

}

Manage services

service {'httpd':

ensure => running,

enable => true,

}

13

What can I do with Puppet?

Install or uninstall software

package {'httpd':

ensure => installed,

}

Execute commands

Create Cron jobs

Manage certificates and Java Keystores

and much, much more …

package {'tcpdump':

ensure => absent,

}

14

Puppet manifest

class ris_ftp::server (

$local_root_dir,

){

package { 'vsftpd':

ensure => installed

}

-> file { '/etc/vsftpd/vsftpd.conf':

content => template("${module_name}/vsftpd.conf.erb"),

notify => Service['vsftpd'],

}

service { 'vsftpd':

ensure => running,

enable => true,

}

}

15

GIT – the place where all the Puppet code is stored

One special GIT repo that connects everything together we call it „control-repo“

GIT branch per environment New environment needed? Simply fork a branch!

16

Dynamic environment creation with r10k

Puppetfile ris_dns => Commit 12 ris_ssh => Commit 3 ris_icinga => Commit 45

Puppetfile ris_dns => Commit 11 ris_ssh => Commit 2 ris_icinga => Commit 40

Puppetfile ris_dns => Commit 11 ris_icinga => Commit 36

Merge changes Merge changes

TEST QA PRODUCTION

17

Puppetfile in real – and that‘s just a part of it

History of our control-repo Including current state of each branch, .i.e., environment

18

control-repo in real

Nearly impossible to manage control-repo + Puppetfile by hand

That‘s why we wrote a Ruby toolset that helps us managing it we call it ris-puppet

Examples: ris-puppet module validate

ris-puppet module deploy --env=test

ris-puppet environment create --env=stefan --from=production

ris-puppet foreman import

Integrated also in GIT server via hooks ex. reject commit if there are syntax errors

19

Now, where‘s the Ruby magic?

20

Questions ?