workflow story: theory versus practice in large enterprises
DESCRIPTION
"Workflow story: Theory versus practice in Large Enterprises" by Marcin Piebiak of Linux Polska at Puppet Camp London 2013.TRANSCRIPT
1
Marcin PiebiakSolutions ArchitectLinux Polska Sp. z o.o.
Workflow story – theory versus practice in Large Enterprises
2
What is it?
package { 'ssh': ensure => 'installed', name => 'openssh-server',}
service { 'ssh': ensure => 'running', name => 'sshd', require => Package['ssh'],}
● ResourcesA lot of ready to use resources:
• user • group • host • cron • exec • file • package • service • ...
• Resources are building blocks.• They can be combined to make larger components.• Together they can model the expected state of your system.
3
package { 'ssh': ensure => 'installed', name => 'openssh-server',}
service { 'ssh': ensure => 'running', name => 'sshd',}
● Resources● Declarative language
Rather than listing a series of steps to carry out we can describe the desired final state only.
What is it?
4
package { 'ssh': ensure => 'installed', name => 'openssh-server',}
root@debian ~]# apt-get install openssh-server
root@redhat ~]# yum install openssh-server
● Resources● Declarative language● Abstraction
Resources in Puppet are abstracted from underlying providers.
What is it?
5
root@redhat ~]# facterarchitecture => x86_64domain => linuxpolska.plfacterversion => 1.5.2fqdn => redhat.linuxpolska.plhardwaremodel => x86_64hostname => redhatinterfaces => eth0ipaddress => 172.16.10.1kernel => Linux...
● Resources● Declarative language● Abstraction● Facts
Puppet uses facter to gather information about the host system.
Custom Facts
Facter.add('role') do setcode do 'cat /etc/role' endend
What is it?
6
$pkg_name = hiera('pkg_name')
package { 'apache': ensure => 'installed', name => $pkg_name,}
● Resources● Declarative language● Abstraction● Facts● Data separation
Puppet uses Hiera as its single source of truth data abstraction layer.
Hiera uses Facter facts to determine a hierarchy.
---:backends: - yaml:yaml: :datadir:/etc/hiera:hierarchy: - %{fqdn} - %{osfamily} - %{environment} - common
What is it?
7
class ssh ( $pkg_name = 'openssh-server', $srv_name = 'sshd',) { package { 'ssh': ensure => 'installed', name => $pkg_name, }
service { 'ssh': ensure => 'running', enable => 'true', name => $srv_name, require => Package['ssh'], }}
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
Classes define a collection of resources that are managed together as a single unit.
What is it?
8
ssh├── manifests│ ├── init.pp│ └── server.pp├── files│ └── ssh_config└── templates └── sshd_config.erb
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
Modules are directories that contain your configuration. They are designed to encapsulate all
of the components related to a given configuration in a single folder hierarchy.
Classes are abstracted by modules:
What is it?
9
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
If a more complex deployment is needed, reusing existing classes saves effort and reduces error.
What is it?
10
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
You own abstraction layers...Profiles:class profiles::application { include tomcat include mysql include myapp}class profiles::base { include ssh include ntp include users}
Roles:class role::webapp { include profiles::base include profiles::customapp include profiles::test_tools}
What is it?
11
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS
Supported OS
What is it?
12
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS
Big environment – no problem
Changes?
base/ntp.yaml --- ntp::local_servers: - 192.168.0.45+ - 192.168.0.1
What is it?
13
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS● Provisioning● Orchestration● Puppet Forge● Live Management● Environments● Reporting● Dry-run mode
And more...
VM/Cloud Provisioning
Orchestration
Live Management
MCollectiveReporting
Custom:- types & providers- facts- functions
Environments
Dry-run mode
What is it?
14
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS● Provisioning● Orchestration● Puppet Forge● Live Management● Environments● Reporting● Dry-run mode
What is it?
Thanks to these all superior features, Puppet is:- fast in deployment- easy to use- universal for a lot of operating systems- with unlimited possibilities- easy to expand- flexible
15
VS.
In large enterprises like banks, telco, insurance, etc. those features are not the most relevant. Implementing Puppet in enterprises we must consider another
priority map, another mindset.
We must answer not trivial questions dealing with the core IT Departments way of work.
16
VS.
So... everyone hasaccess to everything?!
17
VS.
So... can I destroythe whole infrastructure
from one place?
18
SolutionPuppet master installation
on hardened system with limited direct access.
tcp:8140
tcp:443
tcp:22
ssh keys
RBAC
For maintenance.
19
Solution
Git as a communication layer between developers and puppet master.
Release manager
developer
developer
developer
Pull request
Fetch
20
VS.
So... everyone now mustuse puppet?!
21
Solution
On beginning each department can have own environments.
Developers
Systems administrators Databases
administrators
Securitydepartment
22
SolutionEach department can have
many environments and its own idea how to organizework with puppet. Security
department
prodtestdev
Integrator
Releasemanager
Testsintegrator
23
SolutionEach department can have
many environments and its own idea how to organizework with puppet. Security
department
prodtestdev
Integrator
Releasemanager
Testsintegrator
v0.2v0.1
commits
Release
24
Solution
After time some departments will start working together in one environment.
Developers
Systems administrators Databases
administrators
Securitydepartment
integrator
25
SolutionDevelopers
Systems administrators Databases
administrators
Securitydepartment
integrator
integrator
26
Solution
At the end all departments will use one environment.
Developers
Systems administrators Databases
administrators
Securitydepartment
integrator
integrator
27
VS.
How can I find outwho made what change
and who approvedthis modification?
28
Solution
prodtestdev
v0.2v0.1
commits
Release
commit 220938c5a2e51ecd4166eb7d75d14974cbcff897Author: Marcin Piebiak <[email protected]>Date: Fri Jul 5 11:27:43 2013 +0200
Description....
Person who approvedmodifications.
Using git we have:● date● author● description
We can use git as a place for history of the infrastructure.● git status● git log● git diff● git blame
Git history is cryptographically secured.
29
VS.
If I have a lot ofenvironments how can
I use them?
30
Solution
tcp:8140
prodtestdev
We can specify a set of environments for each host to use.
Database testing system, uses three testing environments from different departments.
31
VS.
Great!But... using commandline I can connect to
different environments!
32
Solution
tcp:8140
puppet agent -t --environment
We use imp module, to control puppet agents behavior and their access to environments.
33
Solutionpuppet.conf
[main]modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modulesmanifest = /etc/puppetlabs/manifests/site.pp
[env_sec_prod]modulepath = /var/lib/git/env_sec_prod/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
[env_sec_test]modulepath = /var/lib/git/env_sec_test/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
site.pp
include imp
hiera_include('include')
/etc/puppetlabs/puppet/modules – place for modules from forge and well tested modules and module imp
/etc/puppetlabs/manifests/site.pp - common manifest for all environments
/var/lib/git/$environment/modules – git repository for environment
imp.yaml
---imp::environments: env_sec_prod: order: 'deny,allow' deny: 'all' allow: - 'host1.linuxpolska.lab' - 'www.*.linuxpolska.lab' commiters: - 'john.smith' priority: 1
34
VS.
How can I auditchanges made in
the infrastructure?
35
Solution
Log collector and analyzer.
puppet reports
Puppet reports
When nodes fetch their configurations from the puppet master, they send back inventory data and a report of their run.
36
VS.
How can puppet helpme with audit? How can
I recreate life cycle of each host?
37
Solution
Log collector and analyzer.
● puppet reports● puppet agents catalog● hosts facts● git diffs after commit● hiera configuration for each host● filebuckets● ...
38
Solution
app db system security
file {'/etc/important': ensure => 'file', group => 'apache', mode => '0660',}
file {'/etc/important': ensure => 'file', user => 'root', group => 'root', mode => '0600',}
If we have many environment there is always risk of overwriting someone's changes.
Log collector and analyzer.
39
VS.
How will the modification in puppet
manifestsaffect the wholeinfrastructure?
40
Solution
prodtestdev
v0.2v0.1
commitsLog collector
and analyzer.
Report from puppet normal run.
Report from puppet dry-run.
Using log collector we can analyze the infrastructure modifications before they get to production environment.
41
VS.
All changes are madeautomatically? First I would
like to see what is going to be changed.
42
VS.
How can we rollbackchanges?
43
VS.
After we installpuppet will we know
everything about the infrastructure?
44
THE END
Marcin PiebiakSolutions ArchitectLinux Polska Sp. z o.o.