the saltstack pub crawl

about me ☃  systems engineer @Logicea LLC

☃  I have broken development a few times

☃  I have broken production a few more

☃  I hate databases

about this

The Good Companions Configuration management (CM) is a field of management that focuses on establishing and maintaining consistency of a system. (Wikipedia) Design your infrastructure Systems integrity and consistency Provision and automate Create proper processes Cloud Management Version Control Automations and Remote Execution Monitoring, Alerting, Logging

Configuration Management

kk

YAML

So, SaltStack? Saltstack delivers a dynamic infrastructure communication bus used for orchestration, remote execution, configuration management.

+ Python, YAML, Jinja2, ZeroMQ

+ Event driven

+ Master-agent, Masterless (ssh)

+ Multimaster for HA

+ Cloud/VM/Containers support

+ Orchestration

+ Reactors

+ Runners

+ Guaranteed execution order

So, SaltStack? Saltstack delivers a dynamic infrastructure communication bus used for orchestration, remote execution, configuration management.

- Relatively young project

- Changes rapidly

- Its documentation is… challenging

- Needs some Python background

- Needs design background

- If your saltmaster is compromised, LOL

Terminology Master and Minion The Master controls its Minions :p State Modules Code needed to enforce, set up or change the configuration of a target system Pillars Custom data essential for state execution (e.g. user accounts, lists of virtual hosts etc.) Grains Static minion information SLS Files (.sls) Text files containing list of states to execute or, simply pillar data

Terminology Renderers Render SLS files and pass information to the state system Templates SaltMine Top Files Highstate

Architecture •  Master-Minion Persistent TCP

connection

•  Two listening ports on Master (4505,4506)

•  No listening ports on Minions (yey)

•  ZeroMQ to send/recv messages

•  Encrypted transport (+custom protocol)

•  Minions wait for jobs from a remote Master

•  Minions return job results back to Master

Architecture

Installation # cat /etc/apt/sources.list.d/saltstack.conf:

deb http://debian.saltstack.com/debian jessie-saltstack main

# wget -q -O- "http://debian.saltstack.com/debian-salt-team-joehealy.gpg.key" | apt-key add –

# apt-get update

Saltmaster

master# apt-get install salt-master

.

.

.

.

.

master# salt-key –L

Unaccepted Keys:

minion8

master# salt-key –A

Minion

minion8# apt-get install salt-minion

minion8# cat /etc/salt/minion

master: saltmaster

minion8# systemctl restart \

> salt-minion

Saltmaster saltmaster:/etc/salt# tree

.

├── master

├── pillars

│ ├── defaults

│ │ └── init.sls

│ ├── top.sls

│ └── users

│ ├── init.sls

│ └── gary.sls

│ └── oliver.sls

└── states

├── defaults

│ ├── init.sls

│ ├── linux-debian.sls

│ ├── linux-centos.sls

│ └── sshd_config

├── top.sls

└── users

└── init.sls

Saltmaster Config

# cat /etc/salt/master

file_roots:

base:

- /etc/salt/states

pillar_roots:

base:

- /etc/salt/pillars

Pillars

# cat /etc/salt/pillars/top.sls

base:

'*':

- defaults

- users

# cat /etc/salt/pillars/defaults/init.sls

disabled_services:

- rpcbind

- nfs-common

States

# cat /etc/salt/states/top.sls

base:

'*':

- defaults

- users

# cat /etc/salt/states/defaults/init.sls

basic_pkgs:

pkg.installed:

- install_recommends: False

- pkgs:

- screen:

- lsof:

- ngrep:

openssh-server:

pkg:

- installed

service:

- name: ssh

- running

- enable: True

- watch:

- file: /etc/ssh/sshd_config

file.managed:

- name: /etc/ssh/sshd_config

- source: salt://defaults/sshd_config

- require:

- pkg: openssh-server

{% if grains['os_family'] == "RedHat" %}

disable_selinux:

file.managed:

- name: /etc/selinux/config

- contents: "SELINUX=disabled"

{% endif %}

Saltmaster

Pillars and Grains master:~# salt ‘minion8’ grains.items

minion8: ---------- admins: - manji - mehiel biosversion: Bochs <snip>

id: minion8 init: systemd ip4_interfaces: ---------- eth0: - 10.10.1.115 lo: - 127.0.0.1 os: Debian os_family: Debian <snip>

master:~# salt ‘minion8’ pillar.data

minion8:

----------

disabled_services:

- rpcbind

- nfs-common

users:

----------

manji:

----------

email:

[email protected]

enabled:

True

fullname:

effie mouzeli

home:

/home/manji

pub_keys:

- ssh-rsa AAAAB3NzaC1y

<snip>

master:~# salt ‘minion8’ state.highstate minion8: ------- ID: basic_pkgs Function: pkg.installed Result: True Comment: 12 targeted packages were installed/updated. The following packages were already installed: less, bzip2, wget, ngrep Started: 18:25:18.805716 Duration: 61584.232 ms Changes: ---------- curl: ---------- new: 7.38.0-4+deb8u3 old: <snip> Summary for minion8 ------------- Succeeded: 21 (changed=1) Failed: 0 -------------

Total states run: 21

Salt Minion

Salt Messages Publish job:

salt/job/20160414115046162293/new {

"_stamp": "2016-04-14T08:50:46.166360",

"arg": [

"defaults"

],

"fun": "state.sls",

"jid": "20160414115046162293",

"minions": [

"minion8"

],

"tgt": "minion8",

"tgt_type": "glob",

"user": "root"

}

Return Result:

salt/job/20160414115046162293/ret/minion8 { "_stamp": "2016-04-14T08:50:48.239998",

"cmd": "_return", "fun": "state.sls", "fun_args": [ "defaults" ], "id": "minion8", "jid": "20160414115046162293", "out": "highstate",

"retcode": 0,

"return": {

"pkg_|-basic_pkgs_|-basic_pkgs_|-installed": {

"__run_num__": 5, "changes": {},

"comment": "All specified packages are already installed", "duration": 6.319, "name": "lsof",

"result": true, "start_time": "11:50:47.161264" }, <snip>

"success": true }

Highstate Flow

Advanced Topics Reactors

Runners

Orchestration

Beer Communication

Custom salt modules

Do not Forget •  You need some python and some coding

skills

•  Use a quick solution when requirements are unknown/not clear

•  Refactor when possible

•  Try to make reusable states, don’t repeat yourself (DRY)

•  Try to not over engineer (resist the temptation)

•  Always write documentation

•  Be patient, this is NOT easy

•  Keep your #YOLO moments to a minimum

Sources - Useful Links •  Images from The World’s End (2013) and edgarwright @ flickr.com •  https://docs.saltstack.com/ •  https://github.com/saltstack-formulas •  https://github.com/ministryofjustice/salt-shaker •  https://github.com/harkx/saltstack-cheatsheet •  https://www.digitalocean.com/community/tutorials/an-introduction-to-saltstack-

terminology-and-concepts •  http://bencane.com/2013/09/03/getting-started-with-saltstack-by-example-automatically-

installing-nginx/ •  http://leonardinius.galeoconsulting.com/2014/08/devops-101-on-saltstack-example/ •  https://puppet.com/blog/how-to-choose-right-tools-processes-for-devops •  How sysadmins devalue themselves - https://queue.acm.org/detail.cfm?id=2891413 •  and of course, https://en.wikipedia.org

Thanks to: kargig, kyriakos and andrew for their comments and feedback

Questions?

Thank You !

the saltstack pub crawl

Software