saltstack advanced concepts

www.vishalbiyani.com

SALTSTACK – ADVANCED CONCEPTS



About Me

Vishal Biyani is a DevOps engineer at large. In his past stints he has

done Java development to operations for enterprise systems.

Current interests around Go, Docker, Mesos and plenty of other

stuff. Scribbles at www.vishalbiyani.com. Other interests around

reading, movies and bit of photography. Latest fascination is to

build a DIY Satellite and launch it.




Quick Poll

• How many know anyone of Puppet, Chef, Ansible or Salt

• How many know Salt?

• WHat problems have you faced in Puppet/Chef/Ansible?



Agenda

• Overview of Basics & some Debugging Techniques

• Peer Communication in Salt

• Events, Orchestration & Reactors

• Mine

• Beacons

• Multi-master & Syndic

• Basic Salt Cloud

You can checkout the project at https://github.com/vishal-biyani/saltstack-cluster - this is a

working Salt server+minion setup and has some of examples in today’s talk. Feel free to

file bugs or send pull requests to enhance.

Remind me to show

some exercises when

you see this balloon if I

forget!


https://github.com/vishal-biyani/saltstack-cluster


OVERVIEW OF BASICS & SOME DEBUGGING TECHNIQUES



Quick Basics

• Salt is CM platform based on Python language and YAML data format.

Uses ZeroMQ for scalable and fast communication across large number

of nodes

• Typical use cases you see like any other CM platform:

• Managing OS and platform stack on number of machines

• Remote execution of commands (Remove all data in /temp)

• Managing configuration data

• Orchestration to ensure complex use cases for infrastructure/platform

• Server - Agent based (Called minion-master) though you can use it like

Ansible with salt-ssh module (With some limitations)



Command Line - Remote Execution

•What do you want to target – all web servers for ex.?

Target

• We are using test module and ping function - but there are many more

Module+Command

• In our case no argument - but typically there will be some arguments

ArgumentsHands

on!



State, Pillar, Grains & Formula

• State files is what defines what needs to be done (logic), the extension is

sls

• The configuration data parameters is what is stored in Pillar (data) –

extension is sls. This data is stored on server and provided to target

machines based on role, environment etc.

• There is another type of data item – grains. This is set on individual target machines,

some values are in built by Salt like os, cpu etc. and others can be custom built.

• State files and Pillar+Grain data together can be used to get to desired

state on target machines.

• Reusable modules/playbooks are called Formula

• Of course there are more components – but this is minimal to get going



Once more

• Pillar data is configuration data that is sent by master to

minions.

• Grains data is what minions produce and provide to

master about themselves.



(Some) Development & Debugging Techniques

• When you are developing a piece of logic for certain target (Let’s say for app-server), go

to that machine and run “sudo salt-call state.highstate” – this allowes better debugging.

Yon always add “–l debug” for much more information.

• “state.show_top” can be used to show what does the top state look like for a specific

node instead of guessing.

• In jinja files to inspect variables you can use “ {{ var_name|pprint }}” – based on type and

value of object, you will know what is going wrong. (I am not python guy – is there a

better way?)

• For watching events coming to master you can use command “salt-run state.event

pretty=True” – this will show what data is coming. What happens to event is purely

visible only in debug logs – so enable that in master/minion config.

Are there more techniques I might be missing?

Hands

on!



PEER COMMUNICATION IN SALT



Peer Communication 1/2

Use case• A minion wants to know some details or run some command on other minions

• Communicate between minions for some event coordination

• Get information of other minions for making certain decisions

Before• If you run publish command - it won't work out of the box, Salt by default does

not enable this:



Peer Communication 2/2

After

• You have to enable peer publishing for specific/all modules and minions to enable peer communication. Setting change in master config file:

• The output after enabling peer communication for previous command:

Security?

• What if any minions runs destructive commands using 'cmd.run' module on other minions?

Hands

on!



EVENT, REACTOR AND ORCHESTRATION



Event, Reactor and Orchestration

• Use case

• Events can be sent as part of workflow (Custom events), for example let’s say a

new node is up and needs to be added to load balancer config – so you can

send an event. Internally Salt uses events extensively.

• Reactor is what listens to those events and can invoke another workflow (Call a

SLS file or a formula) etc. You can filter for certain types of events, tags etc.

• Salt internal events can also be listened to and workflows initiated when an

event occurs.

• You can do everything in Reactor that you would do in a normal formula/SLS

call – so fairly complex states can be called as an reaction to certain event.

• You can also pass data with events – which can be retrieved in reactor

execution



SEND AND RECEIVE EVENT

• To send an event through an SLS file:

• To listen to an event and to decide what it should do, you need to add reactor.conf in

/etc/salt/master.d (Or inside master config file):

The tag of the event being sent

You can send data along with event and data can be dynamic as seen from some commands!

This config only tells which event to expect and what action to invoke – like a router sort of. Resides in the master config fil



Processing The EventThe data is within data – hence you see data.data to get data we just passed

We are calling local state - meaning state of that minion. You can call highstate or other things too

Targeting only certain minions based on data received from event!

Passing event data to the SLS so that it can be used there to make more decusions

Hands

on!



Some learnings from reactor

• Debugging reactor is difficult - enabling debug mode on master as well

minion is your best bet

• Be careful of filters you are applying in event sending/reactor

processing and changing state of your machines. Else events will be

filtered and you will wonder what happened?

• Start with a very basic model of working reactor without any conditions

- test it and then grow it.

• Ask on Salt-Users google group - the community is helpful!

• KISS - Keep It Simple Stupid



SALT MINE

Salina Turda, Romania

(https://en.wikipedia.org/wiki/Salina_Turda) is one of most

visited Salt mine with 2M+ visitors so far!

We are going to talk about a virtual version of the same ;)


https://en.wikipedia.org/wiki/Salina_Turda


Salt Mine

• Mine is like grains – but much more real time. Grains is more or less static data about a

minion. Mine can be used for things which are expected to change over time about a

machine/instance and are provided by machine.

• Mine is also a replacement for peer publishing – remember? (Peer publishing is relatively

slow, because it has to reach all nodes and gather data. So in that sense mine is like a

already built data for certain things you need instead of running publish)

Mine functions are run

every X duration and

data is sent to Master

Master stored only

latest data of every

mine execution

A minion which needs data

about other minions can get

using mine.get with

appropriate filters etc.



Salt mine - workingSchedule at which mine functions shoud be run (Goes on minion)

Mine function definition which goes inside Pillar!

Running mine.get will give you info you need!

Hands

on!



Salt Beacon



Salt Beacons

• Salt beacon can “watch” certain things and fire events (What you can watch is limited

things like service, user login, ps etc.).

• These events can be caught by event bus and can be used to invoke reactor actions.

• Some inbuilt actions are already provided - for example clean up the PID file if the

service goes down (In case someone runs kill -9 or service dies for some reason)

• Beacon config can be done in minion config file or via Pillar (We will use pillar).

• We will only watch event - not write reactor for now (Command to watch events: sudo

salt-run state.event pretty=true )

Hands

on!



Salt Syndic & Salt Multimaster



Salt Syndic, Multimaster

• We won’t cover much code in this and next topic, mostly functional capabilities

• Multi master

• One minion can connect to multiple masters and all of them can be ‘hot’

• On all masters, keys for minions must be accepted separately

• If the setting is set to failover - then minions connect to first master that response

in list

• Syndic:

• Salt syndics are basically masters but responsible for only a part of infrastructure. That way

you can set up multiple syndics to split load.

• Syndic eventually point to a master of master and is in sync

• Both of above topologies can be mixed to achieve a highly scalable, highly available

and fault tolerant infrastructure.



Salt Cloud



Salt Cloud

• Salt cloud enables you provisions systems and bring them under Salt management

immediately.

• Roughly on config side:

• Define a provider (LIke Rackspace, AWS etc.)

• Query provider to get options of region, images etc.

• Create profiles with above information

• Use profiles to create instances and query, destroy etc

• Map files are used to create/destroy machines at scale. Map files are basically

multiple machine information combined into a cluster like definition

• You can define grains etc. when creating instances so you can manage them as per

identity.



Questions/Suggestions?

Thank you for your time and attention


saltstack advanced concepts

Technology