Saltstack For DevOpsExtremely fast and simple IT automation andconfiguration management

Aymen El Amri

This book is for sale at http://leanpub.com/saltstackfordevops

This version was published on 2018-04-29

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishingprocess. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools andmany iterations to get reader feedback, pivot until you have the right book and build traction onceyou do.

© 2015 - 2018 Aymen El Amri

Tweet This Book!Please help Aymen El Amri by spreading the word about this book on Twitter!

The suggested tweet for this book is:

I just bought SaltStack For DevOps book. Check it out : https://leanpub.com/saltstackfordevops cc@Salt4DevOps

The suggested hashtag for this book is #Salt4DevOps.

Find out what other people are saying about the book by clicking on this link to search for thishashtag on Twitter:

#Salt4DevOps

Also By Aymen El AmriThe Jumpstart Up

Painless Docker

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Every Book Has A Story, This Story Has A Book . . . . . . . . . . . . . . . . . . . . . . 2To Whom Is This Book Addressed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Conventions Used In This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4How To Properly Enjoy This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4How To Contribute To This Book? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5About The Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Configuration Management And Data Center Automation . . . . . . . . . . . . . . . . . 7DevOps Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12A brief summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Expanding Salt Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Configuration And Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Configuration Of A Masterless Minion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Configuration of salt-master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Configuration of salt-minion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Post Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20General Troubleshooting and Prerequisites Checklist . . . . . . . . . . . . . . . . . . . . 22Troubleshooting salt-master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Troubleshooting salt-minion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Troubleshooting Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Using salt-call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23A Note About Redhat Enterprise Linux 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

CONTENTS 1

Dedication

I dedicate this to my Mother and my Father who were always there for me.

Preface

Every Book Has A Story, This Story Has A Book

I wanted to quickly resign frommy job, my suggestions about working on a continuous delivery andsetting up a deployment pipeline have not been considered by most of my colleagues! Like many ofus, I’m lazy when it comes to repeating tasks manually but passionate when I automate them.

I love automation and in my job, there are hundreds of configuration files and thousands of variablesto copy from text files then to adjust in some platforms, a huge number of poorly-configured serversand hundreds of servers to manage by a very small team.. I wanted to work on the automation ofsome weekly procedures. I was aware that this is a good solution but it was not the priority neitherfor the manager nor for the client.

It was a position within a team of 14 people working on the integration of a number of heavyapplications (mainly Java/Oracle, php/Mysql, Nginx, Python/Jython) with a complex architecture,tens of versions/environments to manage and an infrastructure covering Europe.

In the beginning, I was obliged to follow my boss guidelines and the work methods my team hasadopted which have one goal: satisfying as fast as possible the unceasing demands of the client.

No, but .. wait, this is not good at all!

I spent almost two weeks searching and working on some solutions before I convinced my boss togive me the time to set up an application prototype that will ease the heavy load, accelerate dailyprocedures and reduce human errors.

First, I created a configuration management tool using Python/Sqlite3, automated tests usingSelenium/Python among other procedures I have set up.

Someweeks later, I started learning SaltStack and found it later a good solution to replace my “home-made” applications. It meets the expectations of the integration process, or rather, the continuousintegration, deployments and automatic tests.

I hesitated between several alternatives: Puppet, CFEngine, Chef .. etc. The choice was made basedon several criteria (I was looking for a robust, fast configuration management and remote executiontool that everybody can use without learning a new programming language (YAML) ) .. I have neverregretted my choice.

I found some difficulties and honestly when I started learning Salt, the official documentation wasnot very detailed.

Preface 3

This book is a fruit of long hours of work and self-learning.

Well, in the beginning, I wanted to resign from my job, just a few days after discovering of Salt, Iwas in love with my work, with what I was doing and with what I am learning.

I tried Salt first time when I saw my team taking more than 3 days (sometimes more) to configurehosted platforms at each deployment (we had more than 10 environments per application).

After setting it up, the same procedure was taking less than 1/2 hour.

Through this book, it’s your turn to discover SaltStack, I will be your guide.

I wish you a pleasant reading.

To Whom Is This Book Addressed?

To developers, system administrators and anyone working in one of these teams in collaborationwith the other or simply in an environment that requires knowledge in development, softwareintegration, and system engineering.

Usually, developers think that they are here to serve the machine, or to feed it each period of timewith a fresh code to deploy.

The machine is hungry and should eat, otherwise, it will stop working and the boss will be angry.

System administrators think that machines should be happy.

Feeding the machine will make it sick and angry. Especially with obsolete codes. It will stop workingand the boss will be angry too.

This is an ironical way to describe the conflicts between developers and system administrators butin many cases, it’s true.

Moreover, within the same company there is generally some tension between the two teams:

System administrators accuse developers to write code that consumesmemory, does notmeet systemsecurity standards or not adapted to available machines configuration.

Developers accuse system administrators to be lazy, to lack innovation and to be seriously uncool!

No more mutual accusations, now with the evolution of software development, infrastructure andadopted methodologies (such as Scrum, XP, Kanban), the rise of DevOps culture is the result.

DevOps is more a philosophy and a culture than a job (even if some of the positions I had werecalled “DevOps Engineer” or “DevOps Architect”).

By admitting this, this job seeks closer collaboration and a combination of different roles involvedin software development such as the role of developer, responsible for operations and responsiblefor quality assurance.

The software must be produced at a frenetic pace while at the same time the developing in cascadeseems to have reached its limits.

Preface 4

• If you are a fan of configuration management, automation, and the DevOps culture• If you are a system administrator working on DevOps, SysOps, CloudOps, ResearchOps ..well, Ops in general

• If you are a developer seeking to join the new movement

This book is addressed to you.

Configuration management software are one the most used tools in DevOps environments.

If you are new to configuration management software, this book is also addressed to beginners.

Conventions Used In This Book

Basically, this is a technical book where you will find commands (SaltStack commands) and code(Python, YAML, Jinja2 ..etc).

Commands and code are written in a different format.

Example :

1 python -c 'import urllib; print urllib.urlopen("https://bootstrap.SaltStack.com"\

2 ).read()' | \ sudo sh -s -- git develop

• This book uses italic font for technical words such as libraries, modules, languages names.The goal is to get your attention when you are reading and help you identify them.

• You will find two icons, I have tried to be as simple as possible so I have chosen not to use toomany symbols, you will only find:

To highlight useful and important information.

To highlight a warning or to prevent.

How To Properly Enjoy This Book

This book contains technical explanations and shows in each case an example of a command or aconfiguration to follow.

The explanation gives you a general idea and the code that follows gives you convenience and helpyou to practice what you are reading.

Preface 5

Preferably, you should always look both parts for a maximum of understanding.

Like any new tool or programming language you learned, it is normal to find difficulties andconfusions in the beginning, perhaps even after.

If you are not used to learning new technologies, you can even have a modest understanding whilebeing in an advanced stage of this book. Do not worry, everyone has passed at least once by thiskind of situations.

At the beginning try to make a diagonal reading while focusing on the basic concepts, then try thefirst practical manipulation on your server or just using your laptop/desktop and occasionally comeback to this book for further reading about a specific subject or concept.

This book is neither an encyclopedia nor a documentation but it sets out the most important partsto learn and master, if you find words or concepts that you are not comfortable with, take your timeand do your own online research.

Learning can be serial so understanding a topic requires the understanding of another one.

Through this course, you will learn how to install configure and use SaltStack.

Just before finishing it, you will go through a chapter where good examples of practical use casesare explained.

Through these chapters, try to showcase your acquired understanding, and no, it will not hurt to goback to previous chapters if you are unsure or in doubt.

Finally, try to be pragmatic and have an open mind if you encounter a problem.

The resolution begins by asking the right questions.

How To Contribute To This Book?

This work will be always in progress.

I am an adopter of the lean philosophy so the content will be continuously improved in function ofmany criterions but the most important one is your feedback.

If you have any suggestions please do not hesitate to contact me, you can find me on Twitter1.

This book is not perfect, so you can find a typo here, some punctuation errors there or missing words.

Every line of the used code was tested before but you may find some errors during your learningpath due to a difference of software versions, environment configurations or similar issues.

1https://twitter.com/eon01

Preface 6

About The Author

Aymen is a Cloud Architect, Entrepreneur, Author of Best Selling Trainings and Books, AwardsWinner, CEO of Eralabs2 (A DevOps & Cloud Consulting & Training Company) and Founder ofDevOpsLinks Community3.

He actually lives in Paris and helps companies and startups from everywhere (Europe, US ..) developmodern applications, builds multi-tenant cloud infrastructures, scalable applications, highly stableproduction environments, distributed systems and service-oriented architectures (microservices &PaaS).

You can find Aymen on Twitter4 and join his newsletters DevOpsLinks5 and Shipped6.

For more online training and courses, please visit eralabs.io7.

2http://eralabs.io3http://devopslinks.com4https://twitter.com/eon015http://devopslinks.com6http://shipped.devopslinks.com7http://eralabs.io

Introduction

1 \ ^__^

2 \ (--)\_______

3 (__)\ )\/\

4 ||----w |

5 || ||

Configuration Management And Data CenterAutomation

Configuration management and data center automation are mechanisms to manage a technicaldescription of a system, its components and changes in its configurations.

This happens during the life cycle of a system or during its different processes. Source codedeployment on one or multiple environments is a change and it requires the right configurationmanagement and provisioning tool.

An ecosystem composed of development, test, QA and production environments is an example of aslightly complex ecosystem that can be composed many servers, sometimes thousands or hundredsof servers.

From the development, integration, test, and staging, to the production environment, configurationmanagement becomes a process of normalization of the application configuration according to thestate of infrastructure and functional components and other requirements. The same process ofnormalization should be ensured when changing the environment, the infrastructure or when theconfiguration itself changes.

Imagine that your organization is developing some applications with dependencies between them:databases, different configurations between developer’s, staging and production environments..etc

To accomplish this project with a good time-to-market, you may need tools to:

• A tool to rebuild the same server quickly from provisioned images• Generate dynamic configurations for the same application in different environments• Execute commands on thousands or hundreds of servers from one central machine• Create advanced monitoring and reactors systems

Introduction 8

• Recover quickly from disasters• etc.

Configuration management tools like Salt are useful to the use cases quoted below and can do morethan this.

In fact, with the adoption of agile development methods, the process of development; test anddeployment of a software component has accelerated, therefore methods of management havebecome faster, more automated and more adapted to changes.

DevOps Evolution

Even if many specialists consider provisioning, change management and automation a businessmatter, not an IT one but to make this happen, some special technical skills are required. That’s whynew positions, teams or rather culture in the IT industry have emerged: DevOps.

The illustration below (taken from Wikipedia) shows the essence of the DevOps philosophy.

DevOps as the intersection of development (software engineering), technology operations and quality assurance(QA)

Automation is important to the success of critical IT processes that are part of the life cycle ofa product, including provisioning, change management, release management, patch management,compliance, and security. Therefore, having the right technical skills is important to any “lazy butpragmatic SysAdmin”.

Introduction 9

This book will help you to learn one of the most known IT automation configuration managementand infrastructure automation/orchestration tools.

DevOps Tooling

Currently, several FOSS and proprietary automation and configuration management tools exist.Choosing one of these tools is based on several criteria.

Choice Criteria

• Performance : Among memory consumption, the speed of execution and adaptation toincreasingly complexes architectures, several performance criteria could help you decide tochoose the right tool.

• License : You may choose between FOSS and proprietary software. Most of the existingsoftware are Open Source. It remains to be seen what FOSS license you should choose: GPL,BSD, Apache, MIT..etc

• Programming Language : A such tool is coded using a programming language, but it does notmean that users will manage and automate operations and servers using the same language.For example, SaltStack is written in Python but its users use Jinja and YAML ..etc

Most of the tools are written in Python, Ruby, Java or Golang, but one can also find perl, C and C++ based tools.

• Authentication Methods : A configuration management or a data center automation tool isbased on a model, roughly consisting of clients and a server. The authentication between aclient and a server can be automatic, encrypted, secure, fast .. or not.

• Agents : Some tools use agents that must be installed on target servers (clients), some toolsdo not require remote agents and others offer both choices.

• Scalability : A tool that grows and evolves with the enterprise must provide technical meansand capabilities to ensure scalability at several features and extended functional scopes.

• Portability : Most if not all popular configuration management tools are compatible with nixsystems. Some servers run on *BSD, AIX, HP-UX,Mac OS, Solaris,Windows and other OSs. Inthis case, you must study compatibility.

Introduction 10

AIX BSD HP-UX

Linux MacOS X

Solaris WindowsOthers

Ansible Yes Yes Yes Yes Yes Yes Yes YesBcfg Partial Yes No Yes Partial Yes No NoCFEngineYes Yes Yes Yes Yes Yes Yes Yescdist Yes Yes Yes NoChef Yes Yes Yes Yes Yes Yes Yes YesISconf Yes Yes Yes Yes Yes Yes No NoJuju YesLCFG No No No Partial Partial Partial No NoOCSInven-toryNG

Yes Yes Yes Yes Yes Yes Yes No

Opsi No No No Yes No No Yes NoPIKT Yes Yes Yes Yes Yes Yes No YesPuppet Yes Yes Yes Yes Partial Yes Yes YesQuattor No No No Yes Partial Yes No NoRadmind Yes Yes No Yes Yes Yes Yes NoRex Yes Yes Yes Yes Yes NoRudder Yes Partial No Yes Partial Partial Yes YesRundeck Yes Yes Yes Yes Yes Yes Yes NoSmartFrogNo No Yes Yes Yes Yes Yes NoSalt Yes Yes Partial Yes Yes Yes Yes PartialSpacewalkNo No No Yes No Yes No NoSTAF Yes Yes Yes Yes Yes Yes Yes YesSynctool Yes Yes Yes Yes Yes Yes Yes YesVagrant Yes Yes Yes

Thanks to Wikipedia8 and its contributors for this comparison concerning the portability of thefollowing tools.

• Documentation, Support, and Latest Stable Release: Keep in mind that the quantity andthe quality of the official documentation, forums, groups, and paid support differs from atool to another. A good thing to do is to see the date of the latest stable release, some toolsare no more updated which can cause security risks, lack of support, bugs, and problems ofintegrations with other tools.

Popular tools

Among the popular tools we can find : Ansible, CFEngine, Puppet/Chef and SaltStack.

8http://en.wikipedia.org/wiki/Comparison_of_open-source_configuration_management_software

Introduction 11

Illustration 2: Popular configuration management and automation tools

Ansible : Combines multi-node deployment and ad-hoc task execution. It manages nodes with SSHand requires Python (2.4 or later). It uses JSON and YAML modules and states as descriptions. It isbuilt on Python but its modules can be written in any language. Ansible9 is used by Spotify, Twitter,NASA, and Evernote.

Puppet : Puppet10 is based on a custom declarative language to describe the system configuration, ituses a distributed client-server paradigm and a library for configuration tasks. Puppet requires theinstallation of a master server and client agents on every system that needs to be managed. It is usedby Vmware, Cisco, Paypal, and SalesForce.

SaltStack : Salt11 is what the next chapters of this book will detail. It is used by Rackspace,Photobucket, Nasa, LinkedIn, Hulu, HP Cloud Services, Harvard University, CloudFlare ..etc

9http://www.ansible.com/home10http://puppetlabs.com/11http://saltstack.com/

Getting Started

1 \ ^__^

2 \ (--)\_______

3 (__)\ )\/\

4 ||----w |

5 || ||

Presentation

Salt is an Open Source project, you can read and modify its source code under the Apache license.Its source code is available on github12.

SALSTACK Inc. is the company behind Salt, it was founded by Thomas Hatch, the original creatorof SaltStack.

Salt is used by Apple Inc, Rackspace, Photobucket, NASA, LinkedIn, Hulu, HP Cloud Services, CloudFlare and other know companies.

SaltStack Logo

Salt fundamentally improves the way SysAdmins, integrators, and DevOps configure and manageall aspects of a modern data center infrastructure.

12https://github.com/saltstack/salt

Getting Started 13

It provides a different approach to some existing alternatives such as speed and adaptation to thesize of a cloud. Several recognized businesses use SaltStack to orchestrate and control their cloudservers and infrastructure and automate the “DevOps Toolchain”.

It is built on a platform running relatively fast while enabling remote-controlling distributed infras-tructures, code and data. A layer of security is established while having two-way communicationbetween the different components of the platform (masters, minions ..etc).

The following chapters are conceived for beginners and experienced system administrators, DevOpsprofessionals and developers seeking to manage and configure multiple servers/applications andsoftware platforms easily.

The infrastructure to manage can be on-premise virtual machines, cloud (Amazon EC2 instances,Rackspace ..etc), containers (e.g. Docker) or bare-metal machines as well as hosted applications andplatforms that rely on configuration files.

All you need is a root access, a good understanding of the environment to manage and some basicknowledge (command line, basic commands, Linux ..etc).

Even if it is possible to use a web access to manage Salt but the use of the command line is alwaysmore adapted to our needs for several reasons such as speed and efficiency. If you are familiar withCLIs, understanding Salt commands and its syntax will be easier.

Salt is portable and works with these systems:

• Amazon Linux 2012.09• Arch, CentOS 5/6• Debian 6.x/7.x/8(git installations only)• Fedora 17/18• FreeBSD 9.1/9.2/10• Gentoo• Linaro• Linux Mint 13/14• OpenSUSE 12.x• Oracle Linux 5/5• Red Hat 5/6• Red Hat Enterprise 5/6• Scientific Linux 5/6• SmartOS• SuSE 11 SP1/11 SP2• Ubuntu 10.x/11.x/12.x/13.04/13.10• Elementary OS 0.2

Getting Started 14

According to the official website, other systems and distributions will be compatible in the future.If you want to stay informed just follow the development branches13.

In the following sections, we will be most of the time using Linux. If you are using Macos, there isno real differences for the installation but if you are usingWindows, I recommend using Ubuntu onWindows, this is optional if you would like to keep usingWindows shell.

Windows users, please follow this guide14 to install Ubuntu and Bash.

You can find some differences in the installation of Salt according to your system:Windows, FreeBSDor Solaris..etc. Overall, principles and usage are the same.

You can use Salt installed on an operating system to manage other systems (A Linux to manage aSolaris or a BSD to manage aWindows … etc.).

The installation part of this book will cover Redhat and Debian.

Be sure to check the documentation (docs.saltstack.com) for the installation and the specific use ofyour particular operating system.

A brief summary

SaltStack is based on some special components:

• One or more salt-master, salt-minion and salt-syndic• A key management system salt-key that allows the authentication of a salt-minion on a salt-master

• A system of states to describe configurations• A top.sls that calls states• A system of grain on the minion to manage configurations data• A system of pillars to store other data on the master (such as confidential data)• A transport and data management system called ZeroMQ• An event management system called reactors• Other components like returners, outputters ..etc

A master can manage configurations or execute remote commands on one or more minions. Theseoperations are based on SLS files, and these files are calling Salt modules, grains and/or pillars.

Salt could be used either from the command line (Salt CLI) or in executable scripts (Salt API).

The various components of SaltStack will be explained in this book, some definitions appeal others,that’s why we need - in the first order - to have a global view about Salt.

13https://github.com/saltstack/salt/branches/all14https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/

Getting Started 15

Expanding Salt Use

Some provisioning and testing tools are based on Salt, you may find some or all of them interesting.SaltStack can be interfaced with different cloud providers, integrations tools, and containers .. etc.

Some of the possible integrations are :

Salt + Vagrant

Salty Vagrant is a Vagrant plugin that allows you to use Salt as a provisioning tool. You can useformulas and existing configurations to create and build development environments.

The simplest way to use Salty Vagrant is configuring it to work in masterless mode. Details areexplained in the official Vagrant documentation15.

Through this book, you will learn how to interface Vagrant with Salt in order to automate theprovisioning of virtual machines.

Salt Cloud

Salt Cloud16 is a public cloud provisioning tool created to integrate Salt to the major cloudinfrastructure providers (AWS, Rackspace, GCP, Azure ..etc) in order to facilitate and acceleratethe supply process.

Salt Cloud allows managing a cloud infrastructure based on maps and profiles of virtual machines.This means that many virtual machines in the cloud can be managed easier and faster.

We will see in this book how to integrate Salt with some Cloud providers (e.g. Linode, AWS)

Halite

Halite17 is the client-side web interface (Salt GUI). It connects and operates a SaltStack infrastructure.This tool is a graphical complement, but it is not indispensable for the functioning of Salt. For bestresults, Halite works with Hydrogen and higher versions.

 Salt + Docker

Salt can be used to provision and manage Docker containers (Dockerng / Dockerio). We are goingto see together how to configure and manage Docker containers using Salt.

15https://docs.vagrantup.com/v2/provisioning/salt.html16https://github.com/saltstack/salt-cloud17https://github.com/saltstack/halite

Getting Started 16

Conclusion

The general presentation of Salt is not enough to start using Salt, but it is required if you are notfamiliar with concepts like configuration management and data center automation.

Configuration And Troubleshooting

1 \ ^__^

2 \ (--)\_______

3 (__)\ )\/\

4 ||----w |

5 || ||

Introduction

In the remainder of this guide, we will use a Linux virtual machine where we will install the salt-master and the salt-minion (in the same machine). Basic concepts are explained here so that therewill be practically neither functional nor conceptual differences between having master and minionin the same machine and having them in two different machines.

The only differences that might exist are easily noticeable such as the configuration of IP addresses(interfaces).

For the configuration, we will follow these key steps:

• The configuration of the salt-master• The configuration of the salt-minion• The configuration of salt-key• Functional tests• Diagnostics and troubleshooting

Configuration files for master and minion are in the default configuration directory:

1 /etc/salt

Configuration Of A Masterless Minion

To allow a minion execute Salt commands without a master, the option file_client must beconfigured.

By default, it is set to the remote execution mode so that a minion consults a master before runningany command.

file_client shall be set to local. In the minion configuration file /etc/salt/minion find and uncommentthis line:

Configuration And Troubleshooting 18

1 file_client: remote

Now change it to local.

1 file_client: local

You can find more details in the salt-minion section of the 4th lesson.

Configuration of salt-master

In the configuration file /etc/salt/master the default address for the listening interface (binding)is set to 0.0.0.0:

1 interface: 0.0.0.0

You can adapt it to your specific needs.

e.g: You can change it and restrict it to your localhost (127.0.0.1 or 0.0.0.0):

1 interface: 127.0.0.1

Make sure that your system settings are well configured, think of hosts file for instance.

In our case we will keep the default value of the interface, which means 0.0.0.0 and defaults portsvalues:

1 publish_port: 4505

2 ret_port: 4506

publish_port is the network port and ret_port is the port of the execution and the feedback ofminions.

Same thing for file_roots, we are just keeping its default value:

Configuration And Troubleshooting 19

1 file_roots:

2 base:

3 - /srv/salt/

The base environment is always required and must contain the top file. It is basically the entry pointfor Salt and will be used if no environment is specified. This will be detailed later.

Keep in mind that it is obligatory to configure your base environment.

To deliver files to minions, Salt uses a lightweight file server written in ZeroMQ (or 0mq). This fileserver is running on the master and requires a dedicated port.

The file server uses environments declared in the master configuration file. Each environment canhave multiple root directories:

e.g :

1 file_roots:

2 base:

3 - /srv/salt/

4 dev:

5 - /srv/salt/dev/subfolder1/

6 - /srv/salt/dev/subfolder2/

7 prod:

8 - /srv/salt/prod/subfolder1

9 - /srv/salt/prod/subfolder2

A base environment must have a top file top.sls.

For the sake of simplicity, even if we need to configure different environments, we will keep a singleenvironment in the master configuration file.

1 file_roots:

2 base:

3 - /srv/salt/

Instead, we will focus on how these environments are actually implemented and to do that we willset a state tree for each one of them. We are going to have multiple state trees designating each oneof our environments.

For the time being, restart the master:

Configuration And Troubleshooting 20

1 service salt-master restart

Configuration of salt-minion

The address of master machine should be configured in the minion configuration file /etc/salt/min-ion.

1 master: 127.0.0.1

Keep in mind that the value of master_port should coincide with the value of ret_portpreviously configured on the master.

A minion may have a nickname, you are free to change it by uncommenting id: and changing itsvalue. By default, the minion id/nickname is the name of the machine (in Linux, it is the same nameobtained with hostname command).

This value is stored in the file:

1 /etc/salt/minion_id

Once all settings are made, it is recommended to restart the daemon:

1 service salt-minion restart

Post Installation

A very important thing to do after the installation and before the configuration is to let the masteraccept the minion.

To do this, you should use salt-key.

Salt-key is the public key management system server. A master can communicate with a miniononly when the public key of the latter is processed and accepted by salt-key module.

Salt automatically handles the key generation for running minions, the server will be notified of thepresence of a minion via salt-key.

A minion can have three different states:

Configuration And Troubleshooting 21

• **Accepted **: The public key of the minion is accepted and thus registered on the server.• **Unaccepted **: The public key of the minion has not been accepted yet.• **Rejected **: This is usually a problem of installation or configuration.

After setting up the master and the minion, type the following command to list allminions and theirassociated states:

1 salt-key -L

or

1 salt-key --list all

Accepted minions are ready to receive orders from the master. You can also use the same commandfollowed by the following parameters and the id of the minion:

1 salt-key -a <name_of_the_minion>

The last command adds a minion to the list of accepted minions.

You can also use the following parameter if you want to accept all of the configured minions:

1 salt-key -A

Or

1 salt-key --accept-all

e.g:

To add a minion mynode we should type the following command:

1 salt-key -a 'mynode'

name_of_the_minion andmynode are used as examples. You should change it by the nameof the minion you are using. Remember salt-key -L can help you list the names of all minions.

Configuration And Troubleshooting 22

General Troubleshooting and Prerequisites Checklist

SaltStack offers many possibilities18 to diagnose a problem.

We are going to see the most commonly used methods for commons problems.

Before launching the master and the minion(s), a list of prerequisites should be checked:

• All of your salt-master and salt-minion are installed with the satisfied dependencies• Python version is compatible with the version of SaltStack• salt-master and salt-minion should be running with the root user• Communication and execution ports are well configured and opened• The configuration of any firewall between amaster and aminion should be taken into account(e.g. checking your iptables).

The first test to do is to ping the minion from the master machine:

1 salt <minion_id> test.ping

To ping all the accepted minion use:

1 salt "*" test.ping

A True should appear before each minion that answers.

Troubleshooting salt-master

The first diagnostic step is to launch the salt-master process in debug mode :

1 salt-master -L debug

The output of this command is very important because it contains helpful information about themajority of problems you can encounter when using Salt.

Troubleshooting salt-minion

Like salt-minion, salt-master can be also executed in debug mode.

18http://docs.saltstack.com/en/latest/topics/troubleshooting/

Configuration And Troubleshooting 23

1 salt-minion -L debug

The output will be the set of debug, warning and error messages.

Troubleshooting Ports

For salt-master, both TCP ports 4505 and 4506 should be opened. By running debug mode,information about these ports should be seen in the debug output.

For salt-minion, there are no specific ports to be opened but connectivity must be established:

1 nc -v -z salt.master.ip.address 4505

2 nc -v -z salt.master.ip.address 4506

In the case where a salt-master and a salt-minion are installed in two different production machines,most of the cases your server is behind a firewall (e.g. Security Groups in AWS), check that it allowsthe establishment of the connection between them.

On a Redhat system, iptables must allow the following configuration:

1 -A INPUT -m state --state new -m tcp -p tcp --dport 4505 -j ACCEPT

2 -A INPUT -m state --state new -m tcp -p tcp --dport 4506 -j ACCEPT

If you have troubles with setting your firewall using iptable file, consider using ufw19

(Uncomplicated FireWall).

Using salt-call

salt-call is a good way to debug. The command can be used for example with the states or highstates.

e.g:

1 salt-call -l debug state.highstate

When salt-call is executed on aminion, no salt-master is called during the execution. This commandallows the local execution, development assistance, and more verbose output.

If you want more verbosity, increase the log level.

e.g:

19http://en.wikipedia.org/wiki/Uncomplicated_Firewall

Configuration And Troubleshooting 24

1 salt-call -l debug state.highstate

Like it is said in the SaltStack official documentation:

“The main difference between using salt and using salt-call is that salt-call is run fromthe minion, and it only runs the selected function on that minion. By contrast, salt is runfrom the master, and requires you to specify the minions on which to run the commandusing salt’s targeting system.”

A Note About Redhat Enterprise Linux 5

Salt works with Python 2.6 or 2.7 while on RHEL5, Python 2.4 is installed by default. When youinstall Salt from Git, it is recommended to install its dependencies via EPEL and run Salt executablewith Python26 usually found here: /usr/bin/python26