Download - Puppet at Spotify
April 11, 2023
Erik Dalén
Puppet at Spotify
2Section name
● Over 24 million monthly active users● Launched in 28 countries● Over 20 million songs● More than 1 billion playlists
Growing quickly
Spotify
System Engineer in Site Reliability Engineering at Spotify
Operational system owner for Puppet, playlist system and Cassandra
Community contributor to Puppet
whoami
● More than 450 changes per month● 220 committers to our Puppet git repository● 325 puppet modules
Code review by SRE team using Gerrit
Puppet users since 3 years
Puppet Infrastructure
● Roughly 5500 nodes
● 3 different Puppet installations
● Each with their own CA and PuppetDB
● One or more puppetmasters per data centre
● Run using cron
Git branch = = Puppet environment
Everyone can push to private branches and run puppet against those
Code review mandatory to push to “production” branch
Puppet Infrastructure
Built as Debian Packages
Deployed using Puppet
Backend services
●Client connections are proxied through the accesspoints●Most other backend services are stateless●Storage in Cassandra, PostgreSQL or Tokyo Cabinet
Architecture overview
Accessp
oint
Service 1 DB
Service 2 DB
Service 3
●Puppet module for the service●Deployed and tested in test environment●Hardware requested from SRE team and service
deployed in production
Backend service deployment
Using SRV records to discover services
Puppet module dalen-dnsquery can be used to lookup them from inside Puppet manifests.
Service Discovery
At the moment using different ENCs in different Puppet installations
Will be switching to using Hiera for node classification
Node Classification
Hostname standard:
sto1-puppet-a1
<site>-<role>-<pool><poolid>
Mainly role fact used to determine the classes a machine should have
Node Classification
25 pull requests from Spotify merged to core Puppet and Facter in last 12 months.
Many improvements to puppetlabs modules sent upstream.
Puppet contributions
Tired of writing queries like this?
["and", ["in", "name", ["extract", "certname", ["select-resources", ["and", ["=", "exported", false], ["=", "type", "Class"], ["=", "title", "Cassandra"], ["=", ["parameter", "version"], "1.1"]]]]], ["in", "name", ["extract", "certname", ["select-facts", ["and", ["=", "name", "site"], ["=", "value", "lon"]]]]]]
Querying PuppetDB
Finding nodes using dalen-puppetdbquery:
$ puppet query nodes ‘Class[Cassandra]{version=“1.1”} and site=lon’
$ puppet query nodes ‘processorcount > 16 and manufacturer ~”Dell.*”’
github.com/dalen/puppet-puppetdbquery
Querying PuppetDB
Use PuppetDB as a backend to the datamapper ORM
Node.get(’foo.example.com’).facts.each do |fact|
puts “#{fact.name}: #{fact.value}”
end
dm-puppetdb-adapter
A Puppet face to list files managed by puppet
# puppet ls /etc/systemd/system
nagios-nrpe-server.service
declared in /etc/puppet/environments/production/modules/systemd/manifests/unit.pp:15
content from a "content" parameter
puppet ls
● Splitting the repo out● Remove SRE review requirement on large parts● Support testing using vagrant● Building images using a masterless puppet apply
The future
Consists of a node terminus and a forge implementation
Builds a per node environment dynamically on demand
Will be open sourced Real Soon Now™
Spikor
Questions?
April 11, 2023
Check out spotify.com/jobs or @Spotifyjobs for more information.
Want to join the band?