puppetdb: new adventures in higher-order automation - puppetconf 2013

The life and times of

PuppetDB

Friday, August 23, 13

DEEPAK GIRIDHARAGOPALdeepak@puppetlabs.com@grim_radical

We need to talk!

Puppet agent

Puppet master

Puppet agent

Puppet master

Puppet agent

Puppet master

netmask_lo: 255.0.0.0 augeasversion: 0.10.0 fqdn: pe-debian6.localdomain manufacturer: "VMware, Inc." processorcount: "1" productname: VMware Virtual Platform physicalprocessorcount: 1 facterversion: 1.6.7 boardproductname: 440BX Desktop Reference Platform kernelmajversion: "2.6" hardwareisa: unknown timezone: PDT puppetversion: 2.7.12 (Puppet Enterprise 2.5.1) lsbdistcodename: squeeze is_virtual: "true" operatingsystemrelease: 6.0.2 virtual: vmware type: Other domain: localdomain hostname: pe-debian6 selinux: "false" kernel: Linux

kernelrelease: 2.6.32-5-686 ipaddress: 172.16.245.128 processor0: Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz lsbdistrelease: 6.0.2 uniqueid: 007f0101 hardwaremodel: i686 kernelversion: 2.6.32 operatingsystem: Debian architecture: i386 lsbdistdescription: Debian GNU/Linux 6.0.2 (squeeze) lsbmajdistrelease: "6" interfaces: "eth0,lo" ipaddress_lo: 127.0.0.1 uptime_days: 0 lsbdistid: Debian rubysitedir: /opt/puppet/lib/site_ruby/1.8 rubyversion: 1.8.7 osfamily: Debian memorytotal: &id001 502.57 MB memorysize: *id001 boardmanufacturer: Intel Corporation path: /usr/local/sbin:/usr/local/bin:/

Puppet agent

Puppet master

Puppet agent

Puppet master

Puppet agent

Puppet master

catalog

Puppet agent

Puppet master

catalog

file {“/tmp/foo”: content => “This is a test”}

target: &id063 !ruby/object:Puppet::Resource catalog: *id001 exported: false file: /etc/puppetlabs/puppet/manifests/site.pp line: 44 parameters: !ruby/sym content: This is a test !ruby/sym backup: main reference: "File[/tmp/foo]" tags: - file - node - default - class title: /tmp/foo type: File

file {“/tmp/foo”: content => “This is a test”}

Relationships

Exec[broker_cert_bundle]

File[/etc/puppetlabs/activemq/broker.pem]

Exec[broker_cert_pkcs12]

File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]

Service[mcollective]

File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]

File[/opt/puppet/share/puppet-dashboard/.bashrc]

Service[pe-activemq]

File[/etc/puppetlabs/mcollective/ssl]

File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]

File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]

File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]

Class[Pe_accounts::Data]

Anchor[pe_compliance::end]

File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]

File[/etc/puppetlabs/activemq/broker.ts]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]

Class[Settings] Class[Main]

Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard/.ssh]

Schedule[daily]

File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]

File[/var/lib/peadmin/.vim]

File[/etc/puppetlabs/mcollective/server.cfg]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]

Anchor[pe_accounts::begin]

Class[Pe_accounts::Groups]

Anchor[pe_accounts::end]

Filebucket[main]

File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]

File[/etc/puppetlabs/activemq/broker.ks]

Cron[pe-mcollective-metadata]

Class[Pe_mcollective]

Class[Pe_mcollective::Plugins]

Anchor[pe_mcollective::end]

File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]

File[/etc/puppetlabs/activemq/activemq.xml]

File[/etc/puppetlabs/mcollective/client.cfg]

Exec[mcollective-client-cert]

File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]

File[/opt/puppet/libexec/mcollective/mcollective/agent]

File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]

File[/etc/puppetlabs/activemq/activemq-wrapper.conf]

Schedule[never] Stage[main]Anchor[pe_mcollective::begin]

Class[Pe_mcollective::Posix]

Class[Pe_mcollective::Metadata]

File[/opt/puppet/libexec/mcollective/mcollective/util]

File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]

Pe_accounts::Home_dir[/var/lib/peadmin]

Exec[broker_cert_keystore]

Group[puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]

User[puppet-dashboard]

Schedule[weekly]

Exec[mcollective-server-cert] File[/var/lib/peadmin]

File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]

File[/var/lib/peadmin/.ssh/authorized_keys]

Class[Pe_accounts]

Exec[broker_cert_truststore]

Schedule[hourly]

Class[Pe_compliance::Agent]

Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]

Schedule[monthly] Filebucket[puppet]

Pe_accounts::User[peadmin]

File[/etc/puppetlabs/activemq/broker.p12]

Node[default]

Pe_accounts::User[puppet-dashboard]

Class[Pe_compliance]

File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]

File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]

File[/opt/puppet/libexec/mcollective/mcollective/security]

Group[peadmin]

User[peadmin]

File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]

File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]

Relationships

Schedule[daily]

Filebucket[main]

Schedule[weekly]

Class[Pe_accounts]

Schedule[hourly]

Node[default]

Group[peadmin]

User[peadmin]

Relationships

Schedule[daily]

Filebucket[main]

Schedule[weekly]

Class[Pe_accounts]

Schedule[hourly]

Node[default]

Group[peadmin]

User[peadmin]

Puppet agent

Puppet master

catalog

Puppet agent

Puppet master

catalog

Puppet agent

Puppet master

catalog

Puppet agent

Puppet master

report

Puppet agent

Puppet master

report

"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status change_count: 1 changed: true evaluation_time: 0.001869 events: - !ruby/object:Puppet::Transaction::Event audited: false desired_value: !ruby/sym file historical_value: message: *id006 name: !ruby/sym file_created previous_value: !ruby/sym absent property: ensure status: success time: 2011-10-25 18:51:37.143970 -07:00 failed: false file: *id007 line: 44 out_of_sync: true out_of_sync_count: 1 resource: "File[/tmp/foo]" resource_type: File skipped: false tags: - file - node - default - class time: 2011-10-25 18:51:37.143396 -07:00 title: /tmp/foo

Puppet agent

Puppet master

report

Puppet agent

Puppet master

report

Puppet agent

Puppet master

Puppet agent

Puppet master

Puppet agent

Puppet master PuppetDB

Puppet agent

catalog

Puppet agent

catalog

Puppet agent

catalog facts

Puppet agent

catalog facts

Puppet agent

report

catalog facts

Puppet agent

report

catalog facts

Puppet agent

report

catalog facts

Puppet agent

report

catalog facts

ActiveRecord

Puppet master

catalog

ActiveRecord

Puppet master

catalogcatalogcatalogcatalogcatalogcatalog

ActiveRecord

Puppet master

catalogcatalogcatalogcatalogcatalog catalog

ActiveRecord

Puppet master

catalogcatalogcatalogcatalog catalogcatalog

ActiveRecord

Puppet master

catalogcatalogcatalog catalogcatalogcatalog

ActiveRecord

Puppet master

catalogcatalog catalogcatalogcatalogcatalog

ActiveRecord

Puppet master

catalog catalogcatalogcatalogcatalogcatalog

ActiveRecord

Puppet master

catalog catalog

Puppet master

catalog

Puppet agent

ActiveRecord

Which boxes arerunning nginx?

ActiveRecord

How many serversare running a

vulnerable versionof rails?

ActiveRecord

What are the IPaddresses of my

webservers?

ActiveRecord

Which users have sudo access?

ActiveRecord

LOLWUT

ActiveRecord

LOLWUT

ಠ ಠ_

And now for something completely

different

PuppetDB

/resources/Service/nginx

PuppetDB

resources

/resources/Service/nginx

PuppetDB

/resources/Package/rails

PuppetDB

resources

/resources/Package/rails

PuppetDB

/nodes/foo.com/resources/User/

deepak

PuppetDB

resources

/nodes/foo.com/resources/User/

deepak

PuppetDB

(demo)

We built something quite different

1. Asynchrony

Storage &Querying

CommandQueryResponsibilitySeparation

use a different model to update information than the model you

use to read information

CQRSwrite pipeline

async, parallel, MQ-based, with automatic retry

{ :command "replace catalog" :version 2 :payload {...}}

/commands MQ Parse

Delayed

Dead Letter Office

Process

Command processors must be retry-aware

expect failure, because it *will* happen.

Failures like, oh I don't know,

a database crash?

2. New runtime

Fast,Free,Portable,Multi-core,Popular,

The JVM is all these thingsFriday, August 23, 13

Haters gonna hate!

Tons and tons of high quality libraries

Web servers, concurrency frameworks, databases, fast

parsing/lexing, clustering, debugging, profiling, etc.

Can ship an uberjar, makes deployment straightforward with few moving pieces

And it's fast.

Nobody cares what runtime we use. Users just want stuff to work.

3. AST querying

Queriesare expressed in their own “language”

domain specific, AST-based query language

["and", ["=", "type", "User"], ["=", "title", "deepak"]]

["and", ["=", ["fact", "operatingsystem"], "Debian"], ["<", ["fact", "uptime_seconds"], 10000]]

["and", ["=", "name", "ipaddress"], ["in", "certname", ["extract", "certname", ["select-resources", ["and", ["=", "type", "Class"], ["=", "title", "Apache"]]]]

["or", ["=", "certname", "foo.com"], ["=", "certname", "bar.com"], ["=", "certname", "baz.com"]]

We walk the tree, compiling it to efficient SQL

Haters gonna hate!

AST-based API lets users write their own languages

ah, you’ve got to love open source!

(Package[httpd] and country=fr)or country=us

Package["mysql-server"]and architecture=amd64

Erik Dalén, Spotifyhttps://github.com/dalen/puppet-puppetdbquery

AST-based API lets us more safely manipulate queries

daenny, Puppetboardhttps://github.com/nedap/puppetboard

Puppet Enterprise, Event Inspectorhttps://puppetlabs.com

Foreman Integration (CERN)https://github.com/cernops/puppetdb_foreman

Web UIhttps://github.com/dima-exe/puppetdb-db

Web UIhttps://github.com/gbougeard/puppetdb-frontend

Rubyhttps://github.com/dalen/puppet-puppetdbquery

Ruby (DataMapper)https://github.com/dalen/dm-puppetdb-adapter

Rubyhttps://github.com/ripienaar/ruby-puppetdb

Pythonhttps://github.com/nedap/pypuppetdb

Pythonhttps://github.com/arcus-io/puppetdb-python

Pythonhttps://github.com/JHaals/puppetdb-grep

Javahttps://github.com/thallgren/puppetdb-javaclient

Gohttps://github.com/nightlyone/puppetquery

Scalahttps://github.com/gbougeard/puppetdb-frontend

CoffeeScripthttps://gist.github.com/pmuellr/5591686

Node.jshttps://github.com/nightfly19/minidb

MCollectivehttps://github.com/ploubser/mcollective-puppetdb-discovery

Rundeckhttps://github.com/sirhopcount/puppetdb-rundeck

Rundeckhttps://github.com/martin2110/puppetdb-rundeck

OpenStackhttps://github.com/bodepd/puppet-openstack_puppetdb

Vagranthttps://github.com/grahamgilbert/vagrant-puppetmaster

PowerDNShttps://github.com/evenup/evenup-pdns

4. Boring technology

Relational Database, embedded or PostgreSQL

because they’re actually pretty fantastic at ad-hoc queries,

aggregation, windowing, etc. while maintaining safety

Relational Database, embedded or PostgreSQL

we use arrays, recursive queries, indexing inside complex

structures

5. Weird alien technology

-- Jeff Gagliardi

Thousands of deployments,Hundreds of threads per install,Zero deadlocks,Zero bugs involving mutable state

companion Ruby code has ~10x the defect rate

All with a pretty tiny codebase

6. Conjecturesabout performance

Posit:A resource often

exists across multiple hosts

Feature:Single-instance resource storage

Posit:We’ll often receive the

same catalog for a host

Feature:Single-instance catalog storage

In the field, we almost always see Resource and catalog duplication rates of over 85%.

Monitoring and instrumentation is a big deal. Users want easy ways to consume metrics and analyze performance.

Nagioshttps://github.com/jasonhancock/nagios-puppetdb

Nagioshttps://github.com/favoretti/puppetdb-external-naginator

Muninhttps://github.com/vpetersson/munin_puppetdb

Muninhttps://github.com/dalen/puppetdb-muninplugins

Collectdhttps://gist.github.com/mfournier/5615125

Turns out, people appreciate these

efforts

(how many?)

Thousands of production deployments

Small shops with a dozen hosts,large shops with thousands of hosts, standalone, clustered...

There is a new deployment of PuppetDB every15 minutes.

So...long time since we last spoke

Availability

Available in PE3

On by default, fully supported, and the basis for upcoming reporting and analytics features.

Performance

20% faster storage

Improvements to memoization and caching, eliminate double-serialization, nuked superfluous indexes

Much faster terminus

Better caching and data structures. For a catalog with 10k resources, drops serialization time from ~80s to ~6s.

Resiliance

Death to keystores

Can now use PEM certificates directly, eliminating one of the largest sources of configuration problems.

Configurable HTTPS

Can customize the set of cipher suites and SSL protocols you'd like to use, to match your security needs.

Automatic:

- Recovery from MQ corruption- Compression of the DLO- Purging of inactive node data- DB connection recycling

Backup and restore

Now integrated into the daemon, can restore while PuppetDB is running.

Query changes

V2 API

- No need to ask for only active nodes- Full fact queries (instead of just a list of facts for a node)- Node metadata

Wildcard Accept Headers

curl localhost:8080/v2/nodes

Subqueries

You can now correlate data from resource queries with fact queries with node queries.

"Give me the IP address of all machines with the Nginx service configured"

Report storage

- Comes with a report processing plugin- Store report-level metadata- Can do queries on events that span reports- Basis for PE's Event Inspector

Streamingqueries!

Streaming queries

Stream results to clients on-the-fly, as they come in from the database.

Massively lower latency for first response!

resourceresourceresourceresourceresourceresourceresource

PuppetDB

/v2/resources

PuppetDB

/v2/resources

PuppetDB

(demo)

Coming up!

We will be developing tools to replicate data from one PuppetDB daemon to another. This will help with HA and DR.

PuppetDBDiff &

Mirror PuppetDB

By initially developing an out-of-band mirroring tool, we can create more interesting replication topologies:

PuppetDBDiff &

Mirror PuppetDB

Diff & Mirror

We can also later optimize the process to lower latency, but preserve eventual consistency:

PuppetDB

Diff & Mirror

PuppetDBDirect MQ connection

More flexible routing is coming, allowing for soft failures and read/write splits:

PuppetDB

Puppetmaster

PuppetDB

Replication

Catalogs, Facts,Reports

Collectionqueries

Log error andcontinue

So anyways,

Documented athttp://docs.puppetlabs.com/puppetdb

install, config, upkeep, specs,the works!

Packagedas deb and rpm for open source, part of Puppet Enterprise

available in the Puppet Labspackage repositories

Puppetizedusing the puppetlabs/puppetdb module

available now, on theModule Forge!

Open source

http://github.com/puppetlabs/puppetdb

same license as Puppet itself!

deepakgiridharagopaldeepak@puppetlabs.com@grim_radical [github twitter freenode]

puppetdb: new adventures in higher-order automation - puppetconf 2013

puppet enterprise

custom filevarlibpeadmin

mcollective classpe

false file

keys pe

pedebian6 selinux

file node

debian gnulinux

Technology

juju + puppet (puppetconf 2011)

infrastructure as data - puppetconf 2013

node collaboration - exported resources and puppetdb

open sourcing the cloud - puppetconf 2013

managing and scaling puppet - puppetconf 2014

puppetconf track overview: windows

puppetconf track overview: inside puppet

a practical guide to modules - puppetconf 2014

auditing/security with puppet - puppetconf 2014

puppetconf 2014 killer r10k workflow with notes

package manages and puppet - puppetconf 2015

puppetconf 2013 types and providers

understanding openstack deployments - puppetconf 2014

puppet language 4.0 - puppetconf 2014

getting started with puppet - puppetconf 2013

puppet at github - puppetconf 2013

test driven infrastructure development - puppetconf 2013

puppet for windows users - puppetconf 2014

vagrant + rouster at salesforce.com - puppetconf 2013

node classifier fundamentals - puppetconf 2014