The life and times of
PuppetDB
Friday, August 23, 13
DEEPAK [email protected]@grim_radical
Friday, August 23, 13
We need to talk!
Friday, August 23, 13
Friday, August 23, 13
Friday, August 23, 13
Puppet agent
Puppet master
Friday, August 23, 13
Puppet agent
Puppet master
facts
Friday, August 23, 13
Puppet agent
Puppet master
facts
netmask_lo: 255.0.0.0 augeasversion: 0.10.0 fqdn: pe-debian6.localdomain manufacturer: "VMware, Inc." processorcount: "1" productname: VMware Virtual Platform physicalprocessorcount: 1 facterversion: 1.6.7 boardproductname: 440BX Desktop Reference Platform kernelmajversion: "2.6" hardwareisa: unknown timezone: PDT puppetversion: 2.7.12 (Puppet Enterprise 2.5.1) lsbdistcodename: squeeze is_virtual: "true" operatingsystemrelease: 6.0.2 virtual: vmware type: Other domain: localdomain hostname: pe-debian6 selinux: "false" kernel: Linux
kernelrelease: 2.6.32-5-686 ipaddress: 172.16.245.128 processor0: Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz lsbdistrelease: 6.0.2 uniqueid: 007f0101 hardwaremodel: i686 kernelversion: 2.6.32 operatingsystem: Debian architecture: i386 lsbdistdescription: Debian GNU/Linux 6.0.2 (squeeze) lsbmajdistrelease: "6" interfaces: "eth0,lo" ipaddress_lo: 127.0.0.1 uptime_days: 0 lsbdistid: Debian rubysitedir: /opt/puppet/lib/site_ruby/1.8 rubyversion: 1.8.7 osfamily: Debian memorytotal: &id001 502.57 MB memorysize: *id001 boardmanufacturer: Intel Corporation path: /usr/local/sbin:/usr/local/bin:/
Friday, August 23, 13
Puppet agent
Puppet master
facts
Friday, August 23, 13
Puppet agent
Puppet master
facts
Friday, August 23, 13
Puppet agent
Puppet master
catalog
Friday, August 23, 13
Puppet agent
Puppet master
catalog
Friday, August 23, 13
file {“/tmp/foo”: content => “This is a test”}
Friday, August 23, 13
target: &id063 !ruby/object:Puppet::Resource catalog: *id001 exported: false file: /etc/puppetlabs/puppet/manifests/site.pp line: 44 parameters: !ruby/sym content: This is a test !ruby/sym backup: main reference: "File[/tmp/foo]" tags: - file - node - default - class title: /tmp/foo type: File
file {“/tmp/foo”: content => “This is a test”}
Friday, August 23, 13
Relationships
Exec[broker_cert_bundle]
File[/etc/puppetlabs/activemq/broker.pem]
Exec[broker_cert_pkcs12]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
Service[pe-activemq]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
Anchor[pe_compliance::end]
File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]
File[/etc/puppetlabs/activemq/broker.ts]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Class[Settings] Class[Main]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.ssh]
Schedule[daily]
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]
Anchor[pe_accounts::begin]
Class[Pe_accounts::Groups]
Anchor[pe_accounts::end]
Filebucket[main]
File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
File[/etc/puppetlabs/activemq/broker.ks]
Cron[pe-mcollective-metadata]
Class[Pe_mcollective]
Class[Pe_mcollective::Plugins]
Anchor[pe_mcollective::end]
File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]
File[/etc/puppetlabs/activemq/activemq.xml]
File[/etc/puppetlabs/mcollective/client.cfg]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/etc/puppetlabs/activemq/activemq-wrapper.conf]
Schedule[never] Stage[main]Anchor[pe_mcollective::begin]
Class[Pe_mcollective::Posix]
Class[Pe_mcollective::Metadata]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[broker_cert_keystore]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Schedule[weekly]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Class[Pe_accounts]
Exec[broker_cert_truststore]
Schedule[hourly]
Class[Pe_compliance::Agent]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Schedule[monthly] Filebucket[puppet]
Pe_accounts::User[peadmin]
File[/etc/puppetlabs/activemq/broker.p12]
Node[default]
Pe_accounts::User[puppet-dashboard]
Class[Pe_compliance]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Friday, August 23, 13
Relationships
Exec[broker_cert_bundle]
File[/etc/puppetlabs/activemq/broker.pem]
Exec[broker_cert_pkcs12]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
Service[pe-activemq]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
Anchor[pe_compliance::end]
File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]
File[/etc/puppetlabs/activemq/broker.ts]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Class[Settings] Class[Main]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.ssh]
Schedule[daily]
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]
Anchor[pe_accounts::begin]
Class[Pe_accounts::Groups]
Anchor[pe_accounts::end]
Filebucket[main]
File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
File[/etc/puppetlabs/activemq/broker.ks]
Cron[pe-mcollective-metadata]
Class[Pe_mcollective]
Class[Pe_mcollective::Plugins]
Anchor[pe_mcollective::end]
File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]
File[/etc/puppetlabs/activemq/activemq.xml]
File[/etc/puppetlabs/mcollective/client.cfg]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/etc/puppetlabs/activemq/activemq-wrapper.conf]
Schedule[never] Stage[main]Anchor[pe_mcollective::begin]
Class[Pe_mcollective::Posix]
Class[Pe_mcollective::Metadata]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[broker_cert_keystore]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Schedule[weekly]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Class[Pe_accounts]
Exec[broker_cert_truststore]
Schedule[hourly]
Class[Pe_compliance::Agent]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Schedule[monthly] Filebucket[puppet]
Pe_accounts::User[peadmin]
File[/etc/puppetlabs/activemq/broker.p12]
Node[default]
Pe_accounts::User[puppet-dashboard]
Class[Pe_compliance]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Friday, August 23, 13
Relationships
Exec[broker_cert_bundle]
File[/etc/puppetlabs/activemq/broker.pem]
Exec[broker_cert_pkcs12]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]
Service[mcollective]
File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]
File[/opt/puppet/share/puppet-dashboard/.bashrc]
Service[pe-activemq]
File[/etc/puppetlabs/mcollective/ssl]
File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]
File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]
File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]
Class[Pe_accounts::Data]
Anchor[pe_compliance::end]
File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]
File[/etc/puppetlabs/activemq/broker.ts]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]
Class[Settings] Class[Main]
Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.ssh]
Schedule[daily]
File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]
File[/var/lib/peadmin/.vim]
File[/etc/puppetlabs/mcollective/server.cfg]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d]
File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]
Anchor[pe_accounts::begin]
Class[Pe_accounts::Groups]
Anchor[pe_accounts::end]
Filebucket[main]
File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]
File[/etc/puppetlabs/activemq/broker.ks]
Cron[pe-mcollective-metadata]
Class[Pe_mcollective]
Class[Pe_mcollective::Plugins]
Anchor[pe_mcollective::end]
File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]
File[/etc/puppetlabs/activemq/activemq.xml]
File[/etc/puppetlabs/mcollective/client.cfg]
Exec[mcollective-client-cert]
File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]
File[/opt/puppet/libexec/mcollective/mcollective/agent]
File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]
File[/etc/puppetlabs/activemq/activemq-wrapper.conf]
Schedule[never] Stage[main]Anchor[pe_mcollective::begin]
Class[Pe_mcollective::Posix]
Class[Pe_mcollective::Metadata]
File[/opt/puppet/libexec/mcollective/mcollective/util]
File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]
Pe_accounts::Home_dir[/var/lib/peadmin]
Exec[broker_cert_keystore]
Group[puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard]
File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]
User[puppet-dashboard]
Schedule[weekly]
Exec[mcollective-server-cert] File[/var/lib/peadmin]
File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]
File[/var/lib/peadmin/.ssh/authorized_keys]
Class[Pe_accounts]
Exec[broker_cert_truststore]
Schedule[hourly]
Class[Pe_compliance::Agent]
Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]
Schedule[monthly] Filebucket[puppet]
Pe_accounts::User[peadmin]
File[/etc/puppetlabs/activemq/broker.p12]
Node[default]
Pe_accounts::User[puppet-dashboard]
Class[Pe_compliance]
File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]
File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]
File[/opt/puppet/libexec/mcollective/mcollective/security]
Group[peadmin]
User[peadmin]
File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]
File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]
Friday, August 23, 13
Puppet agent
Puppet master
catalog
Friday, August 23, 13
Puppet agent
Puppet master
catalog
Friday, August 23, 13
Puppet agent
Puppet master
catalog
Friday, August 23, 13
Puppet agent
Puppet master
report
Friday, August 23, 13
Puppet agent
Puppet master
report
"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status change_count: 1 changed: true evaluation_time: 0.001869 events: - !ruby/object:Puppet::Transaction::Event audited: false desired_value: !ruby/sym file historical_value: message: *id006 name: !ruby/sym file_created previous_value: !ruby/sym absent property: ensure status: success time: 2011-10-25 18:51:37.143970 -07:00 failed: false file: *id007 line: 44 out_of_sync: true out_of_sync_count: 1 resource: "File[/tmp/foo]" resource_type: File skipped: false tags: - file - node - default - class time: 2011-10-25 18:51:37.143396 -07:00 title: /tmp/foo
Friday, August 23, 13
Puppet agent
Puppet master
report
Friday, August 23, 13
Puppet agent
Puppet master
report
Friday, August 23, 13
Puppet agent
Puppet master
Friday, August 23, 13
Puppet agent
Puppet master
Friday, August 23, 13
Friday, August 23, 13
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
catalog
facts
catalog
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
catalog
catalog
facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
catalog facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
catalog facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
report
catalog facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
report
catalog facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
report
catalog facts
Friday, August 23, 13
Puppet agent
Puppet master PuppetDB
report
catalog facts
Friday, August 23, 13
Friday, August 23, 13
Friday, August 23, 13
ActiveRecord
Puppet master
catalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalogcatalogcatalogcatalogcatalogcatalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalogcatalogcatalogcatalogcatalog catalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalogcatalogcatalogcatalog catalogcatalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalogcatalogcatalog catalogcatalogcatalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalogcatalog catalogcatalogcatalogcatalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalog catalogcatalogcatalogcatalogcatalog
Friday, August 23, 13
ActiveRecord
Puppet master
catalog catalog
Friday, August 23, 13
Puppet master
catalog
Friday, August 23, 13
Friday, August 23, 13
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Puppet agent
Friday, August 23, 13
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Which boxes arerunning nginx?
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
How many serversare running a
vulnerable versionof rails?
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
What are the IPaddresses of my
webservers?
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Which users have sudo access?
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
Friday, August 23, 13
ActiveRecord
LOLWUT
Friday, August 23, 13
ActiveRecord
LOLWUT
ಠ ಠ_
Friday, August 23, 13
And now for something completely
different
Friday, August 23, 13
PuppetDB
Friday, August 23, 13
/resources/Service/nginx
PuppetDB
Friday, August 23, 13
resources
/resources/Service/nginx
PuppetDB
)
O O
Friday, August 23, 13
PuppetDB
Friday, August 23, 13
/resources/Package/rails
PuppetDB
Friday, August 23, 13
resources
/resources/Package/rails
PuppetDB
)
O O
Friday, August 23, 13
PuppetDB
Friday, August 23, 13
/nodes/foo.com/resources/User/
deepak
PuppetDB
Friday, August 23, 13
resources
/nodes/foo.com/resources/User/
deepak
PuppetDB
)
O O
Friday, August 23, 13
(demo)
Friday, August 23, 13
We built something quite different
Friday, August 23, 13
1. Asynchrony
Friday, August 23, 13
Storage &Querying
Friday, August 23, 13
CQRS
Friday, August 23, 13
CommandQueryResponsibilitySeparation
use a different model to update information than the model you
use to read information
Friday, August 23, 13
CQRSwrite pipeline
async, parallel, MQ-based, with automatic retry
Friday, August 23, 13
{ :command "replace catalog" :version 2 :payload {...}}
Friday, August 23, 13
/commands MQ Parse
Delayed
Dead Letter Office
Process
UUID
Friday, August 23, 13
Command processors must be retry-aware
expect failure, because it *will* happen.
Friday, August 23, 13
Failures like, oh I don't know,
a database crash?
Friday, August 23, 13
2. New runtime
Friday, August 23, 13
Fast,Free,Portable,Multi-core,Popular,
The JVM is all these thingsFriday, August 23, 13
Haters gonna hate!
Friday, August 23, 13
Tons and tons of high quality libraries
Web servers, concurrency frameworks, databases, fast
parsing/lexing, clustering, debugging, profiling, etc.
Friday, August 23, 13
Can ship an uberjar, makes deployment straightforward with few moving pieces
Friday, August 23, 13
And it's fast.
Friday, August 23, 13
Nobody cares what runtime we use. Users just want stuff to work.
Friday, August 23, 13
3. AST querying
Friday, August 23, 13
Queriesare expressed in their own “language”
domain specific, AST-based query language
Friday, August 23, 13
["and", ["=", "type", "User"], ["=", "title", "deepak"]]
Friday, August 23, 13
["and", ["=", ["fact", "operatingsystem"], "Debian"], ["<", ["fact", "uptime_seconds"], 10000]]
Friday, August 23, 13
["and", ["=", "name", "ipaddress"], ["in", "certname", ["extract", "certname", ["select-resources", ["and", ["=", "type", "Class"], ["=", "title", "Apache"]]]]
Friday, August 23, 13
["or", ["=", "certname", "foo.com"], ["=", "certname", "bar.com"], ["=", "certname", "baz.com"]]
Friday, August 23, 13
We walk the tree, compiling it to efficient SQL
Friday, August 23, 13
Haters gonna hate!
Friday, August 23, 13
AST-based API lets users write their own languages
ah, you’ve got to love open source!
Friday, August 23, 13
(Package[httpd] and country=fr)or country=us
Package["mysql-server"]and architecture=amd64
Erik Dalén, Spotifyhttps://github.com/dalen/puppet-puppetdbquery
Friday, August 23, 13
AST-based API lets us more safely manipulate queries
Friday, August 23, 13
daenny, Puppetboardhttps://github.com/nedap/puppetboard
Friday, August 23, 13
Puppet Enterprise, Event Inspectorhttps://puppetlabs.com
Friday, August 23, 13
Foreman Integration (CERN)https://github.com/cernops/puppetdb_foreman
Web UIhttps://github.com/dima-exe/puppetdb-db
Web UIhttps://github.com/gbougeard/puppetdb-frontend
Friday, August 23, 13
Rubyhttps://github.com/dalen/puppet-puppetdbquery
Ruby (DataMapper)https://github.com/dalen/dm-puppetdb-adapter
Rubyhttps://github.com/ripienaar/ruby-puppetdb
Friday, August 23, 13
Pythonhttps://github.com/nedap/pypuppetdb
Pythonhttps://github.com/arcus-io/puppetdb-python
Pythonhttps://github.com/JHaals/puppetdb-grep
Friday, August 23, 13
Javahttps://github.com/thallgren/puppetdb-javaclient
Gohttps://github.com/nightlyone/puppetquery
Scalahttps://github.com/gbougeard/puppetdb-frontend
CoffeeScripthttps://gist.github.com/pmuellr/5591686
Node.jshttps://github.com/nightfly19/minidb
Friday, August 23, 13
MCollectivehttps://github.com/ploubser/mcollective-puppetdb-discovery
Rundeckhttps://github.com/sirhopcount/puppetdb-rundeck
Rundeckhttps://github.com/martin2110/puppetdb-rundeck
Friday, August 23, 13
OpenStackhttps://github.com/bodepd/puppet-openstack_puppetdb
Vagranthttps://github.com/grahamgilbert/vagrant-puppetmaster
PowerDNShttps://github.com/evenup/evenup-pdns
Friday, August 23, 13
4. Boring technology
Friday, August 23, 13
Relational Database, embedded or PostgreSQL
because they’re actually pretty fantastic at ad-hoc queries,
aggregation, windowing, etc. while maintaining safety
Friday, August 23, 13
Friday, August 23, 13
Relational Database, embedded or PostgreSQL
we use arrays, recursive queries, indexing inside complex
structures
Friday, August 23, 13
5. Weird alien technology
Friday, August 23, 13
Friday, August 23, 13
-- Jeff Gagliardi
Friday, August 23, 13
Thousands of deployments,Hundreds of threads per install,Zero deadlocks,Zero bugs involving mutable state
companion Ruby code has ~10x the defect rate
Friday, August 23, 13
All with a pretty tiny codebase
Friday, August 23, 13
6. Conjecturesabout performance
Friday, August 23, 13
Posit:A resource often
exists across multiple hosts
Friday, August 23, 13
Feature:Single-instance resource storage
Friday, August 23, 13
Posit:We’ll often receive the
same catalog for a host
Friday, August 23, 13
Feature:Single-instance catalog storage
Friday, August 23, 13
In the field, we almost always see Resource and catalog duplication rates of over 85%.
Friday, August 23, 13
Monitoring and instrumentation is a big deal. Users want easy ways to consume metrics and analyze performance.
Friday, August 23, 13
Friday, August 23, 13
Nagioshttps://github.com/jasonhancock/nagios-puppetdb
Nagioshttps://github.com/favoretti/puppetdb-external-naginator
Muninhttps://github.com/vpetersson/munin_puppetdb
Muninhttps://github.com/dalen/puppetdb-muninplugins
Collectdhttps://gist.github.com/mfournier/5615125
Friday, August 23, 13
Turns out, people appreciate these
efforts
Friday, August 23, 13
(how many?)
Friday, August 23, 13
Thousands of production deployments
Small shops with a dozen hosts,large shops with thousands of hosts, standalone, clustered...
Friday, August 23, 13
There is a new deployment of PuppetDB every15 minutes.
Friday, August 23, 13
So...long time since we last spoke
Friday, August 23, 13
Availability
Friday, August 23, 13
Available in PE3
On by default, fully supported, and the basis for upcoming reporting and analytics features.
Friday, August 23, 13
Performance
Friday, August 23, 13
20% faster storage
Improvements to memoization and caching, eliminate double-serialization, nuked superfluous indexes
Friday, August 23, 13
Much faster terminus
Better caching and data structures. For a catalog with 10k resources, drops serialization time from ~80s to ~6s.
Friday, August 23, 13
Resiliance
Friday, August 23, 13
Death to keystores
Can now use PEM certificates directly, eliminating one of the largest sources of configuration problems.
Friday, August 23, 13
Configurable HTTPS
Can customize the set of cipher suites and SSL protocols you'd like to use, to match your security needs.
Friday, August 23, 13
Automatic:
- Recovery from MQ corruption- Compression of the DLO- Purging of inactive node data- DB connection recycling
Friday, August 23, 13
Backup and restore
Now integrated into the daemon, can restore while PuppetDB is running.
Friday, August 23, 13
Query changes
Friday, August 23, 13
V2 API
- No need to ask for only active nodes- Full fact queries (instead of just a list of facts for a node)- Node metadata
Friday, August 23, 13
Wildcard Accept Headers
curl localhost:8080/v2/nodes
Friday, August 23, 13
Subqueries
You can now correlate data from resource queries with fact queries with node queries.
"Give me the IP address of all machines with the Nginx service configured"
Friday, August 23, 13
Report storage
- Comes with a report processing plugin- Store report-level metadata- Can do queries on events that span reports- Basis for PE's Event Inspector
Friday, August 23, 13
Streamingqueries!
Friday, August 23, 13
Streaming queries
Stream results to clients on-the-fly, as they come in from the database.
Massively lower latency for first response!
Friday, August 23, 13
resourceresourceresourceresourceresourceresourceresource
PuppetDB
Friday, August 23, 13
resourceresourceresourceresourceresourceresourceresource
/v2/resources
PuppetDB
Friday, August 23, 13
resourceresourceresourceresourceresourceresourceresource
/v2/resources
PuppetDB
)
O O
Friday, August 23, 13
(demo)
Friday, August 23, 13
Coming up!
Friday, August 23, 13
We will be developing tools to replicate data from one PuppetDB daemon to another. This will help with HA and DR.
PuppetDBDiff &
Mirror PuppetDB
Friday, August 23, 13
By initially developing an out-of-band mirroring tool, we can create more interesting replication topologies:
PuppetDBDiff &
Mirror PuppetDB
Diff & Mirror
Friday, August 23, 13
We can also later optimize the process to lower latency, but preserve eventual consistency:
PuppetDB
Diff & Mirror
PuppetDBDirect MQ connection
Friday, August 23, 13
More flexible routing is coming, allowing for soft failures and read/write splits:
PuppetDB
Puppetmaster
PuppetDB
Replication
Catalogs, Facts,Reports
Collectionqueries
Log error andcontinue
Friday, August 23, 13
So anyways,
Friday, August 23, 13
Documented athttp://docs.puppetlabs.com/puppetdb
install, config, upkeep, specs,the works!
Friday, August 23, 13
Packagedas deb and rpm for open source, part of Puppet Enterprise
available in the Puppet Labspackage repositories
Friday, August 23, 13
Puppetizedusing the puppetlabs/puppetdb module
available now, on theModule Forge!
Friday, August 23, 13
Open source
http://github.com/puppetlabs/puppetdb
same license as Puppet itself!
Friday, August 23, 13
[email protected]@grim_radical [github twitter freenode]
Friday, August 23, 13