puppet db: higher-order puppet - deepak giridharagopal - puppetcamp la '12

PuppetDBHigher-order Puppet

Deepak GiridharagopalLead Engineer @ Puppet Labs

[email protected]_radical, #puppet

Monday, May 21, 12

mailto:[email protected]


Let’s talk aboutdata

Monday, May 21, 12

Monday, May 21, 12

Data!Puppet generates a lot of it, in many delicious flavors!

Persisted, ephemeral, machine local, centralized, meticulously structured, totally free-form, human readable, machine optimized...

Monday, May 21, 12

Catalogs“The Graph”

Containment edges, dependency edges, classes, tags, resources, resource parameters, metadata

Monday, May 21, 12

target: &id063 !ruby/object:Puppet::Resource catalog: *id001 exported: false file: /etc/puppetlabs/puppet/manifests/site.pp line: 44 parameters: !ruby/sym content: This is a test !ruby/sym backup: main reference: "File[/tmp/foo]" tags: - file - node - default - class title: /tmp/foo type: File

file {“/tmp/foo”: content => “This is a test”}

Monday, May 21, 12

Relationships

Exec[broker_cert_bundle]

File[/etc/puppetlabs/activemq/broker.pem]

Exec[broker_cert_pkcs12]

File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb]

Service[mcollective]

File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem]

File[/opt/puppet/share/puppet-dashboard/.bashrc]

Service[pe-activemq]

File[/etc/puppetlabs/mcollective/ssl]

File[/etc/puppetlabs/mcollective/ssl/clients]File[mcollective-cert.pem] File[mcollective-public.pem]File[mcollective-private.pem]

File[peadmin-public.pem]File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[puppet-dashboard-public.pem]

File[/var/lib/peadmin/.mcollective] File[/opt/puppet/share/puppet-dashboard/.mcollective]

Class[Pe_accounts::Data]

Anchor[pe_compliance::end]

File[/opt/puppet/share/puppet-dashboard/.ssh/authorized_keys]

File[/etc/puppetlabs/activemq/broker.ts]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem]

Class[Settings] Class[Main]

Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard/.ssh]

Schedule[daily]

File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem]

File[/var/lib/peadmin/.vim]

File[/etc/puppetlabs/mcollective/server.cfg]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d]

File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem]

Anchor[pe_accounts::begin]

Class[Pe_accounts::Groups]

Anchor[pe_accounts::end]

Filebucket[main]

File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb]

File[/etc/puppetlabs/activemq/broker.ks]

Cron[pe-mcollective-metadata]

Class[Pe_mcollective]

Class[Pe_mcollective::Plugins]

Anchor[pe_mcollective::end]

File[credentials] Cron[report_baseline]File[/opt/puppet/sbin/refresh-mcollective-metadata]Exec[broker_cert]

File[/etc/puppetlabs/activemq/activemq.xml]

File[/etc/puppetlabs/mcollective/client.cfg]

Exec[mcollective-client-cert]

File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]

File[/opt/puppet/libexec/mcollective/mcollective/agent]

File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl]File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb]

File[/etc/puppetlabs/activemq/activemq-wrapper.conf]

Schedule[never] Stage[main]Anchor[pe_mcollective::begin]

Class[Pe_mcollective::Posix]

Class[Pe_mcollective::Metadata]

File[/opt/puppet/libexec/mcollective/mcollective/util]

File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb]

Pe_accounts::Home_dir[/var/lib/peadmin]

Exec[broker_cert_keystore]

Group[puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim]File[/opt/puppet/share/puppet-dashboard/.bashrc.custom]

User[puppet-dashboard]

Schedule[weekly]

Exec[mcollective-server-cert] File[/var/lib/peadmin]

File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.bash_profile]File[/var/lib/peadmin/.bashrc]File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.ssh]

File[/var/lib/peadmin/.ssh/authorized_keys]

Class[Pe_accounts]

Exec[broker_cert_truststore]

Schedule[hourly]

Class[Pe_compliance::Agent]

Exec[puppet-dashboard-client-cert]File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb]

Schedule[monthly] Filebucket[puppet]

Pe_accounts::User[peadmin]

File[/etc/puppetlabs/activemq/broker.p12]

Node[default]

Pe_accounts::User[puppet-dashboard]

Class[Pe_compliance]

File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb]

File[/tmp/foo] Schedule[puppet]Anchor[pe_compliance::begin]

File[/opt/puppet/libexec/mcollective/mcollective/security]

Group[peadmin]

User[peadmin]

File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb]

File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb]

Monday, May 21, 12

FactsEverything detected by facter

Facts for hardware, software, networking, CPUs, memory, virtualization, manufacturer info, custom facts...

Coming soon: structured facts!

Monday, May 21, 12

netmask_lo: 255.0.0.0 augeasversion: 0.10.0 fqdn: pe-debian6.localdomain manufacturer: "VMware, Inc." processorcount: "1" productname: VMware Virtual Platform physicalprocessorcount: 1 facterversion: 1.6.7 boardproductname: 440BX Desktop Reference Platform kernelmajversion: "2.6" hardwareisa: unknown timezone: PDT puppetversion: 2.7.12 (Puppet Enterprise 2.5.1) lsbdistcodename: squeeze is_virtual: "true" operatingsystemrelease: 6.0.2 virtual: vmware type: Other domain: localdomain hostname: pe-debian6 selinux: "false" kernel: Linux

kernelrelease: 2.6.32-5-686 ipaddress: 172.16.245.128 processor0: Intel(R) Core(TM) i7-2635QM CPU @ 2.00GHz lsbdistrelease: 6.0.2 uniqueid: 007f0101 hardwaremodel: i686 kernelversion: 2.6.32 operatingsystem: Debian architecture: i386 lsbdistdescription: Debian GNU/Linux 6.0.2 (squeeze) lsbmajdistrelease: "6" interfaces: "eth0,lo" ipaddress_lo: 127.0.0.1 uptime_days: 0 lsbdistid: Debian rubysitedir: /opt/puppet/lib/site_ruby/1.8 rubyversion: 1.8.7 osfamily: Debian memorytotal: &id001 502.57 MB memorysize: *id001 boardmanufacturer: Intel CorporationMonday, May 21, 12

ReportsCatalogs say what you want, reports say what you got.

Desired state, actual state, events, duration, timestamps...

Monday, May 21, 12

"File[/tmp/foo]": !ruby/object:Puppet::Resource::Status change_count: 1 changed: true evaluation_time: 0.001869 events: - !ruby/object:Puppet::Transaction::Event audited: false desired_value: !ruby/sym file historical_value: message: *id006 name: !ruby/sym file_created previous_value: !ruby/sym absent property: ensure status: success time: 2011-10-25 18:51:37.143970 -07:00 failed: false file: *id007 line: 44 out_of_sync: true out_of_sync_count: 1 resource: "File[/tmp/foo]" resource_type: File skipped: false tags: - file - node - default - class time: 2011-10-25 18:51:37.143396 -07:00 title: /tmp/foo

Monday, May 21, 12

Why bother?

Monday, May 21, 12

“There's a war out there, old friend. A world war. And it's not about who's got the most bullets. It's about who controls the information. What we see and hear, how we work, what we think... it's all about the information!”

-- Sneakers Monday, May 21, 12

StoreconfigsCentralized storage of the configuration of all your nodes.

All resources, all parameters, all classes, all tags, all stages...

Enables use of exported resources

Monday, May 21, 12

class exporter { @@file { "/var/lib/puppet/nodes/$fqdn": content => "$ipaddress\n", tag => "ip" }}

node "export1.daysofwonder.com" { include exporter}

node "export2.daysofwonder.com" { include exporter}

node "collector.daysofwonder.com" { File <<| tag == "ip" |>>}

http://www.masterzen.fr/2009/03/08/all-about-puppet-storeconfigs/

Monday, May 21, 12



public key distributionmonitoring checksclustered services

master/slave replicationload balancers

shared filesystemsfirewall rules

...Monday, May 21, 12

QueryInterrogation, investigation, correlation

Use Puppet-generated data in scripts or for integration with other tools

Monday, May 21, 12

Higher order

PuppetMonday, May 21, 12

VolumeEvery node, on every puppet run, generates data

We have customers generating over 750G of data a day. Even storing a small subset of that much information adds up...

Monday, May 21, 12

(demo)

Monday, May 21, 12

Slow = :(When data storage is slow, it makes baby Deepak cry!

Slows down catalog compilation,More quickly saturates a Puppetmaster,Thrashes disk,Bad news!

Monday, May 21, 12

APICurrent APIs are limited

Hard to get at the data, and performance concerns discourage use.

We need better ways of searching, filtering, and correlating data.

Monday, May 21, 12

ParadoxSeemingly contradictory goals

We want to store as much data as we can, and allow for better querying, but without slowing stuff down or reducing reliability.

Monday, May 21, 12

We needAn information clearinghouse

Something that evolves the Puppet Data Library. A scalable, safe place to store the information Puppet collects and generates.

This is a hard problem!

Monday, May 21, 12

PuppetDBDefinitely Better!

Monday, May 21, 12

Grayskull

Monday, May 21, 12

PuppetDB

Monday, May 21, 12

PuppetDB isFast storage of current catalogs and current facts,

100% compatible with storeconfigs and inventory service,

REST APIs for resource, fact, and node retrieval,

...and other things, even!

Monday, May 21, 12

science&

secret alien technology!

Monday, May 21, 12

Message Queue

"new catalog""new facts"

"delete node"

Storeconfigs, Catalogs, Facts

(SCF)Domain objects

Command HandlerParsing

Transformation

Validation

Query handling

REST

Puppetmaster

Compiler

Storeconfigs

Puppet Enterprise Console

CLI & Other Tools

"inventory query""interactive query"

"new catalog""new facts"

"delete node"

Monday, May 21, 12

(export)

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts Catalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

F C

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

F

C

C

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

C

C

Monday, May 21, 12

(collection)

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts Catalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

F

?

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

F?

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

F

?

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

F

C

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

F

C

C

Monday, May 21, 12

(failure)

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts Catalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

FCatalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

F C

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

F

C

C

Monday, May 21, 12

Agent Master

PuppetDB Server

HTTP MQ

DB Workers

DLO

Facts

F

Catalog Resrc

C

C

Monday, May 21, 12

PuppetDB Server

HTTP MQ

DB Workers

DLO

Monday, May 21, 12

PuppetDB Server

HTTP MQ

DBWorkers DLO

Monday, May 21, 12

PuppetDB Server

HTTP MQ

DBWorkers DLOHTTPProxy(SSL)

Monday, May 21, 12

(launch)

Monday, May 21, 12

Reliable!We work very hard to persist everything we accept

Acknowledgements with UUIDS,Checksums,Queueing,Automatic retry and reconnect,and the Dead Letter Office if all else fails!

Monday, May 21, 12

APIs!We don’t cheat

Anything Puppet does with PuppetDB, you can do to

Query your own resources, upload new fact sets, create catalogs, inspect facts...all part of the Puppet Data Library

Monday, May 21, 12

#> curl -H "Accept: application/json" "http://puppetdb/metrics/mbean/ com.puppetlabs.puppetdb.command:type=global,name=processing-time"

{ "50thPercentile": 209.05, "75thPercentile": 236.5865, "95thPercentile": 428.3065999999959, "98thPercentile": 750.53696, "999thPercentile": 1246.722744999993, "99thPercentile": 818.9180600000001, "Count": 3322, "EventType": "calls", "FifteenMinuteRate": 1.1500295609205015e-06, "FiveMinuteRate": 1.387569444096042e-18, "LatencyUnit": "MILLISECONDS", "Max": 26514.032, "Mean": 314.1111032510536, "MeanRate": 0.21577717049577358, "Min": 185.53, "OneMinuteRate": 3.390107448865515e-90, "RateUnit": "SECONDS", "StdDev": 833.6079354075728}

Monday, May 21, 12

https://localhost:8081/metrics/mbean/com.puppetlabs.puppetdb.command:type=global,name=processing-time




curl-H "Accept: application/json"

"http://puppetdb/facts/host.my.net"

Monday, May 21, 12



curl-H "Accept: application/json"

"http://puppetdb/resources?query=..."

Monday, May 21, 12



Transparent!We care about operational visibility

Ships with a real-time dashboard,Dozens of metrics and gauges,Correlate-able logs,Easy to integrate with monitoring systems

Monday, May 21, 12

Speedy!PuppetDB is much, *much* faster than the previous storeconfigs and inventory services

At Puppet Labs, we’ve seen huge reductions in compile times, resource collection times, time to persist catalogs and facts, etc.

Monday, May 21, 12

Design decisions

Monday, May 21, 12

Posit:Hosts are not

entirely unique snowflakes

Monday, May 21, 12

Therefore:A resource often

exists across multiple hosts

Monday, May 21, 12

Feature:Single-instance

resource storage

Monday, May 21, 12

Resource dedupeCompute unique hashes for resources

We quickly hash all the resources in a catalog, and use bulk operations to compare them to hashes stored.

Monday, May 21, 12

Resource dedupeSignificant speed improvement!

Internal to Puppet Labs, we see ~83% resource duplication; this number is consistent with what we’ve seen in most customer environments.

Monday, May 21, 12

Posit:Puppet runs

frequently, but catalogs change

infrequentlyMonday, May 21, 12

Therefore:We’ll often receive

the same catalog for a host

Monday, May 21, 12

Feature:Single-instance catalog storage

Monday, May 21, 12

Catalog dedupeCompute unique hashes for catalogs

We use a Merkle Tree approach (hash tree) for quick comparisons.

Puppet Labs sees ~88% catalog duplication

Big savings!

Monday, May 21, 12

Posit:You have more than

one core, though storeconfigs is

single-threadedMonday, May 21, 12

Therefore:Throughput is not

maximized

Monday, May 21, 12

Feature:Massively parallel

operation

Monday, May 21, 12

ParallelWe can pat our heads and rub our tummies at the same time

Database operations don’t block MQ operations don’t block HTTP operations don’t block hash computation operations don’t block metric calculations don’t block...

Dozens of threads, zero locks

Monday, May 21, 12

Monday, May 21, 12

science&

secret alien technology!

Monday, May 21, 12

The Future

Monday, May 21, 12

http://github.com/puppetlabs/

puppetdb

Monday, May 21, 12

http://github.com/puppetlabs/puppetdb






http://docs.puppetlabs.com/

puppetdb

Monday, May 21, 12







Use it, and tell us about it!

Monday, May 21, 12





PuppetDBThanks for your time!

Deepak GiridharagopalLead Engineer @ Puppet Labs

[email protected]_radical, #puppet

Monday, May 21, 12



puppet db: higher-order puppet - deepak giridharagopal - puppetcamp la '12

Technology