a metadata ocean in chef and puppet

A metadata ocean in Puppet and Chef FOSDEM’14

Marc Cluet DevOps & Automation Advisory Rackspace

http://www.flickr.com/photos/elias_daniel/7470265444/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Who am I?!Marc Cluet (@lynxman)

More than 16 years of experience as a SysAdmin Founding member of Juju and MAAS while at Canonical Leading a DevOps Engineering Team at Rackspace Likes DevOps, programming and long walks on the beach

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

What is Rackspace?!Founded in 1998 in San Antonio, TX by three guys that wanted to create a hosting company Home of Fanatical Support /o/ Second biggest Public Cloud in the world OpenStack Project co-founder

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

What is Metadata?!

/http://www.flickr.com/photos/boojee/3743753784/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

What is Metadata?!

http://www.flickr.com/photos/ggvaidya/8256283184/

Metadata is “data about data”!Structural Metadata!Descriptive Metadata!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Metadata in Chef!Metadata is compiled on the client at execution time!Metadata is always stored in its entirety in the node[] dictionary!$node[var] = val

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Metadata in Chef!You can add metadata in your!

Attribute File!Node/Recipe!Environment!Role!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Organisation in Chef!

http://docs.opscode.com/essentials_environments.html

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Metadata in Puppet!Metadata is always compiled on execution timeMetadata vars can’t be reassigned!$::var = val

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Metadata in Puppet!You can add metadata in your!!Class vars!!Nodes!!Top Scope!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Scoping in Puppet!

http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Hiera in Puppet!Solves metadata fragmentation issues!Defines hierarchy for easy organisation!All in clean and properly styled in either YAML or JSON!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Hiera in Puppet!:backends: - yaml - json:yaml: :datadir: /etc/puppet/hieradata:json: :datadir: /etc/puppet/hieradata:hierarchy: - "%{::clientcert}" - "%{::custom_location}" - common

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/musebrarian/3289649684/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Bad metadata practices!In Chef!

Not keeping consistency between cookbook scope and environment scope!Not removing metadata from nodes when deprecated!Directly assigning metadata in a cookbook without a default attribute!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Bad metadata practices!In Puppet!

Directly assigning metadata in a class without a default value!Assigning vars in nodes.pp!Not assigning global vars in site.pp!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/deepstereo/9616389940/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Good behaviours!Separate vars from execution!

Do not store vars on the execution area (classes/cookbooks)!

High abstraction, good hierarchy!Metadata is all about context!

!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Good behaviours!

Global

Environment

Node

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/monica_andre/3119991125/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Metadata Across Providers!

Provisioner cloud-init

Config Mgmt Puppet

$dog = cute $::dog = big

WHO IS RIGHT?!?!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Metadata Across Providers!

http://www.flickr.com/photos/pss/233841567/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Hard Problem to Solve!

Metadata brain split problem!Gets worse by adding more players!

System view varies!Not same info from bios than from a user level program as facter or ohai!

!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Possible Solutions!Central DB using something fast!!Redis, mongoDB, mySQL!

Metadata translators might fix it?!!Write for everything!

Creates new SPoF!!Hard to solve without throwing $$!

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Hard Problem to Solve!

http://xkcd.com/927/

RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK

Possible Solutions!

Scope vars from top of chain!Pass them over as ext facts!$dog in cloud-init becomes $::dog in puppet!

27 RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.CO.UK

RACKSPACE® HOSTING | 5 MILLINGTON ROAD | HAYES, UNITED KINGDOM UB3 4AZ UK SALES: +44 (0)20 8712 6507 | UK SUPPORT: 0800 988 0300 | WWW.RACKSPACE.CO.UK

@lynxman!

http://slideshare.net/lynxmanuk/!

http://www.rackspace.co.uk/technicalservices/ !

!