a metadata ocean in chef and puppet
DESCRIPTION
What is Metadata and how it is represented in Chef and Puppet? What are the challenges for metadata sharing across providers and possible solutions.TRANSCRIPT
A metadata ocean in Puppet and Chef FOSDEM’14
Marc Cluet DevOps & Automation Advisory Rackspace
http://www.flickr.com/photos/elias_daniel/7470265444/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Who am I?!Marc Cluet (@lynxman)
More than 16 years of experience as a SysAdmin Founding member of Juju and MAAS while at Canonical Leading a DevOps Engineering Team at Rackspace Likes DevOps, programming and long walks on the beach
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
What is Rackspace?!Founded in 1998 in San Antonio, TX by three guys that wanted to create a hosting company Home of Fanatical Support /o/ Second biggest Public Cloud in the world OpenStack Project co-founder
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
What is Metadata?!
/http://www.flickr.com/photos/boojee/3743753784/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
What is Metadata?!
http://www.flickr.com/photos/ggvaidya/8256283184/
Metadata is “data about data”!Structural Metadata!Descriptive Metadata!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Metadata in Chef!Metadata is compiled on the client at execution time!Metadata is always stored in its entirety in the node[] dictionary!$node[var] = val
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Metadata in Chef!You can add metadata in your!
Attribute File!Node/Recipe!Environment!Role!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Organisation in Chef!
http://docs.opscode.com/essentials_environments.html
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Metadata in Puppet!Metadata is always compiled on execution timeMetadata vars can’t be reassigned!$::var = val
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Metadata in Puppet!You can add metadata in your!!Class vars!!Nodes!!Top Scope!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Scoping in Puppet!
http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Hiera in Puppet!Solves metadata fragmentation issues!Defines hierarchy for easy organisation!All in clean and properly styled in either YAML or JSON!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Hiera in Puppet!:backends: - yaml - json:yaml: :datadir: /etc/puppet/hieradata:json: :datadir: /etc/puppet/hieradata:hierarchy: - "%{::clientcert}" - "%{::custom_location}" - common
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/musebrarian/3289649684/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Bad metadata practices!In Chef!
Not keeping consistency between cookbook scope and environment scope!Not removing metadata from nodes when deprecated!Directly assigning metadata in a cookbook without a default attribute!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Bad metadata practices!In Puppet!
Directly assigning metadata in a class without a default value!Assigning vars in nodes.pp!Not assigning global vars in site.pp!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/deepstereo/9616389940/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Good behaviours!Separate vars from execution!
Do not store vars on the execution area (classes/cookbooks)!
High abstraction, good hierarchy!Metadata is all about context!
!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Good behaviours!
Global
Environment
Node
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK http://www.flickr.com/photos/monica_andre/3119991125/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Metadata Across Providers!
Provisioner cloud-init
Config Mgmt Puppet
$dog = cute $::dog = big
WHO IS RIGHT?!?!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Metadata Across Providers!
http://www.flickr.com/photos/pss/233841567/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Hard Problem to Solve!
Metadata brain split problem!Gets worse by adding more players!
System view varies!Not same info from bios than from a user level program as facter or ohai!
!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Possible Solutions!Central DB using something fast!!Redis, mongoDB, mySQL!
Metadata translators might fix it?!!Write for everything!
Creates new SPoF!!Hard to solve without throwing $$!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Hard Problem to Solve!
http://xkcd.com/927/
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
Possible Solutions!
Scope vars from top of chain!Pass them over as ext facts!$dog in cloud-init becomes $::dog in puppet!
27 RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.CO.UK
RACKSPACE® HOSTING | 5 MILLINGTON ROAD | HAYES, UNITED KINGDOM UB3 4AZ UK SALES: +44 (0)20 8712 6507 | UK SUPPORT: 0800 988 0300 | WWW.RACKSPACE.CO.UK
@lynxman!
http://slideshare.net/lynxmanuk/!
http://www.rackspace.co.uk/technicalservices/ !
!