MultiMaster scaling for multiple regions

Greg Cockburn @gergnz

problem:

How do we provide a Puppet Service Globally When WAN pipes suck

what's in our tool box?

VMware ESX LDAP

F5 Load Balancers Puppet Enterprise Edition

Items that need to be addressed •  Puppet Certificate management

•  Node Classification and ENC replication

•  Master Replication

•  Master Availability

•  Master Scalability

•  Reporting and notifications

•  Change Control

One Solution that Worked

Build a Puppeteer:

• This is a Puppet Master Master

• No Client Access

• Acts as a PuppetCA

• Central Point of Entry for Code Updates

• Ensures that the Puppet Masters are in sync

LDAP as an ENC: •  Existing highly available UNIX/Linux backbone service

•  Already replicated to every region

•  Masters are configured to speak with their nearest LDAP

replica

•  Provides an effective audit trail

•  Node definitions are abstracted away from the Puppet

manifests

Replicating Puppet Configuration: •  The Puppet Master is effective at syncing files

•  Use the Puppet Fileserver to replicate the masters o  manifests

o  modules

o  files

o  templates

•  The Puppeteer can 'kick' the other masters to force a run

•  Create a puppet::master class to ensure, masters are

fully controlled

F5 Global Traffic Management (GTM) & DNS:

• Local Puppet Master addresses are returned to

clients based on the DNS server the request

came from

•  If a Master is down then next nearest is returned

• Any Puppet Master globally can answer the

client

F5 Local Traffic Management (LTM):

• On sites with heavy loads this can be used to

rapidly scale the local Puppet Master service

•  If a local Master is taken out of service F5 will

automatically send you to the nearest local

Master

All Tied Together:

Workflow – Adding a New Server

•  Define the client characteristics in the LDAP ENC (eg.

Datacentre, Environment, Server Flavour)

•  Configure the build tools

•  PXE boot then server, OS is installed and puppet

bootstraps

•  Once the client certificate is signed the server is

configured

Workflow (adding a master):

•  Build a 'standard' client

•  Redefine in ENC (LDAP) as a puppetmaster

•  Destroy local certificates

•  generate special certificates on puppetmaster using --

dns_alt_names

•  rerun puppet and Master configurations will sync down

So What’s New:

Since this configuration was deployed Puppet Labs have

been busy:

•  Puppet Sites - Will soon be released and addresses a lot

of the issues here

•  PuppetDB – The new standard for stored configs

Special thanks to Jon Spinks @ Sourced Group

Sourced Group are a Puppet Labs partner providing integration services for Puppet

Enterprise Edition

Q & A

Please go and bother Jon Spinks to find out what Sourced have been doing with Puppet to automate Amazon Web

Services