hashicorp @ just eat - part 2
TRANSCRIPT
JUST EAT“the world’s leading marketplace for online food delivery”
- Amazon Web Services
- Run hundreds of EC2 Instances at peak
- Customers Worldwide
- Microservices > 100
- Multiple accounts, environments
- Using HashiCorp tools in since 2015
Consul - Initial Rollout
- Cluster Per Environment
- Seed key/values from GitHub (git2consul)
- Expand keys : true
- Every EC2 Instance runs Consul Agent
- Each microservice registers a service and has custom ACL
- Used Consul-Template to generate .config files
Consul Key/ValuesFeature Configuration
/config/features/<featureName>/….
Environment Configuration
/config/environment/monitoring/….
Settings
/config/settings/messageRegion/….
Consul-TemplateSetup
- Consul-Template pre-baked on AMIs (Packer)
- Service installed during deployment
- Consul-Template files bundled with feature package
- Generated *.config files (.Net)
Lessons Learnt
- Replacing static files meant more effort to run locally
- Worked well
- However... web.config and connectionstrings.config cannot be changed without cycling AppPool !
Consul Libraries.NET Libraries
- using https://github.com/PlayFab/consuldotnet
JustEatConsul
- Setup Client
- Setup default endpoints - key prefixes
- Create configuration object recursively checking all key/values (remove prefix)
- Return configuration object:
var defaultTimeout = configurationRoot[“timeout”]
Consul WatchesHttpReload
- Endpoint to trigger rebuild of the ConfigurationRoot
Consul Watch
- Windows (NSSM) Service setup during deployment
- “Watches” key prefixes for the feature deployed
- Invokes a WebRequest to local endpoint
- Only rebuild when a value used by the feature is changed
Consul Watches
Lessons Learnt
- Building functionality into the Code vs Static files
- Watch per key prefix…..
- Reliable
Consul ACLs
ACL per Feature
- Generated during Deployment
Lessons Learnt
- Adding new endpoints, meant a deployment to get new ACL
- Use events and triggers
- Avoid different rules across types of environment
- Create Ad-hoc ACLs...and tidy them afterwards!
- Clear browser local storage
Source of Truth
- Launched a Consul Cluster for Just Eat Source of Truth
- Accounts, Environments, Features, Teams
- UI has transformed JSON files to readable content
- Reliable
Use Case : Centralised Data
Use Case : Consul Locks“Unique” Instance Value
- Require unique node name, used for :
- Consul Node
- Monitoring Stats
- Log Files
- Each EC2 needs an instance position
- Use AWS service SimpleDB - being retired….
Deployment
- Instance Locks key/value
- Finds free position,
- Allocates values
- Releases Lock
Lessons Learnt
- Very easy to setup
- Configurable - retries, timeouts, number of locks
- Reliable
Consul Locks001
003
002
Vault“Why aren’t you using Vault?”
- “We plan to….”
Approach
- Launch separate Vault Clusters
- Keep “config” distinct
- Not held in source control
- Backed by Consul
- Manually added passwords
Vault
Challenges
- Mechanism to create “tokens” for EC2 instances
- How to access the secrets
- Keeping it simple vs making it secure
- Backing up the data
- Mechanism to add secrets
- Alerting, Auditing, Logging
.NET Library
- using https://github.com/rajanadar/VaultSharp
JustEatVault
- Use token generated during deployment
var password = configuration[“my_password”]
Vault - Accessing Secrets
Lessons Learnt
- Vault instances didn’t forward requests…..
- Look after your keys
- Read the docs slowly
Vault
- Consul, Packer, Vault, Nomad….
- Use available open source tools
- Newer versions solving workarounds
Summary
Summary
- Some roll-out/migration pain - ports, stats
- Read the docs
- Use the features that suit your needs
- Still making the case for Terraform