dnscluster @ devops krakow 2013
DESCRIPTION
Simple infrastructure DNS service inside Amazon Cloud automated by Ansible, created @ BaseCRMTRANSCRIPT
![Page 1: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/1.jpg)
DNS CLUSTERAutomated Internal DNS Service with Amazon VPC integration
Sławomir Skowron System Engineer (DevOps Team) [email protected]
2013
DevOps Krakow #Meet 1
![Page 2: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/2.jpg)
DNS INTRODUCTION
![Page 3: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/3.jpg)
• Domain Name System is hierarchical and distributed naming system
• Essentially name service for TCP/IP networks
• Allow IP address resolution mechanism
• Adds tree based domain name space,
• Name space is sub-divides into zones and start with root zone
• One of the first NoSQL key-value database
WHAT IS DNS ?
![Page 4: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/4.jpg)
Tree hierarchical structure starting from . (root)
NAMESPACE
![Page 5: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/5.jpg)
ZONESDelegated part of domain name space for administrative responsibility.
![Page 6: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/6.jpg)
DOMAIN NAME SERVERSSoftware on servers that store, manage and serve information about own part of domain namespace called zone
Two types of servers: master and slave
![Page 7: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/7.jpg)
DNS QUERIES
• Recursive - querying other servers until positive response
• Iterative - add local response (cache, local zone) or give info where to look for it.
Two type of external queries: Recursive and Iterative
Cached Queries - DNS Cache - improve latency and throughput
![Page 8: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/8.jpg)
DNS AS A SERVICE INSIDE AMAZON CLOUD
![Page 9: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/9.jpg)
![Page 10: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/10.jpg)
AMAZON EC2 DNS (VPC) PROBLEMS
• Route-53 (right now) is not supporting internal DNS domains
• Amazon VPC Internal DNS support only ec2.internal domains
• Amazon VPC DHCP in default support only AWS DNS
![Page 11: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/11.jpg)
Our own DNS Service
![Page 12: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/12.jpg)
USE CASE
• Available only in LAN and through VPN
• Only A and SRV - infrastructure DNS
• Resolv local and forward if not exist
• No zone transfer, No slaves, No masters
• Updates are simple, secure and fast
Our own DNS Service
![Page 13: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/13.jpg)
SOLUTION
• Clustering for High Availability and Performance
• Integration with our VPC’s DHPC
• Availability in every Amazon Region
• Caching
• Fully Automated and Integrated with Instance Provisioning
• Support for our name space
Our own DNS Service
![Page 14: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/14.jpg)
HIERARCHYHierarchy of private DNS at BaseCRM
![Page 15: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/15.jpg)
DNSCLUSTER RELEASE 1.0
![Page 16: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/16.jpg)
ALMOST :)
![Page 17: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/17.jpg)
SOLUTION• Puppet 3 as Configuration Management solution
• Puppet Hiera, PuppetDB integration
• TheForeman - http://theforeman.org/
• Foreman integrates with BIND
• Unbound as DNSCluster core - local zones, forwarder, cache
• Git for store zones and versioning
![Page 18: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/18.jpg)
DNSCLUSTERIntegration with Puppet and TheForeman
![Page 19: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/19.jpg)
WHAT’S WRONG WITH PUPPET ?
• Puppet is slow
• Hard and slow flow developing with Puppet
• Hard to integrate on running machines before puppet.
• PuppetDB is ok but it’s not scalable enough
• Everything go through Foreman and BIND in our case
![Page 20: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/20.jpg)
PUPPET
![Page 21: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/21.jpg)
ANSIBLERadicaly simple IT orchestration
![Page 22: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/22.jpg)
ANSIBLE• Minimal setup - Python + Libs - pip install ansible
• Use existing auth (root, sudo) on SSH as default transport or accelerated mode
• Ad-hoc operations built in
• async, sync and parallel operations
• Predictable, easy to expand (plugins, connectors, filters, modules)
• Use powerful templates in jinja2
• outputs in json,
• configure in yaml
![Page 24: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/24.jpg)
ANSIBLE @ BASE• Two months of work all in GIT
• 15 playbooks (Universal Flow)
• 25 roles
• 180 yaml files
• 52 template
![Page 25: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/25.jpg)
DNSCLUSTER RELEASE 2.0
![Page 26: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/26.jpg)
SOLUTION• Ansible
• Unbound as DNSCluster core - local zones, forwarder, cache
• Git for store zones and versioning
• Amazon VPC DHCP integration - under development
• ETCD integration - under development
![Page 27: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/27.jpg)
Keep it simple, stupid Core Thinking
KISS
![Page 28: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/28.jpg)
IMPROVEMENT• Simple workflow
• Faster development
• Fast Deploy with low memory/cpu consumption
• No central DB
• All data are stored in 3 places and can be restored from running machines
• Work as push or pull workflow
• Integrated with VPC DHCP if new DNSCluster is created
KISS as core thinking
![Page 29: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/29.jpg)
DNSCLUSTER 2.0Flow for DNSCluster Client
![Page 30: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/30.jpg)
DNSCLUSTER 2.0 - MULTI-REGIONS
![Page 31: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/31.jpg)
DNS CLUSTER PERFORMANCE
![Page 32: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/32.jpg)
DNSCLUSTER PERFORMANCE
1 500 10000
500
1000
1500
2000
2500
AWS DNS
DNSCLUSTER 1 node (1 cpu core – ec2.x1.small)
UNBOUND local cache (forwarders: 3 dnscluster nodes – 3 x ec2.x1.small) 1 pass – 1 unbound thread
UNBOUND local cache (forwarders: 3 dnscluster nodes – 3 x ec2.x1.small) 2 pass – from cache – 1 unbound threads
UNBOUND local cache (forwarders: 3 dnscluster nodes – 3 x ec2.x1.small) 2 pass – from cache – 2 unbound threads
Concurrency
QPS
Queries per second / Concurrency
![Page 33: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/33.jpg)
DNSCLUSTER PERFORMANCELatency / Concurrency
1 500 10000
0.02
0.04
0.06
0.08
0.1
0.12
AWS DNS
DNSCLUSTER 1 node (1 cpu core – ec2.x1.small)
UNBOUND local cache (forwarders: 3 dnscluster nodes – 3 x ec2.x1.small) 1 pass – 1 unbound thread
UNBOUND local cache (forwarders: 3 dnscluster nodes – 3 x ec2.x1.small) 2 pass – from cache – 1 unbound threads
UNBOUND local cache (forwarders: 3 dnscluster nodes – 3 x ec2.x1.small) 2 pass – from cache – 2 unbound threads
Concurrency
Late
nsy
[sec
onds
]
![Page 34: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/34.jpg)
SOON / NEXT TIME ?
Monitoring and Alertingsecond element for our auto scaling
Ansible Universal Template Flow Created @ Base for simple consistent create/destroy instances
![Page 35: Dnscluster @ DevOps Krakow 2013](https://reader033.vdocuments.us/reader033/viewer/2022051314/54b6f36e4a7959fd608b45cc/html5/thumbnails/35.jpg)
THE END