clusternaut: orchestrating percona xtradb cluster with kubernetes
TRANSCRIPT
![Page 1: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/1.jpg)
Clusternaut: Orchestrating Percona XtraDB Cluster
with Kubernetes.Raghavendra Prabhu
Percona Live Data Performance ’[email protected]/[email protected]/@randomsurfer
![Page 2: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/2.jpg)
Yelp’s MissionConnecting people with great
local businesses.
![Page 3: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/3.jpg)
Yelp StatsAs of Q4 2015
86M 3270%95M
![Page 4: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/4.jpg)
MeRaghavendra Prabhu
Software Engineer, Distributed Systems @ Yelp [email protected] / [email protected]
![Page 5: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/5.jpg)
Applicability to any datastore● Derived datastores
○ Elasticsearch○ Redis
● Relational○ MySQL
■ Group replication■ NDB
○ PostgreSQL● MongoDB● Cassandra
![Page 6: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/6.jpg)
Galera - “The Oar boat”
![Page 7: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/7.jpg)
κυβερνήτης “The Helmsman”
![Page 8: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/8.jpg)
Warehouse computing
➔ Mesosphere ➔ Paasta➔ GCE
◆ Reference➔ ECS ➔ Smartcloud➔ Tectonic*
![Page 9: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/9.jpg)
Rationale➔ Nodes v/s Hosts
◆ Resource-based➔ Reusable components
◆ Monitoring and Tracing◆ DNS and Service Discovery◆ Logging◆ Metrics◆ Scheduler
➔ Agnostic ➔ Roles
![Page 10: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/10.jpg)
The Fit➔ Layered
◆ Client - Server◆ Multi-layered
➔ Scaling◆ Horizontal and Vertical
● Preferred?● Need for vertical
![Page 11: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/11.jpg)
The Fit➔ Statelessness
◆ Planes of logic:● Control Plane● Data Plane : Storage
◆ Anti-pattern for containers➔ Elasticity
◆ Elastic Scalability● Scaling down
![Page 12: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/12.jpg)
Declarative vs Imperative➔ Configurable mgmt
◆ Puppet, Nix, Terraform ➔ Microservices
◆ What runs on my laptop ● What runs on server● Reproducibility
➔ 12-factor app➔ Composability ➔ Immutable deployment artifact
![Page 13: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/13.jpg)
Containers ● What is a container and why should I care
○ Operating system virtualization● Isolation
○ Hierarchies of isolation - application, cgroups, namespaces, seccomp…
● Unikernels and VMs○ Role? ○ MirageOS, Rump kernel
● Some - LXC / LXD, Docker*, Rocket*, runc, jails, solaris zones, lmcty, systemd-nspawn
![Page 14: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/14.jpg)
Galera - really short intro! ➔ MySQL and WSREP api➔ Galera plugin➔ Group communication➔ Synchronous replication
◆ ‘Virtually’➔ EVS ➔ Certification-based
◆ Optimistic Concurrency➔ Automatic Node Provisioning
![Page 15: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/15.jpg)
Galera - really short intro! ➔ CAP theorem and Galera
◆ CP➔ How does it fit
◆ Others◆ Idempotency
➔ Stateless? ◆ Symmetric◆ Replicas - Cassandra et.al.◆ MySQL Cluster
➔ Maintenance of Quorum
![Page 16: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/16.jpg)
Orchestration● SOA def
○ ‘Stitching’○ ‘Composing’
● Automation?● Choreography● Best of both worlds
![Page 17: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/17.jpg)
Kubernetes● Started as orchestrator
○ Is an ecosystem for containers● Horizontal Scaling● Self-healing
○ Chaos-monkey● Latest issue
○ Rolling update in clusters○ How K8s solves this
![Page 18: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/18.jpg)
Kubernetes● Bin packing● Automated rollouts and rollbacks● Secret management
○ Elegant● Storage orchestration● Service discovery and load balancing
○ Underrated
![Page 19: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/19.jpg)
Kubernetes: API● Consistent and Versioned
○ Very important glue● Composable● Developed with Swagger● API Groups● Supports both declarative and imperative
○ Rolling-update / Daemon Sets
![Page 20: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/20.jpg)
![Page 21: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/21.jpg)
Kubernetes➔ Components:
◆ Kubelet◆ Pods
● Main service ● Sidekicks
◆ Services● The gcomm:// URL.
◆ Replication Controller
![Page 22: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/22.jpg)
Kubernetes➔ Components:
◆ Labels and selectors● Plumbing / addressing mechanism.● Metadata - docker-machine, MachineMetadata● Set-based and equality-based
![Page 23: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/23.jpg)
Kubernetes➔ Higher Order
◆ Daemon Sets● Logging, Monitoring, Tracing
◆ Replica Sets◆ Deployments
● Rolling updates declarative● Bouncing
○ PaaSTA
![Page 24: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/24.jpg)
Kubernetes➔ Components:
◆ Volumes● Persistent Volumes● External Storage Providers
◆ Secrets / Vault◆ Horizontal Pod Autoscaler
➔ Scheduler◆ Pluggable
![Page 25: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/25.jpg)
Kubernetes: Providers● Bespoke● Google Container Engine (GKE)● AWS● Azure● Determinants:
○ Network - flannel, Weave, calico, GCE.○ OS○ Config Mgmt
![Page 26: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/26.jpg)
Kubernetes: Ecosystem● Deis● Package manager - Helm● Fabric8● Spread
○ From compose to kubernetes● Openshift
![Page 27: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/27.jpg)
Kubernetes➔ Others:
◆ Mesos● Supports k8s too.● Aurora, Chronos, Marathon
◆ Docker Swarm◆ Fleet
➔ Key Differences
![Page 28: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/28.jpg)
Pods
![Page 29: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/29.jpg)
Pods● Herd.. ● What should they contain - containers! ● How is the grouping done ● Pods and nodes
○ Colocation ● Pod communication● Labels
![Page 30: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/30.jpg)
Services
![Page 31: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/31.jpg)
Services● Don’t commingle with `microservices`
○ Think of endpoints. ● Layering architecture
○ Logical address of subset of pods● Communication
○ Environment■ Ordering requirement■ Discovery
○ DNS ■ Issues with DNS
![Page 32: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/32.jpg)
Services: in general● Potential issues
○ Staleness○ Live HUP-ing ○ Propagation
● Haproxy ○ Reload configuration.○ Solved at Yelp with linux qdiscs.
● Flux from Weave
![Page 33: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/33.jpg)
Replication Controller● “Herd Management”● ASG● Pod template: Cookie Cutter
○ Pattern○ Anti-pattern
■ Asymmetric initialisation
![Page 34: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/34.jpg)
Replication Controller● Role
○ Init/Supervisor for cluster○ Rolling updates ○ Multi-version
● Replica Sets
![Page 35: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/35.jpg)
Networking● Docker-style linking● Proxy for Pods● Types
○ Pod to Pod○ Pod to Service○ Intra-Pod○ External to Service
● Providers:○ OpenVSwitch / Flannel / Calico / Weave / Google
![Page 36: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/36.jpg)
External components● Flannel / Others● Etcd● Fluentd● Skydns● Container Registry● REST server● Proxy● cAdvisor / Heapster
![Page 37: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/37.jpg)
PAAS: PaaSTA● Docker● Mesos
○ Chronos○ Marathon
● Sensu● Smartstack
○ Zookeeper● Jenkins● Splunk / Signalfx● Why
![Page 38: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/38.jpg)
Deployment● Declare and build individual Galera/PXC nodes.
○ Keep it minimal and simple○ No assumptions
● Without Kubernetes ○ Docker-compose
■ Possible issues ● Galera node ⇔ Pod
○ Haproxy○ xinetd
![Page 39: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/39.jpg)
Deployment● Basic Steps:
○ Create a ‘flat’ network - 10.0.0.0/24○ Create a ‘cluster’ - zone○ Create a service endpoint.
■ Internal service - 3306/4567/4568.■ External service - 3306/3306(?).■ Expose the external.■ Session affinities.
![Page 40: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/40.jpg)
Deployment● Next:
○ Bootstrap a node Pod from a template.■ Query existing with selector.
○ Start rest of nodes from template.■ Point to Service with selector.■ Replication controller
○ Volumes
![Page 41: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/41.jpg)
Deployment - Implications● Load balancing in state transfers● Respawning of nodes on timeout
○ May not be same nodes.● kubectl to manage
○ Puppet etc. also have modules now.● Separation of client and cluster traffic
![Page 42: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/42.jpg)
Case Study: Safe restarts● Highly available● Unattended autonomous
○ and Imperative● Restarts - services and nodes● More of a orchestration than choreographing● Randomness?● Distributed locks● Low impedance
![Page 43: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/43.jpg)
Service Definition
![Page 44: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/44.jpg)
Dockerfile
![Page 45: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/45.jpg)
```
... | kubectl create -f -
kubectl expose service eclient --port=3306 --target-port=3306 --name=loadbl --type='LoadBalancer’
```
![Page 46: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/46.jpg)
![Page 47: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/47.jpg)
``` .. | kubectl create -f -
kubectl scale --replicas=8 replicationcontrollers controller
```
```kubectl get --no-headers pods -l 'name=pxc' | wc -l2 kubectl stop …kubectl get --no-headers pods -l 'name=pxc' | wc -l2 ```
![Page 48: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/48.jpg)
Credits! ● https://www.pinterest.com/duanejohnson851/star-trek-tng/● https://upload.wikimedia.org/wikipedia/commons/a/a5/CubeSpace.jpg● https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/IUB_Arboretum_-_lotus_pond_-
_dry_seed_pod_-_P1100172.JPG/1280px-IUB_Arboretum_-_lotus_pond_-_dry_seed_pod_-_P1100172.JPG● https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/design/architecture.png● https://pbs.twimg.com/profile_images/511909265720614913/21_d3cvM.png● https://camo.githubusercontent.
com/96468330aba188dbd7d7eeae0caca32d9a6329df/687474703a2f2f656e67696e656572696e67626c6f672e79656c702e636f6d2f696d616765732f70726576696577732f7061617374615f707265766965772e706e67
● http://galeracluster.com/documentation-webpages/_images/replicationapi.png ● https://www.linkedin.com/pulse/containerizing-docker-kubernetes-ramit-surana
![Page 49: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/49.jpg)
Further reading! ● http://kubernetes.io ● https://github.com/ramitsurana/awesome-kubernetes ● https://open.mesosphere.com/frameworks/ ● https://coreos.com/kubernetes/docs/latest/kubernetes-networking.
html● http://paasta.readthedocs.org/en/latest/about/paasta_principles.html● http://12factor.net/ ● http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html
![Page 50: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/50.jpg)
ContactRaghavendra Prabhu
[email protected] / [email protected] Twitter: @randomsurfer
Linkedin: rdprabhuSlideshare: slidunder
Github: ronin13http://rdprabhu.com
http://about.me/raghavendra.prabhu
![Page 51: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/51.jpg)
We are Hiring! Visit
yelp.com/careers
![Page 52: Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042520/5876fcf91a28abf3398b68f9/html5/thumbnails/52.jpg)
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp