load-balancing and scaling in kubernetes
TRANSCRIPT
![Page 1: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/1.jpg)
Load-balancing and Scaling in Kubernetes
Tyler Rowland
Computer Science Senior SeminarDivision of Science and Mathematics
University of Minnesota, MorrisMorris, Minnesota, USA
28 October 2021
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 1 / 27
![Page 2: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/2.jpg)
Introduction
Introduction
What is Kubernetes?Automation of deployment andmanagementCreated by Google in 2015Software companies use own versionnowScaling and load balancing increaseperformance of application
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 2 / 27
![Page 3: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/3.jpg)
Introduction Outline
Outline
Background ConceptsKubernetes HierarchyLoad-balance vs ScalingCentral Processing Unit (CPU)
MethodsLoad-balancingScaling
Conclusion
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 3 / 27
![Page 4: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/4.jpg)
Background Concepts The Kubernetes Hierarchy
Client Server Architecture
UserInternetNodes
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 4 / 27
![Page 5: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/5.jpg)
Background Concepts The Kubernetes Hierarchy
The Kubernetes Hierarchy: Containers
Containers:Run faster and moreconsistent betweenenvironments
Application runs inside ofthe containers
API server
Kubelet
Master Node
Worker node Worker node
Cluster
Container
Pods
Nodes
APP AAPP A APP B APP B
Kube-ProxyKube-Proxy Kubelet
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 5 / 27
![Page 6: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/6.jpg)
Background Concepts The Kubernetes Hierarchy
The Kubernetes Hierarchy: Pods
Containers run inside ofpods
1 or more container per pod
Pods: smallest unit ofexecution
API server
Kubelet
Master Node
Worker node Worker node
Cluster
Container
Pods
Nodes
APP AAPP A APP B APP B
Kube-ProxyKube-Proxy Kubelet
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 6 / 27
![Page 7: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/7.jpg)
Background Concepts The Kubernetes Hierarchy
The Kubernetes Hierarchy: Nodes
Two types of nodes inKubernetes:
Worker NodeMaster node
API server
Kubelet
Master Node
Worker node Worker node
Cluster
Container
Pods
Nodes
APP AAPP A APP B APP B
Kube-ProxyKube-Proxy Kubelet
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 7 / 27
![Page 8: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/8.jpg)
Background Concepts The Kubernetes Hierarchy
The Kubernetes Hierarchy: Cluster
A cluster is made up of:containers, pods, andnodesUnique cluster IP
API server
Kubelet
Master Node
Worker node Worker node
Cluster
Container
Pods
Nodes
APP AAPP A APP B APP B
Kube-ProxyKube-Proxy Kubelet
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 8 / 27
![Page 9: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/9.jpg)
Background Concepts API server
API server
Application ProgrammingInterface (API)
Communicates with Kubeletand Kube-Proxy
Changing the number ofnodes
API server
Kubelet
Master Node
Worker node Worker node
Cluster
Container
Pods
Nodes
APP AAPP A APP B APP B
Kube-ProxyKube-Proxy Kubelet
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 9 / 27
![Page 10: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/10.jpg)
Background Concepts Scaling
Scaling Background
Kubernetes providesHorizontal scaling
Changing number ofpods or nodesDistribute workload
Ex. CPU usage of a podhits 80 percent of capacity
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 10 / 27
![Page 11: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/11.jpg)
Background Concepts Load-Balancing
Load-Balancing Background
Distributes traffic toreplicated applications
Bottleneck?
Traffic
Server
Master Node
Pod Pod Pod
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 11 / 27
![Page 12: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/12.jpg)
Background Concepts CPU
Central Processing Unit (CPU)
CPUs are measured in cores
Measures the compute processing in a machine
Most cases CPU usage in Kubernetes is measured in millicores
Millicore is one thousandth of a core
Ex. 2000 millicores (or 2000m) is equal to 2 cores
This is standard for every unit in Kubernetes
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 12 / 27
![Page 13: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/13.jpg)
Methods Scaling
Server Scalability Using Kubernetes
L. P. Dewi, A. Noertjahyana, H. N. Palit and K. Yedutun, "ServerScalability Using Kubernetes,"
Study uses 3 servicesGet- get dataPost- send dataDelete- delete data
Evaluates CPU usage single vs. multiple server (Using scalability)
3 Kubernetes nodes with multiple servers1 master node2 worker nodes
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 13 / 27
![Page 14: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/14.jpg)
Methods Scaling
Kubernetes Horizontal Pod Autoscaling
What is KHPA?
Increase or decrease based on the number of concurrent users.
KHPA takes the target input as a percentage of CPU usage
The output is the pods target number
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 14 / 27
![Page 15: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/15.jpg)
Methods Scaling
CPU Usage
Simulated behavior using acombination of services
1,650 requests in each task
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 15 / 27
![Page 16: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/16.jpg)
Methods Scaling
CPU Usage
Users send 8000 requests of each service
Evaluates CPU usage (millicores) of single vs. multiple servers
Averages of previous scenario:
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 16 / 27
![Page 17: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/17.jpg)
Methods Scaling
Scaling Results
Single ServerIn both scenarios the CPU usage was much higherWhen the number of requests went up the CPU usage increased
Multiple Servers (Scaling)Servers distribute workloadCPU usage is smaller when there are more requests
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 17 / 27
![Page 18: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/18.jpg)
Methods Load-balancing
Raft Algorithm
Leader selection algorithm forKubernetes
Used to select pod leaders in anode
Majority of traffic goes toleadersCandidates try to become theleaderFollowers are replicaapplications that don’t get themajority of traffic
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 18 / 27
![Page 19: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/19.jpg)
Methods Load-balancing
Toward Highly Scalable Load-Balancing in KubernetesCluster
N. Nguyen and T. Kim, "Toward Highly Scalable Load Balancing inKubernetes Clusters," in IEEE Communications Magazine, vol. 58, no.7, pp. 78-83, July 2020, doi: 10.1109/MCOM.001.1900660.
Concentrated vs. Balanced Leaders5 leaders distributed to 3 nodesBalanced leaders uses the leader electionMajority of traffic is sent to the leaders
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 19 / 27
![Page 20: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/20.jpg)
Methods Load-balancing
Leaders
Concentrated vs. BalancedLeaders
Concentrated: 5leaders in node 1Balanced: 5 leadersdistributed throughout 3nodes
Node 1 Node 2 Node 3
Balanced Leaders Cluster
Node 1 Node 2 Node 3
Concentrated Leaders Cluster
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 20 / 27
![Page 21: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/21.jpg)
Methods Load-balancing
CPU Utilization
4 clients send requests for 150s
The requests are handled by theleader pods
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 21 / 27
![Page 22: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/22.jpg)
Methods Load-balancing
Throughput
Clients increasing from 1 to32
Clients send 2000 requestseach
Throughput- Number ofrequest per second thecluster can recieve
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 22 / 27
![Page 23: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/23.jpg)
Methods Load-balancing
Load-Balancing Results
Concentrated leadersCPU utilization 60 percent higher in node 1 versus the other twonodesOnce 4 clients send requests throughput is constant
Balanced leadersMore balanced CPU utilizationMore requests per second
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 23 / 27
![Page 24: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/24.jpg)
Conclusion
Conclusion
Scaling and load-balancing increase performanceScaling focused on distributing to different nodesLoad-balancing focused on distributing network traffic
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 24 / 27
![Page 25: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/25.jpg)
Acknowledgements
Acknowledgements
I would like to thank Nic McPhee and Elena Machkasova for greatfeedback and guidance throughout this process. Also thanks to theCSCI department for great planning of this event.
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 25 / 27
![Page 26: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/26.jpg)
Questions
Questions
Questions?
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 26 / 27
![Page 27: Load-balancing and Scaling in Kubernetes](https://reader033.vdocuments.us/reader033/viewer/2022042710/6266cb0eed774c3e06676536/html5/thumbnails/27.jpg)
Resources
L. P. Dewi, A. Noertjahyana, H. N. Palit and K. Yedutun, "ServerScalability Using Kubernetes," 2019 4th Technology InnovationManagement and Engineering Science International Conference(TIMES-iCON), 2019, pp. 1-4, doi:10.1109/TIMES-iCON47539.2019.9024501.
N. Nguyen and T. Kim, "Toward Highly Scalable Load Balancing inKubernetes Clusters," in IEEE Communications Magazine, vol. 58, no.7, pp. 78-83, July 2020, doi: 10.1109/MCOM.001.1900660.
Rowland (U of Minn, Morris) Load-balancing and Scaling in Kubernetes 28 October 2021 27 / 27