containers for the enterprise: delivering openshift on openstack for performance and scale
TRANSCRIPT
![Page 1: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/1.jpg)
DELIVERING OPENSHIFT ON OPENSTACK FOR PERFORMANCE AND SCALESTEVE GORDON (@xsgordon)
![Page 2: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/2.jpg)
GOAL
● 1000 OpenShift Container Platform nodes; hosted on● 300 physical Red Hat OpenStack Platform nodes
2
![Page 3: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/3.jpg)
3
APPLICATION CONTAINERS
1. Code: mysqld
2. Configuration: /etc/my.cnf
3. Data: /var/lib/mysql
4. Other stuff :-)
CODE, CONFIGURATION, DATA
![Page 4: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/4.jpg)
4
EXPOSITION OF RESOURCES
![Page 5: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/5.jpg)
5
CONSUMPTION OF RESOURCES
![Page 6: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/6.jpg)
OPENSHIFT ON OPENSTACK
6
Architectural tenets:
● Technical independence● Contextual awareness● Avoiding redundancy● Simplified management
Reference architecture:● red.ht/2ibNmvX
![Page 7: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/7.jpg)
PERFORMANCE AND SCALE TESTING
7
![Page 8: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/8.jpg)
WHERE TO TEST?
8
The CNCF cluster is made up of 1000 nodes deployed at Switch, Las Vegas by Intel for the use of the CNCF community:
Compute Node Spec● 2x Intel E5-2680v3 12-core● 256GB RAM● 2x Intel S3610 400GB SSD● 1x Intel P3700 800GB NVMe PCIe
SSD● 1x QP Intel X710
Storage Node Spec● 2x Intel E5-2680v3 12-core● 128GB RAM● 2x Intel S3610 400GB SSD● 10x Intel 2TB NLSAS HDD● 1x QP Intel X710"
Got a nefarious plan for taking over the world using CNCF related open source projects? Head to https://github.com/cncf/cluster.
![Page 9: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/9.jpg)
WHAT TO TEST?
Goal: 1000 OpenShift Container Platform nodes, on 300 Red Hat OpenStack Platform nodes● Push deployment to its limit, identify:
○ Bottlenecks,○ Config changes, and○ Best practices
● Document, file, and fix issues as appropriate.
9
![Page 10: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/10.jpg)
ARCHITECTURE
10
● OpenStack UnderCloud
![Page 11: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/11.jpg)
ARCHITECTURE
11
● OpenStack UnderCloud● OpenStack Overcloud control
plane
![Page 12: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/12.jpg)
ARCHITECTURE
12
● OpenStack UnderCloud● OpenStack Overcloud control
plane● OpenStack Overcloud compute
![Page 13: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/13.jpg)
ARCHITECTURE
13
● OpenStack UnderCloud● OpenStack Overcloud control
plane● OpenStack Overcloud compute● OpenShift/Kubernetes master node
VMs● OpenShift routing, registry, and
metrics● OpenShift/Kubernetes minions● Load Generators
![Page 14: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/14.jpg)
ARCHITECTURE
14
● OpenStack UnderCloud● OpenStack Overcloud control
plane● OpenStack Overcloud compute● OpenShift/Kubernetes master node
VMs● OpenShift routing, registry, and
metrics● OpenShift/Kubernetes minions
![Page 15: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/15.jpg)
ARCHITECTURE
15
● OpenStack UnderCloud● OpenStack Overcloud control
plane● OpenStack Overcloud compute● OpenShift/Kubernetes master node
VMs● OpenShift routing, registry, and
metrics● OpenShift/Kubernetes minions● Load generators
![Page 16: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/16.jpg)
SYSTEM VERIFICATION TEST SUITE
● Red Hat OpenShift Performance and Scalability team’s upstream test suites● Main tests are
○ cluster-loader○ Networking/synthetic○ Workload Generator○ Reliability/Longevity
● https://github.com/openshift/svt
16
![Page 17: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/17.jpg)
17
CLUSTER-LOADER ARCHITECTURE
Start Parse args & config
ConfigObj
End
False
Create Namespace
True
XExists
?Items < N
False
Create X
Iterate ItemCount
True True
False
X can be:● Quota● Template● Service● User● Pod● RC
![Page 18: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/18.jpg)
CLUSTER LOADER TEST
18
OpenShift (Kubernetes) Object Quantity
Nodes 1,000
Namespaces (projects) 13,000
Pods 52,000
Build Configs 39,000
Templates 78,000
Image Streams 13,000
Deployment Configs and Services 39,000 (incl. 13,000 Replication Controllers)
Secrets 260,000
Routes 39,000
Component # of bugs filed/encountered
Kubernetes 8
Installer 8
Images 5
Docker 5
Containers 3
Storage 2
Ansible 2
Management 1
iptables 1
Deployments 1
Command 1
Builds 1
Auth 1
![Page 19: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/19.jpg)
NOTABLE SCALE ISSUES ENCOUNTERED
19
etcd Disk Utilization API Server CPU
![Page 20: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/20.jpg)
NOTABLE SCALE ISSUES ENCOUNTERED
20
Ansible Installer https://github.com/ansible/ansible/issues/16749
![Page 21: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/21.jpg)
FUTURE OPPORTUNITIES
● Reference architecture available at red.ht/2ibNmvX● Containers and OpenStack: A platform for distributed applications, paper
available at http://red.ht/2hSfIPs● Learn more about Red Hat Summit at redhat.com/summit
21
![Page 22: Containers for the Enterprise: Delivering OpenShift on OpenStack for Performance and Scale](https://reader031.vdocuments.us/reader031/viewer/2022030214/5899dd491a28ab4a0b8b7085/html5/thumbnails/22.jpg)
THANK YOU
22