kubernetes on openstack - hpc advisory council
TRANSCRIPT
![Page 2: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/2.jpg)
© 2018 SWITCH | 2
Infrastructure & Data Services
Our offer
Our customers
Your added value
![Page 3: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/3.jpg)
© 2018 SWITCH | 3
Your added value
SWITCH made – Swiss made
• Swiss law and data location• Scalable storage and computing power on
demand with immediate availability in accordance to the need of – and controlled by –the institutions
• Flexible usage and charging model, no up-front investment
• Simple administration; integrated into the academic network of SWITCH;security and identity services included
• Support for academic use cases • Created together with you
![Page 4: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/4.jpg)
© 2018 SWITCH | 4
Our customers
Higher education• Cantonal universities• ETH domain with research institutions• Universities of applied sciences• Universities of teacher education
University-related organizations• Spin-Offs • Research institutions • eLearning Center • University hospitals
![Page 5: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/5.jpg)
© 2018 SWITCH | 5
Our offer
• SWITCHengines• Virtual Private Cloud (VPC)• SCALE-UP (Project)*
* For developing academic services with 9 universities, as part of the „Scientific Information“ projects mandated by swissuniversities.
![Page 6: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/6.jpg)
© 2018 SWITCH | 6
SWITCHengines
Customer tailored computing and storage performance for universities, research and teaching –further developed in the
SCALE-UP project mandated by swissuniversities.
Your benefits• Your data in Switzerland
• Integrated network and security
• Support for academic use cases
• Simple administration and billing
• Created together with you
Customers• Universities
• Research institutions
• eLearning Center
• University
hospitals
• Spin-Offs
Services• SWITCHengines (IaaS)
• Virtual Private Cloud (VPC)
• SCALE-UP (academic project)
![Page 7: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/7.jpg)
© 2018 SWITCH | 7
Openstack Cloud Engineer at SWITCH. He works in NRENs since 2011, first in Italy and then in Switzerland. He has significant experience in running critical infrastructures using Open Source software. He works together with the Infrastructure & Data team, to deliver an Openstack based cloud to the Swiss Universities.
Saverio Proto
![Page 8: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/8.jpg)
© 2018 SWITCH | 8
Container orchestration.Open Source system to deploy containerized applications.
• What you should already know:• Docker, build docker containers• Run your application in a container
• What Kubernentes will do for me ?• Manage the lifecycle of containers• Schedule the containers to the hosts• Attach storage volumes to the containers• Setup networking
Kubernetes (K8s): what is it ?
![Page 9: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/9.jpg)
© 2018 SWITCH | 9
• K8s can interact with the Openstack API
• Use keystone for authentication• Create volumes• Create load balancers• Set routes in Neutron routers
Kubernetes on Openstack
![Page 10: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/10.jpg)
© 2018 SWITCH | 10
• https://github.com/switch-ch/k8s-on-openstack/• Forked from https://github.com/infraly/k8s-on-openstack
This ansible playbook will create Openstack instances, and will install Kubernetes on them.
export KEY=keynameexport IMAGE="Ubuntu Xenial 16.04 (SWITCHengines)"export NETWORK=k8s( read REAME for complete variables list)
ansible-playbook site.yaml
Deploy K8s on Openstack
![Page 11: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/11.jpg)
© 2018 SWITCH | 11
ubuntu@k8s-master:~$ kubectl get nodesNAME STATUS AGE VERSIONk8s-1 Ready 3m v1.9.4k8s-2 Ready 3m v1.9.4k8s-3 Ready 3m v1.9.4k8s-master Ready 4m v1.9.4
Start !
![Page 12: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/12.jpg)
© 2018 SWITCH | 12
The playbook returns an admin.conf that looks like this:apiVersion: v1
clusters:
- cluster:
certificate-authority-data: [..CUT..]
server: https://<ipaddress>:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: [..CUT..]
client-key-data: [..CUT..]
Client config
![Page 13: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/13.jpg)
© 2018 SWITCH | 13
• kubectl is the command line tool• ~/.kube/config is the config file
• Supports Openstack Keystone authentication• kubectl is able to read usual Openstack OS_ env vars
• kubectl config set-credentials \
osuser --auth-provider=openstack
• source ~/openrc
Client config
![Page 14: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/14.jpg)
© 2018 SWITCH | 14
Create a context to use the Openstack User
kubectl config set-context \
--cluster=kubernetes \
--user=osuser osuser@kubernetes
Lets use the contextkubectl config use-context osuser@kubernetes
Client config
![Page 15: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/15.jpg)
© 2018 SWITCH | 15
Keystone authentication for k8s
Kubectlclient
OpenstackKeystone
KubeApiserver
WebHook
2. Requestwith bearer token
1. Token issue
3. Calling WebHook for token validation
4. Token validation with keystone
![Page 16: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/16.jpg)
© 2018 SWITCH | 16
kube-apiserver--authentication-token-webhook-config-file=webhook.kubeconfig.yaml
---
apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: https://localhost:8443/webhook
name: webhook
contexts:
- context:
cluster: webhook
user: webhook
name: webhook
current-context: webhook
kind: Config
preferences: {}
users:
- name: webhook
Configure the webhook
![Page 17: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/17.jpg)
© 2018 SWITCH | 17
It is just a container:https://hub.docker.com/r/zioproto/k8s-keystone-auth/
More recent source code can be found here:https://github.com/kubernetes/cloud-provider-openstack
make image-k8s-keystone-auth
Start the webhook container
![Page 18: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/18.jpg)
© 2018 SWITCH | 18
• Keystone tested for Authentication
• Default RBAC for Kubernetes implements Authorization
• Keystone projects are mapped to Kubernetes Groups
Authorization via RBAC
![Page 19: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/19.jpg)
© 2018 SWITCH | 19
/etc/kubernetes/cloud-config[Global]auth-url = https://keystone.cloud.switch.ch:5000/v2.0username = [email protected] = ******tenant-name = [email protected] = ZH
[BlockStorage]trust-device-path = falsebs-version = v2
[Route]router-id = 3a6cd142-91cd-4a04-9a86-a73455a0155d
[LoadBalancer]lb-version = v2floating-network-id = 3cc83f7d-9119-475b-ba17-f3510c7902e8subnet-id = 5e18c72c-1902-4846-bd84-ec54cf028375
K8s Master interaction with Openstack
![Page 20: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/20.jpg)
© 2018 SWITCH | 20
/etc/kubernetes/manifests/kube-controller-manager.yaml
spec:
containers:
- command:
- kube-controller-manager
- --cloud-provider=openstack
- --cloud-config=/etc/kubernetes/cloud-config
Interaction with Openstack
![Page 21: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/21.jpg)
© 2018 SWITCH | 21
I changed my the way of doing Networking after the Openstack summit in Sydney
Use --network-plugin=kubenet
• https://github.com/zioproto/k8s-on-openstack/commit/f4506ed202ecc6fc4ff5ac603fd28f3664cb2871
• https://www.openstack.org/videos/sydney-2017/kubernetes-on-openstack-the-technical-details
Kubernetes Neutron Networking
![Page 22: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/22.jpg)
© 2018 SWITCH | 22
Kubernetes Neutron Networking
neutronrouter
VMaka k8s node
VMaka k8s node
Default gateway is the neutron router
Has static routes to reach Pod Networks
Default gateway is the neutron router
Pod Pod
Pod network assigned by the k8s master
Pod Pod
Pod network assigned by the k8s master
![Page 23: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/23.jpg)
© 2018 SWITCH | 23
• Docker containers run into Openstack instances• Persistent storage is cinder volumes
• K8s will attach the Cinder Volume to the right Openstackinstance were the docker container is scheduled
volumes:
- name: mysql-persistent-storage
cinder:
volumeID: <uuid>
fsType: ext4
Persistent Storage
![Page 24: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/24.jpg)
© 2018 SWITCH | 24
• This method is not practical• we have to manage Cinder volumes on the side• Export UUID
• Solution• Define a Storage Class
Persistent storage
![Page 25: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/25.jpg)
© 2018 SWITCH | 25
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: defaultannotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/cinder
parameters:
type: fast
availability: nova
Storage Class
![Page 26: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/26.jpg)
© 2018 SWITCH | 26
kind: PersistentVolumeClaimapiVersion: v1metadata:
name: mysql-pvcannotations:
volume.beta.kubernetes.io/storage-class: defaultspec:
accessModes:- ReadWriteOnce
resources:requests:
storage: 30Gi
Persistent Storage pvc
![Page 27: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/27.jpg)
© 2018 SWITCH | 27
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pvc
Persistent Storage with pvc
![Page 28: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/28.jpg)
© 2018 SWITCH | 28
I suggest not to use it, at least in Openstack Newton
• Both DNAT and SNAT are necessary• you lose the client IP information
• IPv6 support status is unknown to me
• It is a pure L4 loadbalancer, cannot terminate TLS connections or rewrite HTTP headers
Openstack LBaaSv2 with K8s
![Page 29: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/29.jpg)
© 2018 SWITCH | 29
Openstack LBaaSv2
neutronrouter
VMaka k8s node
VMaka k8s node
Default gateway is the neutron router
Default gateway is the neutron router
Pod Pod
Pod network assigned by the k8s master
Pod Pod
Pod network assigned by the k8s master
neutronLBaaS
Svc FloatingIP configured here
Openstack project network
Both DNAT and SNAT are necessary here.
![Page 30: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/30.jpg)
© 2018 SWITCH | 30
• An API object that manages external access to the services in a cluster, typically HTTP.
• Ingress can provide load balancing, TLS termination and name-based virtual hosting.
• https://kubernetes.io/docs/concepts/services-networking/ingress/
• At least two known implementations• Nginx based• HAProxy
Kubernetes NGINX Ingress
![Page 31: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/31.jpg)
© 2018 SWITCH | 31
Kubernetes NGINX Ingress
VM
aka k8s node
VM
aka k8s node
Pod Pod
Pod network assigned
by the k8s master
Pod Pod
Pod network assigned
by the k8s master
VM k8s
master
Kubernetes-nginx-ingress
Default network namespace
(docker –net=host)
• It is a special privileged Docker container running NGINX
• On the k8s master (s) or dedicated VM
• Has access to external IP addresses and cluster IP addresses
Kubernetes cluster network
![Page 32: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/32.jpg)
© 2018 SWITCH | 32
How I use the Ingress ?
![Page 33: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/33.jpg)
© 2018 SWITCH | 33
Slack channel #sig-openstack• Join via http://slack.k8s.io/
Follow our cloud blog:• https://cloudblog.switch.ch/
Get in touch with me• [email protected]
Getting involved
![Page 34: Kubernetes on Openstack - HPC Advisory Council](https://reader031.vdocuments.us/reader031/viewer/2022012414/616e1e008cf2e1362e462e60/html5/thumbnails/34.jpg)
© 2018 SWITCH | 34
Thank you ! Questions ?