of running kubernetes for publication deep dive: the value · decoupling the os from the hardware...

#vmworld

Deep Dive: The Valueof Running Kubernetes

on vSphereFrank Denneman, VMware, Inc.

@FrankDennemanMichael Gasch, VMware, Inc.

@embano1

CNA1553BE

#CNA1553BE

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


Kubernetes Primer

Customer Scenario – Making the Case for Bare Metal

Experience Report – Kubernetes on Bare Metal vs. vSphere

QnA



Kubernetes Primer


#CNA1553BE 5©2018 VMware, Inc.

Google Search(late 1990s)

The Origin of Kubernetes



Revolutionizing the Way we build Distributed(cloud-native) Applications today.

Google Search Pillars:• Commodity• Fault-Tolerant Software• Fraction of the Cost from High-End Servers

The Origin of KubernetesGoogle Search

Source: https://ai.google/research/pubs/pub49VMworld 2018 Content: Not for publication or distribution

https://ai.google/research/pubs/pub49


Platform Engineering Responsibilities



“We must treat the Datacenter itself as one massive Warehouse-scale Computer.”

The Origin of KubernetesThe Datacenter as a Computer

SSH




Borg(~2003)





Borg(~2003)

Cgroups(2007)





Borg(~2003)

Cgroups(2007)

Omega(~2012)




The Origin of KubernetesContainers become Mainstream

In Search for a Common Language



The Origin of KubernetesSo what is a Container, really?

Kernel Mode

Cgroups

Namespaces

Security Capabilities

Scheduler

Syscall

task_struct

…

Scheduling Entity (se)

“running”

syscall.Exec(ENTRYPOINT/CMD)*

A Structure in Kernel Memory. The Kernel has no Notion of a “Container”. It’s yet another Executable.

User Mode

Docker Engine

ContainerCreate()

* After Container Sandbox Initialization(nsenter.go/nsexec.c)

sched_classfair.c (CFS)




Borg(~2003)

Cgroups(2007)

Omega(~2012)

Docker(2013)





Borg(~2003)

Cgroups(2007)

Omega(~2012)

Docker(2013)


Kubernetes(2014)



"Kubernetes is an open-source System for automating Deployment, Scaling, and

Management of containerized Applications."

The Origin of KubernetesContainer Orchestration



Kubernetes Cluster

KubernetesHigh-Level Architecture

Infrastructure(Compute, Storage, Networking)

Control Plane Worker

Pod Pod Pod Pod PodAPI

Kubernetes Cloud Provider



Customer ScenarioMaking the Case for Bare Metal



ABC Inc. is the Leader in Manufacturing Wayback Machines

Enterprise IT Organization with separate Infrastructure, Linux/Middleware and Development Teams (Silos)

>90% standardized and virtualized on VMware vSphere

Going through Digital Transformation to become more Customer and Feedback driven• Need to develop (iterate) faster with an agile Approach

Technical Vehicle: Containers and “cloud-native” Application Architectures• Kubernetes as the Framework to build and run these new Applications• Embrace and contribute to Open Source Software

Meet ABC Inc.



Linux Team at ABC Inc. decided to deploy Kubernetes on Bare Metal

Justification:• New (cloud-native) Applications don’t need vSphere Features like HA and vMotion• Containers are more lightweight, replacing VM’s and the Hypervisor• Kubernetes provides Hypervisor Functionality, e.g. Resource Management and HA• Virtualization reduces Performance of containerized Applications• Reduce Complexity and Costs by eliminating the Hypervisor from the Stack• IT Infrastructure not agile enough (no Self-Service)

ABC Inc.’s vSphere Team reached out to VMware for Help

ABC Inc.’s Decision to go Bare Metal



Back to 2005

merchoid.com



Experience ReportKubernetes on Bare Metal vs. vSphere



Day 0 Planning and “Green Lights”

Day 1 Experiences with first Deployments

Day 2 Container and Cluster Sprawl

Day 3 Maintenance & Availability

Terminology



Day 0Planning and “Green Lights”



Day 0Planning and “Green Lights”

Kubernetes Cluster

Infrastructure(Compute, Storage, Networking)

Cloud Provider(Custom)

External Dependencies

DNS DBs IPAM

Images CA Auth

Secrets Monitoring Logging

CustomIntegrations

Label: AZ=AZ-1VMworld 2018 Content: Not for publication or distribution


Day 0Realization: Managing Bare Metal Systems is hard



How vSphere Can Help



Average Time to get HW in DC – Unpredictable Process



Average Time to get Hardware in Data Center

86 Days



Hardware CompatibilityDecoupling the OS from the Hardware reduces operational Overhead

a simple NIC revision change can directly impact the Kubernetes host

Virtualized hardware decouples the OS from the underlying hardware.Hardware abstraction reduces operational overhead for supported firmware versions of components.

Configuration management done at the physical layer (firmware, drivers, etc). (Drift?)(Supported?)



Non Disruptive PatchingvMotion Workload away for Hardware, Firmware or Driver Update


Need to Patch, vMotion workloadNo disruption

Need to patch – Kill workload



Kill Doesn’t matter for Stateless WorkloadsETCD isn’t stateless, and what about top 10 Workloads in Containers today?



Strong Security IsolationStrong Isolation between Workloads with efficient Resource Usage


VMs provides strong isolation between guest, allowing multi tenancy. Efficient use of resources

Containers are processes in Linux Kernel, security concerns can lead to reduced resource utilization

Tenant A Tenant B



Modern DCs operate various workloads in different packaging formats

vSphere provides unified platform for these workloads

Use your current tool and skillset to manage this workloads

Focus on creating value

Functional Use of HardwareGeneral vs Dedicated

general purpose allows mixed workloads and superior resource utilization

dedicated hardware to a particular function hinders resource optimization



Day 1Experiences with first Deployments



Physical Host

Day 1In the old Bare Metal Days (Pre-Virtualization Era)

Kernel

App

M

Hardware

16 Cores 128GB RAID NIC-Teaming

G G W W W

NUMA sysctl nice IRQ-Balance

Bins/Libs

Almost exclusive Access to Host Resources for this App (1:1)

Best Practices for this Deployment Type were developed

App uses Host Information for Runtime Tuning

Downside: Utilization & Agility

M

G

W

OS Thread: main()

OS Thread: GC

OS Thread: Worker/PoolVMworld 2018 Content: Not for publication or distribution


Physical Host

Day 1Containers and Kubernetes on Bare Metal to the Rescue?

Kernel

Hardware

64 Cores(HT)

384GB RAID NIC-Teaming

NUMA sysctl Cgroups IRQ-Balance

Container Runtime

Kubelet

Not all Runtimes are Cgroup-aware!

How to tune per Workload?

Resource Contention and Workload Interference!

Isolation (Security)?

Utilization & Agility kube-scheduler

Node: BM001Capacity:

cpus: 64memory: 384GB

Allocatable:cpus: 60memory: 360GB

How much to reserve vs. Waste?



Hyperthreading



Hyperthreading in vSphere


‹#› 44©2018 VMware, Inc.

Consistent performance is obtained by avoiding NUMA boundaries



NUMA Architecture



Day 2Container and Cluster Sprawl




Org

aniz

atio

n

EnvironmentVMworld 2018 Content: Not for publication or distribution



Imb

alan

ce

CostVMworld 2018 Content: Not for publication or distribution


Multi-Tenancy


VM

vSphere Cluster

VM VM VM VM VM VM VMVM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod



Multi-TenancyIncrease in Utilization: Scale out & redistribute


VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod



Day 3Maintenance & Availability



Day 3Maintenance & Availability (Control Plane)

Admission Control and Failover Capacity (MTTR)?Proactive HA?Impact of Host Maintenance/

Failure on Control Plane?



Day 3Maintenance & Availability (Workloads)

Kubernetes Control Plane

Controller Manager Scheduler

“QA”“Dev” “Prod” “QA”

4 CPUs 4 CPUs 4 CPUs

“Prod” “QA”“Dev”

2 CPUs 1 CPU 1 CPU 2 CPUs 1 CPU1 CPU 1 CPU

Only considering beta/stable and in-tree Kubernetes FeaturesDisruptive Pod Priority & Preemption, incl. Priority Queue, beta in v1.11

Example assumes shared File System for Persistent Volumes

Queue: “QA”“Dev”“Prod”pick

* Default, configurable** Fixed

pod-eviction-timeout

5min*ReconcilerMaxWaitForUnmountDuration

6min**

“Dev”

1 CPU

“QA”

1 CPU



Multi Cluster ConfigurationPriority

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

Important Important Important Important

More Important More Important More Important More Important

Meh Meh Meh



HA Restart PriorityEnsure “Prod” Systems get restarted first



Restart Dependency

Works based on VM to VM rules

Only 1 level, so for A-B-C create two rules

• VM Group B depends on A• VM Group C depends on B• ETCD-Masters-Workers

Specify when to start next batch

• Resources allocated• Powered On• Guest Heartbeat• App Heartbeat

Or “HA Orchestrated Restart” as it is also called



Kubernetes Cluster Node RolesControl Plane (Masters) and Workers

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

(Master) (Master) (Master) (Workers) (Worker)



Multi-TenancyDRS Affinity Rules

VM

vSphere Cluster


K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod




Multiple Fault DomainsQuorum dictates Design

VM

Fault Domain A


K8S Prod

VM

K8S Prod

V

K8S Prod

VM

K8S Prod

VM

K8S Test

VM

K8S Test

VM

K8S Test

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

VM

K8S Prod

(Master) (Master)

Fault Domain B

(Worker)

VM

K8S Prod(Master) (Worker)

VM

K8S Prod

(Worker) (Worker)

(VM Anti-Affinity)

Host-VM Rules



DRS proactively avoid needing the use of HA• Integrates with server vendor’s monitoring software• Health states are passed to DRS• DRS reacts based on health state of hardware

None of DRS affinity/anti-affinity rules are violated

Quarantine Mode accepts workloads if performance degradation is imminent

Proactive HAMoving Workloads away at first Signs of Trouble



VM Latency SensitivityCPU Core Isolation


DON’T FORGET TO FILL OUT YOUR SURVEY.

#vmworld #CNA1553BE


THANK YOU!

#vmworld #CNA1553BE



Global Services: Reimagining Support Giving you a more proactive, personalized, effortless experience

Read All About ItSupport Insider Blog

https://blogs.vmware.com/kb/

Meet the Team Connect at the

VMVillage’s Listening Post and the Global

Services Meeting Center

Download VMware Skyline™

Visit the VMware Skyline station in the Solutions

Exchange VMworld 2018 Content: Not for publication or distribution

https://blogs.vmware.com/kb/

More Sessions on Kubernetes

Try HOL

VMware Cloud-Native Apps

Follow Us

https://blogs.vmware.com/cloudnativehttps://www.youtube.com/c/VMwareCloudNativeApps

@cloudnativeapps

Tuesday, Nov 6CNA1656BE Put a Lid on It: Securing Containers and Kubernetes on vSphereCNA1634BE Container Portfolio at VMwareCNA1816BE Container and Kubernetes 101 for Admins

Wednesday, Nov 7CNA2755BE Architecting PKS for Production Lessons Learned from PKS DeploymentsCNA2084BE Intro to VMware Kubernetes Engine – Managed K8sService on Public CloudDC3845KE Cloud and Developer Keynote: Public Clouds and Kubernetes at ScaleCNA1674BE Deep Dive: Run Kubernetes in Production with PKS

Thursday, Nov 8CNA3124BE Deep Dive: VMware Kubernetes Engine – Kubernetes as a Service on Public CloudCNA2009BE Run Stateful Apps on Kubernetes with PKS: Highlight WebLogic Server

1932 VMware Kubernetes Engine – Getting Started

1931 VMware Pivotal Container Service and Kubernetes – Getting Started1935 VMware Pivotal Container Service on VMware NSX-T


of running kubernetes for publication deep dive: the value · decoupling the os from the hardware...

Documents