for publication lessons learned for production...lessons learned from pks deployments romain decker,...

#vmworld

Architecting PKSfor Production:

Lessons Learnedfrom PKS Deployments

Romain Decker, VMware, Inc.Dominic Foley, VMware, Inc.

CNA2755BE

#CNA2755BE

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


Containers & Kubernetes

PKS Architecture

PKS Deployment



Containers/Kubernetes Levelset



By 2020, 75% Applications Supporting Digital Business will be

“Built” not “Bought”

Modern Apps Digital TransformationApp Velocity & Customization Drive Competitive Advantage

Software Customization Innovation/Disruption

Source: Gartner

Speed

Boot EnvironmentsRapidly

Portability

Ability to MoveContainers Freely

Lightweight

Minimal Resources Needed

Containers are at the forefront of

what enables app velocity and

customizationVMworld 2018 Content: Not for publication or distribution


Cloud-Native application (CNA) - a methodology of building and running applications that fully exploits the power of the cloud computing, offers on-demand, limitless computing power, whether on public or private cloud.

Microservices architecture with small code-base packaged in containers is what enables CNA

Modern Applications are based on Distributed Microservices …that require containers to run

REST API

REST API

REST API

REST API

REST API

REST API

APIGateway

WEBUI

WEBUI

DeveloperDeveloper

Microservices• Small, modular code base • Easier to update, scale • Runs in containers

Monoliths • Single, large code base • Slower to update, scale • Runs in VMs or containers



Application

Operating System & Dependencies

PhysicalInfrastructure

Application

Operating System & Dependencies

PhysicalInfrastructure

Containers and VMsA practical comparison

OS Dependencies

Compute | Net | Sec | Storage

OS Abstraction

Container Host OS

Container

Compute | Net | Sec | Storage

Hard Problem

Easier

Ubiquitous

Ubiquitous

Configuration Management

https://youtu.be/L1ie8negCjc

How does “someone” create this?


https://youtu.be/L1ie8negCjc


Business App

Docker and Kubernetes

Core Docker functionality is ability run containers • Manual, no fault tolerance, coordinating

scale/upgrades, etc

Scheduling, provisioning, and resource management of multiple containers

• Major Container Solutions Kubernetes Support

• Public Clouds Kubernetes Container Service

$docker run container_web

$docker run container_web

$docker run container_LB

$docker run container_DB

$kubctl create –f App.yaml

The “App”

Wanted: Container Orchestrator!

Kubernetes Cluster

ContainersOne at a Time

Kubernetes (aka K8s) Orchestrating Multiple Containers

Kubernetes in 5 minhttps://youtu.be/PH-2FfFD2PU


https://youtu.be/PH-2FfFD2PU


PlatformOps

Infrastructure

Compute Network Monitoring

Security Storage

Architecting w/Specific Application Requirements

Kubernetes Cluster

vSphere NSX Wavefront

NSX Datastores

Platform Operator

Focus on mappingKubernetes constructs to

a given infrastructure

Load Balancer

Persistent Volumes

Resources / Availability Zone

Security Policy

Application Metrics

ELK Spark Nth App

K8s API

Common

App Devarchitects with native

Kubernetes constructs

the SDDC & public clouds

AppDev

Mapping Kubernetes to SDDChttps://youtu.be/ex8jY7HNnUY


https://youtu.be/ex8jY7HNnUY


Challenges of Running Kubernetes in Production

Source: Cloud native Computing Foundation User Survey 2017

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Difficultychoosing an

orchestrationsolution

Reliability ScalingDeployments

Logging Complexity Networking Monitoring Storage Security

Addressed by VMware PKS



Desired state of Application

VMware PKS & Kubernetes

11

Container/Appscheduling, scale,

resiliency, and Day 2

Desired state of Kubernetes

Clusters

Kubernetes cluster creation, scale, resiliency, and Day 2

AppsApps

AppsApps

AppsApps

VMware PKS



Kubernetes is Only One Layer of the Container Service Stack

Image Registry

Framework Lifecycle Management

Security and Networking

Persistence

Virtual Infrastructure

Physical Infrastructure

Mo

nito

ring

, Lo

gg

ing

, Ana

lyti

cs

Cluster Health Monitoring, Healing and Lifecycle Management

Scheduling, Orchestration, Service Creation

vSAN, Hatchway

vSphere


NSX-T

PKS Control Plane

BOSH (cluster LCM)

Kubernetes

Harbor

vRe

aliz

e S

uite

PKS



PKS Architecture



VMware PKS on SDDC Rapidly deliver and operationalize next-generation apps


BOSH

NSX-T

Service Broker

vSANvSphere

etcd worker

Container Registry

master etcd workermaster

PKS Control Plane

Kubernetes Cluster Kubernetes Cluster

vRealizeAutomation

vRealizeLog Insight

vRealizeOperations

vRealizeNetwork Insight

Wavefrontby VMware



Identifying PKS Components

OPS MANAGER: provide UI to install Bosh Director and PKS Control Plane VM

BOSH: deploy and manage Kubernetes clusters

PKS: front end API for users to interact with PKS

HARBOR: private container registry

Management and data planes

vSphere

Physical Hardware

workermaster

Kubernetes Cluster

OPS MANAGER

BOSH

HARBOR

P

PKSworkermaster

Kubernetes Cluster

PKS DATA PLANE PKS MANAGEMENT PLANE



Availability zones allow you to provide high-availability and load balancing to VMs.

Ops Manager will balance the instances across all of the configured availability zones (AZ’s).

Availability Zones (AZ)vSphere CPI Tile Configuration

Resource Pool could be left blank if using a vSphere cluster as the AZ, or could be used to limit resources consumption for example.

Add new Availability Zones• Minimum = 2 (Management x1, Kubernetes Node VMs x1)• Recommended = 4 to 7 (Management x1, multiple choices

for Kubernetes Masters and Workers)



Availability Zones (AZ)PKS Tile Configuration (Assign AZ’s)

SINGLETON OPS MANAGER JOBS

• A service where only a single instance (VM) is deployed, e.g. Ops Manager, BOSH VM, Services Broker VM, Harbor.

• Singleton jobs are commonly the infrastructure/management VM’s and usually reside in the Management AZ.

• Otherwise singleton jobs can share the same AZ as the balanced jobs.

BALANCED OPS MANAGER JOBS

• A balanced job will have multiple instances deployed, e.g. a Kubernetes cluster with 3x Master nodes would be balanced across 3x AZ’s.

• This is why it is important to map your physical infrastructure to your Availability Zones!



COMPUTE CLUSTER 2 COMPUTE CLUSTER 3COMPUTE CLUSTER 1MANAGEMENT CLUSTER

Topology Example – Multi Compute ClustersAZ are used to set locality of a VM against different locations

AZ-MGMT

P

AZ-COMP-01 AZ-COMP-02 AZ-COMP-03

workermaster

worker

workermaster

worker

workermaster

worker

workermaster

worker

worker

worker

worker

worker

worker

worker

worker

worker

PKS CLUSTER 1Medium plan, multi-master

PKS CLUSTER 2Small plan, single master

Singletons placed in AZ-MGMT

VDS

1:1 mapping between AZ and vSphere Clusters

Each compute cluster can resides in a dedicated rack or room

Storage must be accessible by all ESXi servers hosting Kubernetes Node VMs

Cluster doesn’t participate inNSX-T Fabric

NSX-T Transport Nodes (Geneve)

STORAGE STORAGEVMworld 2018 Content: Not for publication or distribution


COMPUTE CLUSTERMANAGEMENT CLUSTER

Topology Example – Single Compute ClusterResource pools used to segment availability zones

STORAGE

AZ-MGMT

AZ-COMP-01AZ-RES-01

AZ-COMP-02AZ-RES-02

AZ-COMP-03AZ-RES-03

workermaster

worker

workermaster

worker

workermaster

worker

workermaster

worker

worker

worker

worker

worker

worker

worker

worker

worker

PKS CLUSTER 1Medium plan, multi-master

PKS CLUSTER 2Small plan, single master

Singletons placed in AZ-MGMT

VDS

Mapping between AZ and Resource Pools

P

As Resource Pools are used to define AZ, there is no guarantee that Kubernetes Master Nodes will land on different ESXi hosts.

Storage must be accessible by all ESXi servers hosting Kubernetes Node VMs. vSAN can be used in this scenario.

Cluster doesn’t participate inNSX-T Fabric

STORAGE

NSX-T Transport Nodes (Geneve)



Non-routable (internal to NSX-T) doesn’t imply non unique subnets

Requirements (scale, troubleshooting)

Dependencies

Deployment philosophies

CHOICES BASED ON

ADDITIONAL CONSIDERATIONS

Networking TopologyNO-NAT and NAT choices

EXTERNAL TO NSX-T(ROUTABLE / NO-NAT)

INTERNAL TO NSX-T (ROUTABLE / NO-NAT)

INTERNAL TO NSX-T (NON ROUTABLE / NAT)

INTERNAL TO NSX-T | (ROUTABLE / NO-NAT)

INTERNAL TO NSX-T(NON-ROUTABLE / NAT)

> PKS MANAGEMENT NETWORK <

> POD NETWORKS <

INTERNAL TO NSX-T | (NON-ROUTABLE / NAT)

> NODE NETWORKS <



PKS & NSX-T Networking IntegrationDesign considerations

NAT MODE

Enable NAT mode for node network

POD IP BLOCK

Will be carved out to create networks to host Kubernetes pods belonging to the same namespace

Should be a multiple of /24

POOL ID

Used for: K8S Master VIP, SNAT from pods, Kubernetes Service kind (LoadBalancer L4),

Kubernetes Ingress kind (L7)

Cannot be on the same subnetas the uplink/transit network

T0 MAPPING

PKS supports only a single T0 currently

T0 must be configured in Active-Standbyregardless of networking topology

NODE IP BLOCK

Will be carved out to create networks to host Kubernetes cluster node VMs

Should be a multiple of /24

Scale is directly impacted by IP Blocks and Pool configurationVMworld 2018 Content: Not for publication or distribution


Networking Topology: Option #1

CONSIDERATIONS

• PKS Management external to NSX-T, deployed on a classic vSphere port group

• PKS Management and vSphere / NSX Management networks can be combined

PKS Management external to NSX-T + NO-NAT

POD NETWORK – ‚PKS-INFRASTRUCTURE‘

POD NETWORK – ‚KUBE-SYSTEM‘

KUBERNETES NODES

POD NETWORK – ‚DEFAULT‘

P

T0

T1

VIP

PHYSICAL NETWORK

T1

T1

T1

T1

MASTER W W W W

PKS MANAGEMENT

MANAGEMENT NA

T

NO NAT

NO

NA

T

ROUTABLE IP




CONSIDERATIONS

• PKS Management internal to NSX-T, deployed on a logical switch

• The tier-1 logical router and logical switch required for the PKS Management network must be created upfront

PKS Management internal to NSX-T + NO-NAT



KUBERNETES NODES


P

T1

VIP

PHYSICAL NETWORK

T1

T1

T1

T1

MASTER W W W W

PKS MANAGEMENT

MANAGEMENT NA

TN

O N

AT

ROUTABLE IP

T0

NO NAT

T1VMworld 2018 Content: Not for publication or distribution



CONSIDERATIONS

• PKS Management internal to NSX-T, deployed on a logical switch

• The tier-1 logical router and logical switch required for the PKS Management network must be created upfront

• DNAT rules required for PKS Management

PKS Management internal to NSX-T + NAT



KUBERNETES NODES


P

T1

VIP

PHYSICAL NETWORK

T1

T1

T1

T1

MASTER W W W W

PKS MANAGEMENT

MANAGEMENT NA

T

ROUTABLE IP

T0

NAT

NA

T

T1VMworld 2018 Content: Not for publication or distribution


Kubernetes Cluster Nodes VM Storage & Persistent Volumes Hatchway solving persistent storage challenges in Kubernetes

worker workermaster

VMDK

vSAN Considerations• Availability zones do not map with vSAN Fault Domain• PKS with vSAN stretched cluster is not supported• vSAN is a vSphere cluster construct

worker

VMFSNFS

VSAN

Persistent Volumes Considerations• SDRS (Storage DRS) not supported on VMs hosting

Kubernetes Clusters• Datastore must be accessible by all ESXi servers

hosting Kubernetes VMs

PERSISTENT VOLUMEMapped to a VMDK usingProject Hatchwayhttps://vmware.github.io/hatchway/

worker


https://vmware.github.io/hatchway/


PKS Design Highlights

InfrastructureUnderstand how

elements relate between themselves and build for

scale

NetworkingThe network topology depends on scale and

deployment philosophy

StorageHatchway is your friend, if the underlying storage

aligns to your Kubernetes clusters



PKS Deployment



RequirementsPlanning is crucial for a successful deployment

SOFT HARD INFRA

INFRASTRUCTURE READINESS

Core infrastructure

vSphere (topology, permissions)

vSphere HA & DRS

NTP / DNS (forward and reverse)

Co-existence with NSX-V or PAS

SOFTWARE

vSphere 6.5 U1, 6.5 U2, 6.7

NSX-T 2.2, 2.3

HARDWARE

Hardware Compatibility List (HCL)

Resource Requirements



Networking Requirements

NET

PKS

Network assignments

Reserved IP range for PKS (10.100.200.0/24)

Reserved IP ranges for Docker & Harbor (172.17.0.1/16, 172.18.0.1/16, 172.19.0.1/16, 172.20.0.1/16, 172.21.0.1/16, 172.22.0.1/16)

PHYSICAL CONNECTIVITY

Transport VLAN for Geneve, MTU (≥ 1700)

Transit VLAN(s)

Dynamic routing using eBGP (BFD recommended) or static routing (HA VIP recommended)

NSX-T Large Edge VM

• OpsManager communicates with vCenter and ESXi hosts

• Bosh communicates with vCenter, ESXi hosts, and Kubernetes master and worker nodes

• NCP should be able to reach NSX Manager

• Kube-DNS (on K8s worker node) should be able to reach K8s master node.

• K8s worker nodes should be able to reach vCenter (for persistent volumes – Hatchway project)

FIREWALL



Deployment WorkflowNSX-T dial tone: infrastructure readiness for PKS

FOUNDATION

NSX-T Manager, Controller and Edge appliances OVA images deployment

Controller Cluster configuration and registration

OBJECTS AND PROFILES

Uplink profiles

IP Pools, IP Blocks

Transport Zones / N-VDS

Logical Switches (Overlay and VLAN for external connectivity)

EXTERNAL CONNECTIVITY

T0 router creation (Active-Standby), peering with physical router

T1 routers creation

** NAT rules creation

PREPARATION

Certificates (creation and registration against NSX-T Manager using FQDN): Super User Principal Identity Certificate and CA Certificate

Compute Managers

FABRIC

Edges and ESXi configured as transport nodes

Edge cluster creation

** If required, based on the network topology implemented



Deployment WorkflowPivotal Container Service

FOUNDATION

Ops Manager OVA deployment

Configure Authentication System

PKS CONFIGURATION

Import PKS Tile

Assign AZ and Networks

Define PKS API FQDN

Configure Plans

Kubernetes Cloud Provider: vSphere IaaS

NSX-T integration: Container Networking Interface

OPS MANAGER FOR VSPHERE

vCenter config (DC, datastores, networking)

Availability Zones creation

Networks creation

PKS CONFIGURATION

Monitoring (Wavefront integration)

Errands: NSX-T Validation required

Upload Stemcell

FOUNDATION

Import Harbor Tile

Configure Harbor

Apply Changes



Don’t Forget

BOSH DIRECTOR

Enable VM Resurrector Plugin: checked

Enable Post Deploy Scripts: checked

ERRANDS

NSX-T Validation errand: On



TIP – Create a PKS Management VMCLI Tools

UUAC

PKS

KUBECTL

BOSH

OM

BOSH

BOSH CLI

Manage and troubleshoot PKS deployments (tasks, etc.)

Provide information of VMs that BOSH manages

USER AUTHENTICATION AND AUTHORIZATION CLI

Create and manage PKS users

Grant PKS cluster access to users

PKS CLI

Create, delete or scale-out PKS clusters

Get PKS credentials

OPS MANAGER CLI

Interact with Ops ManagerKUBECTL

Interact with Kubernetes by controlling the cluster manager

Deploy applications

Application dev

Application ops

Platform/Site ReliabilityEngineer or vSphere Admin



Preparation• Understand the versions you want to run• Get infrastructure & development teams in a room together – understand the solution• Infrastructure readiness: requirements, NSX certificates replacement• Create a (dedicated) management box with CLI tools

Design• Availability Zones: Clusters vs Resource Pools• vSphere Topology – Shared or individual clusters (Management/Edge & Compute or

Management, Edge, Compute)• Networking: planning (reserved CIDR), understand how subnets (/24) will be used per

K8S cluster & namespace, Virtual Switch design (VDS & N-VDS)• Avoid cross data center Kubernetes clusters unless you REALLY know what you are

doing!

Takeaways



Deployment• Use FQDN and not IP when configuring the connection to NSX from Ops Manager and

PKS tile• Ensure NTP is configured and time sync between vCenter, ESXi hosts, Ops Manager,

BOSH and PKS• Follow documentation

Operations• Before deploying a new PKS cluster, make sure that enough resources are still

available, otherwise the deployment will fail: Node and Pod IP blocks, IP Pool.• Basic troubleshooting – E.G. Failed cluster deployment

Takeaways



ADDITIONNAL SESSIONS• NET1677BE – Kubernetes Container Networking with NSX-T Data Center Deep Dive

– Thursday, Nov 08, 3:00 p.m. – 4:00 p.m.

• CNA2009BE – Run Stateful Apps on Kubernetes with PKS: Highlight WebLogic Server– Thursday, Nov 08, 10:30 a.m. – 11:30 a.m.

• NET1561BE – Next-Generation Reference Design with NSX-T Data Center– Part 1: Thursday, Nov 08, 9:00 a.m. – 10:00 a.m.– Part 2: Thursday, Nov 08, 10:30 a.m. – 11:30 a.m.

HANDS-ON LABS• SPL-1931-01-CNA – VMware Pivotal Container Service and Kubernetes• SPL-1935-01-NET – VMware Pivotal Container Service on VMware NSX-T• SPL-1926-01-NET – VMware NSX-T Data Center – Getting Started

Find out more


PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.

#vmworld #CNA2755BE


THANK YOU!

#vmworld #CNA2755BE


for publication lessons learned for production...lessons learned from pks deployments romain decker,...

Documents