cna2006be deep dive: architecting container services with ... · deep dive: architecting container...

62
Merlin Glynn (VMware) Ramiro Salas (Pivotal) CNA2006BE #VMworld #CNA2006BE Deep Dive: Architecting Container Services with VMware & Pivotal Developer- Ready Infrastructure VMworld 2017 Content: Not for publication or distribution

Upload: others

Post on 22-May-2020

12 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Merlin Glynn (VMware)Ramiro Salas (Pivotal)

CNA2006BE

#VMworld #CNA2006BE

Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 2: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 3: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Agenda

1 Pivotal Cloud Foundry 101Why do my Developers want it?

2 Kubernetes 101Why do my Developers want it?

3 Ops: Architecture for Containers 101

4 Ops: Network & Security Controls

5 Ops: Monitoring & Logging

6 Ops: Platform as Code{}

7 Ops: PCF+PKS

3

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 4: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Pivotal Cloud Foundry 101Why do my Developers want It?

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 5: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Pivotal Cloud Foundry 101

5

war

Availability Zone 1 Availability Zone 2 Availability Zone 3

Staging

Root

FS

Build

Pack

war

`cf push`

Drop

let

A

I

A

Imyapp.foo.com

*.foo.com = NSX Edge Vip

NSX Edge

PCF Routing PCF Routing PCF Routing

LB Pool Members

“Here is my source code

Run it on the cloud for me

I do not care how”

URL Request:

myapp.foo.com

Developer

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 6: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Kubernetes 101Why do my Developers want It?

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 7: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Kubernetes 101

7

K8s Cluster

Worker

`kubectl apply –f myapp.yml`

Developer

Worker

kube-proxyMaster

etcd

kube-proxy

Service: nodeport | ingress

POD POD

Load Balancer

URL Request:

myapp.foo.com/k8siscool

Docker

Registry

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 8: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Architecting for Containers 101

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 9: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DRI … Architect for Agility

Virtual Data Center

• Architect the right Abstractions

• Automate Everything

• Build for Failure

Control Agility

vSphere NSX vSAN

Pivotal Cloud Foundry

PCF

PKS

BOSH powered Kubernetes

BO

SH

Platform

Operator

Developer

Wavefront

Self Service

• Automation

• Day 2 Operations

• Control

• Application Services or Container Services

• Application Logging & Monitoring

vRLI (Dev)

vRops

vRLI (Ops)

vRNI

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 10: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ops: Architecting for Availability & Scale

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 11: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

vSphere Fundamentals for PCF

11

BOSH

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

Developer

PCF OrgPCF Space

App App

Architecting for Availability & Scale

Virtual Data Center

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 12: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Physical Fault Domains

12

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

Developer

PCF OrgPCF Space

App App

Virtual Data Center

Cluster Design Best Practices

• Enable vSphere HA

• Enable & Tune BOSH HealthMonitor Resurrection

vSphere HA

vSphere HA

BOSH Agent(s)

BOSH

BOSH Hlth

Monitor

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 13: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Physical Fault Domains

13

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

Developer

PCF OrgPCF Space

App App

Virtual Data Center

Cluster Design Best Practices

• Enable vSphere HA

• Enable & Tune BOSH HealthMonitor Resurrection

• Plan For Singletons

– Externalize

– DR (vDP, Image, Snapshot, pgdump)

BOSH Agent(s)

BOSH

webdav

(blob)

BOSH

S3 Compat

Storage

PCF BlobStore

DR

DR DR

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 14: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

IaaS Multi Tenancy

14

AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

vSAN/NFS/VMFS

Resource Pool

AZ1 Foundation 1

ESX Cluster

vSAN/NFS/VMFS

ESX Cluster

vSAN/NFS/VMFS

ESX Cluster

Virtual Data Center

Cluster Design Best Practices

• Enable vSphere HA

• Enable & Tune BOSH HealthMonitor Resurrection

• Plan For Singletons

– Externalize

– DR (vDP, Image, Snapshot, pgdump)

• Use Resource Pools & Scale Clusters as needed

BOSH

Resource Pool

AZ2 Foundation 1

Resource Pool

AZ3 Foundation 1

Resource Pool

AZ1 Foundation 2Resource Pool

AZ2 Foundation 2

Resource Pool

AZ3 Foundation 2

Dev|Test|UAT

Foundation

Prod

Foundation

C

P

I

C

P

I

CPI Acct 1 Assigned

vCenter PermsPool Limits & Shares

CPI Acct 2 Assigned

vCenter PermsPool Limits & Shares

AC

L

Quota

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 15: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Recovering the Platform

15

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

PCF OrgPCF Space

App App

BC/DR Best Practices

• Platform as Code{}

BOSH Agent(s)

BOSH

BOSH

S3 Compat

Storage

PCF BlobStore

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 16: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Backup Job

Recovering the Platform

16

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

PCF OrgPCF Space

App App

BC/DR Best Practices

• Platform as Code{}

• Backup Services for Platform Persistent Data

• Backup Services for App Service Persistent Data

– Don’t Forget External App Data not managed by PCF

BOSH Agent(s)

BOSH

BOSH

S3 Compat

Storage

PCF BlobStore

MySql PCF Service

Tile

mysql mysql mysql

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 17: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Backup Job

Recovering the Platform

17

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

PCF OrgPCF Space

App App

BC/DR Best Practices

• Platform as Code{}

• Backup Services for Platform Persistent Data

• Backup Services for App Service Persistent Data

– Don’t Forget External App Data not managed by PCF

• VMotion (Yes)

• SVMotion (NO)

BOSH Agent(s)

BOSH

BOSH

S3 Compat

Storage

PCF BlobStore

*

*

vmdk

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 18: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

{}

NSXEdge LTM

Multi-Site Platforms

18

AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

BC/DR Best Practices

• Business Continuity w/ Multi Site

– GSLB

BOSH

GSLB

NSXEdge LTM

Health Checks

Health Checks

{}{}

Common ServiceMeshData

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 19: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

19

VMware PKS

Kubernetes on BOSH (Kubo)

BOSH

NSX

Analytics Automation

SecurityOperations

Monitoring

GCP

Service Brokermasteretcd worker

Logging

vSANvSphere

masteretcd workerContainer

Registry

(PKS)

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 20: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

What about PKS?

20

BOSH Deploys KUBO

• Same BOSH Availability Zone Constructs are available

• Spread Core K8S Jobs across BOSH Availability Zones

– Master

– ETCD

– Workers

• Multi Site can be GSLB in much the same way as PCF

• BOSH Makes Kubernetes Day 1 & Day 2 easy.

• Does NOT require PCF

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 21: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Architecting the Platform

21

BC/DR Best Practices

• Platform as Code{}

• Backup Services for Platform Persistent Data

• Backup Services for App Service Persistent Data

• Business Continuity w/ Multi Site

Cluster Design Best Practices

• Enable vSphere HA

• Enable & Tune BOSH HealthMonitorResurrection

• Plan For Singletons

• Use Resource Pools & Scale Clusters as needed

• VMotion (Yes)

• SVMotion (NO)

DEVELOPER-READY

INFRASTRUCTURE

Deliver innovation faster

to customers

Architectural Resource(s) Link(s)

VMware VVD (Validated Design) In Progress

Pivotal ’Lite’ Reference Architecture https://docs.pivotal.io/pivotalcf/1-11/refarch/vsphere/vsphere_ref_arch.html

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 22: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ops: Network & Security Controls

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 23: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

http://myapp.default-apps.foo.com

Network Fundamentals for PCF

23

Network Design Best Practices

• Get Wildcard Certs & DNS Approved

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

AP

PS

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub)

Cell

Cell

SS

HTC

P

TC

P

SS

H

AP

PS

A

I

A

I

External Services

Internal Apps

LS: OSPF

app.public-apps.foo.com

CF

ASG

{}

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 24: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

AP

PS

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub)

Cell

Cell

SS

HTC

P

TC

P

SS

H

AP

PS

A

I

A

I

External Services

Internal Apps

LS: OSPF

CF

ASG

Network Security & Controls

24

Network Design Best Practices

• Get Wildcard Certs & DNS Approved

• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs

– On Demand: Developer trigger VM provision

– Pre-Provisioned: Ops triggers VM provision

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 25: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

AP

PS

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub)

Cell

Cell

SS

HTC

P

TC

P

SS

H

AP

PS

A

I

A

I

External Services

Internal Apps

LS: OSPF

CF

ASG

Network Security & Controls

25

Network Design Best Practices

• Get Wildcard Certs & DNS Approved

• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs

– On Demand: Developer trigger VM provision

– Pre-Provisioned: Ops triggers VM provision

• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 26: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

AP

PS

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub)

Cell

Cell

SS

HTC

P

TC

P

SS

H

AP

PS

A

I

A

I

External Services

Internal Apps

LS: OSPF

CF

ASG

Network Security & Controls

26

Network Design Best Practices

• Get Wildcard Certs & DNS Approved

• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs

– On Demand: Developer trigger VM provision

– Pre-Provisioned: Ops triggers VM provision

• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges

• Use NSX Edge for Load Balancing, SSL Termination, & Perimeter FW ACLs

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 27: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

LS: Isolation_A

PCF

PCF Isolation Segment

GO

RTRCELL CELL CELL

/22

Public Apps

AP

PS

DNS:

*.public-apps.foo.com

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)

Cell

Cell

SS

HTC

P

ISO

ISO

TC

P

SS

H

AP

PS

A

I

A

I

CF

ASG

External Services

Internal Apps

LS: OSPF

Network Security & Controls

27

Network Design Best Practices

• Get Wildcard Certs & DNS Approved

• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs

– On Demand: Developer trigger VM provision

– Pre-Provisioned: Ops triggers VM provision

• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges

• Use NSX Edge for Load Balancing, SSL Termination, & Perimeter FW ACLs

• Use NSX DLR for PCF Org & Space level segmentation

– Multiple Isolation Segments

– Isolation segments allow Operators to group Diego cells and attach to multiple Logical Swicthes.

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 28: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

LS: Isolation_A

PCF

PCF Isolation Segment

GO

RTRCELL CELL CELL

/22

Public Apps

AP

PS

DNS:

*.public-apps.foo.com

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)

Cell

Cell

SS

HTC

P

ISO

ISO

TC

P

SS

H

AP

PS

A

I

A

I

CF

ASG

External Services

Internal Apps

LS: OSPF

Network Security & Controls

28

Network Design Best Practices …

• Use NSX Security Groups for dynamic security principals

– BOSH Integrated NSX (Dynamic Membership)

– Ingress & Egress PCF Org/Space Specific FW

– Dynamic LB Pool Membership

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

BOSH

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 29: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

LS: Isolation_A

PCF

PCF Isolation Segment

GO

RTRCELL CELL CELL

/22

Public Apps

AP

PS

DNS:

*.public-apps.foo.com

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)

Cell

Cell

SS

HTC

P

ISO

ISO

TC

P

SS

H

AP

PS

A

I

A

I

CF

ASG

External Services

Internal Apps

LS: OSPF

Network Security & Controls

29

Network Design Best Practices …

• Use NSX Security Groups for dynamic security principals

– BOSH Integrated NSX (Dynamic Membership)

– Ingress & Egress PCF Org/Space Specific FW

– Dynamic LB Pool Membership

• Use Distributed Firewall Policy

– Leverage PCF Integrated Dynamic Security Groups

– Control East+West from single policy engine

– Control App to App at the Org/Space level with Isolation Segments

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

BOSH

{}

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 30: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

LS: Isolation_A

PCF

PCF Isolation Segment

GO

RTRCELL CELL CELL

/22

Public Apps

AP

PS

DNS:

*.public-apps.foo.com

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)

Cell

Cell

SS

HTC

P

ISO

ISO

TC

P

SS

H

AP

PS

A

I

A

I

CF

ASG

External Services

Internal Apps

LS: OSPF

Network Security & Controls

30

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

{}

Network Design Best Practices …

• Use NSX Security Groups for dynamic security principals

– BOSH Integrated NSX (Dynamic Membership)

– Ingress & Egress PCF Org/Space Specific FW

– Dynamic LB Pool Membership

• Use Distributed Firewall Policy

– Leverage PCF Integrated Dynamic Security Groups

– Control East+West from single policy engine

– Control App to App at the Org/Space level with Isolation Segments

• Use RFC 1918 for Repeatability

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 31: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Network Security & Controls

31

Platform

Operator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

Network Design Best Practices …

• Platform as Code{} to automate Day 1 & Day 2 ops

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *A

PP

S

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub)

Cell

Cell

SS

HTC

P

TC

P

SS

H

AP

PS

A

I

A

I

External Services

Internal Apps

LS: OSPF

CF

ASG

DNS:

*.sys.pcf.foo.com

*.default-apps.foo.com

Single PCF ‘Foundation’

PCF

Elastic Runtime

** *

LS: Services #

LS: Infra

NSX Edge

LS: ERTLS: Services #

LS: Services #

CF Control

Plane

Ops

Mgr

(PCF)

BOSH GO

RTR

Diego

Brain

TCP

RTR

/26 /22/24(s)

PCF

PCF

Rabbit

PCF

Mysql

* * *

AP

PS

Logical Routing (DLR)

IaaS: vSphere Security Zone A (Hub)

Cell

Cell

SS

HTC

P

TC

P

SS

H

AP

PS

A

I

A

I

External Services

Internal Apps

LS: OSPF

CF

ASG

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 32: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Network Security & Controls

PCF OrgPCF Space

AppA AppB AppC

cf create-security-group SECURITY-GROUP PATH-TO-RULES-FILE cf create-security-group dev-mssql mssql.json

PCF Application Security Groups (ASG):

– Uses iptables in the Diego Cell Server

– Controls Egress only at the container source level

– Can control any IP address as the target

• Operator Declares in the Platform

[ {

"protocol": "tcp",

"destination": "10.0.11.0/24",

"ports": "1-65535"

},

{

"protocol": "udp",

"destination": "10.0.11.0/24",

"ports": "1-65535"

} ]

Platform

Operator

Prod Mssql

192.168.11.10

Prod Mssql

10.0.11.10

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 33: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Network Security & Controls

PCF OrgPCF Space

AppA AppB AppC

cf allow-access SOURCE-APP DESTINATION-APP --protocol PROTOCOL --port PORT• cf allow-access “AppA” “Appc” --protocol TCP --port 443

Developer

PCF Container to Container Networking:

– Creates and Overlay (VXLAN)

– Controls ingress & egress between Ais(containers)

– Uses CNI

• Today Flannel

• Tomorrow NSX-T

– Developer can Declare in CI/CD

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 34: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

What about PKS?

34

KUBO Networking is less Complex

• Typically multiple smaller K8s Deployments

• The core Kubernetes components need to route to each other

• Services Deployed on an Overlay Network

– NSX-T

• Enterprise Security Policy

• Enterprise Tools & Logging

• Common Ingress Paths:

– kube-proxy running on external gateway

– Load Balance to kube-proxy

Image source: https://github.com/cloudfoundry-incubator/kubo-deployment/blob/master/docs/images/kubo-network.png

Load Balancer

EXTERNAL

SVC

Request

External

Service Gateway

kubeproxy

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 35: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Network Security & Controls

35

DEVELOPER-READY

INFRASTRUCTURE

Deliver innovation faster

to customers

Resource(s) Link(s)

KUBO Git Repo https://github.com/cloudfoundry-incubator/kubo-deployment

VMware PCF & NSX Design Guide Coming Soon

Network Design Best Practices

• Use NSX Security Groups for dynamic security principals

• Use Distributed Firewall Policy

– Control East+West from single policy engine

– Control App to App at the Org/Space level with Isolation Segments

• Use Container to Container Networking to allow developer to define fine grained App level security

• Use RFC 1918 Repeatability

• Platform as Code{} to automate Day 1 & Day 2 ops

Network Design Best Practices …

• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs

• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges

• Use NSX Edge for Load Balancing, SSL Termination, & Perimeter FW ACLs

• Use NSX DLR for PCF Org & Space level segmentation

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 36: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ops: PCF Monitoring & Logging

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 37: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Monitoring & Logging

37

Developer

Virtual Data Center

– I need to keep my apps healthy

– I need self service to my Apps Log’s

– I need to instrument my Apps (APM)

Platform

Operator

– I need to keep the Platform healthy

– I need to plan capacity

– I need to watch & Alert on KPIs

– I need to audit

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 38: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Monitoring & Logging

38

Developer

Virtual Data Center

– I need to keep my apps healthy

– I need self service to my Apps Log’s

– I need to instrument my Apps (APM)

PCF Metrics

`cf logs appA`

https://metrics.sys.pcf-foundation.io

Nozzle

vRLIhttps://vrli.pcf-foundation.io

Developer Log Access Routes

– `cf logs`: streams single app’s log events for dev to redirect where needed

– PCF Metrics: PCF app correlating App logs, and container Metrics, ~2week retention

– vRLI: Longer term scalable log storage and indexing, dashboards, & alerts

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 39: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Agents Added to Buildpacks

Future !!!

Monitoring & Logging

39

Developer

Virtual Data Center

– I need to keep my apps healthy

– I need self service to my Apps Log’s

– I need to instrument my Apps (APM)

App & App execution specific Metrics

• tc_server: jdbc_query_failed

• custom_app_metric: transaction_response_time

Platform

Operator

Exposed to developers via CF Service Broker

`cf create service my-apm-endpoint`

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 40: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Monitoring & Logging

40

Platform

Operator

– I need to keep the Platform healthy

– I need to plan capacity

– I need to watch & Alert on KPIs

– I need to audit

vRops

vRops Nozzle

Cloud Foundry Metrics (KPIs)

vSphere & NSX Metrics (KPIs)VMworld 2017 Content: Not fo

r publication or distri

bution

Page 41: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Monitoring & Logging

41

Platform

Operator

– I need to keep the Platform healthy

– I need to plan capacity

– I need to watch & Alert on KPIs

– I need to audit

vRLI

vRops

vRops Nozzle

Cloud Foundry Metrics (KPIs)

vSphere & NSX Metrics (KPIs)

Syslog Nozzle

vSphere & NSX Events

CF Platform Events

Thre

shold

s

Ale

rts

Da

sh

bo

ard

s VMworld 2017 Content: Not fo

r publication or distri

bution

Page 42: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Monitoring & Logging

42

Platform

Operator

– I need to keep the Platform healthy

– I need to plan capacity

– I need to watch & Alert on KPIs

– I need to audit

vRLI

vRops

vRops Nozzle

Cloud Foundry Metrics (KPIs)

vSphere & NSX Metrics (KPIs)

Syslog Nozzle

vSphere & NSX Events

CF Platform Events

Thre

shold

s

Ale

rts

Da

sh

bo

ard

s

All App Events

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 43: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Deamon

Set

Deamon

Set

What about PKS?

POD vRLI

POD

vRLI

• App Logging

• System Logging

– OS & Processes not run in Containers

App Logging

• Per App Only

Sidecar

• App Logging @ Pod level

POD

Daemon

Set

(PODs)

vRLI

POD

LOGGER

DOCKERDDOCKERD

vRLI

DaemonSet

• App Logging @ Cluster level

• Cluster Logging

Dockerd

• App Logging @ Cluster level

• Cluster Logging

• Not handled in K8s API

SyslogD

Platform

Operator

Developer

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 44: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

What about PKS?

K8s Monitoring Integration w/ Wavefront by VMware

Wavefront Integration can be deployed as containers within the K8s Cluster

– Proxy

– Heapster

• Comprehensive Dashboards

– SaaS

• APM for the Developer

• Cluster KPIs for the Operator

• Integrated with PKS

Image source: https://www.wavefront.com/surf-container-wave-join-wavefront-container-world-santa-clara/

Platform

Operator

Developer

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 45: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

45

Platform

Operator

What about PKS?

vRealize Operations & K8s

• Operator KPIs

• Single Pane for SDDC & K8s clusters monitoring

• vRLI Integrated

• Alert on K8s KPIs

• Entity Relationship

• Capacity Planning

• Integrated with PKS

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 46: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ops: Monitoring & Logging

46

DEVELOPER-READY

INFRASTRUCTURE

Deliver innovation faster

to customers

Resource(s) Link(s)

Wavefront: KUBO Integration https://community.wavefront.com/docs/DOC-1204

Blue Medora : vRops MP https://marketplace.vmware.com/vsx/solutions/blue-medora-mp-for-pivotal-cloud-foundry

Blue Medora : vRLI Pack https://marketplace.vmware.com/vsx/solutions/content-pack-for-pivotal-cloud-foundry

Developer

Virtual Data Center

– I need to keep my apps healthy

– I need self service to my Apps Log’s

– I need to instrument my Apps (APM)

Platform

Operator

– I need to keep the Platform healthy

– I need to plan capacity

– I need to watch & Alert on KPIs

– I need to audit

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 47: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ops: Platform as Code{}

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 48: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

BOSH 101

48

• Built for Platform Operators

• Deploys Complex Distributed Systems

– PCF

– Kubo

• Day 1 & Day 2 Ops

– Initial Deployment

– Updates/Patches

– Maintains Health

Platform

Operator

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 49: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

NSX_Config:

edge_vip_1:3

nsxmgr_endpoint: nsxmgr.vmware.io

lswicth_ert_cidr: 192.168.10.0/22

49

AZ1 AZ2 AZ3

Platform

Operator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

Drives NSX-V

NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)

Platform as Code{}

• Declarative

Day 1 & Day 2

YAML

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 50: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ert_config:

diego_database_instances:3

diego_brain_instances: 3

diego_cell_instances: 9

50

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

PCF OrgPCF Space

App App

BOSH

Drives NSX-V

NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)

Platform as Code{}

• Declarative

Day 1 & Day 2

YAML

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 51: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ert_config:

diego_database_instances:3

diego_brain_instances: 3

diego_cell_instances: 12

51

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

PCF OrgPCF Space

App App

BOSH

Drives NSX-V

NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)

Platform as Code{}

• Declarative

Day 1 & Day 2

YAML

Cell_3 Cell_3 Cell_3

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 52: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

52

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

cc uaa brain cc uaa brain cc uaa brain

mysql mysql mysql

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

loggre

gator

loggre

gator

loggre

gator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

PCF OrgPCF Space

App App

BOSH

Drives NSX-V

NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)

Platform as Code{}

• Declarative

• Change Controlled

• Archived

• Audited

Day 1 & Day 2

Cell_3 Cell_3 Cell_3

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 53: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

53

Platform

Operator

Ops Manager

(OVA) vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

BOSH

Drives NSX-V

Platform as Code{}

• Declarative

• Change Controlled

• Archived

• Audited

Day 1 & Day 2

NSXEdge LTM

NSXEdge LTM

• Repeat

– Scale

• Repair

– Recovery

• Repave

– Rotate Creds

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 54: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

54

Platform

Operator

CVE & Update Patching

The New Stack

• Patch at ANY Layer of the Application Stack

• Address CVE in minutes/hours versus days/weeks

• Simply re-stage all apps when any layer is patched

• Platform as Code{}

Day 1 & Day 2

Developer

PCF ERT Tile

PCF Stemcells

PCF OrgPCF Space

App App

CVE in Root File

System of Container

CVE Exec Layer: TC

Server

CVE on the Container

Host OS

Vulnerability in

Code{}

Restage Applications

PCF BuildPack

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 55: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

55

Platform

Operator

CVE & Update Patching

The New Stack

• Stemcells still there …

• Harbor Scans Images for Vulnerability (Clair)

• Address CVE in minutes/hours versus days/weeks

• Platform as Code{}

Developer

Stemcells

CVE in Root File

System of Container

CVE Exec Layer: TC

Server

CVE on the Container

Host OS

Vulnerability in

Code{}

Restage Applications

What about PKS?

Docker

Registry

CVE

FOUND

!!!

BOSH

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 56: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

What about PKS?

KUBO Can scale …. A lot

BOSH allows for a repeatablepattern of K8S Clusters as well.

• Many Development teams

• Multiple Security Zones for Applications

• Multi Cluster HA within a DC

• CI/CD Pattern similar to PCF

Platform

Operator

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

vSAN/NFS/VMFS

Resource Pool

ESX Cluster

VCF

BOSH

Developer

Developer

A

BvRA

PKS

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 57: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Ops: Platform As Code{}

57

DEVELOPER-READY

INFRASTRUCTURE

Deliver innovation faster

to customers

Resource(s) Link(s)

Pivotal NSX + PCF Pipeline https://github.com/cf-platform-eng/nsx-ci-pipeline

Pivotal Generic PCF Install & Upgrade pipelines https://github.com/pivotal-cf/pcf-pipelines

Virtual Data Center

CVE & Update Patching

The New Stack

• Patch at ANY Layer of the Application Stack

• Address CVE in minutes/hours versus days/weeks

• Simply re-stage all apps when any layer is patched

• Platform as Code{}

Day 1 & Day 2

• Declarative

• Change Controlled

• Archived

• Audited

• Repeat

– Scale

• Repair

– Recovery

• Repave

– Rotate Creds

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 58: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Wrapping It up …

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 59: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

Developer Ready Infrastructure

vSphere NSX vSAN

Pivotal Cloud Foundry

PCF

PKS

BOSH powered KubernetesB

OS

H

Platform

Operator

Developer

Wavefront

Self Service

• Automation

• Day 2 Operations

• Control

• Application Services or Container Services

• Application Logging & Monitoring

Solves for DevOps Reqs …

vRLI (Dev)

vRops

vRLI (Ops)

vRNI

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 60: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

60

VMworld US Key Focus Description

CNA1509BU DRI Developer-Ready Infrastructure from VMware & Pivotal

CNA1612BU PCF & KuboUse Cases: Deploying real-world workloads on Kubernetes and Pivotal Cloud

Foundry

CNA2006BU DRIDeep Dive: Architecting Container Services with VMware and Pivotal

Developer Ready Infrastructure

CNA2080BU Kubo Deep Dive: How to Deploy and Operationalize Kubernetes

CNA3429BU KuboBasics of Kubernetes on BOSH: Run Production-grade Kubernetes on the

SDDC

CNA3430BU PCFYour Enterprise Cloud-Native App Platform: An Introduction to Pivotal Cloud

Foundry

MGT2871BUPCF & vRops,

vRLI

Bridging the Operations Gap Between the Software-Defined Data Center

and Pivotal CF for VMware Deployments

NET1523BU PCF & NSX Integrating NSX and Cloud Foundry

PAR4411PU DRIEmerging Technologies with VMware and Pivotal - presented jointly by

VMware, Pivotal and Special Guest Speakers from Cognizant and WWT

Developer Ready Infrastructure @ VMworld

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 61: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

VMworld 2017 Content: Not fo

r publication or distri

bution

Page 62: CNA2006BE Deep Dive: Architecting Container Services with ... · Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure VMworld 2017 ... Pivotal

VMworld 2017 Content: Not fo

r publication or distri

bution