pivotal container service (pks) documentation · your pks-provisioned kubernetes cluster does not...

PivotalContainerService

(PKS)

Version1.1

Published:17July2019

©2019PivotalSoftware,Inc.AllRightsReserved.

247

1516192023252728293032395657596268728791

108109111112115130132134135137140146149150151153157159162163165166171174175

TableofContents

TableofContentsPivotalContainerService(PKS)PKSReleaseNotesPKSConceptsPKSClusterManagementPKSAPIAuthenticationLoadBalancersinPKSVMSizingforPKSClustersPKSTelemetryPASandPKSDeploymentswithOpsManagerInstallingPKSvSpherevSpherePrerequisitesandResourceRequirementsPreparingvSphereBeforeDeployingPKSInstallingPKSonvSpherevSpherewithNSX-TIntegrationvSpherewithNSX-TPrerequisitesandResourceRequirementsDeploymentTopologiesPreparingNSX-TBeforeDeployingPKSDeployingOpsManageronvSpherewithNSX-TConfiguringOpsManageronvSpherewithNSX-TIntegrationGeneratingandRegisteringCertificatesInstallingPKSonvSpherewithNSX-TIntegrationGoogleCloudPlatform(GCP)GCPPrerequisitesandResourceRequirementsCreatingServiceAccountsinGCPforPKSConfiguringaGCPLoadBalancerforthePKSAPIInstallingPKSonGCPInstallingthePKSCLIInstallingtheKubernetesCLIUpgradingPKSOverviewWhatHappensDuringPKSUpgradesUpgradingPKSUpgradingPKSwithNSX-TMaintainingWorkloadUptimeConfiguringtheUpgradePipelineManagingPKSConfiguringPKSAPIAccessManagingUsersinPKSwithUAAManagingPKSDeploymentswithBOSHConfiguringaGCPLoadBalancerforPKSClustersAddingCustomWorkloadsVerifyingDeploymentHealthDownloadingClusterLogsViewingandExportingUsageDataServiceInterruptionsDeletingPKSUsingPKS

©CopyrightPivotalSoftwareInc,2013-2019 2 1.1

176179180181182183184185186188189190191192193194199202203204205206207213

CreatingClustersRetrievingClusterCredentialsandConfigurationViewingClusterListsViewingClusterDetailsViewingClusterPlansUsingDynamicPersistentVolumesScalingExistingClustersAccessingDashboardDeployingandAccessingBasicWorkloadsDeletingClustersLoggingOutofthePKSEnvironmentUsingHelmwithPKSConfiguringTillerBackingUpandRestoringPKSInstallingBOSHBackupandRestoreBackingupthePKSControlPlaneRestoringthePKSControlPlaneBBRLoggingPKSSecurityPKSSecurityDisclosureandReleaseProcessDiagnosingandTroubleshootingPKSDiagnosticToolsTroubleshootingPKSCLI


PivotalContainerService(PKS)Pagelastupdated:

PivotalContainerService(PKS)enablesoperatorstoprovision,operate,andmanageenterprise-gradeKubernetesclustersusingBOSHandPivotalOpsManager.

OverviewPKSusestheOn-DemandBroker todeployCloudFoundryContainerRuntime ,aBOSHreleasethatoffersauniformwaytoinstantiate,deploy,andmanagehighlyavailableKubernetesclustersonacloudplatformusingBOSH.

AfteroperatorsinstallthePKStileontheOpsManagerInstallationDashboard,developerscanprovisionKubernetesclustersusingthePKSCommandLineInterface(PKSCLI),andruncontainer-basedworkloadsontheclusterswiththeKubernetesCLI,kubectl.

PKSisavailableaspartofPivotalCloudFoundry orasastand-aloneproduct.

WhatPKSAddstoKubernetesThefollowingtabledetailsthefeaturesthatPKSaddstotheKubernetesplatform.

Feature IncludedinK8s IncludedinPKS

Singletenantingress ✓ ✓

Securemulti-tenantingress ✓

Statefulsetsofpods ✓ ✓

Multi-containerpods ✓ ✓

Rollingupgradestopods ✓ ✓

Rollingupgradestoclusterinfrastructure ✓

Podscalingandhighavailability ✓ ✓

Clusterprovisioningandscaling ✓

MonitoringandrecoveryofclusterVMsandprocesses ✓

Persistentdisks ✓ ✓

Securecontainerregistry ✓

Embedded,hardenedoperatingsystem ✓

FeaturesPKShasthefollowingfeatures:

Kubernetescompatibility:ConstantcompatibilitywithcurrentstablereleaseofKubernetes

Production-ready:Highlyavailablefromapplicationstoinfrastructure,withnosinglepointsoffailure

BOSHadvantages:Built-inhealthchecks,scaling,auto-healingandrollingupgrades

Fullyautomatedoperations:Fullyautomateddeploy,scale,patch,andupgradeexperience

Multi-cloud:Consistentoperationalexperienceacrossmultipleclouds

GCPAPIsaccess:TheGoogleCloudPlatform(GCP)ServiceBrokergivesapplicationsaccesstotheGoogleCloudAPIs,andGoogleContainerEngine(GKE)consistencyenablesthetransferofworkloadsfromortoGCP

OnvSphere,PKSsupportsdeployingandrunningKubernetesclustersinair-gappedenvironments.

PKSComponents


https://docs.pivotal.io/svc-sdk/odb/index.htmlhttps://docs-kubo.cfapps.io/https://docs.pivotal.io

ThePKScontrolplanecontainsthefollowingcomponents:

AnOn-DemandBroker thatdeploysCloudFoundryContainerRuntime (CFCR),anopen-sourceprojectthatprovidesasolutionfordeployingandmanagingKubernetes clustersusingBOSH .

AServiceAdapter

ThePKSAPI

FormoreinformationaboutthePKScontrolplane,seePKSClusterManagement.

ForadetailedlistofcomponentsandsupportedversionsbyaparticularPKSrelease,seethePKSReleaseNotes.

PKSConceptsForconceptualinformationaboutPKS,seePKSConcepts.

PKSPrerequisitesForinformationabouttheresourcerequirementsforinstallingPKS,seethetopicthatcorrespondstoyourcloudprovider:

vSpherePrerequisitesandResourceRequirements

vSpherewithNSX-TPrerequisitesandResourceRequirements

GCPPrerequisitesandResourceRequirements

PreparingtoInstallPKSToinstallPKS,youmustdeployOpsManagerv2.1orv2.2.YouuseOpsManagertoinstallandconfigurePKS.

IfyouareinstallingPKStovSphere,youcanalsoconfigureintegrationwithNSX-TandHarbor.

Consultthefollowingtableforcompatibilityinformation:

IaaS OpsManagerv2.1orv2.2 NSX-T Harbor

vSphere Required Available Available

GCP Required NotAvailable Available

Formoreinformationaboutcompatibilityandcomponentversions,seethePKSReleaseNotes.

ForinformationaboutpreparingyourenvironmentbeforeinstallingPKS,seethetopicthatcorrespondstoyourcloudprovider:

vSphere

vSpherewithNSX-TIntegration

GCP

InstallingPKSForinformationaboutinstallingPKS,seeInstallingPKSforyourIaaS:

vSphere


GCP

UpgradingPKSForinformationaboutupgradingthePKStileandPKS-deployedKubernetesclusters,seeUpgradingPKS.


https://docs.pivotal.io/svc-sdk/odb/https://docs-kubo.cfapps.iohttps://kubernetes.io/docs/home/https://bosh.io/docs

ManagingPKSForinformationaboutconfiguringauthentication,creatingusers,andmanagingyourPKSdeployment,seeManagingPKS.

UsingPKSForinformationaboutusingthePKSCLItocreateandmanageKubernetesclusters,seeUsingPKS.

BackingUpandRestoringPKSForinformationaboutusingBOSHBackupandRestore(BBR)tobackupandrestorePKS,seeBackingUpandRestoringPKS.

PKSSecurityForinformationaboutsecurityinPKS,seePKSSecurity.

DiagnosingandTroubleshootingPKSForinformationaboutdiagnosingandtroubleshootingissuesinstallingorusingPKS,seeDiagnosingandTroubleshootingPKS.

[email protected].


mailto:[email protected]

PKSReleaseNotesPagelastupdated:

ThistopiccontainsreleasenotesforPivotalContainerService(PKS)v1.1.x.

v1.1.6ReleaseDate:September24,2018

ProductSnapshot

Element Details

Version v1.1.6

Releasedate September24,2018

CompatibleOpsManagerversions v2.1.x,v2.2.x

Stemcellversion 3586.42

Kubernetesversion v1.10.7

NSX-Tversion v2.1,v2.2

NCPversion v2.2.1

What’sNewUpdatesstemcelltov3586.42.

UpdatesKubernetestov1.10.7.

ThedefaultfortheWorkerPersistentDiskTypehasbeenupdatedto50GB.

ThedefaultfortheMaster/ETCDandWorkerVMTypehasbeenupdatedto32GBdisk.

KnownIssuesThedefaultfortheMaster/ETCDVMTypeonPlan2shouldbeupdatedtohaveaminimumdisksizeof32GB.

v1.1.5ReleaseDate:August31,2018

ProductSnapshot

Element Details

Version v1.1.5

Releasedate August31,2018




NSX-Tversion v2.1,v2.2

NCPversion v2.2.1


What’sNewUpdatesstemcellto3586.36.

AddssupportforNSX-Tv2.2.

UpdatesNCPtov2.2.1.

NSX-TArchitecturalChanges

KnownIssuesYoucannotenterwhitespaceintoanyofthefieldsinthePKStile,includingleadingandtrailingspacesandspacesbetweencharacters.UsingaspaceinanyfieldcausesthePKSdeploymenttofail.

ThefollowingknownissuesapplytoPKSdeploymentsonvSpherewithNSX-T:

UpdatingloadbalancerrulesfailsfromTLSingresstonon-TLSingresswithNCPrestart.

Stalepoolfoundwhendeletinganingressrulewhichisupdatedfromnon-TLStoTLS.

DeletionofHTTPSVSpoolfailsafterupdatingNCP.

NCPcrashesonrestartiftheloadbalancerhasmaxvirtualservers.

TLSingresscertificateisnotremovedafterdeletingallrelatedTLSingressobjects.

SNIcertificateisnotupdatedafterchangingnon-TLSingresstoTLSingresswithNCPrestart.

NCPerrorannotationsarenotfoundwhenupdatingtheLBIPPoolfromavalidtononexistentIPPool.

NSXcleanupoperationdoesnotreleasetheexternalIPordeleteSNATrulesontheT0router.

ThefollowingknownissueappliestoPKSdeploymentsonGCP:

Ifyouusestemcellv3586.18orlaterinthe3586lineofLinuxstemcellswhendeployingPKSonGCP,youmayseethefollowing:

Theoutputofthe bosh vms commandshowsanerrormessagethatincludes unresponsive agent .YourPKS-provisionedKubernetesclusterdoesnotrespondtoanyPKSCLIcommands,suchas pks get-credentials or pks delete-cluster .

Untilthisissueisresolved,usestemcellv3586.16whendeployingPKSonGCP.

NSX-TArchitecturalChanges

PKSv1.1.5includesarchitecturalchangesrelatedtoitsintegrationwithNSX-TandNCP.PKSusesNCPtointegratewithNSX-T.FormoreinformationaboutNCP,seeOverviewofNSX-TContainerPlug-in intheVMwaredocumentation.

NSX-TNodeAgentandKubeProxy

InPKSv1.1.4andearlier,theNSX-TNodeAgentandNSX-TKubeProxyrunasadaemonsetoneachworkernode.InPKSv1.1.5,boththeNSX-TNodeAgentandtheKubeProxyrunasBOSH-managedprocessesoneachworkernode.

NSX-TContainerPlugin(NCP)

InPKSv1.1.4andearlier,NCPrunsasaKubernetespodonasingleworkernode.WithPKSv1.1.5,NCPrunsasaBOSH-managedprocessontheKubernetesmasternode.

InPKSv1.1.5,ifyoudeployamulti-mastercluster,theNCPprocessrunsonallmasternodesbutisactiveononlyasinglemaster.IftheNCPprocessonan

Note:TheissueslistedbelowpertaintoNSX-Tv2.2andNCPv2.2.1.NSX-Tv2.3andNCPv2.3includefixesfortheseissues.PKSsupportfortheseversionsisunderdevelopmentforafuturerelease.

Note:ThechangesinthissectionapplytoPKSdeploymentsonvSpherewithNSX-T.

Note:YoudonotneedtoinstallorconfigureNCP.NCPisautomaticallyinstalledandconfiguredwhenyoudeployPKSinanNSX-Tenvironment.


https://docs.vmware.com/en/VMware-NSX-T/2.0/com.vmware.nsxt.ncp_kubernetes.doc/GUID-52A92986-0FDF-43A5-A7BB-C037889F7559.html

activemasterisunresponsive,BOSHactivatesanotherNCPprocess.

PKSLogsforNSX-TandNCP

InPKSv1.1.4andearlier,youaccessNSX-TandNCPlogsusing kubectl commands.InPKSv1.1.5,NSX-TandNCPareBOSH-managedprocesses,andyouaccessthelogsforthesecomponentsusingBOSH.

BOSHjobsrelatedtoNSX-TintegrationwithNCPasaBOSHprocess:

Location BOSHJobs

MasterNode

/var/vcap/sys/log/ncp

/var/vcap/sys/log/pks-nsx-t-prepare-master-vm

/var/vcap/sys/log/pks-nsx-t-ncp

WorkerNodes

/var/vcap/sys/log/nsx-kube-proxy

/var/vcap/sys/log/openvswitch

/var/vcap/sys/log/nsx-cni

/var/vcap/sys/log/nsx-node-agent

RuntheBOSHcommand bosh–dMY-DEPLOYMENTlogs

tocollecttheselogs,replacing MY-DEPLOYMENT withthenameofyourPKSdeployment.For

moreinformation,seeUsingLogs intheBOSHdocumentation.

WhenyouupgradetoPKSv1.1.5,theexistinglogsforNSX-TandNCParedeleted.Beforeyouupgrade,youmaywanttobacktheselogsup.Forexample,youmayneedtoanalyzetheselogsifyouexperienceproblemswithyourPKSdeploymentbeforeupgrading,orproblemsrelatedtoafailedupgrade.

v1.1.4ReleaseDate:August8,2018

ProductSnapshot

Element Details

Version v1.1.4

Releasedate August8,2018




NSX-Tversion v2.1

NCPversion v2.2


UpdatesKubernetestov1.10.5.

Includessecurityenhancements.

KnownIssuesIfyouusestemcellv3586.18orlaterinthe3586lineofLinuxstemcellswhendeployingPKSonGCP,youmayseethefollowing:



https://bosh.io/docs/job-logs/


v1.1.3ReleaseDate:July30,2018

ProductSnapshot

Element Details

Version v1.1.3

Releasedate July30,2018





Telemetryinformationisnowsentlessfrequently.





ProductSnapshot

Element Details

Version v1.1.2





SecurityFixesThisreleaseincludesthefollowingsecurityfix:

HighCVE-2018-11047:UAAacceptsrefreshtokenasaccesstokenonadminendpoints


https://www.cloudfoundry.org/blog/cve-2018-11047/





ProductSnapshot

Element Details

Version v1.1.1





What’sNewUAAandsecurityenhancements

NSX-Tpatches

Telemetrypatch

Kubernetes1.10.4

BugFixesOpsManagerv2.1.7andlaterisnowsupportedinPKSv1.1.1.However,PivotalrecommendsusingOpsManagerv2.2todeployPKS.

UpgradeProcedureToupgradetoPKSv1.1.1,youmustupgradefromPKSv1.0.2orlater.

ToupgradetoPKSv1.1.1,followtheproceduresinUpgradingPKS.PivotalrecommendsusingOpsManagerv2.2todeployPKS.

ForaddedsecurityinOpsManagerv2.2,disabletheAllowLegacyAgentsoptionintheDirectorConfigpaneoftheBOSHDirectortile.Formoreinformation,seetheOpsManagerconfigurationtopicforyourcloudprovider.Forexample,ConfiguringBOSHDirectoronvSphere .




Note:PKSv1.1.1andlatercanbedeployedonOpsManagerv2.1orv2.2.PivotalrecommendsusingOpsManagerv2.2todeployPKS.ForaddedsecurityinOpsManagerv2.2,disabletheAllowLegacyAgentsoptionintheDirectorConfigpaneoftheBOSHDirectortile.Formoreinformation,seetheOpsManagerconfigurationtopicforyourcloudprovider.Forexample,ConfiguringBOSHDirectoronvSphere .


https://docs.pivotal.io/pcf/om/2-2/vsphere/config.html#dir-confighttps://docs.pivotal.io/pcf/om/2-2/vsphere/config.html#dir-config

v1.1.0

ReleaseDate:June28,2018

UpgradeProcedure

ToupgradetoPKSv1.1.0,followtheproceduresinUpgradingPKS.

FeaturesThissectiondescribesnewfeaturesintroducedinPKSv1.1.0.

GeneralFeaturesAddssupportforKubernetes1.10.3.

AddssupportforbackingupandrestoringPKSusingBOSHBackupandRestore(BBR).Formoreinformation,seeBackingUpandRestoringPKS.

AddssupportforgrantingPKScontrolplaneaccesstoclientsandexternalLDAPgroups.Formoreinformation,seetheGrantClusterAccesssectionofManageUsersinUAA.

AddssupportforallowingworkerstobedeployedacrossAvailabilityZones(AZs).

Addssupportfornetworkautomationandnodenetworkisolation.

AddssupportforNFSbyenablingrpcbindonworkernodes.

Addssupportforkube-controller-managertoissuecertificates.

AddssupportforconfiguringHTTP/HTTPSproxytobeusedbytheKubernetescontrolplane.

AddssupportforconfiguringtheSecurityContextDenyadmissioncontroller.Formoreinformation,seeUsingAdmissionControllers intheKubernetesdocumentation.

EnablestheMutatingAdmissionWebhookadmissioncontroller.Formoreinformation,seeUsingAdmissionControllers intheKubernetesdocumentation.

EnablesauditloggingfortheAPIserver.

Createslogsfordelete-all-clustererrandsinthe/var/vcap/sys/log/delete-all-clustersfolderonthePKScontrolplaneVM.

AddsBOSHinstanceIDstoworkernodelabels.

HardenssecuritybyremovingtheABACauthorizationoptionforclusters.

HardenssecuritybyusingserviceaccountIDsinsteadofserviceaccountkeysforGCPdeployments.

HardenssecurityforKubernetessystemcomponents.Forexample,kube-dnsnowusesitsownconfigurationinsteadofthekubeletconfiguration.

vSphereFeaturesAddssupportforNO-NATdeploymenttopologiesforPKSinstallationsonNSX-T.Formoreinformation,seeDeploymentTopologies.

AddssupportforPKSintegrationwithVMwareWavefront tocapturemetricsforclustersandpods.Formoreinformation,seethe(Optional)LoggingsectionofInstallingPKSforyourIaaS.Forexample,seeInstallingPKSonvSphere.

AddssupportfornodenetworkaccessusingHTTPproxyforvSpheredeployments.Formoreinformation,seetheNetworkingsectionofInstallingPKSonvSphere.

AddssupportforPKSintegrationwithVMwarevRealizeLogInsight(vRLI) fortaggedloggingofthecontrolplane,clusters,andpods.Formoreinformation,seethe(Optional)MonitoringsectionofInstallingPKSforyourIaaS.Forexample,seeInstallingPKSonvSphere.

AddssupportforintegrationwithVMwareAnalyticsCloud(VAC) tocapturetelemetryinformation.

HardenssecuritybyremovingVMchangepermissionsfromworkernodesforvSpheredeployments.

HardenssecuritybyremovingvCenterusercredentialsfromworkernodesforvSpheredeployments.

WARNING:PKSv1.1.0isnolongeravailablefordownloadfromPivotalNetwork.

Note:TheonlysupportedupgradepathforPKSv1.1.0isfromPKSv1.0.2andlater.DonotupgradedirectlytoPKSv1.1.0fromv1.0.0.Instead,firstupgradePKSv1.0.0tov1.0.2;thenupgradePKSv1.0.2tov1.1.0.Alternatively,doacleaninstallofPKSv1.1.0.


https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/https://www.wavefront.comhttps://www.vmware.com/products/vrealize-log-insight.htmlhttps://codepen.io/didkobravo/project/live/AYdRpX

AddssupportforHarborRegistry integrationenhancements:updatedHarbortile,abilitytouseNFSandGoogleBucketsasanimagestore,andHTTP/HTTPSproxyserversforClair.

BugFixesPreventsunnecessaryroutecreationinthekube-controller-manager.

RetainstheoriginalsourceIPwhenusingFlannel.

Disablestheread-onlyportinthekubeletconfiguration.

DisablescAdvisorinthekubeletconfiguration.

Foraddedsecurity,theKubernetesAPIservernolongertriestofixmalformedrequests.

TheKubernetesAPIservernowcleansupterminatedpodsmoreoftentoavoidrunningoutofdiskspace.

TheKubernetesAPIservernowunmountsvolumesofterminatedpodsforsecurityreasons.

OperatorsnolongerhavetomanuallydeleteNSX-Tobjectscreatedduringthelifeoftheproduct.InPKSv1.1,runningthepks delete-clustercommanddeletesallNSXobjects.

BetaComponentsAddssupportfordeployingmultipleKubernetesmasternodesacrossAZs.Forinformationaboutconfiguringmultiplemasters,seethePlanssectionofInstallingPKSforyourIaaS.Forexample,seeInstallingPKSonvSphere.

ComponentVersionsPKSv1.1.0includesorsupportsthefollowingcomponentversions:

ProductComponent VersionSupported Notes

PivotalCloudFoundryOperationsManager(OpsManager)

2.1.0-2.1.6 SeparatedownloadavailablefromPivotalNetwork

Stemcell 3586.24

Kubernetes 1.10.3 PackagedinthePKSTile(CFCR)

CFCR(Kubo) 0.17 PackagedinthePKSTile

Golang 1.9.7 PackagedinthePKSTile

NCP 2.2 PackagedinthePKSTile

KubernetesCLI 1.10.3SeparatedownloadavailablefromthePKSsectionofPivotalNetwork

PKSCLI 1.1SeparatedownloadavailablefromthePKSsectionofPivotalNetwork

VMwarevSphere

6.5U2and6.5U1.Editions:vSphereEnterprisePlusEdition

vSpherewithOperationsManagementEnterprisePlus

vSphereversionssupportedforPivotalContainerService(PKS)

VMwareNSX-T 2.1-AdvancedEditionNSX-TversionssupportedforPivotalContainerService(PKS)

WARNING:Thisfeatureisabetacomponentandisintendedforevaluationandtestpurposesonly.Donotusethisfeatureinaproductionenvironment.Productsupportandfutureavailabilityarenotguaranteedforbetacomponents.

WARNING:Youcannotchangethenumberofmasternodesforexistingclusters.Tousethemulti-masterfeature,youmustcreateanewplanthatusesmultiplemaster/etcdnodesanddeployanewcluster.IfyouarealreadyusingallthreeplanconfigurationsinthePKStile,youmustdeleteaplanandallclustersyoudeployedusingthatplanbeforeyoucandeployamulti-mastercluster.

WARNING:OpsManagerv2.1.7andlaterisnotsupportedinPKSv1.1.0.


https://vmware.github.io/harbor/

VMwareHarborRegistry 1.5.0 SeparatedownloadavailablefromPivotalNetwork

VMwarevRealizeLogInsight(forvSpheredeployments)

4.6 SeparatedownloadavailablefromPivotalNetwork

*Componentsmarkedwithanasteriskhavebeenpatchedtoresolvesecurityvulnerabilitiesorfixcomponentbehavior.

KnownIssuesThissectionincludesknownissueswithPKSv1.1.0andcorrespondingworkarounds.

PKSv1.1.0doesnotsupportOpsManagerv2.1.7andlater.Formoreinformation,seeError:DuplicateVariableNameintheTroubleshootingtopic.

IfyouusePKSCLIv1.0.xwithPKStilev1.1.x,youmustloginevery600secondstomanuallyrefreshtheCLItoken.PivotalrecommendsupgradingtoPKSCLIv1.1.xtosolvethisissue.

IfyouupgradePKSfromv1.0.xtov1.1,youmustenabletheUpgradeAllClusterserrandinthePKStileconfiguration.Thisensuresexistingclusterscanperformresizeordeleteactionsaftertheupgrade.

Ifyouusestemcellv3586.18orlaterinthe3586lineofLinuxstemcellswhendeployingPKSonGCP,youmayseethefollowing:



ClusterSecurityRecommendations

ToreducetheriskofcompromisedclustersinyourPKSdeployment,thefollowingpoliciesarerecommended:

Ensurethatonlytrustedoperatorsandsystemshaveaccesstoclusters.

Ensurethatonlytrustedimagesaredeployedtoclusters.

Maintaintrustedimagestoconsistentlyincludecurrentsecurityfixes.

Donotexposenetworkportstountrustednetworksunlessstrictlyrequired.

ReconfigureGCPLoadBalancersAfterMasterVMRecreation

IfKubernetesmasternodeVMsarerecreatedforanyreason,youmustreconfigureyourclusterloadbalancerstopointtothenewmasterVMs.Forexample,afterastemcellupgrade,BOSHrecreatestheVMsinyourdeployment.

ToreconfigureyourGCPclusterloadbalancertousethenewmasterVM,followtheprocedureintheReconfiguringaGCPLoadBalancersectionofConfiguringaGCPLoadBalancerforPKSClusters.

ExistingABACClusters

Attribute-basedaccesscontrol(ABAC)isnolongersupportedinv1.1.DeleteanyABACclustersbeforeupgradingtov1.1.

NewDefaultVMType

IntheResourceConfigpane,thedefaultVMTypeisnowlarge.ThisistoensurethatPKScontrolplaneVMhassufficientresources.

IftheVMsinyourPKSinstallationusethedefaultVMtype,yourVMswillusethenewlargeVMtypeafterupgradingtoPKSv1.1.0.

IftheVMsinyourPKSinstallationuseacustomVMtype,yourconfigurationremainsthesameafterupgradingtoPKSv1.1.0.

[email protected].



PKSConceptsPagelastupdated:

ThistopicdescribesPivotalContainerService(PKS)concepts.Seethefollowingsections:

PKSClusterManagement

PKSAPIAuthentication

LoadBalancersinPKS

VMSizingforPKSClusters

[email protected].



PKSClusterManagementThistopicdescribeshowPivotalContainerService(PKS)managesthedeploymentofKubernetesclusters.

OverviewUsersinteractwithPKSandPKS-deployedKubernetesclustersintwoways:

DeployingKubernetesclusterswithBOSHandmanagingtheirlifecycle.ThesetasksareperformedusingthePKScommandlineinterface(CLI)andthePKScontrolplane.

Deployingandmanagingcontainer-basedworkloadsonKubernetesclusters.ThesetasksareperformedusingtheKubernetesCLI, kubectl .

ClusterLifecycleManagementThePKScontrolplaneenablesuserstodeployandmanageKubernetesclusters.

ForcommunicatingwiththePKScontrolplane,PKSprovidesacommandlineinterface,thePKSCLI.SeeInstallingthePKSCLIforinstallationinstructions.

PKSControlPlaneOverviewThePKScontrolplanemanagesthelifecycleofKubernetesclustersdeployedusingPKS.ThecontrolplaneallowsuserstodothefollowingthroughthePKSCLI:

Viewclusterplans

Createclusters

Viewinformationaboutclusters

Obtaincredentialstodeployworkloadstoclusters

Scaleclusters

Deleteclusters

Inaddition,thePKScontrolplanecanupgradeallexistingclustersusingtheUpgradeallclustersBOSHerrand.Formoreinformation,seeUpgradeKubernetesClustersinUpgradingPKS.

PKSControlPlaneArchitectureThePKScontrolplaneisdeployedonasingleVMthatincludesthefollowingcomponents:

ThePKSAPIserver

ThePKSBroker

AUserAccountandAuthentication(UAA)server

Thefollowingillustrationshowshowthesecomponentsinteract:


ThePKSAPILoadBalancerisusedforGCPandvSpherewithoutNSX-Tdeployments.IfPKSisdeployedonvSpherewithNSX-T,aDNATruleisconfiguredforthePKSAPIhostsothatitisaccessible.Formoreinformation,seetheSharethePKSAPIEndpointsectioninInstallingPKSonvSpherewithNSX-TIntegration.

UAA

WhenauserlogsintoorlogsoutofthePKSAPIthroughthePKSCLI,thePKSCLIcommunicateswithUAAtoauthenticatethem.ThePKSAPIpermitsonlyauthenticateduserstomanageKubernetesclusters.Formoreinformationaboutauthenticating,seePKSAPIAuthentication.

UAAmustbeconfiguredwiththeappropriateusersanduserpermissions.Formoreinformation,seeManagingUsersinPKSwithUAA.

PKSAPI

ThroughthePKSCLI,usersinstructthePKSAPIservertodeploy,scaleup,anddeleteKubernetesclustersaswellasshowclusterdetailsandplans.ThePKSAPIcanalsowriteKubernetesclustercredentialstoalocalkubeconfigfile,whichenablesuserstoconnecttoaclusterthrough kubectl .

ThePKSAPIsendsallclustermanagementrequests,exceptread-onlyrequests,tothePKSBroker.

PKSBroker

WhenthePKSAPIreceivesarequesttomodifyaKubernetescluster,itinstructsthePKSBrokertomaketherequestedchange.

ThePKSBrokerconsistsofanOn-DemandServiceBroker andaServiceAdapter.ThePKSBrokergeneratesaBOSHmanifestandinstructstheBOSHDirectortodeployordeletetheKubernetescluster.

ForPKSdeploymentsonvSpherewithNSX-T,thereisanadditionalcomponent,thePKSNSX-TProxyBroker.ThePKSAPIcommunicateswiththePKSNSX-TProxyBroker,whichinturncommunicateswiththeNSXManagertoprovisiontheNodeNetworkingresources.ThePKSNSX-TProxyBrokerthenforwardstherequesttotheOn-DemandServiceBrokertodeploythecluster.


https://docs.pivotal.io/svc-sdk/odb/index.html

ClusterWorkloadManagementPKSusersmanagetheircontainer-basedworkloadsonKubernetesclustersthrough kubectl .Formoreinformationabout kubectl ,seeOverviewofkubectl intheKubernetesdocumentation.

[email protected].


https://kubernetes.io/docs/reference/kubectl/overview/mailto:[email protected]

PKSAPIAuthenticationPagelastupdated:

ThistopicdescribeshowthePivotalContainerService(PKS)APIworkswithUserAccountandAuthentication(UAA)tomanageauthenticationandauthorizationinyourPKSdeployment.

AuthenticatingPKSAPIRequestsBeforeuserscanloginandusethePKSCLI,youmustconfigurePKSAPIaccesswithUAA.FOrmoreinformation,seeConfiguringPKSAPIAccess.YouusetheUAACommandLineInterface(UAAC)totargettheUAAserverandrequestanaccesstokenfortheUAAadminuser.Ifyourrequestissuccessful,theUAAserverreturnstheaccesstoken.TheUAAadminaccesstokenauthorizesyoutomakerequeststothePKSAPIusingthePKSCLIandgrantclusteraccesstoneworexistingusers.Formoreinformation,GrantClusterAccessinManagingUsersinPKSwithUAA.

WhenauserwithclusteraccesslogsintothePKSCLI,theCLIrequestsanaccesstokenfortheuserfromtheUAAserver.Iftherequestissuccessful,theUAAserverreturnsanaccesstokentothePKSCLI.WhentheuserrunsPKSCLIcommands,forexample, pksclusters ,theCLIsendstherequesttothePKSAPIserverandincludestheuser’sUAAtoken.

ThePKSAPIsendsarequesttotheUAAservertovalidatetheuser’stoken.IftheUAAserverconfirmsthatthetokenisvalid,thePKSAPIusestheclusterinformationfromthePKSbrokertorespondtotherequest.Forexample,iftheuserruns pksclusters ,theCLIreturnsalistoftheclustersthattheuserisauthorizedtomanage.

RoutingtothePKSAPIControlPlaneVMThePKSAPIserverandtheUAAserverusedifferentportnumbersonthecontrolplaneVM.Forexample,ifyourPKSAPIdomainis api.pks.example.com ,youcanreachyourPKSAPIandUAAserversatthefollowingURLs:

Server URL

PKSAPI api.pks.example.com:9021

UAA api.pks.example.com:8443

RefertoOpsManager>PivotalContainerService>PKSAPI>APIHostname(FQDN)foryourPKSAPIdomain.

Loadbalancerimplementationsdifferbydeploymentenvironment.ForPKSdeploymentsonGCPoronvSpherewithoutNSX-T,whenyouinstallthePKStile,youconfigurealoadbalancertoaccessthePKSAPI.Formoreinformation,seetheConfigureExternalLoadBalancersectionofInstallingPKSforyourIaaS.

ForproceduresthatdescriberoutingtothePKScontrolplaneVM,seetheConfigureExternalLoadBalancersectionofInstallingPKSforyourIaaS.

ForoverviewinformationaboutloadbalancersinPKS,seeLoadBalancersinPKSDeploymentswithoutNSX-T.

[email protected].



LoadBalancersinPKSPagelastupdated:

ThistopicdescribesthetypesofloadbalancersthatareusedinPivotalContainerService(PKS)deployments.Loadbalancersdifferbythetypeofdeployment.

LoadBalancersinPKSDeploymentswithoutNSX-TForPKSdeploymentsonGCPorvSpherewithoutNSX-T,youcanconfigureloadbalancersforthefollowing:

PKSAPI:ConfiguringthisloadbalancerallowsyoutorunPKSCommandLineInterface(CLI)commandsfromyourlocalworkstation.

KubernetesClusters:ConfiguringaloadbalancerforeachnewclusterallowsyoutorunKubernetesCLI(kubectl)commandsonthecluster.

Workloads:Configuringaloadbalancerforyourapplicationworkloadsallowsexternalaccesstotheservicesthatrunonyourcluster.

ThefollowingdiagramshowswhereeachoftheaboveloadbalancerscanbeusedwithinyourPKSdeploymentonGCPoronvSpherewithoutNSX-T:

IfyouuseeithervSpherewithoutNSX-TorGCP,youareexpectedtocreateyourownloadbalancerswithinyourcloudproviderconsole.Ifyourcloudproviderdoesnotofferloadbalancing,youcanuseanyexternalTCPorHTTPSloadbalancerofyourchoice.

AboutthePKSAPILoadBalancerForPKSdeploymentsonGCPandonvSpherewithoutNSX-T,theloadbalancerforthePKSAPIallowsyoutoaccessthePKSAPIfromoutsidethenetwork.


Forexample,configuringaloadbalancerforthePKSAPIallowsyoutorunPKSCLIcommandsfromyourlocalworkstation.

ForinformationaboutconfiguringthePKSAPIloadbalancer,seetheConfigureExternalLoadBalancersectionofInstallingPKSforyourIaaS.

AboutKubernetesClusterLoadBalancersForPKSdeploymentsonGCPandonvSpherewithoutNSX-T,whenyoucreateacluster,youmustconfigureexternalaccesstotheclusterbycreatinganexternalTCPorHTTPSloadbalancer.TheloadbalancerallowstheKubernetesCLItocommunicatewiththecluster.

Ifyoucreateaclusterinanon-productionenvironment,youcanchoosenottousealoadbalancer.Toallowkubectltoaccesstheclusterwithoutaloadbalancer,youcandooneofthefollowing:

CreateaDNSentrythatpointstothecluster’smasterVM.Forexample:

my-cluster.example.com A 10.0.0.5

Ontheworkstationwhereyourunkubectlcommands,addthemasterIPaddressofyourclusterand kubo.internal tothe /etc/hosts file.Forexample:

10.0.0.5 kubo.internal

Forinformationaboutconfiguringaclusterloadbalancer,seeCreatingClusters.

AboutWorkloadLoadBalancersForPKSdeploymentsonGCPandonvSpherewithoutNSX-T,toallowexternalaccesstoyourapp,youcaneithercreatealoadbalancerorexposeastaticportonyourworkload.

Forinformationaboutconfiguringaloadbalancerforyourappworkload,seeDeployingandAccessingBasicWorkloads.

LoadBalancersinPKSDeploymentsonvSpherewithNSX-TPKSdeploymentsonvSpherewithNSX-TdonotrequirealoadbalancerconfiguredtoaccessthePKSAPI.TheyrequireonlyaDNATruleconfiguredsothatthePKSAPIhostisaccessible.Formoreinformation,seeRetrievethePKSEndpointinInstallingPKSonvSpherewithNSX-TIntegration.

NSX-Thandlesloadbalancercreation,configuration,anddeletionautomaticallyaspartoftheKubernetesclustercreate,update,anddeleteprocess.WhenanewKubernetesclusteriscreated,NSX-Tcreatesandconfiguresadedicatedloadbalancertiedtoit.Theloadbalancerisasharedresourcedesignedtoprovideefficienttrafficdistributiontomasternodesaswellasservicesdeployedonworkernodes.Eachapplicationserviceismappedtoavirtualserverinstance,carvedoutfromthesameloadbalancer.Formoreinformation,seeLogicalLoadBalancer intheNSX-Tdocumentation.

Virtualserverinstancesarecreatedontheloadbalancertoprovideaccesstothefollowing:

KubernetesAPIandUIservicesonaKubernetescluster.Thisallowsrequeststobeloadbalancedacrossmultiplemasternodes.

Ingresscontroller.ThisallowsthevirtualserverinstancetodispatchHTTPandHTTPSrequeststoservicesassociatedwithIngressrules.

type:loadbalancer services.ThisallowstheservertohandleTCPconnectionsorUDPflowstowardexposedservices.

Loadbalancersaredeployedinhigh-availabilitymodesothattheyareresilienttopotentialfailuresandabletorecoverquicklyfromcriticalconditions.

ResizingLoadBalancersWhenanewKubernetesclusteriscreatedusingthePKSAPI,NSX-Tcreatesadedicatedloadbalancerforthatnewcluster.Bydefault,thesizeoftheloadbalancerissettoSmallinNSXManager.ASmallsizedloadbalancerislimitedtoamaximumof10NSX-Tvirtualservers.

Note:The NodePort ServicetypeisnotsupportedforPKSdeploymentsonvSpherewithNSX-T.Only type:LoadBalancer ServicesandServicesassociatedwithIngressrulesaresupportedonvSpherewithNSX-T.

Note:PivotalrecommendschangingthesizeofyourNSX-TloadbalancerfromSmalltoMediuminNSXManager.Doingsoincreasesyourvirtualserverlimitfrom10to100.


https://docs.vmware.com/en/VMware-NSX-T/2.1/com.vmware.nsxt.admin.doc/GUID-46567C8D-A5C5-4793-8CDF-858E58FDE3C4.html

[email protected].



VMSizingforPKSClustersPagelastupdated:

ThistopicdescribeshowPivotalContainerService(PKS)recommendsyouapproachthesizingofVMsforclustercomponents.

OverviewWhenyouconfigureplansinthePKStile,youprovideVMsizesforthemasterandworkernodeVMs.Formoreinformationaboutconfiguringplans,seethePlanssectionofInstallingPKSforyourIaaS:

vSphere


GoogleCloudPlatform(GCP)

PKSdeterminesthesizeofthemasternodeVMsautomaticallybasedonthenumberofworkernodeVMs.Youselectthenumberofmasternodeswhenyouconfiguretheplan.

ForworkernodeVMs,youselectthenumberandsizebasedontheneedsofyourworkload.ThesizingofmasterandworkernodeVMsishighlydependentonthecharacteristicsoftheworkload.Adapttherecommendationsinthistopicbasedonyourownworkloadrequirements.

MasterNodeVMSizeThemasternodeVMsizeislinkedtothenumberofworkernodes.TheVMsizingshowninthefollowingtableispermasternode:

NumberofWorkers CPU RAM(GB)

1-5 1 3.75

6-10 2 7.5

11-100 4 15

101-250 8 30

251-500 16 60

500+ 32 120

WorkerNodeVMNumberandSizeAmaximumof100podscanrunonasingleworkernode.TheactualnumberofpodsthateachworkernoderunsdependsontheworkloadtypeaswellastheCPUandmemoryrequirementsoftheworkload.

TocalculatethenumberandsizeofworkerVMsyourequire,determinethefollowingforyourworkload:

Maximumnumberofpodsyouexpecttorun[ p ]

Memoryrequirementsperpod[ m ]

CPUrequirementsperpod[ c ]

Usingthevaluesabove,youcancalculatethefollowing:

Minimumnumberofworkers[ W ]= p / 100

MinimumRAMperworker= m * 100

MinimumnumberofCPUsperworker= c * 100

Thiscalculationgivesyoutheminimumnumberofworkernodesyourworkloadrequires.Werecommendthatyouincreasethisvaluetoaccountfor

Note:Iftherearemultiplemasternodes,allmasternodeVMsarethesamesize.Toconfigurethenumberofmasternodes,seethePlanssectionofInstallingPKSforyourIaaS.


failuresandupgrades.

Forexample,increasethenumberofworkernodesbyatleastonetomaintainworkloaduptimeduringanupgrade.Additionally,increasethenumberofworkernodestofityourownfailuretolerancecriteria.

ThemaximumnumberofworkernodesthatyoucancreateforaPKS-provisionedKubernetesclusteris50.

ExampleWorkerNodeRequirementCalculationAnexampleapphasthefollowingminimumrequirements:

Numberofpods[ p ]=1000

RAMperpod[ m ]=1GB

CPUperpod[ c ]=0.10

TodeterminehowmanyworkernodeVMstheapprequires,dothefollowing:

1. Calculatethenumberofworkersusing p / 100 :

1000/100 = 10 workers

2. CalculatetheminimumRAMperworkerusing m * 100 :

1 * 100 = 100 GB

3. CalculatetheminimumnumberofCPUsperworkerusing c * 100 :

0.10 * 100 = 10 CPUs

4. Forupgrades,increasethenumberofworkersbyone:

10 workers + 1 worker = 11 workers

5. Forfailuretolerance,increasethenumberofworkersbytwo:

11 workers + 2 workers = 13 workers

Intotal,thisappworkloadrequires13workerswith10CPUsand100GBRAM.

[email protected].



PKSTelemetryPagelastupdated:

ThistopicdescribesthemetricsthatthePivotalContainerService(PKS)tilesendswhenyouenabletheVMwareCustomerExperienceImprovementProgram(CEIP)orthePivotalTelemetryProgram(Telemetry).YoucanoptinoroptoutofeitherprogramintheUsageDatapaneofthePKStile.

Formoreinformation,seetheInstallingPKStopicforyourIaaS:

vSphere


GoogleCloudPlatform(GCP)

EventEnvelopePropertiesWhenPKSsendsmetricstoCEIPorTelemetry,thetilepackagesthedatawiththefollowingdeploymentinformation:

PropertyName PropertyDescription ExampleData AddedinPKSVersion

event Thetypeofevent create_cluster v1.1

product_version PKStileversion 1.2.0-build.40 v1.1

cloud_provider CloudproviderforthePKSinstallation GCP v1.1

vcenter_id vCenterID 00000a11-22bb-3333-4c4c-555566667777 v1.1

ClusterEventsPKSsendsmetricsfortheclustermanagementeventsshowninthetablebelow:

EventName EventDescription PropertyName PropertyDescription AddedinPKSVersion

create_cluster Thiseventisgeneratedwhenausercreatesacluster.

user_id Ahashedvalueoftheusername. v1.1

timestamp Thetimewhentheusercreatedthecluster. v1.1

plan_nameThenameofthePKSplanthatwasusedtocreatethecluster.

v1.1

plan_id TheIDofthePKSplanthatwasusedtocreatethecluster.

v1.1

cluster_name Thenameofthecluster. v1.1

cluster_id TheIDofthecluster. v1.1

number_of_workersThenumberofworkernodeVMsinthecluster.

v1.1

resize_clusterThiseventisgeneratedwhenaclusterisresized.




v1.1

plan_idTheIDofthePKSplanthatwasusedtocreatethecluster.

v1.1



old_number_of_workersThenumberofworkernodeVMsintheclusterbeforetheresizeevent.

v1.1

new_number_of_workersThenumberofworkernodeVMsintheclusteraftertheresizeevent.

v1.1




delete_clusterThiseventisgeneratedwhenauserdeletesacluster.



v1.1

plan_idTheIDofthePKSplanthatwasusedtocreatethecluster.

v1.1



ClusterMetricsPKSsendsbothagentmetricsandclusterpodmetricsforeachcluster.

Thefollowingtabledescribesclusteragentmetrics:

AgentMetricName AgentMetricDescription Example AddedinPKSVersion

agentid TheuniqueBOSH-generateddeploymentnameforthecluster.service-instance_00000a11-22bb-3333-4c4c-555566667777

v1.1

isvrlienabledIfvRealizeLogInsight(vRLI)isenabled,thisvalueistrue.IfvRLIisdisabled,thisvalueiffalse.

true v1.1

isvropsenabledIfvRealizeOperations(vROps)isenabled,thisvalueistrue.IfvROpsisdisabled,thisvalueisfalse.

false v1.1

iswavefrontenabledIfWavefrontisenabled,thisvalueistrue.IfWavefrontisdisabled,thisvalueisfalse.

true v1.1

vcenter_id ThisisyourvCenterID.00000a11-22bb-3333-4c4c-555566667777

v1.1

Thefollowingtabledescribesclusterpodmetrics:

ClusterPodMetricName ClusterPodMetricDescription Example AddedinPKSVersion

collected_atThistimestamprepresentsthemetriccollectiontimeontheagent.

2018-05-3121:45:27.681UTC v1.1

cpu_usedThisvaluerepresentshowmuchCPUwasinuseatthetimewhentheeventhappened.

11412427 v1.1

memory_usedThisvaluerepresentshowmuchmemorywasinuseatthetimewhentheeventhappened.

4816896 v1.1

pkst_kubernetesclusterinfo__fkThisvalueisaforeignkeythatpointstoanentryinthepkst_kubernetesclusterinfodatabase.

77777a66-55bb-4444-3c3c-222211110000

v1.1

[email protected].



PASandPKSDeploymentswithOpsManagerPagelastupdated:

OpsManagerisawebappthatyouusetodeployandmanagePivotalApplicationService(PAS)andPivotalContainerService(PKS).ThistopicexplainswhyPivotalrecommendsusingseparateinstallationsofOpsManagerforPASandPKS.

FormoreinformationaboutdeployingPKS,seeInstallingPKS.

SecurityOpsManagerdeploysthePASandPKSruntimeplatformsusingBOSH.Forsecurityreasons,PivotaldoesnotrecommendinstallingPASandPKSonthesameOpsManagerinstance.Forevenstrongersecurity,PivotalrecommendsdeployingeachOpsManagerinstanceusingauniquecloudprovideraccount.

TileConfigurationandTroubleshootingSeparateinstallationsofOpsManagerallowyoutocustomizeandtroubleshootruntimetilesindependently.YoumaychoosetoconfigureOpsManagerwithdifferentsettingsforyourPASandPKSdeployments.

Forexample,PKSandmanyPASfeaturesdependonBOSHDNS.IfyoudeployPAStoaseparateOpsManagerinstance,youcandisableBOSHDNSfortroubleshootingpurposes.PAScanrunwithoutBOSHDNS,butkeyfeaturessuchassecureservicecredentialswithCredHub,servicediscoveryforcontainer-to-containernetworking,andNSX-TintegrationdonotworkwhenBOSHDNSisdisabled.

IfyoudeployPASandPKStothesameOpsManagerinstance,youcannotdisableBOSHDNSwithoutbreakingyourPKSinstallationalongwiththePASfeaturesthatdependonBOSHDNS.

[email protected].



InstallingPKSPagelastupdated:

YoucaninstallPivotalContainerService(PKS)onGoogleCloudPlatform(GCP)orvSphere.Forinstallationinstructions,seethefollowing:

vSphere


GCP

[email protected].



vSphereThistopicliststhestepstofollowwheninstallingPivotalContainerService(PKS)onvSphere.

InstallingPKSToinstallPKS,followtheinstructionsbelow:

PrerequisitesandResourceRequirements

PreparingvSphereBeforeDeployingPKS

DeployingOpsManageronvSphere:

DeployingBOSHandOpsManagerv2.1tovSphere DeployingBOSHandOpsManagerv2.2tovSphere

ConfiguringOpsManageronvSphere:

ConfiguringBOSHDirectorv2.1onvSphere ConfiguringBOSHDirectorv2.2onvSphere

InstallingPKSonvSphere

(Optional)IntegratingVMwareHarborwithPKS

InstallingthePKSandKubernetesCLIsThePKSandKubernetesCLIshelpyouinteractwithyourPKS-provisionedKubernetesclustersandKubernetesworkloads.ToinstalltheCLIs,followtheinstructionsbelow:

InstallingthePKSCLI

InstallingtheKubernetesCLI

[email protected].


https://docs.pivotal.io/pcf/om/2-1/vsphere/deploy.htmlhttps://docs.pivotal.io/pcf/om/2-2/vsphere/deploy.htmlhttps://docs.pivotal.io/pcf/om/2-1/vsphere/config.htmlhttps://docs.pivotal.io/pcf/om/2-2/vsphere/config.htmlhttps://docs.pivotal.io/partners/vmware-harbor/integrating-pks.htmlmailto:[email protected]

vSpherePrerequisitesandResourceRequirementsPagelastupdated:

ThistopicdescribestheprerequisitesandresourcerequirementsforinstallingPivotalContainerService(PKS)onvSphere.

ForprerequisitesandresourcerequirementsforinstallingPKSonvSpherewithNSX-Tintegration,seevSpherewithNSX-TPrerequisitesandResourceRequirements.

PKSsupportsair-gappeddeploymentsonvSpherewithorwithoutNSX-Tintegration.

YoucanalsoconfigureintegrationwiththeHarbortile,anenterprise-classregistryserverforcontainerimages.Formoreinformation,seeVMwareHarborRegistry inthePivotalPartnerdocumentation.

PrerequisitesBeforeinstallingPKS,youmustinstallOpsManager.YouuseOpsManagertoinstallandconfigurePKS.

ToprepareyourvSphereenvironmentforinstallingOpsManagerandPKS,reviewthesectionsbelowandthenfollowtheinstructionsinPreparingvSphereBeforeDeployingPKS.

vSphereVersionRequirementsOpsManagerandPKSsupportthefollowingvSpherecomponentversions:

Versions Editions

VMwarevSphere6.5U2

VMwarevSphere6.5U1

vSphereEnterprisePlus

vSpherewithOperationsManagementEnterprisePlus

PKSv1.1.2andlaterarecompatiblewithvSphere6.5U2.

ResourceRequirementsInstallingOpsManagerandPKSrequiresthefollowingvirtualmachines(VMs):

VM CPU RAM Storage

PivotalContainerService 2 8GB 16GB

PivotalOpsManager 1 8GB 160GB

BOSHDirector 2 8GB 16GB

EachPKSdeploymentrequiresephemeralVMsduringinstallationandupgradesofPKS.AfteryoudeployPKS,BOSHautomaticallydeletestheseVMs.

ToenablePKStodynamicallycreatetheephemeralVMswhenneeded,ensurethatthefollowingresourcesareavailableinyourvSphereinfrastructurebeforedeployingPKS:

EphemeralVM Number CPUCores RAM EphemeralDisk

BOSHCompilationVMs 4 4 4GB 32GB

EachKubernetesclusterprovisionedthroughPKSdeploystheVMslistedbelow.IfyoudeploymorethanoneKubernetescluster,youmustscaleyourallocatedresourcesappropriately.

VM Number CPUCores RAM EphemeralDisk PersistentDisk

master 1or3 2 4GB 8GB 5GB

worker 1ormore 2 4GB 8GB 50GB

errand(ephemeral) 1 1 1GB 8GB none

*

*


https://docs.pivotal.io/partners/vmware-harbor/index.html

[email protected].



PreparingvSphereBeforeDeployingPKSPagelastupdated:

BeforeyouinstallPivotalContainerService(PKS)onvSpherewithoutNSX-Tintegration,youmustprepareyourvSphereenvironment.InadditiontofulfillingtheprerequisitesspecifiedinvSpherePrerequisitesandResourceRequirements,youmustcreatethefollowingtwoserviceaccountsinvSphere:

MasterNodeServiceAccount:YoumustcreateaserviceaccountforKubernetesclustermasterVMs.

BOSH/OpsManagerServiceAccount:YoumustcreateaserviceaccountforBOSHandOpsManager.

Afteryoucreatetheserviceaccountslistedabove,youmustgrantthemprivilegesinvSphere.Pivotalrecommendsconfiguringeachserviceaccountwiththeleastpermissiveprivilegesanduniquecredentials.

Forthemasternodeserviceaccount,youcancreateacustomroleinvSpherebasedonyourstorageconfiguration.KubernetesmasternodeVMsrequirestoragepermissionstocreateloadbalancersandattachpersistentdiskstopods.CreatingacustomroleallowsvSpheretoapplythesameprivilegestoallKubernetesmasternodeVMsinyourPKSinstallation.

WhenyouconfiguretheKubernetesCloudProviderpaneofthePKStile,youenterthemasternodeserviceaccountcredentialsinthevSphereMasterCredentialsfields.

Formoreinformation,seetheKubernetesCloudProvidersectionofInstallingPKSonvSphere.

FortheBOSH/OpsManagerserviceaccount,youcanapplyprivilegesdirectlytotheserviceaccountwithoutcreatingarole.YoucanalsoapplythedefaultVMwareAdministratorSystemRole totheserviceaccounttoachievetheappropriatepermissionlevel.

Step1:CreatetheMasterNodeServiceAccount1. FromthevCenterconsole,createaserviceaccountforKubernetesclustermasterVMs.

2. GrantthefollowingVirtualMachineObjectprivilegestotheserviceaccount:

Privilege(UI) Privilege(API)

VirtualMachine>Configuration>Advanced VirtualMachine.Configuration.Advanced

VirtualMachine>Configuration>Settings VirtualMachine.Configuration.Settings

Step2:GrantStoragePermissionsKubernetesmasternodeVMserviceaccountsrequirethefollowing:

Readaccesstothefolder,host,anddatacenteroftheclusternodeVMs

PermissiontocreateanddeleteVMswithintheresourcepoolwherePKSisdeployed

Grantthesepermissionstothemasternodeserviceaccountbasedonyourstorageconfigurationusingoneoftheproceduresbelow:

StaticOnlyPersistentVolumeProvisioning

DynamicPersistentVolumeProvisioning(withStoragePolicy-BasedVolumePlacement)

DynamicPersistentVolumeProvisioning(withoutStoragePolicy-BasedVolumePlacement)

FormoreinformationaboutvSpherestorageconfigurations,seevSphereStorageforKubernetes intheVMwarevSpheredocumentation.

StaticOnlyPersistentVolumeProvisioningToconfigureyourKubernetesmasternodeserviceaccountusingstaticonlyPersistentVolume(PV)provisioning,dothefollowing:

1. CreateacustomrolethatallowstheserviceaccounttomanageKubernetesnodeVMs.Givethisroleaname.Forexample,manage-k8s-node-vms .FormoreinformationaboutcustomrolesinvCenter,seeCreateaCustomRole intheVMwarevSpheredocumentation.

Note:IfyourKubernetesclustersspanmultiplevCenters,youmustsettheserviceaccountprivilegescorrectlyineachvCenter.


http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.pg.doc/PG_Authenticate_Authorize.8.6.html#1110514https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/index.htmlhttps://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-41E5E52E-A95B-4E81-9724-6AD6800BEF78.html

a. GrantthefollowingprivilegesattheVMFolderlevelusingeitherthevCenterUIorAPI:


VirtualMachine>Configuration>Addexistingdisk VirtualMachine.Config.AddExistingDisk

VirtualMachine>Configuration>Addnewdisk VirtualMachine.Config.AddNewDisk

VirtualMachine>Configuration>Addorremovedevice VirtualMachine.Config.AddRemoveDevice

VirtualMachine>Configuration>Removedisk VirtualMachine.Config.RemoveDisk

b. SelectthePropagatetoChildObjectscheckbox.

2. (Optional)CreateacustomrolethatallowstheserviceaccounttomanageKubernetesvolumes.Givethisroleaname.Forexample,manage-k8s-volumes .

a. GrantthefollowingprivilegeattheDatastorelevelusingeitherthevCenterUIorAPI:


Datastore>Lowlevelfileoperations Datastore.FileManagement

b. ClearthePropagatetoChildObjectscheckbox.

3. GranttheserviceaccounttheexistingRead-onlyrole.ThisroleincludesthefollowingprivilegesatthevCenter,Datacenter,DatastoreCluster,andDatastoreStorageFolderlevels:


Read-only System.Anonymous

System.Read

System.View

4. ContinuetoStep3:CreatetheBOSH/OpsManagerServiceAccount.

DynamicPersistentVolumeProvisioning(withStoragePolicy-BasedVolumePlacement)ToconfigureyourKubernetesmasternodeserviceaccountusingdynamicPVprovisioningwithstoragepolicy-basedplacement,dothefollowing:


a. GrantthefollowingprivilegesattheCluster,Hosts,andVMFolderlevelsusingeitherthevCenterUIorAPI:


VirtualMachine>Resource>Assignvirtualmachinetoresourcepool Resource.AssignVMToPool





VirtualMachine>Inventory>Createnew VirtualMachine.Inventory.Create

VirtualMachine>Inventory>Remove VirtualMachine.Inventory.Delete


2. CreateacustomrolethatallowstheserviceaccounttomanageKubernetesvolumes.Givethisroleaname.Forexample,manage-k8s-volumes .



Datastore>Allocatespace Datastore.AllocateSpace



Note:ThisroleisrequiredifyoucreateaPersistentVolumeClaim(PVC)tobindwithastaticallyprovisionedPV,andthereclaimpolicyissettodelete.WhenthePVCisdeleted,thestaticallyprovisionedPVisalsodeleted.


https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-41E5E52E-A95B-4E81-9724-6AD6800BEF78.html

3. CreateacustomrolethatallowstheserviceaccounttoreadtheKubernetesstorageprofile.Givethisroleaname.Forexample, k8s-system-read-and-spbm-profile-view .

a. GrantthefollowingprivilegeatthevCenterlevelusingeitherthevCenterUIorAPI:


Profile-drivenstorageview StorageProfile.View





System.Read

System.View

5. ContinuetoStep3:CreatetheBOSH/OpsManagerServiceAccount.

DynamicVolumeProvisioning(withoutStoragePolicy-BasedVolumePlacement)ToconfigureyourKubernetesmasternodeserviceaccountusingdynamicPVprovisioningwithoutstoragepolicy-basedplacement,dothefollowing:


a. GrantthefollowingprivilegesattheCluster,Hosts,andVMFolderlevelsusingeitherthevCenterUIorAPI:







2. CreateacustomrolethatallowstheserviceaccounttomanageKubernetesvolumes.Givethisroleaname.Forexample,manage-k8s-volumes .



Datastore>Allocatespace Datastore.AllocateSpace






System.Read

System.View

Step3:CreatetheBOSH/OpsManagerServiceAccount1. FromthevCenterconsole,createaserviceaccountforBOSHandOpsManager.

2. GrantthepermissionsbelowtotheBOSHandOpsManagerserviceaccount.

Note:TheprivilegeslistedinthissectiondescribetheminimumrequiredpermissionstodeployBOSH.Youcanalsoapplythedefault


https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-41E5E52E-A95B-4E81-9724-6AD6800BEF78.html

vCenterRootPrivilegesGrantthefollowingprivilegesontherootvCenterserverentitytotheserviceaccount:



System.Read

System.View

Managecustomattributes Global.ManageCustomFields

vCenterDatacenterPrivilegesGrantthefollowingprivilegesonanyentitiesinadatacenterwhereyoudeployPKS:

RoleObject


UsersinherittheRead-OnlyrolefromthevCenterrootlevel System.Anonymous

System.Read

System.View

DatastoreObject

Grantthefollowingprivilegesmustatthedatacenterleveltouploadanddeletevirtualmachinefiles:


Allocatespace Datastore.AllocateSpace

Browsedatastore Datastore.Browse

Lowlevelfileoperations Datastore.FileManagement

Removefile Datastore.DeleteFile

Updatevirtualmachinefiles Datastore.UpdateVirtualMachineFiles

FolderObject


Deletefolder Folder.Delete

Createfolder Folder.Create

Movefolder Folder.Move

Renamefolder Folder.Rename

GlobalObject


Setcustomattribute Global.SetCustomField

VMwareAdministratorSystemRole totheserviceaccounttoachievetheappropriatepermissionlevel,butthedefaultroleincludesmoreprivilegesthanthoselistedbelow.


http://pubs.vmware.com/vsphere-51/index.jsp#com.vmware.wssdk.pg.doc/PG_Authenticate_Authorize.8.6.html#1110514

HostObject


Modifycluster Host.Inventory.EditCluster

InventoryServiceObject


vSphereTagging>CreatevSphereTag InventoryService.Tagging.CreateTag

vSphereTagging>DeletevSphereTag InventoryService.Tagging.EditTag

vSphereTagging>EditvSphereTag InventoryService.Tagging.DeleteTag

NetworkObject


Assignnetwork Network.Assign

ResourceObject


Assignvirtualmachinetoresourcepool Resource.AssignVMToPool

Migratepoweredoffvirtualmachine Resource.ColdMigrate

Migratepoweredonvirtualmachine Resource.HotMigrate

vAppObject

Granttheseprivilegesattheresourcepoollevel.


Import VApp.Import

vAppapplicationconfiguration VApp.ApplicationConfig

VirtualMachineObject

Configuration


Addexistingdisk VirtualMachine.Config.AddExistingDisk

Addnewdisk VirtualMachine.Config.AddNewDisk

Addorremovedevice VirtualMachine.Config.AddRemoveDevice

Advanced VirtualMachine.Config.AdvancedConfig

ChangeCPUcount VirtualMachine.Config.CPUCount

Changeresource VirtualMachine.Config.Resource

ConfiguremanagedBy VirtualMachine.Config.ManagedBy

Diskchangetracking VirtualMachine.Config.ChangeTracking

Disklease VirtualMachine.Config.DiskLease

Displayconnectionsettings VirtualMachine.Config.MksControl

Extendvirtualdisk VirtualMachine.Config.DiskExtend

Memory VirtualMachine.Config.Memory

Modifydevicesettings VirtualMachine.Config.EditDevice


Rawdevice VirtualMachine.Config.RawDevice

Reloadfrompath VirtualMachine.Config.ReloadFromPath

Removedisk VirtualMachine.Config.RemoveDisk

Rename VirtualMachine.Config.Rename

Resetguestinformation VirtualMachine.Config.ResetGuestInfo

Setannotation VirtualMachine.Config.Annotation

Settings VirtualMachine.Config.Settings

Swapfileplacement VirtualMachine.Config.SwapPlacement

Unlockvirtualmachine VirtualMachine.Config.Unlock

GuestOperations


GuestOperationProgramExecution VirtualMachine.GuestOperations.Execute

GuestOperationModifications VirtualMachine.GuestOperations.Modify

GuestOperationQueries VirtualMachine.GuestOperations.Query

Interaction


Answerquestion VirtualMachine.Interact.AnswerQuestion

ConfigureCDmedia VirtualMachine.Interact.SetCDMedia

Consoleinteraction VirtualMachine.Interact.ConsoleInteract

Defragmentalldisks VirtualMachine.Interact.DefragmentAllDisks

Deviceconnection VirtualMachine.Interact.DeviceConnection

GuestoperatingsystemmanagementbyVIXAPI VirtualMachine.Interact.GuestControl

Poweroff VirtualMachine.Interact.PowerOff

Poweron VirtualMachine.Interact.PowerOn

Reset VirtualMachine.Interact.Reset

Suspend VirtualMachine.Interact.Suspend

VMwareToolsinstall VirtualMachine.Interact.ToolsInstall

Inventory


Createfromexisting VirtualMachine.Inventory.CreateFromExisting

Createnew VirtualMachine.Inventory.Create

Move VirtualMachine.Inventory.Move

Register VirtualMachine.Inventory.Register

Remove VirtualMachine.Inventory.Delete

Unregister VirtualMachine.Inventory.Unregister

Provisioning


Allowdiskaccess VirtualMachine.Provisioning.DiskRandomAccess

Allowread-onlydiskaccess VirtualMachine.Provisioning.DiskRandomRead

Allowvirtualmachinedownload VirtualMachine.Provisioning.GetVmFiles

Allowvirtualmachinefilesupload VirtualMachine.Provisioning.PutVmFiles

Clonetemplate VirtualMachine.Provisioning.CloneTemplate

Clonevirtualmachine VirtualMachine.Provisioning.Clone

Customize VirtualMachine.Provisioning.Customize


Deploytemplate VirtualMachine.Provisioning.DeployTemplateMarkastemplate VirtualMachine.Provisioning.MarkAsTemplate

Markasvirtualmachine VirtualMachine.Provisioning.MarkAsVM

Modifycustomizationspecification VirtualMachine.Provisioning.ModifyCustSpecs

Promotedisks VirtualMachine.Provisioning.PromoteDisks

Readcustomizationspecifications VirtualMachine.Provisioning.ReadCustSpecs

SnapshotManagement


Createsnapshot VirtualMachine.State.CreateSnapshot

Removesnapshot VirtualMachine.State.RemoveSnapshot

Renamesnapshot VirtualMachine.State.RenameSnapshot

Revertsnapshot VirtualMachine.State.RevertToSnapshot

NextStepsAfteryoucompletetheinstructionsprovidedinthistopic,installoneofthefollowing:

PivotalOpsManagerv2.1.x

PivotalOpsManagerv2.2.x

ToinstallanOpsManagerversionthatiscompatiblewiththePKSversionyouintendtouse,followtheinstructionsinthecorrespondingversionoftheOpsManagerdocumentation.

Version

OpsManagerv2.1DeployingBOSHandOpsManagertovSphere

ConfiguringBOSHDirectoronvSphere

OpsManagerv2.2DeployingBOSHandOpsManagertovSphere

ConfiguringBOSHDirectoronvSphere

[email protected].

Note:YouuseOpsManagertoinstallandconfigurePKS.EachversionofOpsManagersupportsmultipleversionsofPKS.ToconfirmthatyourOpsManagerversionsupportstheversionofPKSthatyouinstall,seePKSReleaseNotes.


https://docs.pivotal.io/pcf/om/2-1/vsphere/deploy.htmlhttps://docs.pivotal.io/pcf/om/2-1/vsphere/config.htmlhttps://docs.pivotal.io/pcf/om/2-2/vsphere/deploy.htmlhttps://docs.pivotal.io/pcf/om/2-2/vsphere/config.htmlmailto:[email protected]

InstallingPKSonvSpherePagelastupdated:

ThistopicdescribeshowtoinstallandconfigurePivotalContainerService(PKS)onvSphere.

PrerequisitesBeforeperformingtheproceduresinthistopic,youmusthavedeployedandconfiguredOpsManager.Formoreinformation,seevSpherePrerequisitesandResourceRequirements.

IfyouuseaninstanceofOpsManagerthatyouconfiguredpreviouslytoinstallotherruntimes,confirmthefollowingsettingsbeforeyouinstallPKS:

1. NavigatetoOpsManager.

2. OpentheDirectorConfigpane.

3. SelecttheEnablePostDeployScriptscheckbox.

4. CleartheDisableBOSHDNSserverfortroubleshootingpurposescheckbox.

5. ClicktheInstallationDashboardlinktoreturntotheInstallationDashboard.

6. ClickApplyChanges.

Step1:InstallPKSToinstallPKS,dothefollowing:

1. DownloadtheproductfilefromPivotalNetwork .

2. Navigateto https://YOUR-OPS-MANAGER-FQDN/ inabrowsertologintotheOpsManagerInstallationDashboard.

3. ClickImportaProducttouploadtheproductfile.

4. UnderPivotalContainerServiceintheleftcolumn,clicktheplussigntoaddthisproducttoyourstagingarea.

Step2:ConfigurePKSClicktheorangePivotalContainerServicetiletostarttheconfigurationprocess.

AssignAZsandNetworksPerformthefollowingsteps:

1. ClickAssignAZsandNetworks.


https://network.pivotal.io

2. Selecttheavailabilityzone(AZ)whereyouwanttodeploythePKSAPIVMasasingletonjob.

3. UnderNetwork,selecttheinfrastructuresubnetyoucreatedforthePKSAPIVM.

4. UnderServiceNetwork,selecttheservicessubnetyoucreatedforKubernetesclusterVMs.

5. ClickSave.

PKSAPIPerformthefollowingsteps:

1. ClickPKSAPI.

2. UnderCertificatetosecurethePKSAPI,provideyourowncertificateandprivatekeypair.

Note:YoumustselectanadditionalAZforbalancingotherjobsbeforeclickingSave,butthisselectionhasnoeffectinthecurrentversionofPKS.


ThecertificatethatyousupplyshouldcoverthedomainthatroutestothePKSAPIVMwithTLSterminationontheingress.

Ifyoudonothaveacertificateandprivatekeypair,PKScangenerateoneforyoubyperformingthefollowingsteps.

a. SelecttheGenerateRSACertificatelink.b. EnterthewildcarddomainforyourAPIhostname.Forexample,ifyourPKSAPIdomainis api.pks.example.com ,thenenter

*.pks.example.com .c. ClickGenerate.

3. UnderAPIHostname(FQDN),enterafullyqualifieddomainname(FQDN)toaccessthePKSAPI.Forexample, api.pks.example.com .

4. ClickSave.

PlansToactivateaplan,performthefollowingsteps:

1. ClickthePlan1,Plan2,orPlan3tab.

2. SelectActivetoactivatetheplanandmakeitavailabletodevelopersdeployingclusters.

Note:IfyouconfiguredOpsManagerFrontEndwithoutacertificate,youcanusethisnewcertificatetocompleteOpsManagerconfiguration.ToconfigureyourOpsManagerFrontEndcertificate,seeConfigureFrontEnd .

Note:Aplandefinesasetofresourcetypesusedfordeployingclusters.Youcanconfigureuptothreeplans.YoumustconfigurePlan1.


https://docs.pivotal.io/pcf/om/2-2/gcp/prepare-env-manual.html#config-frontend

3. UnderName,provideauniquenamefortheplan.

4. UnderDescription,editthedescriptionasneeded.TheplandescriptionappearsintheServicesMarketplace,whichdeveloperscanaccessbyusingPKSCLI.

5. UnderMaster/ETCDNodeInstances,selectthedefaultnumberofKubernetesmaster/etcdnodestoprovisionforeachcluster.Youcanentereither1 or 3 .Forincreasedmasternodeavailability,setthisvalueto 3 .

6. UnderMaster/ETCDVMType,selectthetypeofVMtouseforKubernetesmaster/etcdnodes.Formoreinformation,seetheMasterNodeVMSizesectionofVMSizingforPKSClusters.

7. UnderMasterPersistentDiskType,selectthesizeofthepersistentdiskfortheKubernetesmasternodeVM.

WARNING:Tochangethenumberofmaster/etcdnodesforaplan,youmustensurethatnoexistingclustersusetheplan.PKSdoesnotsupportchangingthenumberofmaster/etcdnodesforplanswithexistingclusters.

WARNING:Thisfeatureisabetacomponentandisintendedforevaluationandtestpurposesonly.Donotusethisfeatureinaproductionenvironment.Productsupportandfutureavailabilityarenotguaranteedforbetacomponents.


8. UnderMaster/ETCDAvailabilityZones,selectoneormoreAZsfortheKubernetesclustersdeployedbyPKS.IfyouselectmorethanoneAZ,PKSdeploysthemasterVMinthefirstAZandtheworkerVMsacrosstheremainingAZs.

9. UnderWorkerNodeInstances,selectthedefaultnumberofKubernetesworkernodestoprovisionforeachcluster.Forhighavailability,createclusterswithaminimumofthreeworkernodes,ortwoperAZifyouintendtousepersistentvolumes.Forexample,ifyoudeployacrossthreeAZs,youshouldhavesixworkernodes.Formoreinformationaboutpersistentvolumes,seePersistentVolumesinMaintainingWorkloadUptime.Provisioningaminimumofthreeworkernodes,ortwonodesperAZisalsorecommendedforstatelessworkloads.

10. UnderWorkerVMType,selectthetypeofVMtouseforKubernetesworkernodeVMs.Formoreinformation,seetheWorkerNodeVMNumberandSizesectionofVMSizingforPKSClusters.

11. UnderWorkerPersistentDiskType,selectthesizeofthepersistentdiskfortheKubernetesworkernodeVMs.

12. UnderWorkerAvailabilityZones,selectoneormoreAZsfortheKubernetesworkernodes.PKSdeploysworkernodesequallyacrosstheAZsyouselect.

13. UnderErrandVMType,selectthesizeoftheVMthatcontainstheerrand.Thesmallestinstancepossibleissufficient,astheonlyerrandrunningonthisVMistheonethatappliestheDefaultClusterAppYAMLconfiguration.

14. (Optional)Under(Optional)Add-ons-Usewithcaution,enteradditionalYAMLconfigurationtoaddcustomworkloadstoeachclusterinthisplan.Youcanspecifymultiplefilesusing --- asaseparator.Formoreinformation,seeAddingCustomWorkloads.

Note:IfyouinstallPKSv1.1.5orlaterinanNSX-Tenvironment,werecommendthatyouselectaWorkerVMTypewithaminimumdisksizeof16GB.Thediskspaceprovidedbythedefault“medium”WorkerVMTypeisinsufficientforPKSwithNSX-Tv1.1.5orlater.


15. (Optional)Toallowuserstocreatepodswithprivilegedcontainers,selecttheEnablePrivilegedContainers-Usewithcautionoption.Formoreinformation,seePods intheKubernetesdocumentation.

16. (Optional)Todisabletheadmissioncontroller,selecttheDisableDenyEscalatingExeccheckbox.Ifyouselectthisoption,clustersinthisplancancreatesecurityvulnerabilitiesthatmayimpactothertiles.Usethisfeaturewithcaution.

17. ClickSave.

Todeactivateaplan,performthefollowingsteps:

1. ClickthePlan1,Plan2,orPlan3tab.

2. SelectPlanInactive.

3. ClickSave.

KubernetesCloudProviderIntheprocedurebelow,youusecredentialsforvCentermasterVMs.Youmusthaveprovisionedtheserviceaccountwiththecorrectpermissions.Formoreinformation,seeCreatetheMasterNodeServiceAccountinPreparingvSphereBeforeDeployingPKS.

ToconfigureyourKubernetescloudprovidersettings,followtheprocedurebelow:

1. ClickKubernetesCloudProvider.

2. UnderChooseyourIaaS,selectvSphere.

3. EnsurethevaluesinthefollowingprocedurematchthoseinthevCenterConfigsectionoftheOpsManagertile.

a. EnteryourvCenterMasterCredentials.Entertheusernameusingtheformat [email protected] .Formoreinformationaboutthemasternodeserviceaccount,seePreparingvSphereBeforeDeployingPKS.

b. EnteryourvCenterHost.Forexample, vcenter.CF-EXAMPLE.com .c. EnteryourDatacenterName.Forexample, CF-EXAMPLE-dc .d. EnteryourDatastoreName.Forexample, CF-EXAMPLE-ds .e. EntertheStoredVMFoldersothatthepersistentstoresknowwheretofindtheVMs.Toretrievethenameofthefolder,navigatetoyourBOSHDirectortile,clickvCenterConfig,andlocatethevalueforVMFolder.Thedefaultfoldernameis pcf_vms .

Note:Werecommendusingashareddatastoreformulti-AZandmulti-clusterenvironments.


https://kubernetes.io/docs/concepts/workloads/pods/pod/#privileged-mode-for-pod-containers

4. ClickSave.

(Optional)LoggingYoucandesignateanexternalsyslogendpointforPKScomponentandclusterlogmessages.

TospecifythedestinationforPKSlogmessages,dothefollowing:

1. ClickLogging.

2. Toenablesyslogforwarding,selectYes.

3. UnderAddress,enterthedestinationsyslogendpoint.

4. UnderPort,enterthedestinationsyslogport.

5. Selectatransportprotocolforlogforwarding.

6. (Optional)PivotalstronglyrecommendsthatyouenableTLSencryptionwhenforwardinglogsastheymaycontainsensitiveinformation.Forexample,theselogsmaycontaincloudprovidercredentials.ToenableTLS,performthefollowingsteps:

a. UnderPermitterPeer,providetheacceptedfingerprint(SHA1)ornameofremotepeer.Forexample, *.YOUR-LOGGING-SYSTEM.com .b. UnderTLSCertificate,provideaTLScertificateforthedestinationsyslogendpoint.

7. YoucanmanagelogsusingVMwarevRealizeLogInsight(vRLI) .TheintegrationpullslogsfromallBOSHjobsandcontainersrunninginthecluster,includingnodelogsfromcoreKubernetesandBOSHprocesses,Kuberneteseventlogs,andPODstdoutandstderr.

Bydefault,vRLIloggingisdisabled.ToenableandconfigurevRLIlogging,underEnableVMwarevRealizeLogInsightIntegration?,selectYesand

Note:YoudonotneedtoprovideanewcertificateiftheTLScertificateforthedestinationsyslogendpointissignedbyaCertificateAuthority(CA)inyourBOSHcertificatestore.

Note:BeforeyouconfigurethevRLIintegration,youmusthaveavRLIlicenseandvRLImustbeinstalled,running,andavailableinyourenvironment.Youneedtoprovidetheliveinstanceaddressduringconfiguration.Forinstructionsandadditionalinformation,seethevRealizeLogInsightdocumentation .


https://www.vmware.com/products/vrealize-log-insight.htmlhttps://docs.vmware.com/en/vRealize-Log-Insight/index.html

thenperformthefollowingsteps:

a. UnderHost,entertheIPaddressorFQDNofthevRLIhost.b. (Optional)SelecttheEnableSSL?checkboxtoencryptthelogsbeingsenttovRLIusingSSL.c. ChooseoneofthefollowingSSLcertificatevalidationoptions:

ToskipcertificatevalidationforthevRLIhost,selecttheDisableSSLcertificatevalidationcheckbox.Selectthisoptionifyouareusingaself-signedcertificateinordertosimplifysetupforadevelopmentortestenvironment.

ToenablecertificatevalidationforthevRLIhost,cleartheDisableSSLcertificatevalidationcheckbox.

d. (Optional)IfyourvRLIcertificateisnotsignedbyatrustedCArootorotherwellknowncertificate,enterthecertificateintheCAcertificatefield.LocatethePEMoftheCAusedtosignthevRLIcertificate,copythecontentsofthecertificatefile,andpastethemintothefield.CertificatesmustbeinPEM-encodedformat.

e. UnderRatelimiting,enteratimeinmillisecondstochangetherateatwhichlogsaresenttothevRLIhost.Theratelimitspecifiestheminimumtimebetweenmessagesbeforethefluentdagentbeginstodropmessages.Thedefaultvalue(0)meanstherateisnotlimited,whichsufficesformanydeployments.

8. ClickSave.ThesesettingsapplytoanyclusterscreatedafteryouhavesavedtheseconfigurationsettingsandclickedApplyChanges.IftheUpgradeallclusterserrandhasbeenenabled,thesesettingarealsoappliedtoexistingclusters.

NetworkingToconfigurenetworking,dothefollowing:

1. ClickNetworking.

Note:Disablingcertificatevalidationisnotrecommendedforproductionenvironments.

Note:Ifyourdeploymentisgeneratingahighvolumeoflogs,youcanincreasethisvaluetolimitnetworktraffic.Considerstartingwithalowernumber,suchas10,andtuningtooptimizeforyourdeployment.Alargenumbermightresultindroppingtoomanylogentries.

Note:ThePKStiledoesnotvalidateyourvRLIconfigurationsettings.Toverifyyoursetup,lookforlogentriesinvRLI.


2. UnderContainerNetworkingInterface,selectFlannel.

3. (Optional)ConfigureaglobalproxyforalloutgoingHTTPandHTTPStrafficfromyourKubernetesclusters.ThissettingwillnotsettheproxyforrunningKubernetesworkloadsorpods.

ProductionenvironmentscandenydirectaccesstopublicInternetservicesandbetweeninternalservicesbyplacinganHTTPorHTTPSproxyinthenetworkpathbetweenKubernetesnodesandthoseservices.

IfyourenvironmentincludesHTTPorHTTPSproxies,configuringPKStousetheseproxiesallowsPKS-deployedKubernetesnodestoaccesspublicInternetservicesandotherinternalservices.FollowthestepsbelowtoconfigureaglobalproxyforalloutgoingHTTP/HTTPStrafficfromyourKubernetesclusters:

a. UnderHTTP/HTTPSproxy,selectEnabled.b. UnderHTTPProxyURL,entertheURLofyourHTTP/HTTPSproxyendpoint.Forexample, http://myproxy.com:1234 .c. (Optional)Ifyourproxyusesbasicauthentication,entertheusernameandpasswordunderHTTPProxyCredentials.d. UnderNoProxy,entertheservicenetworkCIDRwhereyourPKSclusterisdeployed.ListanyadditionalIPaddressesthatshouldbypasstheproxy.

4. UnderAllowoutboundinternetaccessfromKubernetesclustervms(IaaS-dependent),ignoretheEnableoutboundinternetaccesscheckbox.

5. ClickSave.

Note:Bydefault,the .internal , 10.100.0.0/8 ,and 10.200.0.0/8 IPaddressrangesarenotproxied.ThisallowsinternalPKScommunication.


UAAToconfiguretheUAAserver,dothefollowing:

1. ClickUAA.

2. UnderPKSCLIAccessTokenLifetime,enteratimeinsecondsforthePKSCLIaccesstokenlifetime.

3. UnderPKSCLIRefreshTokenLifetime,enteratimeinsecondsforthePKSCLIrefreshtokenlifetime.

4. Selectoneofthefollowingoptions:

TouseaninternaluseraccountstoreforUAA,selectInternalUAA.ClickSaveandcontinueto(Optional)Monitoring.TouseanexternaluseraccountstoreforUAA,selectLDAPServerandcontinuetoConfigureLDAPasanIdentityProvider.

ConfigureLDAPasanIdentityProvider

TointegrateUAAwithoneormoreLDAPservers,configurePKSwithyourLDAPendpointinformationasfollows:

1. UnderUAA,selectLDAPServer.

2. ForServerURL,entertheURLsthatpointtoyourLDAPserver.IfyouhavemultipleLDAPservers,separatetheirURLswithspaces.EachURLmustincludeoneofthefollowingprotocols:


ldap:// :UsethisprotocolifyourLDAPserverusesanunencryptedconnection.ldaps:// :UsethisprotocolifyourLDAPserverusesSSLforanencryptedconnection.Tosupportanencryptedconnection,theLDAPservermustholdatrustedcertificateoryoumustimportatrustedcertificatetotheJVMtruststore.

3. ForLDAPCredentials,entertheLDAPDistinguishedName(DN)andpasswordforbindingtotheLDAPserver.Forexample,cn=administrator,ou=Users,dc=example,dc=com .Ifthebinduserbelongstoadifferentsearchbase,youmustusethefullDN.

4. ForUserSearchBase,enterthelocationintheLDAPdirectorytreewhereLDAPusersearchbegins.TheLDAPsearchbasetypicallymatchesyourdomainname.

Forexample,adomainnamed cloud.example.com mayuse ou=Users,dc=example,dc=com asitsLDAPusersearchbase.

5. ForUserSearchFilter,enterastringtouseforLDAPusersearchcriteria.ThesearchcriteriaallowsLDAPtoperformmoreeffectiveandefficientsearches.Forexample,thestandardLDAPsearchfilter cn=Smith returnsallobjectswithacommonnameequalto Smith .

IntheLDAPsearchfilterstringthatyouusetoconfigurePKS,use {0} insteadoftheusername.Forexample,use cn={0} toreturnallLDAPobjectswiththesamecommonnameastheusername.

Inadditionto cn ,othercommonattributesare mail , uid and,inthecaseofActiveDirectory, sAMAccountName .

6. ForGroupSearchBase,enterthelocationintheLDAPdirectorytreewheretheLDAPgroupsearchbegins.

Forexample,adomainnamed cloud.example.com mayuse ou=Groups,dc=example,dc=com asitsLDAPgroupsearchbase.

FollowtheinstructionsintheGrantClusterAccesstoanExternalLDAPGroupsectionofManagingUsersinPKSwithUAAtomapthegroupsunderthissearchbasetorolesinPKS.

7. ForGroupSearchFilter,enterastringthatdefinesLDAPgroupsearchcriteria.Thestandardvalueis member={0} .

8. ForServerSSLCert,pasteintherootcertificatefromyourCAcertificateoryourself-signedcertificate.

Note:WerecommendthatyouprovideLDAPcredentialsthatgrantread-onlypermissionsontheLDAPsearchbaseandtheLDAPgroupsearchbase.

Note:ForinformationabouttestingandtroubleshootingyourLDAPsearchfilters,seeConfiguringLDAPIntegrationwithPivotalCloudFoundry .


https://community.pivotal.io/s/article/Configuring-LDAP-Integration-with-Pivotal-Cloud-Foundry

9. ForServerSSLCertAltName,dooneofthefollowing:

Ifyouareusing ldaps:// withaself-signedcertificate,enteraSubjectAlternativeName(SAN)foryourcertificate.Ifyouarenotusing ldaps:// withaself-signedcertificate,leavethisfieldblank.

10. ForFirstNameAttribute,entertheattributenameinyourLDAPdirectorythatcontainsuserfirstnames.Forexample, cn .

11. ForLastNameAttribute,entertheattributenameinyourLDAPdirectorythatcontainsuserlastnames.Forexample, sn .

12. ForEmailAttribute,entertheattributenameinyourLDAPdirectorythatcontainsuseremailaddresses.Forexample, mail .

13. ForEmailDomain(s),enteracomma-separatedlistoftheemaildomainsforexternaluserswhocanreceiveinvitationstoAppsManager.

14. ForLDAPReferrals,choosehowUAAhandlesLDAPserverreferralstootheruserstores.UAAcanfollowtheexternalreferrals,ignorethemwithoutreturningerrors,orgenerateanerrorforeachexternalreferralandaborttheauthentication.

15. ClickSave.

(Optional)MonitoringYoucanmonitorKubernetesclustersandpodsmetricsexternallyusingtheintegrationwithWavefrontbyVMware .

Bydefault,monitoringisdisabled.ToenableandconfigureWavefrontmonitoring,dothefollowing:

1. UnderWavefrontIntegration,selectYes.

Note:BeforeyouconfigureWavefrontintegration,youmusthaveanactiveWavefrontaccountandaccesstoaWavefrontinstance.YouprovideyourWavefrontaccesstokenduringconfigurationandenablingerrands.Foradditionalinformation,seePivotalContainerServiceIntegrationDetails intheWavefrontdocumentation.


https://docs.wavefront.comhttps://docs.wavefront.com/integrations_pks.html

2. UnderWavefrontURL,entertheURLofyourWavefrontsubscription.Forexample, https://try.wavefront.com/api .

3. UnderWavefrontAccessToken,entertheAPItokenforyourWavefrontsubscription.

4. ToconfigureWavefronttosendalertsbyemail,enteremailaddressesorWavefrontTargetIDsseparatedbycommasunderWavefrontAlertRecipient.Forexample: [email protected],Wavefront_TargetID .Tocreatealerts,youmustenableerrands.

5. IntheErrandstab,enableCreatepre-definedWavefrontalertserrandandDeletepre-definedWavefrontalertserrand.

6. ClickSave.YoursettingsapplytoanyclusterscreatedafteryouhavesavedtheseconfigurationsettingsandclickedApplyChanges.

Note:ThePKStiledoesnotvalidateyourWavefrontconfigurationsettings.Toverifyyoursetup,lookforclusterandpodmetricsin


UsageDataVMware’sCustomerExperienceImprovementProgram(CEIP)andthePivotalTelemetryProgram(Telemetry)providesVMwareandPivotalwithinformationthatenablesthecompaniestoimprovetheirproductsandservices,fixproblems,andadviseyouonhowbesttodeployanduseourproducts.AspartoftheCEIPandTelemetry,VMwareandPivotalcollecttechnicalinformationaboutyourorganization’suseofthePivotalContainerService(“PKS”)onaregularbasis.SincePKSisjointlydevelopedandsoldbyVMwareandPivotal,wewillsharethisinformationwithoneanother.InformationcollectedunderCEIPorTelemetrydoesnotpersonallyidentifyanyindividual.

ForinformationaboutthemetricsPKSsendswhenyouoptintoCEIPorTelemetry,seePKSTelemetry.

RegardlessofyourselectionintheUsageDatapane,asmallamountofdataissentfromCloudFoundryContainerRuntime(CFCR)tothePKStile.However,thatdataisnotsharedexternally.

ToconfiguretheUsageDatapane:

1. SelecttheUsageDataside-tab.

2. ReadtheUsageDatadescription.

3. Makeyourselection.

a. Tojointheprogram,selectYes,IwanttojointheCEIPandTelemetryProgramforPKS.b. Todeclinejoiningtheprogram,selectNo,IdonotwanttojointheCEIPandTelemetryProgramforPKS.

4. ClickSave.

ErrandsErrandsarescriptsthatrunatdesignatedpointsduringaninstallation.

Toconfigurewhenpost-deployandpre-deleteerrandsforPKSarerun,makeaselectioninthedropdownnexttotheerrand.ForatypicalPKSdeployment,werecommendthatyouleavethedefaultsettings.

Wavefront.

Note:IfyoujointheCEIPandTelemetryProgramforPKS,openyourfirewalltoallowoutgoingaccessto https://vcsa.vmware.com/ph-prd onport 443.


Formoreinformationabouterrandsandtheirconfigurationstate,seeManagingErrandsinOpsManager .

ResourceConfigTomodifytheresourceusageofPKS,clickResourceConfigandeditthePivotalContainerServicejob.

Step3:ApplyChanges

WARNING:BecausePKSusesfloatingstemcells,updatingthePKStilewithanewstemcelltriggerstherollingofeveryVMineachcluster.Also,updatingotherproducttilesinyourdeploymentwithanewstemcellcausesthePKStiletorollVMs.ThisrollingisenabledbytheUpgradeallclusterserrand.WerecommendthatyoukeepthiserrandturnedonbecauseautomaticrollingofVMsensuresthatalldeployedclusterVMsarepatched.However,automaticrollingcancausedowntimeinyourdeployment.

IfyouupgradePKSfrom1.0.xto1.1,youmustenabletheUpgradeAllClustererrand.Thisensuresexistingclusterscanperformresizeordeleteactionsaftertheupgrade.

Note:IfyouexperiencetimeoutsorslownesswheninteractingwiththePKSAPI,selectaVMTypewithgreaterCPUandmemoryresourcesforthePivotalContainerServicejob.


https://docs.pivotal.io/pivotalcf/customizing/managing_errands.html

Afterconfiguringthetile,returntotheOpsManagerInstallationDashboardandclickApplyChangestodeploythetile.

Step4:RetrievethePKSAPIEndpointYoumustsharethePKSAPIendpointtoallowyourorganizationtousetheAPItocreate,update,anddeleteclusters.SeeCreatingClustersformoreinformation.

ToretrievethePKSAPIendpoint,dothefollowing:

1. NavigatetotheOpsManagerInstallationDashboard.

2. ClickthePivotalContainerServicetile.

3. ClicktheStatustabandlocatethePivotalContainerServicejob.TheIPaddressofthePivotalContainerServicejobisthePKSAPIendpoint.

Step5:ConfigureExternalLoadBalancerAfteryouinstallthePKStile,configureanexternalloadbalancertoaccessthePKSAPIfromoutsidethenetwork.Youcanuseanyexternalloadbalancer.

YourexternalloadbalancerforwardstraffictothePKSAPIendpointonports8443and9021.ConfiguretheexternalloadbalancertoresolvetothedomainnameyousetinthePKSAPIsectionofthetileconfiguration.

Configureyourloadbalancerwiththefollowinginformation:

IPaddressfromRetrievePKSAPIEndpoint

Ports8443and9021

HTTPSorTCPprotocol

Step6:InstallthePKSandKubernetesCLIsThePKSandKubernetesCLIshelpyouinteractwithyourPKS-provisionedKubernetesclustersandKubernetesworkloads.ToinstalltheCLIs,followtheinstructionsbelow:

InstallingthePKSCLI

InstallingtheKubernetesCLI

Step7:ConfigurePKSAPIAccessFollowtheproceduresinConfiguringPKSAPIAccess.

Step8:ConfigureAuthenticationforPKSConfigureauthenticationforPKSusingUserAccountandAuthentication(UAA).ForinformationaboutmanagingusersinPKSwithUAA,seeManagingUsersinPKSwithUAA.

NextStepsAfterinstallingPKSonvSphere,youmaywanttodothefollowing:

IntegrateVMwareHarborwithPKStostoreandmanagecontainerimages.Formoreinformation,seeIntegratingVMwareHarborRegistrywithPKS .

CreateyourfirstPKScluster.Formoreinformation,seeCreatingClusters.

[email protected].

©CopyrightPivotalSo

pivotal container service (pks) documentation · your pks-provisioned kubernetes cluster does not...

Documents