Table of Contents

Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1What does Programming Kubernetes Mean? 1A Motivational Example 3Extension Patterns 4Controllers and Operators 5

The Controller Loop 6Events 7Edge Versus Level Driven Triggers 9Changing The Objects In-Cluster Or The External World 12Optimistic Concurrency 14Operators 17

2. Kubernetes API basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21The API Server 21

The HTTP Interface of the API Server 22API Terminology 23Kubernetes API Versioning 27Declarative State Management 27

Using the API from the Command Line 28How The API Server Processes Requests 32

3. Basics of client-go. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37The Repositories 37

The Client Library 37Kubernetes API Types 39API Machinery 40

iii

Creating and Using a Client 41Versioning and Compatibility 43API Versions and Compatibility Guarantees 46

Kubernetes Objects in Go 48TypeMeta 50ObjectMeta 16Spec and Status 53

Client sets 54Status Subresources 56Listings And Deletions 56Watches 57Client Expansion 58Client options 58

Informers and Caching 60Work Queue 64

API Machinery in Depth 66Kinds 66Resources 66REST Mapping 67Scheme 68

Vendoring 70Glide 70Dep 71Go Modules 72

4. Using Custom Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Discovery Information 77Type definitions 78Advanced Features of Custom Resources 81

Validating Custom Resources 81Short Names And Categories 83Printer Columns 84Subresources 86

A Developers View on Custom Resources 90Dynamic Client 91Typed Clients 92Controller-runtime Client of Operator SDK and Kubebuilder 97

5. Automating Code Generation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101Why Code Generation 101Calling the Generators 101Controlling the generators with tags 103

iv | Table of Contents

Global Tags 104Local Tags 105deepcopy-gen tags 106runtime.Object and DeepCopyObject 107client-gen tags 108informer-gen and `lister-gen 110

Further Material 110

6. Solutions For Writing Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111Preparation 111Following sample-controller 112Kubebuilder 120The Operator SDK 128Other Approaches 133Uptake And Future Directions 134

7. Shipping Controllers And Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135Lifecycle Management and Packaging 135

Packaging: The Challenge 135Helm 136Kustomize 138Other Packaging Options 140Packaging Good Practices 141Lifecycle Management 141

Production-Ready Deployments 142Getting The Permissions Right 142Automated Builds And Testing 146Custom Controllers And Observability 146

8. Custom API Servers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151Use Cases For Custom API Servers 151Example: A Pizza Restaurant 153The Architecture: Aggregation 154

API Services 156Inner Structure of a Custom API Server 158Delegated Authentication and Trust 160Delegated Authorization 161

Writing Custom API Servers 163Options and Config Pattern and Startup Plumbing 164The First Start 170Internal Types and Conversion 172Writing the API Types 175

Table of Contents | v

Conversions 177Defaulting 180Roundtrip Testing 181Validation 183Registry and Strategy 185API Installation 190Admission 193

Deploying Custom API Servers 202Deployment Manifests 202Setting up RBAC 204Running the Custom API Server Insecurely 206Certificates and Trust 208Sharing etcd 211

Summary 212

9. Advanced Custom Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213Custom Resources Versioning 213

An example: The Pizza Restaurant – again 214Conversion Webhook Architecture 217Conversion Webhook Implementation 221Setting up the HTTPS Server 221Deploying the Conversion Webhook 227Seeing Conversion in Action 228

Admission Webhooks 231Admission Requirements in The Restaurant Example 232Admission Webhook Architecture 232Registering Admission Webhooks 234Implementation an Admission Webhook 236Admission Webhook in Action 240

Structural Schemas and the future of CustomResourceDefinitions 241Structural Schemas 242Pruning versus preserving unknown fields 244Controlling pruning 245IntOrString and RawExtensions 246Default Values 81

Summary 248

A. Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255

vi | Table of Contents

Index

Aadmission, 35, 159, 166

chain, 194, 199configuration, 199initializers, 166mutating, 183, 193order, 194plugin, 35, 193, 208plugin initializer, 199register, 198validating, 183, 193webhook, 153, 159, 174, 194, 196, 205, 231

aggregated API server (see aggregation)aggregation, 75, 151, 155, 202, 208, 214, 227aggregator (see aggregation)alpha version, 46API

aggregation, 155group, 24, 43invoke via command line, 28Machinery, 40proxy, 28resource, 24server, 24

HTTP interface, 22kind, 23processing requests, 32

version, 24API server, 21apiextensions-apiserver, 76, 81, 158APIService, 202apps group, 43auditing, 159, 160, 165, 196authentication, 159, 165

delegated, 161, 205authorization, 161, 165

chain, 163delegated, 161, 205SubjectAccessReview, 160, 186

Bbearer token, 161beta version, 46builder pattern, 43

CCA bundle, 208category, 84cient-gen, 102client

client-gen, 90controller-runtime, 90dynamic, 90, 91internal, 177loopback, 201set, 42typed, 90

client certificate, 160client-gen, 69, 90, 95, 97client-go, 37clientcmd, 41clientset, 42, 54, 95cloud provier, 5cluster scoped, 56cobra command, 169code examples from this book, xcohabitation, 152component

255

kube-aggregator, 202config, 166

in-cluster, 42control plane, 21, 28

API server, 21cloud controller manager, 5controller manager, 5, 21etcd, 21scheduler, 21

controllerchanging resource state, 12definition, 5deployment, 8events, 7loop, 6optimistic concurrency, 14ReplicaSet, 8triggers, 9

controller-runtime, 90conversion, 24, 48, 159, 172, 177, 190

function, 178naming pattern, 178

webhook, 214, 218conversion-gen, 102, 176, 177ConversionReview, 218core group, 52CoreDNS, 28CR (see custom resource)CRD (see custom resource)crd-gen, 83crd-schema-gen, 83CRUD, 35custom API server, 151custom resource

admission webhook, 231conversion, 214conversion webhook, 218definition, 43, 75, 76, 125, 151, 176versioning, 213

CustomResourceDefinition (see customresource)

Ddecoding, 159deep copy, 49, 95deepcopy-gen, 102defaulter-gen, 102, 176, 180defaulting, 175, 190defaulting function naming pattern, 180

DeferredDiscoveryRESTMapper, 68delegated authentication, 161delegated authorization, 161dep, 71deserializer, 223desired state, 27discovery, 77, 84, 97

endpoint, 171RESTMapper, 68, 78

dynamic client, 90

Eedge-driven trigger, 9encoding, 159error

conflict, 16etcd, 163etcd operator, 211etcdproxy-controller, 211external version, 174

Ffeature gate, 165, 247field selector, 56fuzzer, 182

GGA, 46generator

client-gen, 69, 90, 97, 102conversion-gen, 102, 176, 177crd-gen, 83crd-schema-gen, 83deepcopy-gen, 102default-gen, 180defaulter-gen, 102, 176go-to-protobuf, 152informer-gen, 102lister-gen, 102

generators, 125generic registry, 186Gengo, 101Git, viiiglide, 70Go (build system), viiigo modules, 72go-to-protobuf, 152go.mod (see go modules)

256 | Index

graceful termination, 170Grafana, 148GroupVersion, 55GroupVersionKind, 25, 49, 66, 197GroupVersionKinds, 177GroupVersionResource, 25, 66, 91GVK, 25, 49, 66, 177, 197GVR, 25, 66, 91

Hhandler chain, 159Helm, 136

chart, 137HTTP/2, 152hub version, 172

Iimpersonation, 159in-cluster config, 42informer, 60

factory, 201relist, 60resync, 60

informer-gen, 102internal client, 55, 177internal version, 172IntOrString, 246

JJSON, 22

schema, 82

Kkind, 66

Kubernetes in Docker, viiiklog, 147kube-aggregator, 155kube-apiserver, 155kube-dns, 28Kubebuilder, 83, 97, 120, 143kubeconfig, 41KUBECONFIG, 42kubectl, 42, 133Kubernetes

releases, viiiKUDO, 133Kustomize, 138kutil, 133

Llabel selector, 56legacy group, 52level-driven trigger, 9lifecycle management, 135lister-gen, 102logger, 147long-running, 59, 59

Mman-in-the-middle attack, 156manifests

GitHub repository, xmaster node, 21meta/v1, 41Metacontroller, 133metrics, 148Minikube, viii

NNameGenerator, 189namespace scoped, 56

OObjectMeta, 43, 53, 93, 185ObjectTyper, 189OpenAPI, 85, 169

schema, 81, 82OpenShift, 152, 152operator

advanced, 213alternatives, 133basics, 17build, 146CI, 146definition, 5integration testing, 146logging, 147monitoring, 148observability, 146permissions, 142production ready, 142scalability, 146SDK, 128testing, 146

Operator SDK, 53, 97optimistic concurrency, 88option-config pattern, 164, 222

Index | 257

optionsrecommended, 164

Ppackage management, viiipackaging, 135

Ansible, 140awk, 140challenge, 135Chef, 140cloud native languages, 140good practices, 141Helm, 136jq, 140Ksonnet, 141Kustomize, 138Puppet, 140Salt, 140sed, 140shell scripts, 140YAML, 136ytt, 140

post start hook, 169ProcessInfo, 166Prometheus, 148protobuf (see protocol buffer)protocol buffer, 22, 43, 52, 58, 151, 176, 181,

225prune, 245pseudo version, 73

QQPS, 59

Rrate limiting, 58RawExtension, 246RBAC, 86, 135, 142, 161, 204 (see role based

access control)cluster role, 205, 206cluster role binding), 205role, 205

recommended options, 164reflection, 68registry, 168relist period, 60remote procuedure call, 8request

long-running, 59request header, 160request header client CA, 160resource, 66, 91resource version, 16

conflict error, 16REST, 22, 35, 186

client, 55config, 42, 54, 61, 91endpoint, 66endpoints, 66interface

CollectionDeleter, 186Creater, 186CreaterUpdater, 186Exporter, 187Getter, 187GracefulDeleter, 187Lister, 187Patcher, 187Scoper, 187Updater, 187Watcher, 187

mapper, 55mapping, 26, 67, 158, 159storage constructor, 189verbs, 37

RESTful, 22, 35RESTMapper, 26, 55, 67, 91

discovery, 68resync period, 60role based access control, 34, 141 (see RBAC)Rook Operator kit, 133roundtrip, 175, 181

test, 181RPC (see remote procedure call)runtime.Object, 68

SSAR (see subjet access review)scale subresource, 88, 109scheduler, 8schema

structural, 242scheme, 52, 68, 91, 97, 107selector

field, 56, 64label, 56, 64

semantic versioning, 38

258 | Index

semver, 46semver, 38server-side printing, 84service account, 204shared informer factory, 61short name, 83SIG API Machinery, 120Simple Go client, 133spec, 27spec-status split, 86, 108, 127status, 27status subresource, 56, 108storage version, 48, 173Store, 60strategy, 187, 192structural schema, 242structured logging, 147subject access review, 160, 161, 186subresource, 25, 59, 86, 235

exec, 86logs, 86portforward, 86scale, 86, 88, 109status, 56, 86, 86, 108, 145

Tthrottling, 58

burst, 59queries per second, 59

timeout, 58token, 160

access review, 159, 161bearer, 161

tokenmaccess review, 161

typed client, 90TypeMeta, 50, 93, 93, 93

UUnstructured, 91UserAgent, 58

Vvalidation, 35, 183

client-side, 83vendoring

dep, 71glide, 70Go modules, 72

verb, 37create, 37delete, 38get, 37list, 37, 64patch, 38update, 37watch, 38, 60, 60, 64

versioninternal, 227

Wwatch, 57

event, 57webhook, 166

admission, 174, 194, 196conversion, 218, 231

WebSocket, 25, 152

Index | 259