kubecon eu 2016: custom volume plugins
TRANSCRIPT
![Page 2: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/2.jpg)
Just one thing
![Page 3: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/3.jpg)
It's easy!
![Page 4: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/4.jpg)
Easier than you might think
![Page 5: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/5.jpg)
What's a
volume?
![Page 6: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/6.jpg)
Persistentvs
Non persistent
![Page 7: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/7.jpg)
User POV
![Page 8: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/8.jpg)
spec: containers: - name: web image: nginx ports: - name: web containerPort: 80 volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html"
volumes: - name: www-root flocker: datasetName: my-flocker-vol
![Page 9: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/9.jpg)
... volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html"
volumes: - name: www-root flocker: datasetName: my-flocker-vol
![Page 10: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/10.jpg)
Available4 AWS EBS, GCE PD, Azure File
4 Git Repo
4 NFS
4 GlusterFS, Cinder
4 Flocker
4 Secrets
![Page 11: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/11.jpg)
Is this
new?
![Page 12: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/12.jpg)
HTTP API
![Page 13: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/13.jpg)
/VolumeDriver.Create .Mount .Path .Unmount .Remove
![Page 14: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/14.jpg)
Go interface{}
![Page 15: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/15.jpg)
![Page 16: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/16.jpg)
1. Add it to the API (pkg/api/{,v1/}types.go):
type VolumeSource struct { EmptyDir *EmptyDirVolumeSource `json:"emptyDir,omitempty"` Flocker *FlockerVolumeSource `json:"flocker,omitempty"` ...
![Page 17: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/17.jpg)
2. Add it to the Kubelet (cmd/kubelet/app/plugins.go):
func ProbeVolumePlugins(pluginDir string) []volume.VolumePlugin { allPlugins := []volume.VolumePlugin{} allPlugins = append(allPlugins, empty_dir.ProbeVolumePlugins()...) allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...) ...
![Page 18: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/18.jpg)
3. Implement the volume plugin (pkg/volume/...):
type VolumePlugin interface { Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool
NewBuilder(spec *Spec, podRef *api.Pod, opts VolumeOptions) (Builder, error) NewCleaner(name string, podUID types.UID) (Cleaner, error)}
![Page 19: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/19.jpg)
type VolumePlugin interface { Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool ...
![Page 20: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/20.jpg)
... NewBuilder( spec *Spec, podRef *api.Pod, opts VolumeOptions, ) (Builder, error)
NewCleaner( name string, podUID types.UID, ) (Cleaner, error)}
![Page 21: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/21.jpg)
Simplified example of NewBuilder
func (p *plg) NewBuilder(spec, pod, opts) (volume.Builder, error) { source, _ := p.getFlockerVolumeSource(spec) builder := flockerBuilder{ flocker: &flocker{ datasetName: source.DatasetName, pod: pod, ... }, ... } return &builder, nil}
![Page 22: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/22.jpg)
4. Implement the builder
type Builder interface { Volume
SetUp(fsGroup *int64) error SetUpAt(dir string, fsGroup *int64) error GetAttributes() Attributes}
![Page 23: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/23.jpg)
Simplified example of SetUpAt for git repo
![Page 24: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/24.jpg)
I lied
![Page 25: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/25.jpg)
![Page 26: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/26.jpg)
func (b *gitRepoVolumeBuilder) SetUpAt(dir string, fsGroup *int64) error { if volumeutil.IsReady(b.getMetaDir()) { return nil }
wrapped, err := b.plugin.host.NewWrapperBuilder(b.volName, wrappedVolumeSpec, &b.pod, b.opts) if err != nil { return err } if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err }
args := []string{"clone", b.source}
if len(b.target) != 0 { args = append(args, b.target) } if output, err := b.execCommand("git", args, dir); err != nil { return fmt.Errorf("failed to exec 'git %s': %s: %v", strings.Join(args, " "), output, err) }
files, err := ioutil.ReadDir(dir) if err != nil { return err }
if len(b.revision) == 0 { // Done! volumeutil.SetReady(b.getMetaDir()) return nil }
var subdir string
switch { case b.target == ".": // if target dir is '.', use the current dir subdir = path.Join(dir) case len(files) == 1: // if target is not '.', use the generated folder subdir = path.Join(dir, files[0].Name()) default: // if target is not '.', but generated many files, it's wrong return fmt.Errorf("unexpected directory contents: %v", files) }
if output, err := b.execCommand("git", []string{"checkout", b.revision}, subdir); err != nil { return fmt.Errorf("failed to exec 'git checkout %s': %s: %v", b.revision, output, err) } if output, err := b.execCommand("git", []string{"reset", "--hard"}, subdir); err != nil { return fmt.Errorf("failed to exec 'git reset --hard': %s: %v", output, err) }
volumeutil.SetReady(b.getMetaDir()) return nil}
![Page 27: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/27.jpg)
1. Check meta: was it called before? Is it ready?
2. Use empty dir to prepare the path
3. git clone it
4. Checkout the revision sent on the pod definition
5. Job done? Mark as ready
![Page 28: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/28.jpg)
Important bit of empty_dir
switch ed.medium { case api.StorageMediumDefault: err = ed.setupDir(dir) case api.StorageMediumMemory: err = ed.setupTmpfs(dir, securityContext) default: err = fmt.Errorf("unknown storage medium %q", ed.medium) }
volume.SetVolumeOwnership(ed, fsGroup)
![Page 29: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/29.jpg)
Another simplified example for Flocker
![Page 30: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/30.jpg)
I liedagain
![Page 31: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/31.jpg)
![Page 32: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/32.jpg)
func (b flockerBuilder) SetUpAt(dir string, fsGroup *int64) error { if volumeutil.IsReady(b.getMetaDir()) { return nil }
if b.client == nil { c, err := b.newFlockerClient() if err != nil { return err } b.client = c }
datasetID, err := b.client.GetDatasetID(dir) if err != nil { return err }
s, err := b.client.GetDatasetState(datasetID) if err != nil { return fmt.Errorf("The volume '%s' is not available in Flocker. You need to create this manually with Flocker CLI before using it.", dir) }
primaryUUID, err := b.client.GetPrimaryUUID() if err != nil { return err }
if s.Primary != primaryUUID { if err := b.updateDatasetPrimary(datasetID, primaryUUID); err != nil { return err } }
b.flocker.path = s.Path volumeutil.SetReady(b.getMetaDir()) return nil}
![Page 33: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/33.jpg)
1. Was it called? Is it ready?
2. Is the dataset ready? If not, not my problem mate
3. If primary UUIDs doesn't match (rescheduled pod), update them
4. Wait until ready
5. Mark as ready
![Page 34: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/34.jpg)
5. Implement the Persistent Volume Plugin
type PersistentVolumePlugin interface { VolumePlugin
GetAccessModes() []api.PersistentVolumeAccessMode}
![Page 35: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/35.jpg)
func (p *plg) GetAccessModes() []api.PersistentVolumeAccessMode { return []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }}
![Page 37: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/37.jpg)
"Problems"
![Page 38: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/38.jpg)
4 CLAs
4 Shippable & Jenkins
4 hack/ scripts
4 PR
![Page 39: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/39.jpg)
Summary1. Add API
2. Enable the plugin on the Kubelet
3. Implement VolumePlugin interface: NewBuilder & NewCleaner
4. Implement the builder itself: SetUpAt
5. Persistent?
6. ❤
![Page 40: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/40.jpg)
![Page 41: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/41.jpg)
Just one (more) thing
![Page 42: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/42.jpg)
![Page 43: KubeCon EU 2016: Custom Volume Plugins](https://reader031.vdocuments.us/reader031/viewer/2022021814/58f154ab1a28abba388b45c5/html5/thumbnails/43.jpg)
We are hiring at