data focused docker clustering
DESCRIPTION
See also the more detailed blog post at https://clusterhq.com/blog/data-focused-docker-clustering/ In this talk from the July Docker Edinburgh Meetup Luke Marsden, from ClusterHQ, explains how a data driven approach to clustering can improve robustness and and scalability, while reducing complexity.TRANSCRIPT
![Page 1: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/1.jpg)
Data focused Docker clustering
Luke MarsdenFounder, ClusterHQ
clusterhq.comgithub.com/clusterhq/flocker
![Page 2: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/2.jpg)
Background:
Container clusters where volumes are a first class citizen
![Page 3: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/3.jpg)
Motivations:
robustnessscalability
eliminate SPOFsreduce complexity
![Page 4: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/4.jpg)
Modern applications
are complex distributed systems
![Page 5: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/5.jpg)
Containers and PaaS today
capture the easy, data-free bits
![Page 6: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/6.jpg)
We should aim to capture the entire app
enabling larger class of apps to move to containers
![Page 7: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/7.jpg)
Goals
Multi-node/clustered
![Page 8: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/8.jpg)
Ease of operational manageability
Homogeneous serversNo single point of failure
Multi-regionData services the cluster depends on must
be protected by/within the system itself
![Page 9: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/9.jpg)
Containers should be portable, even when they have state
(corollary: the system should expose portable, resilient volumes)
![Page 10: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/10.jpg)
Dependent services or applications should not need to be reconfigured when a service
moves around
![Page 11: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/11.jpg)
Stateful things scale vertically, stateless
things scale horizontally
![Page 12: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/12.jpg)
Proposed implementation
![Page 13: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/13.jpg)
Management layer
Strongly consistent group messaging semantics
Distributed consensus protocolsNo single master
![Page 14: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/14.jpg)
Storage
ZFS: storage analogue to containers for compute
Independently snapshotted and replicatedSharding container volumes across a cluster
Redundancy invariantAutomatic slave allocation
![Page 15: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/15.jpg)
Networking
AwesomeProxyany request to any node
Routes vs. Links
![Page 16: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/16.jpg)
Live migration
Pausing in flight requestsQuiesce filesystem
Replicate last snapshotUnleash requests
![Page 17: Data focused docker clustering](https://reader036.vdocuments.us/reader036/viewer/2022062406/558ca0b2d8b42a256d8b4734/html5/thumbnails/17.jpg)
Thank you - questions?
Check it out!github.com/clusterhq/flocker
clusterhq.com