leadership election with spring cloud cluster
TRANSCRIPT
![Page 1: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/1.jpg)
Locks and Leaders with SpringIntegration
Dave Syer, 2016 Twitter: @david_syer (see also @artem_bilan, @gprussell) Email: [email protected]
![Page 2: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/2.jpg)
AgendaWhat is a distributed lock?
What is a leader election?
What is it good for?
How do I do it?
Some tools for thinking about failure
Warning: Here be Dragons!
![Page 3: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/3.jpg)
A Simple Distributed Operation
![Page 4: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/4.jpg)
Two Clients
![Page 5: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/5.jpg)
Shuffle!
![Page 6: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/6.jpg)
Shuffle!
![Page 7: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/7.jpg)
Shuffle!
![Page 8: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/8.jpg)
Shuffle!
![Page 9: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/9.jpg)
Shuffle!
![Page 10: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/10.jpg)
Shuffle!
![Page 11: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/11.jpg)
Scary, Eh?
![Page 12: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/12.jpg)
LocksExample code using java.util.concurrent.locks.Lock:
boolean acquired = false; try acquired = lock.tryLock(10, TimeUnit.SECONDS); if (acquired) // Do something unique! catch (InterruptedException e) Thread.currentThread().interrupt(); throw new RuntimeException("Interrupted"); finally if (acquired) lock.unlock();
![Page 13: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/13.jpg)
Spring Integration: LockRegistrypublic interface LockRegistry Lock obtain(Object lockKey);
![Page 14: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/14.jpg)
Locks with Spring IntegrationExample code using LockRegistry:
boolean acquired = false; try acquired = lock.tryLock(10, TimeUnit.SECONDS); if (acquired) // Do something unique! catch (InterruptedException e) Thread.currentThread().interrupt(); throw new RuntimeException("Interrupted"); finally if (acquired) lock.unlock();
(same code)
![Page 15: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/15.jpg)
Locks and LeasesA distributed lock nearly always has a shelf life (it expires).
Technically, that makes it a "lease".
Without expiry system can't make progress when a lock holder dies.
![Page 16: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/16.jpg)
Dragons(At least) two problems are lurking:
1. Acquiring a lock requires consensus.
2. Leases expire and holder can never be sure when that happens.
Read this: http://martin.kleppmann.com/2016/02/08/howtododistributedlocking.html
TL;DR If using a lock for "correctness" not "efficiency" be very careful.
Important: you can tune the system to adjust the probability, or how long it lasts, but fundamentally you cannotprevent the system from ever allowing more than one holder of a lock.
![Page 17: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/17.jpg)
Leader Elections
Simple idea: if you hold a lock you are the leader.
What can you do with it?
Highly available globally unique things, often with messages
sequences
message aggregation
scheduling, e.g. cron service
![Page 18: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/18.jpg)
Spring Integration: Leader InitiatorImplementations of leader election need to be able to start an election and fire events on granted and revoked.
Zookeeper
Hazelcast
Etcd(*)
Generic (lockbased)
For a user it looks like this (create a new bean which is a SmartLifecycle):
@Bean public LeaderInitiator leaderInitiator(CuratorFramework client, Candidate candidate) return new LeaderInitiator(client, candidate);
(*) No support for etcd v3. Probably dead.
![Page 19: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/19.jpg)
Spring Integration: CallbacksCallbacks on leadership events:
public interface Candidate void onGranted(Context ctx) throws InterruptedException; void onRevoked(Context ctx); ...
See also:
@EventListener(OnGrantedEvent.class) public void start()
@EventListener(OnRevokedEvent.class) public void stop()
![Page 20: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/20.jpg)
ClosingWhen to use locks and leaders: HA active/passive failover
How to avoid it: latency from restarting a failed app might be fine
Otherwise, be idempotent. For correctness, deduplication store has to be ACID.
Relational databases can be really useful, as is Zookeeper.
Careful with the physics.
![Page 21: Leadership Election with Spring Cloud Cluster](https://reader035.vdocuments.us/reader035/viewer/2022082205/58f29f171a28ab6d7c8b459d/html5/thumbnails/21.jpg)
Linkshttp://presos.dsyer.com/decks/locksandleaders.html
Sample code: https://github.com/SpringOnePlatform2016/dsyerlocksandleaders
Spring Integration: https://github.com/springprojects/springintegration
Spring Cloud Cluster: https://github.com/springcloud/springcloudcluster