![Page 1: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/1.jpg)
Death Stars & River Deltas
Toward a Functional Programming Analogy for Microservices
![Page 2: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/2.jpg)
Hi, I’m Bobby
I’m on the Technology Fellows team at
I dislike accidental complexity
[email protected] @bobbycalderwood https://github.com/bobby
![Page 3: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/3.jpg)
Disclaimers
• What follows is not a rigorous attempt to extend the formal models of OO and FP to the distributed case
• Rather, it is an informal extension of basic intuition and principles
• Plus I’m intentionally trolling a little, relax :-)
![Page 4: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/4.jpg)
Microservices!
• Adopt!
• Abandon!
• Assumptions…
![Page 5: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/5.jpg)
AWS Death Star diagram, circa 2008 as per Werner Vogels tweet https://twitter.com/Werner/status/741673514567143424
![Page 6: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/6.jpg)
Object-Oriented• Encapsulated data access via synchronous calls
• Mutable state change via synchronous calls
• Dependency web
• Imperative, sequence-oriented orchestration
• Referentially opaque
![Page 7: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/7.jpg)
Functional Programming• Data access via ubiquitous access to immutable
values
• State change via mapping an identity to different immutable values over time
• Data Flow graph
• Declarative orchestration
• Referential transparency
![Page 8: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/8.jpg)
Contrasting PrinciplesObject Oriented Functional
Data Access Encapsulated Ubiquitous
State Change Mutable, in-place Immutable, values over time
Organization Dependency graph Data-flow graph
Orchestration Imperative, sequential Declarative, parallelizable
Referentially… Opaque Transparent
![Page 9: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/9.jpg)
Object-Oriented
• Might be fine within single memory space (let’s talk…)
• Does not scale well to distributed case!
![Page 10: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/10.jpg)
Distributed Objects• Latency grows with depth of dependency graph
• Temporal liveness coupling
• Synchronous, pull-oriented by default
• Cascading failure modes
• Inconsistent, imperative orchestration
• Hidden narrative: can only see the nouns, verbs ephemeral
• Complexity: how to reason about state of system at any point in time?
![Page 11: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/11.jpg)
Source: NASA, Public Domain
![Page 12: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/12.jpg)
Deltas and Lambdas
• OO : Death Star :: FP : River Delta
• Functional Programming analogy scales well to the distributed case!
![Page 13: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/13.jpg)
Distributed Functions• Low latency at both read and write time (with eventual
consistency in between)
• Temporal de-coupling
• Isolated failure modes
• (Eventually) consistent, declarative orchestration
• Reified narrative: event stream
• Clear reasoning about (and replay of!) state over time
![Page 14: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/14.jpg)
Example: Balance Calculation
• Customer-defined weekly spending limits/notifications on a particular account
• Aggregate the debits, possibly emit event when balance exceeds limit for time period
• React to event in customer-defined way (prevent additional transactions until end of period, send notification, etc.)
• Allow customer to see transactions and balances
![Page 15: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/15.jpg)
OO-Analogy µServices
![Page 16: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/16.jpg)
FP-Analogy µServices
![Page 17: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/17.jpg)
But how?
• Architecture informed by
• Techniques
• Rules
• Tools
![Page 18: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/18.jpg)
FP-Analogy µServices
![Page 19: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/19.jpg)
Techniques• REST (at the edge)
• CQRS
• Event Sourcing (storage)
• Pub/Sub (conveyance)
• Sagas
• Serverless
![Page 20: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/20.jpg)
Rules1. Capture all observations and changes at the edge
(carefully!) to an immutable event stream
2. Reactively calculate streams of derived state from the event stream
3. Aggregate state wherever and however it provides value
4. Manage outgoing reactions (“side-effects”) to state and events carefully
![Page 21: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/21.jpg)
Capture changes at the edge to immutable event stream
• A (very) few authorized teams capture all raw observations (Events) and customer requests for action (Commands)
• at the edge of bounded context
• with minimal processing
• immutably and durably
• Causally related events go on same log
• One (logical) writer per command/event stream
• No change gets into bounded context via any other means!
![Page 22: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/22.jpg)
Capture changes at the edge to immutable event stream
![Page 23: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/23.jpg)
Reactively calculate derived state from the event log
• A few authorized teams process Commands into Events (probably using aggregate state)
• A few (more) authorized teams calculate state streams of general interest derived from Events
• Single-entity state changes
• Regulated or audited calculations
• Could be recursive, i.e. certain state changes might trigger further events downstream
![Page 24: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/24.jpg)
Reactively calculate derived state from the event log
![Page 25: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/25.jpg)
Aggregate state wherever and however it provides value• Many teams across orgs aggregate streams into views for their
respective use-cases
• Into whatever data store makes sense for required data access pattern
• Possibly emitting views back onto streams (a la Kafka Streams’ KTable)
• Facilitates stateful computations: joins, windowed aggregates, command processing
• Everyone traces lineage all the way back to edge Command/Event
![Page 26: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/26.jpg)
Aggregate state wherever and however it provides value
![Page 27: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/27.jpg)
Manage “side-effects” carefully
• A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”
• A “side-effect” in this case is any action not associated with reading or writing the streams or aggregates, e.g.
• Sending email, text message
• Making a call to an outside web service
• Calling a service to write a command/event to a stream you don’t own
• Log results of attempt to stream to facilitate retry, Saga/rollback/compensating action
![Page 28: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/28.jpg)
Tools• Kafka
• Kafka Streams
• Kafka Connect
• Various data stores
• Serverless integrations (e.g. OpenWhisk Kafka package)
![Page 29: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/29.jpg)
Example
![Page 30: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/30.jpg)
Questions?
![Page 31: Death Stars & River Deltas - Amazon S3€¦ · Manage “side-effects” carefully • A few authorized teams react to Events by reaching outside of bounded context to cause “side-effects”](https://reader034.vdocuments.us/reader034/viewer/2022050419/5f8f23e850c4c203d40d0f74/html5/thumbnails/31.jpg)
References• Sagas: https://www.cs.cornell.edu/andru/cs711/2002fa/reading/
sagas.pdf
• Ben Stopford on Events/Microservices: https://www.confluent.io/blog/build-services-backbone-events/
• Rich Hickey:
• Clojure’s approach to state: https://clojure.org/about/state
• The Value of Values: https://www.infoq.com/presentations/Value-Values
• The Language of the System: https://www.youtube.com/watch?v=ROor6_NGIWU