how events are reshaping modern systems

How Events Are Reshaping Modern Systems

Jonas Bonér @jboner

Why Events?

1. Events drive autonomy 2. Events help reduce risk 3. Events help you move faster 4. Events Increase Loose coupling 5. Events increase stability 6. Events increase scalability 7. Events increase resilience 8. Events increase traceability 9. Events allow for time-travel

Why Should you care?

1. Cloud and multicore architectures 2. Microservices and distributed systems 3. Data-centric applications 4. “We want more, of everything, and we want it now.” - Your Customers

Why Now?1. Cloud and multicore architectures 2. Microservices and distributed systems 3. Data-centric applications 4. “We want more, of everything, and we want it now.” - Your Customers

Do Not Settle For

Microliths

“Event-Driven Architecture (EDA) is a design paradigm in which a software component executes in response

to receiving one or more event notifications.”- Gartner

Event Driven Architecture To The Rescue

What is an

Event?

Event“Something that happens.”

- Merriam Webster

The Nature of Events

✴ Events represent Facts of information ➡ Facts are immutable

➡ Facts Accrue - Knowledge can only grow




✴ Events/Facts can be disregarded/Ignored




✴ Events/Facts can be disregarded/Ignored✴ Events/Facts Can not be retracted (once accepted)




✴ Events/Facts can be disregarded/Ignored✴ Events/Facts Can not be retracted (once accepted)✴ Events/Facts Can not be deleted (once accepted)

➡ Might be needed for legal or moral reasons




✴ Events/Facts can be disregarded/Ignored✴ Events/Facts Can not be retracted (once accepted)✴ Events/Facts Can not be deleted (once accepted)

➡ Might be needed for legal or moral reasons

✴ Events/Facts (new) can invalidate existing Facts


Event Loop What Makes Event Tick

✴ Queue + anonymous Callbacks

✴ The Hollywood principle

✴ At-Most-Once delivery guarantees

✴ Local context only

✴ Sync or AsynC

✴ Limited and quite powerless

The Event Loop

✴ Queue + anonymous Callbacks

✴ The Hollywood principle

✴ At-Most-Once delivery guarantees

✴ Local context only

✴ Sync or AsynC

✴ Limited and quite powerless

The Event Loop

*

* Too limited as a general model of computation, and programming model for modern distributed systems. But is a useful building block.

✴ Ephemeral and anonymous

✴ No sane failure model Error Channel - console.log(‘4/0=err’, err)

✴ No composition Signature: Input => Side-effect - (Any => Unit)

✴ Hard to Debug

✴ Hard to reason about

✴ Leads to Callback hell

The Problem With Event Callbacks

https://techbrunch.gousto.co.uk/2016/04/22/callback-hell/

✴ Patterns Observer, Pub-Sub, Broadcast, …

✴ Futures & PRomises

✴ Dataflow Variables

✴ Stream Processing ✴ Event-driven Microservices ✴ FaaS

We Need High Level

Abstractions

Unified Event Driven Abstractions Powered By

Message Driven Fabric

Ladder of abstraction



Network Boundary




Unified API (Event-driven Microservices, Patterns, Streams, Futures, FaaS,…)

Network Boundary





Network Boundary

Event 1 Ladder of abstraction




Local Event Loop

Network Boundary





Local Event Loop

Distribution Fabric

Network Boundary





Local Event Loop

Distribution Fabric Distribution Fabric

Network Boundary

Message Passing





Local Event Loop

Distribution Fabric

Local Event Loop

Distribution Fabric

Network Boundary

Message Passing





Local Event Loop

Distribution Fabric

Local Event Loop

Distribution Fabric

Network Boundary

Message Passing

Event 1 Event 1 Ladder of abstraction




Local Event Loop

Distribution Fabric

Local Event Loop

Distribution Fabric

Network Boundary

Message Passing

Event 1 Event 1

Reactive Systems is doing the heavy lifting





Local Event Loop

Distribution Fabric

Local Event Loop

Distribution Fabric

Network Boundary

Message Passing

Event 1 Event 1

Reactive Systems is doing the heavy lifting

Ladder of abstraction Reactive Programming

1. REceive and react to facts (or not), that are coming its way

2. Publish new facts (as events) to the rest of the world

3. Invert the control flow to minimize coupling and increase autonomy

Event Driven

Services

Mutable State Needs To Be Contained And Non Observable

Publish Facts To Outside World

Event Stream

Event Stream

Use The

as the integration fabric

Event Driven

Services

Event Driven

Services

Command

Event Driven

Services

Command

Event

Event Stream

Event Driven

Services

Command

Event

Event Event Event Stream

Event Driven

Services

EventualConsistency

Command

Event

Event Event Event Stream

Eventual Consistency

Eventual Consistency

No One Wants

It’s a necessery evil

But Relax It Is How The World Works

Information Has Latency

Information Is Always

From the Past

Welcome To The Wild Ocean Of

Non Determinism Distributed Systems

“In a system which cannot count on distributed transactions, the

management of uncertainty must be implemented in the business logic.”

- Pat Helland

Life Beyond Distributed Transactions - An Apostate’s Opinion, Pat Helland (2007)

We Need To Model

Uncertainty

Exploit Reality

Exploit Reality

We need to

In our design

Events Can Lead To Greater

Certainty

Events Can Help Us Craft Autonomous Islands Of Determinism

What does it mean to be

Automonous?

From greek: Auto-nomos

✴ Auto meaning self

✴ Nomos meaning Law

Autonomy

Promise Theory

Promise Theory

Let

Lead the way

✴Impositions diverge into unpredictable outcomes from definite beginnings ⇒ Distributed information ⇒ Decreased certainty

Convergence vs.

Divergence of information

✴Promises converge towards a definite outcome from unpredictable beginnings ⇒ Local information ⇒ Improved certainty

“Autonomy makes information local, leading to greater certainty and stability.”

“An autonomus component can only promise its own behavior.”

- Mark Burgess

In Search of Certainty, Thinking in Promises - Mark Burgess

Benefits of Thinking in Promises

✴ If a service only promises its own behavior Then all information needed to ➡ Resolve a conflict ➡ Repair under failure ➡ Is available within the service itself


✴ If a service only promises its own behavior Then all information needed to ➡ Resolve a conflict ➡ Repair under failure ➡ Is available within the service itself

✴ Promises remove the need for needless 1. Communication 2. Coordination 3. Consensus


“The first principle of successful scalability is to batter the consistency

mechanisms down to a minimum.”- James Hamilton

As transcribed in: Toward a Cloud Computing Research Agenda. SIGACT News, 40(2):68–80, June 2009.

Events First Design Drives Autonomy

Events First Domain Driven

Design

“When you start modeling events, it forces you to think about the behavior of the system. As opposed to thinking

about the structure of the system.”- Greg Young

A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)

✴ Don’t focus on the things The Nouns The Domain Objects

✴ Focus on what happens The Verbs The Events

Mine the

Facts

Event Storming

Event Driven Design

✴ IntentS ➡ Communication ➡ Conversations ➡ Expectations ➡ Contracts ➡ Control Transfer

Event Driven Design


Event Driven Design

✴ Facts ➡ State ➡ History ➡ Causality ➡ Notifications ➡ State Transfer


Event Driven Design


Commands


Event Driven Design


Commands Events

Event Driven Design

✴Commands ➡ Object form of method/Action request ➡ Imperative: CreateOrder, ShipProduct

Event Driven Design


✴Reactions ➡ Represents side-effects

Event Driven Design


✴Reactions ➡ Represents side-effects

✴Events ➡ Represents something that has happened ➡ Past-tense: OrderCreated, ProductShipped

Event Driven Design

Commands Eventsvs

Commands Eventsvs1. All about intent 1. Intentless

Commands Eventsvs1. All about intent2. Directed

1. Intentless2. Anonymous

Commands Eventsvs1. All about intent2. Directed3. Single addressable destination

1. Intentless2. Anonymous3. Just happens - for others (0-N) to observe


4. Models personal communication


4. Models broadcast (speakers corner)



5. Distributed focus



5. Local focus



5. Distributed focus6. Command & Control



5. Local focus6. Autonomy

Let the Events Define the

Bounded Context

Principles For Promise Theory Driven

Protocol Design

1. Convergency Promises should reach a desired state, a stable outcome - idempotent behavior


Protocol Design


2. Embracing Failure Expect things to go wrong Promises may or may not be kept


Protocol Design


2. Embracing Failure Expect things to go wrong Promises may or may not be kept

3. Autonomy You can’t rely on other services


Protocol Design

✴ Maintains Integrity & consistency

✴ Is our Unit of Consistency

✴ Is our Unit of Failure

✴ Is our Unit of Determinism

✴ Is fully Autonomous

The Aggregate

Enough Talk Show Me The

Code

Sample in Java: https://gist.github.com/jboner/4361c710d2e2386be8bddc39edb0528a Sample in Scala: https://gist.github.com/jboner/dea4aa819e127d543d684208d74081b5

https://gist.github.com/jboner/4361c710d2e2386be8bddc39edb0528a

https://gist.github.com/jboner/dea4aa819e127d543d684208d74081b5

Are we done now?

Are we done now?Perhaps. We have come a long way.

Are we done now?Perhaps. We have come a long way.But events can also be used for:


➡ Persistence


➡ Persistence➡ Managing time

Event Based Persistence

“Update-in-place strikes systems designers as a cardinal sin: it violates

traditional accounting practices that have been observed for hundreds of years.”

- jim Gray

The Transaction Concept, Jim Gray (1981)

Event Logging The Bedrock

Event Sourcing A Cure For the Cardinal Sin

“The truth is the log. The database is a cache of a subset of the log.”

- Pat Helland

Immutability Changes Everything, Pat Helland (2015)

Event Sourced Services

Happy Path


Happy Path


1) Receive and verify Command (“ApprovePayment”)

Happy Path


2) Create new Event (“PaymentApproved”)


Happy Path




3) Append Event to Event Log

Happy Path





4) Update internal component state

Happy Path


5) Run side-effects (approve the payment)





SAD Path - recovering from failure

Happy Path








Happy Path







1) Rehydrate Events from Event Log


Happy Path










Happy Path









Memory Image

Event Sourcing

✴ One single Source of Truth with All history ✴ Allows for Memory Image - durable in-memory state ✴ Avoids the Object-relational mismatch ✴ Allows others to Subscribe to state changes ✴ Mechanical sympathy (Single Writer Principle etc.)

✴ Unfamiliar model ✴ Versioning of events

✴ Deletion of events (legal or moral reasons)

Disadvantages Of Using Event Sourcing

Events Allow Us To Manage

Time

“Modeling events forces you to have a temporal focus on what’s going on in the system.

Time becomes a crucial factor of the system.”- Greg Young

A Decade of DDD, CQRS, Event Sourcing, Greg Young (Presentation from 2016)

But, What is Time

Really?

Causality

How We Make Sense Of The World

Time Is the Succession Of Causally Related Events

✴ Event is a snapshot in time ✴ Event ID is an index for time ✴ Event Log is our full history

The database of Our past

The path to Our present

Event Sourcing Allows Us To

Model Time

Event Sourcing Allows For

Time Travel

Event Sourcing Allows For

Time Travel

✴Replay the log for historic debugging ✴Replay the log for auditing & traceability ✴Replay the log on failure

✴Replay the log for replication

“The future is a function of the past.”- A. P. Robertson

“The (local) present is a merge function of multiple concurrent pasts.”

- me

val newLocalPresent = observedPasts. foldLeft(oldLocalPresent) { _ merge _ }

We need to explicitly model the

local present as facts derived from the

merging of multiple

concurrent pasts

We Can Even Fork the Past

...Or Join Two Distinct Pasts

Let Make Our Demo

Event Sourced

Untangle Your Read and Write Models

With CQRS

CQRS

Write Side Model

Commands

CQRS

Write Side Model

Write to Event Log

Commands

CQRS

Write Side Model

Events

Write to Event Log

Commands

CQRS

Write Side Model

Events

Events

Write to Event Log

Commands

CQRS

Read Side Model

Write Side Model

Events

Events

Read Data Store

Write to Event Log

Commands

CQRS

Read Side Model

Write Side Model

Events

Queries

Events

Read Data Store

Write to Event Log

Commands

CQRS

Read Side Model

Write Side Model

Events

Queries

Events

Read Data Store

Write to Event Log

EventualConsistency

Commands

✴ Temporal Decoupling ✴ Increased Resilience ✴ Increased Scalability

✴ Allows for Multiple Event subscribers ✴ Allows for Polyglot Persistence

Advantages Of Using CQRS

✴ More infrastructure to manage ✴ 2 data models (or more) to maintain ✴ Eventual Consistency

Disadvantages Of Using CQRS

Key Takeaways

Key Takeaways

Event driven design helps you to: ✴ Design autonomous services ✴ Move Faster towards a Reactive architecture ✴ reduce risk when modernizing applications ✴ Balance Certainty and Uncertainty

Key Takeaways

Event driven design helps you to: ✴ Design autonomous services ✴ Move Faster towards a Reactive architecture ✴ reduce risk when modernizing applications ✴ Balance Certainty and Uncertainty

Event Logging allows you to: ✴ take control of your system’s history ✴ time-travel

✴ Balance Strong and eventual consistency

Learn MoreDownload my latest book for free at: bit.ly/reactive-microsystems

http://bit.ly/reactive-microsystems

how events are reshaping modern systems

Engineering