@crichardson

Events on the outside, on the inside and at the core

Chris Richardson

Founder of the original CloudFoundry.com Author of POJOs in Action

@crichardson chris@chrisrichardson.net http://plainoldobjects.com http://microservices.io http://eventuate.io

@crichardson

Presentation goal

How event sourcing enables the event-driven

enterprise

@crichardson

About Chris

@crichardson

About Chris

Consultant and trainer focusing on microservices (http://www.chrisrichardson.net/)

@crichardson

About Chris

Founder of a startup that is creating a platform that makes it easy for

application developers write microservices

(http://bit.ly/trialeventuate)

@crichardson

For more information

https://github.com/cer/event-sourcing-examples

http://microservices.io

http://plainoldobjects.com/

https://twitter.com/crichardson

http://eventuate.io/

@crichardson

Agenda

Events on the outside

Events on the inside

Events at the core with event sourcing

Designing event-centric domain model

@crichardson

What’s an event?

http://www.merriam-webster.com/dictionary/event

@crichardson

Examples of events

@crichardson

Instacart: event-driven grocery shopping

Who wants to be notified?Humans via

Email

SMS

Mobile push

…

Applications

Applications that implement other parts of the business process, e.g. order fulfillment system

Analytics dashboards, monitoring, …

Applications that deliver notifications to humans: Twilio, SendGrid, …

…

@crichardson

How: Polling for events

HTTP

Periodically poll for events

AtomPub

Based on HTTP

Head is constantly changing

Tail is immutable and can be efficiently cached

High-latency, inefficient

@crichardson

Using WebSockets

Browser

Web Socket

STOMP

Client application

Service

Web Socket

STOMP

ApplicationSUBSCRIBE

MESSAGEMESSAGE

Low latency, more efficient, but what about past events?

@crichardson

Webhooks = user-defined HTTP callback

Client Service

register(events, callbackUrl)

POST callbackUrl

POST callbackUrl

…

https://en.wikipedia.org/wiki/WebhookLow latency, more efficient, but what

about past events?

@crichardson

Github webhookshttps://developer.github.com/webhooks/

Installed on an organization or repository

e.g. POST /repos/:owner/:repo/hooks

Available events:

push - push to a repository

fork - repository is forked

pull_request - assigned, unassigned, …

push - push to a repository

…

Twilio - Telephony and SMS as a service

REST API

Allocate phone numbers

Make and receive phone calls

Send and receive SMS messages

Pay per use:

Phone calls - per-minute

SMS – per SMS sent or received

Phone number – per month

Examples

OpenVBX is a web-based, open source phone system

StubHub – notifies sellers of a pending sale via phone

SurveyMonkey – interactive polling

Salesforce – SMS-based voting for 19,000 conference attendees

@crichardson

Using Twilio

Twilio Your Application

TwiML doc

HTTP GET/POST

REST API

Manage resourcesSend SMS

Initiate voice calls

Webhooks handle incoming SMS and voice calls

VoiceSMS

Phone number ⇒ SMS URL + VOICE URL

@crichardson

Integration hubs - Zapier, IFTTTApplication abstraction:

Triggers - events published by application: polling or Webhooks

Action - operation supported by application, e.g. REST API end points

@crichardson

WebHooks =

web friendly publish/subscribe

@crichardson

Enterprise integration patterns

http://www.enterpriseintegrationpatterns.com/patterns/messaging/

@crichardson

messaging system

Messaging-based IPC

Sender RecipientChannelmessage

@crichardson

Example messaging systems

@crichardson

The event-driven enterprise

App A App BApp X

App Y

Firewall

Message Broker

webhooks

WebSockets

AtomPub

App Z App C

MsgBrokerClient Msg

BrokerClient

MsgBrokerClient

App D

MsgBrokerClient

Inside the firewallOutside the firewall

@crichardson

Agenda

Events on the outside

Events on the inside

Events at the core with event sourcing

Designing event-centric domain model

@crichardson

Traditional monolithic architecture

Simple to develop

test deploy scale

Shopping Cart

CustomersOrders

Mobile client RESTAPI

Browser WebUI

MessageBroker

Adapter

MySQLAdapter

RDBMS

…

Message Broker

@crichardson

But that leads* to monolithic hell

For large and/or complex applications…

@crichardson

Today: use a microservice, polyglot architecture

Orders

Customers

…Shopping UI

Mobile Client

Browser

API Gateway

Order management

RESTAPI

CustomerManagement

RESTAPI

….

RESTAPI

OrderDatabase

(MongoDB)MongoDBAdapter

CustomerDatabase(Sharded MySQL)

MySQLAdapter

@crichardson

But now we have distributed data management

problems

@crichardson

Example: placing an order

Order Service Customer Service

Order Database

Customer Database

Order #1 Customer #1

No 2PC

No ACID

NoSQL SQL

@crichardson

Customer management

How to maintain invariants?

Order management

Order Service

placeOrder()

Customer Service

updateCreditLimit()

Customer

creditLimit ...

has ordersbelongs toOrder

total

Invariant: sum(open order.total) <= customer.creditLimit

?

@crichardson

Use an event-driven architecture

Services publish events when something important happens, e.g. state changes

Services subscribe to events and update their state

Maintain eventual consistency across multiple aggregates (in multiple datastores)

Synchronize replicated data

@crichardson

Event-driven application architectureApplication

Service A Service B

Message Broker

Service C

MsgBrokerClient

MsgBrokerClient

MsgBrokerClient

Service D

MsgBrokerClient

@crichardson

Order ManagementOrder

id : 4567 total: 343 state = CREATED

Customer Management

Customer creditLimit : 12000 creditReservations: {}

Customer creditLimit : 12000 creditReservations: { 4567 -> 343}

Order id : 4567 total: 343 state = OPEN

Eventually consistent credit checking

Message Bus

createOrder()

Publishes:Subscribes to:

Subscribes to:

publishes:

OrderCreatedEvent

CreditReservedEvent

OrderCreatedEvent CreditReservedEvent

@crichardson

Now there are two problems to solve….

@crichardson

Problem #1: How to design eventually consistent business logic?

More on that later….

@crichardson

Problem #2: How to atomically update database and publish an event

Order Service

Order Database

Message Broker

insert Order

publish OrderCreatedEvent

dual write problem

?

@crichardson

Update and publish using 2PC

Guaranteed atomicity BUT

Need a distributed transaction manager

Database and message broker must support 2PC

Impacts reliability

Not fashionable

2PC is best avoided

@crichardson

Transaction log tailingHow:

Read the database “transaction log” = single source of truth

Publish events to message broker

LinkedIn databus https://github.com/linkedin/databus

Supports Oracle and MySQL

Publish as events

AWS DynamoDB streams

Ordered sequence of creates, updates, deletes made to a DynamoDB table

Last 24 hours

Subscribe to get changes

MongoDB

Read the oplog

Transaction log tailing: benefits and drawbacks

Benefits

No 2PC

No application changes required

Guaranteed to be accurate

Drawbacks

Immature

Database specific solutions

Low-level DB changes rather business level events = need to reverse engineer domain events

@crichardson

Use database triggers

Track changes to tables

Insert events into an event table

Use datastore as a message queue

Pull events from event table and write to message broker

Database triggers: benefits and drawbacks

Benefits

No 2PC

No application changes required

Drawbacks

Requires the database to support them

Database specific solutions

Low-level DB changes rather business level events = need to reverse engineer domain events

Error-prone, e.g. missing trigger

@crichardson

Application created events

Use datastore as a message queue

Txn #1: Update database: new entity state & event

Txn #2: Consume event

Txn #3: Mark event as consumed

Eventually consistent mechanism (used by eBay)

See BASE: An Acid Alternative, http://bit.ly/ebaybase

Application created eventsBenefits

High-level domain events

No 2PC

Drawbacks

Requires changes to the application

Only works for SQL and some NoSQL databases

Error-prone

@crichardson

Agenda

Events on the outside

Events on the inside

Events at the core with event sourcing

Designing event-centric domain model

@crichardson

Just publish events

Application

Database

Message Broker

update

publish

X

@crichardson

Event sourcingFor each aggregate (business entity):

Identify (state-changing) domain events

Define Event classes

For example,

ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent

Order: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped

@crichardson

Persists events NOT current state

Order

status ….

Event table

101 ACCEPTED

Order tableXOrderCreated101 901 …

OrderApproved101 902 …

OrderShipped101 903 …

@crichardson

Replay events to recreate state

Order

state

OrderCreated(…) OrderAccepted(…) OrderShipped(…)

Events

Periodically snapshot to avoid loading all events

@crichardson

The present is a fold over history

currentState = foldl(applyEvent, initialState, events)

@crichardson

Domain logic = event-driven aggregates

CustomerAggregateOrder

Aggregate

Order CreatedEvent Check Credit

Command

Credit ReservedEvent

Approve OrderCommand

Create OrderCommand

External request

Emits events Event ⇒ Command

@crichardson

Request handling in an event-sourced application

HTTP Handler

Event Store

pastEvents = findEvents(entityId)

Order

new()

applyEvents(pastEvents)

newEvents = processCmd(SomeCmd)

saveEvents(newEvents)

Microservice A

(optimistic locking)

@crichardson

Event Store publishes events - consumed by other services

Event Store

Event Subscriber

subscribe(EventTypes)

publish(event)

publish(event)

Aggregate

CQRS View

update()

update()

Microservice B

send notifications

…

Event store = database + message broker

Hybrid database and message broker

Implementations:

Home-grown/DIY

geteventstore.com by Greg Young

http://eventuate.io (mine)

Event Store

Save aggregate

events

Get aggregate

events

Subscribe to events

@crichardson

Benefits of event sourcingSolves data consistency issues in a Microservice/NoSQL-based architecture

Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…

Eliminates O/R mapping problem (mostly)

Reifies state changes:

Built-in, reliable audit log,

temporal queries

Preserved history ⇒ More easily implement future requirements

@crichardson

Drawbacks of event sourcing

Weird and unfamiliar

Events = a historical record of your bad design decisions

Handling duplicate events can be tricky

Application must handle eventually consistent data

Event store only directly supports PK-based lookup => use Command Query Responsibility Segregation (CQRS) to handle queries

@crichardson

Agenda

Events on the outside

Events on the inside

Events at the core with event sourcing

Designing event-centric domain model

@crichardson

Use the familiar building blocks of DDD

Entity

Value object

Services

Repositories

Aggregates ⟸ essential

Aggregate designGraph consisting of a root entity and one or more other entities and value objects

Each core business entity = Aggregate: e.g. customer, Account, Order, Product, ….

Reference other aggregate roots via primary key

Often contains partial copy of other aggregates’ data

Order

OrderLine Item

quantity productId productName productPrice

customerId

Address

street city …

@crichardson

Partition the domain model into Aggregates

Order

OrderLine Item

quantity

…

Addressstreet city …

Customer

Productname price

@crichardson

Transaction = processing one command by one aggregate

No opportunity to update multiple aggregates within a transaction

Aggregate granularity is important

If an update must be atomic (i.e. no compensating transaction) then it must be handled by a single aggregate

e.g. scanning boarding pass at security checkpoint or when entering jetway

@crichardson

Aggregate granularity

Consistency Scalability/ User experience

Customer

Order

Product

Customer

Order

Product

Customer

Order

Product

Designing domain eventsRecord state changes for an aggregate

Part of the public API of the domain model ProductAddedToCart

id : TimeUUID senderId: UUID productId productName productPrice …

Required by aggregate

Enrichment: Useful for

consumers

Event metadata

@crichardson

Example event

@crichardson

Designing commandsCreated by a service from incoming request

Processed by an aggregate

Immutable

Contains value objects for

Validating request

Creating event

Auditing user activity

@crichardson

Example command

@crichardson

Hybrid OO/FP domain objects

@crichardson

OO = State + Behavior

creditLimit creditReservations : Map[OrderId, Money]

Customer

List<Event> processCommand ( Command aCommand) { … }

void applyEvent (Event anEvent) { … }

State

Behavior

@crichardson

Aggregate traits

Map Command to Events

Apply event returning updated Aggregate

Used by Event Store

to reconstitute aggregate

@crichardson

ReflectiveMutableCommandProcessingAggregate

@crichardson

Customer - command processing

@crichardson

Customer - applying events

@crichardson

Event Store APIT is a subclass of Aggregate[T]

Rx Observable = Future++

@crichardson

Creating an order

save() concisely specifies: 1.Creates Customer aggregate 2.Processes command 3.Applies events 4.Persists events

@crichardson

Event handling in Customers

1.Load Customer aggregate 2.Processes command 3.Applies events 4.Persists events

Triggers BeanPostProcessor Durable subscription name

@crichardson

Summary

Events are a central to modern applications

Events integrate applications

Events maintain data consistency in a microservices architecture

⇒

Build events into the core of your application using event sourcing

@crichardson

@crichardson chris@chrisrichardson.net

http://plainoldobjects.com http://microservices.io

Questions?