developing microservices with aggregates (devnexus2017)

@crichardson

Developing Microservices with Aggregates

Chris Richardson

Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action

@crichardson chris@chrisrichardson.net http://eventuate.io

@crichardson

Presentation goal

Show how Domain Driven Design Aggregates

and Microservices are a perfect match

@crichardson

About Chris

@crichardson

About Chris

Consultant and trainer focusing on modern

application architectures including microservices

(http://www.chrisrichardson.net/)

@crichardson

About Chris

Founder of a startup that is creating an open-source/SaaS platform

that simplifies the development of transactional microservices

(http://eventuate.io)

@crichardson

For more information

http://learnmicroservices.io

@crichardson

Agenda

The problem with Domain Models and microservices

Overview of aggregates

Maintaining consistency between aggregates

Using event sourcing with Aggregates

Customers and Orders example

Using explicit saga orchestration

@crichardson

The Microservice architecture tackles complexity through

modularization

@crichardson

Microservice =

Business capability

@crichardson

Microservice architecture

Browser

Mobile Device

Store Front UI

API Gateway

Catalog Service

Review Service

Order Service

… Service

Catalog Database

Review Database

Order Database

… Database

HTML

REST

REST

Database per service

@crichardson

But there are challenges…

@crichardson

Product service

Customer serviceOrder service

Domain model = tangled web of classes

Order

OrderLine Item

quantity

…

Addressstreet city …

Customer

Productname price

?

?

creditLimit

@crichardson

Reliance on ACID transactions to enforce invariantsBEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION

Simple and ACID

@crichardson

But it violates encapsulation…BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION

Private to the Order Service

Private to the Customer Service

@crichardson

.. and requires 2PCBEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION

@crichardson

2PC is not an optionGuarantees consistency

BUT

2PC coordinator is a single point of failure

Chatty: at least O(4n) messages, with retries O(n^2)

Reduced throughput due to locks

Not supported by many NoSQL databases (or message brokers)

CAP theorem ⇒ 2PC impacts availability

….

@crichardson

Doesn’t fit with the NoSQL DB “transaction” model

@crichardson

Agenda

The problem with Domain Models and microservices

Overview of aggregates

Maintaining consistency between aggregates

Using event sourcing with Aggregates

Customers and Orders example

Using explicit saga orchestration

@crichardson

Domain Driven Design - building blocks

Entity

Value object

Services

Repositories

Aggregates

Adopted

Ignored (at least by me)

About AggregatesCluster of objects that can be treated as a unit

Graph consisting of a root entity and one or more other entities and value objects

Typically business entities are Aggregates, e.g. customer, Account, Order, Product, ….

Order

OrderLine Item

quantity productId productName productPrice

customerId

Address

street city …

Aggregate: rule #1

Reference other aggregate roots via

identity

(primary key)

Order

OrderLine Item

quantity productId productName productPrice

customerId

Address

street city …

@crichardson

Foreign keys in a Domain Model?!?

@crichardson

Domain model = collection of loosely connected aggregates

Order

OrderLine Item

quantity

…

Addressstreet city …

Customer

Productname price

@crichardson

Product service

Customer serviceOrder service

Easily partition into microservices

Order

OrderLine Item

quantity

…

Addressstreet city …

Customer

Productname price

@crichardson

Aggregate rule #2

Transaction =

processing one command by one aggregate

@crichardson

Product service

Customer serviceOrder service

Transaction scope = service

Order

OrderLine Item

quantity

…

Addressstreet city …

Customer

Productname price

@crichardson

Transaction scope =

NoSQL database “transaction”

@crichardson

Agenda

The problem with Domain Models and microservices

Overview of aggregates

Maintaining consistency between aggregates

Using event sourcing with Aggregates

Customers and Orders example

Using explicit saga orchestration

@crichardson

Customer management

How to maintain data consistency between aggregates?

Order management

Order Service

placeOrder()

Customer Service

updateCreditLimit()

Customer

creditLimit ...

has ordersbelongs toOrder

total

Invariant: sum(open order.total) <= customer.creditLimit

?

@crichardson

Saga

Using event-driven Sagas instead of 2PCDistributed transaction

Order Customer

Local transaction

Order

Local transaction

Customer

Local transaction

Order

Event Event

@crichardson

Order Service

Saga-based, eventually consistent order processing

Customer Service

Order created

Credit Reserved

Credit Check Failed

Create Order

OR Customer

creditLimit creditReservations ...

Order

state total …

approve()/reject()

Event Handler

Event Handler

reserveCredit()

@crichardson

Order ServiceOrder

id : 4567 total: 343 state = CREATED

Customer Service

Customer creditLimit : 12000 creditReservations: {}

Customer creditLimit : 12000 creditReservations: { 4567 -> 343}

Order id : 4567 total: 343 state = APPROVED

Eventually consistent credit checking

Message Bus

createOrder()

Publishes:Subscribes to:

Subscribes to:

publishes:

OrderCreatedEvent

CreditReservedEvent

OrderCreatedEvent CreditReservedEvent

@crichardson

Complexity of compensating transactions

ACID transactions can simply rollback

BUT Developer must write application logic to “rollback” eventually consistent transactions

For example:

CreditCheckFailed => cancel Order

Money transfer destination account closed => credit source account

Careful design required!

@crichardson

How atomically update database and publish an event

Order Service

Order Database

Message Broker

insert Order

publish OrderCreatedEvent

dual write problem

?

@crichardson

Failure = inconsistent system

Order Service

Order Database

Message Broker

insert Order

publish OrderCreatedEvent

X

@crichardson

2PC is not an option

@crichardson

Reliably publish events when state changes

@crichardson

Agenda

The problem with Domain Models and microservices

Overview of aggregates

Maintaining consistency between aggregates

Using event sourcing with Aggregates

Customers and Orders example

Using explicit saga orchestration

@crichardson

Event sourcing = event-centric persistence

Application

Database

Event store

update

publish

X

@crichardson

Event sourcing

For each DDD aggregate:

Identify (state changing) domain events

e.g. use Event Storming

Define Event classes

For example, Order events: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped

@crichardson

Persists events NOT current state

Order

status ….

101 ACCEPTED

Order tableX

@crichardson

Event-centric schema

@crichardson

Persists events NOT current state

Event table

Entity type Event id

Entity id

Event data

Order 902101 …OrderApproved

Order 903101 …OrderShipped

Event type

Order 901101 …OrderCreated

@crichardson

Replay events to recreate state

Order

state

OrderCreated(…) OrderAccepted(…) OrderShipped(…)

Events

Periodically snapshot to avoid loading all events

@crichardson

The present is a fold over history

currentState = foldl(applyEvent, initialState, events)

@crichardson

Event Store

Application architecture

Order 123 Customer 456

OrderCreated OrderApproved …

CustomerCreated CustomerCreditReserved …

CreateOrder UpdateOrder GetOrder

Subscribe

Order Service

CreateCustomer UpdateCustomer GetCustomer

Subscribe

Customer Service

@crichardson

Request handling in an event sourced application

HTTP Handler

Event Store

pastEvents = findEvents(orderId)

Order

new()

applyEvents(pastEvents)

newEvents = processCmd(someCmd)

saveEvents(newEvents) (optimistic locking)

Order Service

applyEvents(newEvents)

@crichardson

Event Store publishes events consumed by other services

Event Store

Event Subscriber

subscribe(EventTypes)

publish(event)

publish(event)

Customer

update()

Customer Service

@crichardson

Event Store publishes events consumed by other services

Event Store

Event Subscriber

subscribe(EventTypes)

publish(event)

publish(event)

CQRS View

update()

Service Xyz

send notifications

…

Event store = database + message broker

Home grown/DIY

.NET geteventstore.com

https://www.lightbend.com/lagom

http://eventuate.io (mine)

…

Event Store

Save aggregate

events

Get aggregate

events

Subscribe to events

@crichardson

Benefits of event sourcingSolves data consistency issues in a Microservice/NoSQL based architecture

Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…

Eliminates O/R mapping problem (mostly)

Reifies state changes:

Built in, reliable audit log

temporal queries

Preserved history ⇒ More easily implement future requirements

@crichardson

Drawbacks of event sourcing…

Requires application rewrite

Weird and unfamiliar style of programming

Events = a historical record of your bad design decisions

Must detect and ignore duplicate events

Idempotent event handlers

Track most recent event and ignore older ones

…

@crichardson

… Drawbacks of event sourcing

Querying the event store can be challenging

Some queries might be complex/inefficient, e.g. accounts with a balance > X

Event store might only support lookup of events by entity id

Must use Command Query Responsibility Segregation (CQRS) to handle queries ⇒ application must handle eventually consistent data

@crichardson

Agenda

The problem with Domain Models and microservices

Overview of aggregates

Maintaining consistency between aggregates

Using event sourcing with Aggregates

Customers and Orders example

Using explicit saga orchestration

@crichardson

Orders and Customers example

Kafka

Rest API

Event Store

Event Store

Adapter

Customer Service

Rest APIOrder Service

Customer Aggregate

Order Aggregate

Rest APIOrder HistoryService

MongoDB

CQRS View

Event Store

Adapter

Event Store

Adapter

Used to notify Order Service of invalid

customer ids!

@crichardson

Customer microservice

Customer Service

CustomerController

Spring Cloud

StreamKafka

Customer Aggregate

Rest API

Customer Event Handlers

EventStore

Event Store

Adapter

Order*Event

POST /customers

@crichardson

The Customer aggregate

Money creditLimit Map<OrderId, Money> creditReservations

Customer

List<Event> process(CreateCustomerCommand cmd) { … } List<Event> process(ReserveCreditCommand cmd) { … } … void apply(CustomerCreatedEvent anEvent) { … } void apply(CreditReservedEvent anEvent) { … } …

State

Behavior

@crichardson

Familiar concepts restructured

class Customer {

public void reserveCredit( orderId : String, amount : Money) {

// verify

// update state this.xyz = … }

public List<Event> process( ReserveCreditCommand cmd) { // verify … return singletonList( new CreditReservedEvent(…); ) }

public void apply( CreditReservedEvent event) { // update state this.xyz = event.xyz }

@crichardson

Customer command processing

@crichardson

Customer applying events

@crichardson

Customer Controller

@crichardson

Creating a Customer

save() concisely specifies: 1.Creates Customer aggregate 2.Processes command 3.Applies events 4.Persists events

@crichardson

Event handling in CustomersDurable subscription name

1. Load Customer aggregate 2. Processes command 3. Applies events 4. Persists events

Triggers BeanPostProcessor

Publish message to Spring Cloud Stream (Kafka) when Customer not found

@crichardson

Event handler in Orders

@crichardson

Agenda

The problem with Domain Models and microservices

Overview of aggregates

Maintaining consistency between aggregates

Using event sourcing with Aggregates

Customers and Orders example

Using explicit saga orchestration

@crichardson

Order Service

Order Service ⬄ Customer Service: tightly coupled, cyclic dependency 😢

Customer Service

Order created

Credit Reserved

Credit Check FailedOR

Must subscribe to all Order events that

affect available credit!Order …

@crichardson

Order Service

Decouple by maintaining available credit in OrderService

Customer ServiceAvailable Credit Changed

Customer CreatedCustomer

creditLimit availableCredit ...

Order

state total …

AvailableCredit Tracker

creditLimit availableCredit ...

Simpler 😄 Still cyclic 😢

Use an explicit sagaSaga object that orchestrates the flow

Created in response to an initial event, e.g. OrderCreated

A state machine

Receives events from aggregates

Sends commands to aggregates

Implemented as an event-sourced aggregate

Saga

Event

Command

@crichardson

Customer Service

Order Service

Using an OrderCreationSaga

Order

state total …

AvailableCredit Tracker

creditLimit availableCredit ...

Order Creation

Saga

create()create()

Reserve Credit

Credit Reserved

Approve Order

Customer

Update Available

Credit

OrderCreated

Acyclic 😄

@crichardson

Order saga state machine

NEW

SUCCEEDED

FAILED

OrderCreatedEvent/ reserveCredit()

CustomerCreditReservedEvent /approveOrder()

EntityNotFoundCommandExecutionErrorCustomerCreditLimitExceededEvent /

rejectOrder()

@crichardson

Saga as a singleton state machine

A state machine state e.g. enum { NEW, …}

A data object

@crichardson

Event processing

(State + Data, Event) ⇒

(State’+ Data’, Commands)

@crichardson

(OrderSagaState, OrderSagaData, Event) ⇒

(Commands, OrderSagaState’, OrderSagaData’)

@crichardson

OrderCreationSaga state machine definition

event ⇒ saga id

starting + state transitions

@crichardson

Initial actionTriggers saga

creation

1. Send command 2. Transition to new state 3. Create data

@crichardson

Event handlerTriggering

event

1. Send command 2. Transition to new state

@crichardson

Kafka-based command bus

Order Saga

Credit Tracking

Aggregate

CreditTrackingCommand Topic

OrderSagaCommandFailure Topic

Reserve Credit

CustomerNotFound

Per aggregate

type

Per saga type

@crichardson

Summary

Aggregates are the building blocks of microservices

Use event-driven sagas to maintain consistency between aggregates

Event sourcing is a good way to implement a event-driven architecture

Use explicit sagas to orchestrate transactions

@crichardson

@crichardson chris@chrisrichardson.net

http://learnmicroservices.io

Questions?