developing microservices with aggregates (devnexus2017)
TRANSCRIPT
@crichardson
Developing Microservices with Aggregates
Chris Richardson
Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action
@crichardson [email protected] http://eventuate.io
@crichardson
Presentation goal
Show how Domain Driven Design Aggregates
and Microservices are a perfect match
@crichardson
About Chris
@crichardson
About Chris
Consultant and trainer focusing on modern
application architectures including microservices
(http://www.chrisrichardson.net/)
@crichardson
About Chris
Founder of a startup that is creating an open-source/SaaS platform
that simplifies the development of transactional microservices
(http://eventuate.io)
@crichardson
For more information
http://learnmicroservices.io
@crichardson
Agenda
The problem with Domain Models and microservices
Overview of aggregates
Maintaining consistency between aggregates
Using event sourcing with Aggregates
Customers and Orders example
Using explicit saga orchestration
@crichardson
The Microservice architecture tackles complexity through
modularization
@crichardson
Microservice =
Business capability
@crichardson
Microservice architecture
Browser
Mobile Device
Store Front UI
API Gateway
Catalog Service
Review Service
Order Service
… Service
Catalog Database
Review Database
Order Database
… Database
HTML
REST
REST
Database per service
@crichardson
But there are challenges…
@crichardson
Product service
Customer serviceOrder service
Domain model = tangled web of classes
Order
OrderLine Item
quantity
…
Addressstreet city …
Customer
Productname price
?
?
creditLimit
@crichardson
Reliance on ACID transactions to enforce invariantsBEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION
Simple and ACID
@crichardson
But it violates encapsulation…BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION
Private to the Order Service
Private to the Customer Service
@crichardson
.. and requires 2PCBEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION
@crichardson
2PC is not an optionGuarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)
CAP theorem ⇒ 2PC impacts availability
….
@crichardson
Doesn’t fit with the NoSQL DB “transaction” model
@crichardson
Agenda
The problem with Domain Models and microservices
Overview of aggregates
Maintaining consistency between aggregates
Using event sourcing with Aggregates
Customers and Orders example
Using explicit saga orchestration
@crichardson
Domain Driven Design - building blocks
Entity
Value object
Services
Repositories
Aggregates
Adopted
Ignored (at least by me)
About AggregatesCluster of objects that can be treated as a unit
Graph consisting of a root entity and one or more other entities and value objects
Typically business entities are Aggregates, e.g. customer, Account, Order, Product, ….
Order
OrderLine Item
quantity productId productName productPrice
customerId
Address
street city …
Aggregate: rule #1
Reference other aggregate roots via
identity
(primary key)
Order
OrderLine Item
quantity productId productName productPrice
customerId
Address
street city …
@crichardson
Foreign keys in a Domain Model?!?
@crichardson
Domain model = collection of loosely connected aggregates
Order
OrderLine Item
quantity
…
Addressstreet city …
Customer
Productname price
@crichardson
Product service
Customer serviceOrder service
Easily partition into microservices
Order
OrderLine Item
quantity
…
Addressstreet city …
Customer
Productname price
@crichardson
Aggregate rule #2
Transaction =
processing one command by one aggregate
@crichardson
Product service
Customer serviceOrder service
Transaction scope = service
Order
OrderLine Item
quantity
…
Addressstreet city …
Customer
Productname price
@crichardson
Transaction scope =
NoSQL database “transaction”
@crichardson
Agenda
The problem with Domain Models and microservices
Overview of aggregates
Maintaining consistency between aggregates
Using event sourcing with Aggregates
Customers and Orders example
Using explicit saga orchestration
@crichardson
Customer management
How to maintain data consistency between aggregates?
Order management
Order Service
placeOrder()
Customer Service
updateCreditLimit()
Customer
creditLimit ...
has ordersbelongs toOrder
total
Invariant: sum(open order.total) <= customer.creditLimit
?
@crichardson
Saga
Using event-driven Sagas instead of 2PCDistributed transaction
Order Customer
Local transaction
Order
Local transaction
Customer
Local transaction
Order
Event Event
@crichardson
Order Service
Saga-based, eventually consistent order processing
Customer Service
Order created
Credit Reserved
Credit Check Failed
Create Order
OR Customer
creditLimit creditReservations ...
Order
state total …
approve()/reject()
Event Handler
Event Handler
reserveCredit()
@crichardson
Order ServiceOrder
id : 4567 total: 343 state = CREATED
Customer Service
Customer creditLimit : 12000 creditReservations: {}
Customer creditLimit : 12000 creditReservations: { 4567 -> 343}
Order id : 4567 total: 343 state = APPROVED
Eventually consistent credit checking
Message Bus
createOrder()
Publishes:Subscribes to:
Subscribes to:
publishes:
OrderCreatedEvent
CreditReservedEvent
OrderCreatedEvent CreditReservedEvent
@crichardson
Complexity of compensating transactions
ACID transactions can simply rollback
BUT Developer must write application logic to “rollback” eventually consistent transactions
For example:
CreditCheckFailed => cancel Order
Money transfer destination account closed => credit source account
Careful design required!
@crichardson
How atomically update database and publish an event
Order Service
Order Database
Message Broker
insert Order
publish OrderCreatedEvent
dual write problem
?
@crichardson
Failure = inconsistent system
Order Service
Order Database
Message Broker
insert Order
publish OrderCreatedEvent
X
@crichardson
2PC is not an option
@crichardson
Reliably publish events when state changes
@crichardson
Agenda
The problem with Domain Models and microservices
Overview of aggregates
Maintaining consistency between aggregates
Using event sourcing with Aggregates
Customers and Orders example
Using explicit saga orchestration
@crichardson
Event sourcing = event-centric persistence
Application
Database
Event store
update
publish
X
@crichardson
Event sourcing
For each DDD aggregate:
Identify (state changing) domain events
e.g. use Event Storming
Define Event classes
For example, Order events: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
@crichardson
Persists events NOT current state
Order
status ….
101 ACCEPTED
Order tableX
@crichardson
Event-centric schema
@crichardson
Persists events NOT current state
Event table
Entity type Event id
Entity id
Event data
Order 902101 …OrderApproved
Order 903101 …OrderShipped
Event type
Order 901101 …OrderCreated
@crichardson
Replay events to recreate state
Order
state
OrderCreated(…) OrderAccepted(…) OrderShipped(…)
Events
Periodically snapshot to avoid loading all events
@crichardson
The present is a fold over history
currentState = foldl(applyEvent, initialState, events)
@crichardson
Event Store
Application architecture
Order 123 Customer 456
OrderCreated OrderApproved …
CustomerCreated CustomerCreditReserved …
CreateOrder UpdateOrder GetOrder
Subscribe
Order Service
CreateCustomer UpdateCustomer GetCustomer
Subscribe
Customer Service
@crichardson
Request handling in an event sourced application
HTTP Handler
Event Store
pastEvents = findEvents(orderId)
Order
new()
applyEvents(pastEvents)
newEvents = processCmd(someCmd)
saveEvents(newEvents) (optimistic locking)
Order Service
applyEvents(newEvents)
@crichardson
Event Store publishes events consumed by other services
Event Store
Event Subscriber
subscribe(EventTypes)
publish(event)
publish(event)
Customer
update()
Customer Service
@crichardson
Event Store publishes events consumed by other services
Event Store
Event Subscriber
subscribe(EventTypes)
publish(event)
publish(event)
CQRS View
update()
Service Xyz
send notifications
…
Event store = database + message broker
Home grown/DIY
.NET geteventstore.com
https://www.lightbend.com/lagom
http://eventuate.io (mine)
…
Event Store
Save aggregate
events
Get aggregate
events
Subscribe to events
@crichardson
Benefits of event sourcingSolves data consistency issues in a Microservice/NoSQL based architecture
Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…
Eliminates O/R mapping problem (mostly)
Reifies state changes:
Built in, reliable audit log
temporal queries
Preserved history ⇒ More easily implement future requirements
@crichardson
Drawbacks of event sourcing…
Requires application rewrite
Weird and unfamiliar style of programming
Events = a historical record of your bad design decisions
Must detect and ignore duplicate events
Idempotent event handlers
Track most recent event and ignore older ones
…
@crichardson
… Drawbacks of event sourcing
Querying the event store can be challenging
Some queries might be complex/inefficient, e.g. accounts with a balance > X
Event store might only support lookup of events by entity id
Must use Command Query Responsibility Segregation (CQRS) to handle queries ⇒ application must handle eventually consistent data
@crichardson
Agenda
The problem with Domain Models and microservices
Overview of aggregates
Maintaining consistency between aggregates
Using event sourcing with Aggregates
Customers and Orders example
Using explicit saga orchestration
@crichardson
Orders and Customers example
Kafka
Rest API
Event Store
Event Store
Adapter
Customer Service
Rest APIOrder Service
Customer Aggregate
Order Aggregate
Rest APIOrder HistoryService
MongoDB
CQRS View
Event Store
Adapter
Event Store
Adapter
Used to notify Order Service of invalid
customer ids!
@crichardson
Customer microservice
Customer Service
CustomerController
Spring Cloud
StreamKafka
Customer Aggregate
Rest API
Customer Event Handlers
EventStore
Event Store
Adapter
Order*Event
POST /customers
@crichardson
The Customer aggregate
Money creditLimit Map<OrderId, Money> creditReservations
Customer
List<Event> process(CreateCustomerCommand cmd) { … } List<Event> process(ReserveCreditCommand cmd) { … } … void apply(CustomerCreatedEvent anEvent) { … } void apply(CreditReservedEvent anEvent) { … } …
State
Behavior
@crichardson
Familiar concepts restructured
class Customer {
public void reserveCredit( orderId : String, amount : Money) {
// verify
// update state this.xyz = … }
public List<Event> process( ReserveCreditCommand cmd) { // verify … return singletonList( new CreditReservedEvent(…); ) }
public void apply( CreditReservedEvent event) { // update state this.xyz = event.xyz }
@crichardson
Customer command processing
@crichardson
Customer applying events
@crichardson
Customer Controller
@crichardson
Creating a Customer
save() concisely specifies: 1.Creates Customer aggregate 2.Processes command 3.Applies events 4.Persists events
@crichardson
Event handling in CustomersDurable subscription name
1. Load Customer aggregate 2. Processes command 3. Applies events 4. Persists events
Triggers BeanPostProcessor
Publish message to Spring Cloud Stream (Kafka) when Customer not found
@crichardson
Event handler in Orders
@crichardson
Agenda
The problem with Domain Models and microservices
Overview of aggregates
Maintaining consistency between aggregates
Using event sourcing with Aggregates
Customers and Orders example
Using explicit saga orchestration
@crichardson
Order Service
Order Service ⬄ Customer Service: tightly coupled, cyclic dependency 😢
Customer Service
Order created
Credit Reserved
Credit Check FailedOR
Must subscribe to all Order events that
affect available credit!Order …
@crichardson
Order Service
Decouple by maintaining available credit in OrderService
Customer ServiceAvailable Credit Changed
Customer CreatedCustomer
creditLimit availableCredit ...
Order
state total …
AvailableCredit Tracker
creditLimit availableCredit ...
Simpler 😄 Still cyclic 😢
Use an explicit sagaSaga object that orchestrates the flow
Created in response to an initial event, e.g. OrderCreated
A state machine
Receives events from aggregates
Sends commands to aggregates
Implemented as an event-sourced aggregate
Saga
Event
Command
@crichardson
Customer Service
Order Service
Using an OrderCreationSaga
Order
state total …
AvailableCredit Tracker
creditLimit availableCredit ...
Order Creation
Saga
create()create()
Reserve Credit
Credit Reserved
Approve Order
Customer
Update Available
Credit
OrderCreated
Acyclic 😄
@crichardson
Order saga state machine
NEW
SUCCEEDED
FAILED
OrderCreatedEvent/ reserveCredit()
CustomerCreditReservedEvent /approveOrder()
EntityNotFoundCommandExecutionErrorCustomerCreditLimitExceededEvent /
rejectOrder()
@crichardson
Saga as a singleton state machine
A state machine state e.g. enum { NEW, …}
A data object
@crichardson
Event processing
(State + Data, Event) ⇒
(State’+ Data’, Commands)
@crichardson
(OrderSagaState, OrderSagaData, Event) ⇒
(Commands, OrderSagaState’, OrderSagaData’)
@crichardson
OrderCreationSaga state machine definition
event ⇒ saga id
starting + state transitions
@crichardson
Initial actionTriggers saga
creation
1. Send command 2. Transition to new state 3. Create data
@crichardson
Event handlerTriggering
event
1. Send command 2. Transition to new state
@crichardson
Kafka-based command bus
Order Saga
Credit Tracking
Aggregate
CreditTrackingCommand Topic
OrderSagaCommandFailure Topic
Reserve Credit
CustomerNotFound
Per aggregate
type
Per saga type
@crichardson
Summary
Aggregates are the building blocks of microservices
Use event-driven sagas to maintain consistency between aggregates
Event sourcing is a good way to implement a event-driven architecture
Use explicit sagas to orchestrate transactions