building microservices with scala, functional domain models and spring boot
DESCRIPTION
In this talk you will learn about a modern way of designing applications that’s very different from the traditional approach of building monolithic applications that persist mutable domain objects in a relational database.We will talk about the microservice architecture, it’s benefits and drawbacks and how Spring Boot can help. You will learn about implementing business logic using functional, immutable domain models written in Scala. We will describe event sourcing and how it’s an extremely useful persistence mechanism for persisting functional domain objects in a microservices architecture.TRANSCRIPT
@crichardson
Building microservices with Scala, functional domain models and Spring Boot
Chris Richardson
Author of POJOs in ActionFounder of the original CloudFoundry.com
@[email protected]://plainoldobjects.com
@crichardson
Presentation goal
Share my experiences with building an application using Scala, functional domain
models, microservices, event sourcing, CQRS, and Spring Boot
@crichardson
About Chris
@crichardson
About Chris
Founder of a buzzword compliant (stealthy, social, mobile, big data, machine learning, ...) startup
Consultant helping organizations improve how they architect and deploy applications using cloud, micro services, polyglot applications, NoSQL, ...
@crichardson
Agenda
Why build event-driven microservices?
Overview of event sourcing
Designing microservices with event sourcing
Implementing queries in an event sourced application
Building and deploying microservices
@crichardson
Let’s imagine that you are building a banking app...
@crichardson
Domain model
Account
balance
open(initialBalance)debit(amount)credit(amount)
TransferTransaction
fromAccountIdtoAccountIdamount
@crichardson
Tomcat
Traditional application architecture
Browser/Client
WAR/EAR
RDBMS
Customers
Accounts
Transactions
Banking Banking UI
developtest
deploy
Simple to
Load balancer
scale
Spring MVC
SpringHibernate
...
HTMLREST/JSON
@crichardson
Problem #1: monolithic architecture
Intimidates developers
Obstacle to frequent deployments
Overloads your IDE and container
Obstacle to scaling development
Requires long-term commitment to a technology stack
@crichardson
Solution #1: use a microservice architecture
Banking UI
Account Management Service
Transaction Management Service
Account Database
Transaction Database
Standaloneservices
@crichardson
Problem #2: relational databases
Scalability
Distribution
Schema updates
O/R impedance mismatch
Handling semi-structured data
@crichardson
Solution #2: use NoSQL databases
Avoids the limitations of RDBMS
For example,
text search ⇒ Solr/Cloud Search
social (graph) data ⇒ Neo4J
highly distributed/available database ⇒ Cassandra
...
@crichardson
But now we have problems with data consistency!
@crichardson
Problem #3: Microservices = distributed data management
Each microservice has it’s own database
Some data is replicated and must be kept in sync
Business transactions must update data owned by multiple services
Tricky to implement reliably without 2PC
@crichardson
Problem #4: NoSQL = ACID-free, denormalized databases
Limited transactions, i.e. no ACID transactions
Tricky to implement business transactions that update multiple rows, e.g. http://bit.ly/mongo2pc
Limited querying capabilities
Requires denormalized/materialized views that must be synchronized
Multiple datastores (e.g. DynamoDB + Cloud Search ) that need to be kept in sync
@crichardson
Solution to #3/#4: Event-based architecture to the rescue
Microservices publish events when state changes
Microservices subscribe to events
Synchronize replicated data
Maintains eventual consistency across multiple aggregates (in multiple datastores)
@crichardson
Eventually consistent money transfer
Message Bus
Transaction management service
Account management
service
transferMoney()
Publishes:Subscribes to:
Subscribes to:
publishes:TransferTransactionCreatedEvent
AccountDebitedEvent
DebitRecordedEvent
AccountCreditedEventTransferTransactionCreatedEvent
DebitRecordedEvent
AccountDebitedEventAccountCreditedEvent
@crichardson
But reliably generating events is difficult
Must atomically update datastore and publish event(s)
Non-option: datastore and message broker use 2PC
Use datastore as message queue
Local transaction updates state and publishes message
See BASE: An Acid Alternative, http://bit.ly/ebaybase
But
Business logic and event publishing code intertwined
Tricky to implement with aggregate-oriented NoSQL database
@crichardson
Agenda
Why build event-driven microservices?
Overview of event sourcing
Designing microservices with event sourcing
Implementing queries in an event sourced application
Building and deploying microservices
@crichardson
Event sourcingFor each aggregate:
Identify (state-changing) domain events
Define Event classes
For example,
Account: AccountOpenedEvent, AccountDebitedEvent, AccountCreditedEvent
ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent
@crichardson
Persists events NOT current state
Account
balance
open(initial)debit(amount)credit(amount)
AccountOpened
Event table
AccountCredited
AccountDebited
101 450
Account tableX101
101
101
901
902
903
500
250
300
@crichardson
Replay events to recreate state
Account
balance
AccountOpenedEvent(balance)AccountDebitedEvent(amount)AccountCreditedEvent(amount)
Events
@crichardson
Aggregate traits
Map Command to Events
Apply event returning updated Aggregate
@crichardson
Account - command processing
Prevent overdraft
@crichardson
Account - applying eventsImmutable
@crichardson
Request handling in an event-sourced application
HTTPHandler
EventStore
pastEvents = findEvents(entityId)
Account
new()
applyEvents(pastEvents)
newEvents = processCmd(SomeCmd)
saveEvents(newEvents)
Microservice A
@crichardson
Event Store publishes events - consumed by other services
EventStore
EventSubscriber
subscribe(EventTypes)
publish(event)
publish(event)
Aggregate
NoSQLmaterialized
view
update()
update()
Microservice B
@crichardson
Event Store APItrait EventStore {
def save[T <: Aggregate[T]](entity: T, events: Seq[Event], assignedId : Option[EntityId] = None): Future[EntityWithIdAndVersion[T]]
def update[T <: Aggregate[T]](entityIdAndVersion : EntityIdAndVersion, entity: T, events: Seq[Event]): Future[EntityWithIdAndVersion[T]]
def find[T <: Aggregate[T] : ClassTag](entityId: EntityId) : Future[EntityWithIdAndVersion[T]]
def findOptional[T <: Aggregate[T] : ClassTag](entityId: EntityId) Future[Option[EntityWithIdAndVersion[T]]]
def subscribe(subscriptionId: SubscriptionId): Future[AcknowledgableEventStream]}
In case you are wondering: Akka Persistence is too Akka-centric
@crichardson
Benefits of event sourcingBusiness:
Built in audit log
Enables temporal queries
Technical:
Solves data consistency issues in a Microservice/NoSQL-based architecture:
Atomically save and publish events
Event subscribers update other aggregates ensuring eventual consistency
Event subscribers update materialized views in SQL and NoSQL databases
Eliminates O/R mapping problem
@crichardson
Drawbacks of event sourcing
Weird and unfamiliar
Events = a historical record of your bad design decisions
Handling duplicate events can be tricky
Idempotent commands
Duplicate detection, e.g. track most recently seen event
@crichardson
Agenda
Why build event-driven microservices?
Overview of event sourcing
Designing microservices with event sourcing
Implementing queries in an event sourced application
Building and deploying microservices
@crichardson
The anatomy of a microservice
Event Store
HTTP Request
HTTP Adapter
Aggregate
Event Adapter
Cmd
Cmd
EventsEvents
Xyz Adapter
Xyz Request
microservice
@crichardson
Asynchronous Spring MVC controller
@crichardson
AccountTransactionService
DSL concisely specifies:1.Creates new TransferTransaction2.Processes command3.Applies events4.Persists events
@crichardson
TransferTransaction Aggregate
@crichardson
Handling events published by Accounts
1.Load TransferTransaction2.Processes command3.Applies events4.Persists events
@crichardson
Agenda
Why build event-driven microservices?
Overview of event sourcing
Designing microservices with event sourcing
Implementing queries in an event sourced application
Building and deploying microservices
@crichardson
Let’s imagine that you want to display an account and it’s recent transactions...
@crichardson
Displaying balance + recent transactions
We need to do a “join: between the Account and the corresponding TransferTransactions
(Assuming Debit/Credit events don’t include other account, ...)
BUTEvent Store = primary key lookup of individual aggregates, ...
⇒Use Command Query Responsibility Separation
Define separate “materialized” query-side views that implement those queries
@crichardson
Query-side microservices
Event Store
Updater - microservice
View UpdaterService
EventsReader - microservice
HTTP GET Request
View Query Service
ViewStore
e.g. MongoDB
Neo4JCloudSearch
update query
@crichardson
Persisting account balance and recent transactions in MongoDB
{ id: "298993498", balance: 100000, transactions : [
{"transactionId" : "4552840948484", "fromAccountId" : 298993498, "toAccountId" : 3483948934, "amount" : 5000}, ...
], changes: [ {"changeId" : "93843948934", "transactionId" : "4552840948484", "transactionType" : "AccountDebited", "amount" : 5000}, ... ] }
Denormalized = efficient lookup
Transactions that update the account
The sequence of debits and credits
@crichardson
Updating MongoDB using Spring Data
class AccountInfoUpdateService (mongoTemplate : MongoTemplate, ...) extends CompoundEventHandler {
@EventHandler def recordDebit(de: DispatchedEvent[AccountDebitedEvent]) = { ... val ci = AccountChangeInfo(...)
mongoTemplate.updateMulti( new Query(where("id").is(de.entityId.id).and("version").lt(changeId)), new Update(). dec("balance", amount). push("changes", ci). set("version", changeId), classOf[AccountInfo]) }
@EventHandler def recordTransfer(de: DispatchedEvent[TransferTransactionCreatedEvent]) = ...
}
insert/In-place update
duplicate event detection
updates to and from accounts
@crichardson
Retrieving account info from MongoDB using Spring Data
class AccountInfoQueryService(accountInfoRepository : AccountInfoRepository) {
def findByAccountId(accountId : EntityId) : AccountInfo = accountInfoRepository.findOne(accountId.id)
}
case class AccountInfo(id : String, balance : Long, transactions : List[AccountTransactionInfo], changes : List[ChangeInfo], version : String)
case class AccountTransactionInfo(changeId : String, transactionId : String,
transactionType : String, amount : Long, balanceDelta : Long)
trait AccountInfoRepository extends MongoRepository[AccountInfo, String]
Implementation generated by Spring Data
@crichardson
Agenda
Why build event-driven microservices?
Overview of event sourcing
Designing microservices with event sourcing
Implementing queries in an event sourced application
Building and deploying microservices
@crichardson
Building microservices with Spring Boot
Makes it easy to create stand-alone, production ready Spring Applications
Automatically configures Spring using Convention over Configuration
Externalizes configuration
Generates standalone executable JARs with embedded web server
@crichardson
Spring Boot simplifies configuration
Spring Container
Application components
Fully configured application
ConfigurationMetadata
•Typesafe JavaConfig•Annotations•Legacy XML
Default Configuration
Metadata
Spring BootYou write less
of this
Inferred from CLASSPATH
@crichardson
Tiny Spring configuration for Account microservice
@Configuration@EnableAutoConfiguration@Import(classOf[JdbcEventStoreConfiguration]))@ComponentScanclass AccountConfiguration {
@Bean def accountService(eventStore : EventStore) = new AccountService(eventStore)
@Bean def accountEventHandlers(eventStore : EventStore) = EventHandlerRegistrar.makeFromCompoundEventHandler( eventStore, "accountEventHandlers", new TransferWorkflowAccountHandlers(eventStore)) @Bean @Primary def scalaObjectMapper() = ScalaObjectMapper
}
Service
Event handlers
Scan for controllers
Customize JSON serialization
@crichardson
The Main program
object BankingMain extends App {
SpringApplication.run(classOf[AccountConfiguration], args :_ *)
}
@crichardson
Building with Gradle
buildscript { repositories { mavenCentral() } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.0.0.RELEASE") }}
apply plugin: 'scala'apply plugin: 'spring-boot'
...
@crichardson
Running the microservice
$ java -jar build/libs/banking-main-1.0-SNAPSHOT.jar --server.port=8081...11:38:04.633 INFO n.c.e.e.bank.web.main.BankingMain$ - Started BankingMain. in 8.811 seconds (JVM running for 9.884)
$ curl localhost:8081/health{"status":"UP", "mongo":{"status":"UP","version":"2.4.10"}, "db":{"status":"UP","database":"H2","hello":1}}
Built in health checks
Command line arg processing
@crichardson
Jenkins-based deployment pipeline
Build & Testmicro-service
Build & TestDockerimage
Deploy Docker image
to Repository
One pipeline per micro-service
@crichardson
Summary
Event Sourcing solves key data consistency issues with:
Microservices
Partitioned/NoSQL databases
Spring and Scala play nicely together
Spring Boot makes it very easily to build production ready microservices
