BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Building event-driven Microservices with Kafka EcosystemGuido SchmutzLondon, 30.5.2018
@gschmutz guidoschmutz.wordpress.com
Guido Schmutz
Working at Trivadis for more than 21 years
Oracle ACE Director for Fusion Middleware and SOA
Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data
Head of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 30 years of software development experience
Contact: [email protected]
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Building event-driven Microservices with Kafka Ecosystem
Agenda
1. Where do we come from?
2. What are Microservices?
3. Why not Event Driven?
4. What about streaming sources?
5. What about integrating legacy applications?
6. What about (historical) data analytics?
7. Why Kafka for Event-Driven Microservices?
8. Summary
Building event-driven Microservices with Kafka Ecosystem
Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UIDataGUI
Data Storage
Shared Database
Building event-driven Microservices with Kafka Ecosystem
sync request/response
Shop UI App
Business Activity Service
SOA Approach
Building event-driven Microservices with Kafka Ecosystem
Contract-first Web Services
Technical layers offer their own interfaces
Reuse on each level
Lower layer often wraps legacy code
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI LogicGUI
Business Entity ServiceShop Web App
Shop UI UI LogicGUI
Data Storage
CustomerDatabase
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order andProduct DBSOAP
SOAP
SOAP SOAP
SOAP
SOAP
SOAP
Shop UI App
Business Activity Service
Virtualized SOA Approach
Search BASCustomer DAO
Order DAO
Order BAS
Shop UI UI LogicGUI
Business Entity Service
Shop Web App
Shop UI UI LogicGUI
Data Storage
CustomerDatabase
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order andProduct DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
Building event-driven Microservices with Kafka Ecosystem
What are Microservices?
Building event-driven Microservices with Kafka Ecosystem
Tightly Scoped behind clear interfaces
Responsible for managing their own data (not necessarily the infrastructure)
Should be highly decoupled
Independently deployable, self-contained and autonomous
SOA done right ?!
Microservice Approach
Building event-driven Microservices with Kafka Ecosystem
Tightly Scoped
Responsible for managing their own data
Highly decoupled
Independently deployable, self-contained and autonomous
{ }Customer API
CustomerCustomer Logic
Order Microservice
{ }Order API OrderOrder Logic
Product Microservice
{ }Product API ProductProduct Logic
Stock Microservice
{ }Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
Microservice Approachwith API Gateway
Customer Microservice
{ }Customer API
CustomerCustomer Logic
Order Microservice
{ }Order API OrderOrder Logic
Product Microservice
{ }Product API ProductProduct Logic
Stock Microservice
{ }Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
API Gateway
Building event-driven Microservices with Kafka Ecosystem
Synchronous World of Request-Response leads to tight, point-to-point couplings
Building event-driven Microservices with Kafka Ecosystem
problem in lower end of chain have a ripple effect on other service• crash of service• overloaded service / slow response time• change of interface
Service 2Service 1
{ }
APILogic
{ }
API Logic
StateState
Service 3
{ }
API Logic
State
Service 4
{ }
API Logic
State
Service 5
{ }
API Logic
State
Service 7
{ }
API Logic
State
Service 6
{ }
API Logic
State
3 mechanisms through which services interact
Request-Driven (Imperative) Event Driven (Functional)
Service
Logic
State
Event Consume
OrderEvent{iPad}
Command”Order IPad”boolean order(IPad)
EventPublish
OrderValidatedEvent{iPad}
Query”Retrieve my OrdersList<Orders> getAllOrders(for)
Building event-driven Microservices with Kafka Ecosystem
Event-Driven (Async)Microservice Approach
Customer Microservice
{ }Customer API
CustomerCustomer Logic
Order Microservice
{ }Order API OrderOrder Logic
Product Microservice
{ }Product API ProductProduct Logic
Stock Microservice
{ }Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
API Gateway
Building event-driven Microservices with Kafka Ecosystem
Event Store
sync request/response
async request/response
async, event pub/sub
How to work with streaming data sources
Customer Microservice
{ }Customer API
CustomerCustomer Logic
Order Microservice
{ }Order API OrderOrder Logic
Product Microservice
{ }Product API ProductProduct Logic
Stock Microservice
{ }Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
Event Store
Location
Social
Clickstream
Sensor Data
MobileApps
WeatherData
Event Stream
Building event-driven Microservices with Kafka Ecosystem
Hadoop ClusterdHadoop Cluster
Stream Processing Cluster
Streaming Analytics ArchitectureBI Tools
SQL
Search / Explore
Online & Mobile Apps
Search
ServiceEvent Stream
ResultsStream Analytics
Reference /Models
Dashboard
Location
Social
Clickstream
Sensor Data
MobileApps
WeatherData
Microservice Cluster
Microservice State
{ }
API
EventStream
EventStream
Event Store
Service
Building event-driven Microservices with Kafka Ecosystem
What about integrating legacy applications?
Building event-driven Microservices with Kafka Ecosystem
Data Store
Integrate existing systems through CDC
Customer
Event Store
Integration
Microservice
StateLogic
CDCCDC Connector
Customer Fat Client App
Customer BOCustomer UI
Stream Processing
ResultsStream Processor
Reference /Models
Dashboard
Capture changes directly on database
Change Data Capture (CDC) => think like a global database trigger
Transform existing systems to event producer
Building event-driven Microservices with Kafka Ecosystem
Hadoop ClusterdHadoop Cluster
Stream Processing Cluster
Integrate existing systems through CDCBI Tools
SQL
Search / Explore
Online & Mobile Apps
Search
Service
Event Stream
ResultsStream Processor
Reference /Models
DashboardLocation
Social
Clickstream
Sensor Data
MobileApps
WeatherData Microservice Cluster
Microservice State
{ }
API
EventStream
EventStream
Event Store
Service
Billing &Ordering
CRM / Profile
MarketingCampaigns
Change DataCapture
Building event-driven Microservices with Kafka Ecosystem
Streaming & (Big) Data Analytics Architecture
Event Stream
Hadoop ClusterdHadoop ClusterBig Data Cluster
Data Fl
ow
Parallel Processing
Storage
Storage
Raw
Ref
ined
Results
Microservice Cluster
Microservice State
{ }
API
Stream Processing Cluster
StreamProcessor
State
{ }
API
EventStream
EventStream
SQL
Search
Service
BI Tools
Enterprise Data Warehouse
Search / Explore
Online & Mobile Apps
SQL Export
SearchEvent Store
Service
Location
Social
Clickstream
Sensor Data
MobileApps
WeatherData
Billing &Ordering
CRM / Profile
MarketingCampaigns
Change DataCapture
File Import / SQL Import
Building event-driven Microservices with Kafka Ecosystem
Apache Kafka – A Streaming Platform
Building event-driven Microservices with Kafka Ecosystem
High-Level Architecture
Distributed Log at the Core
Scale-Out Architecture
Logs do not (necessarily) forget
Apache Kafka – Schema Registry
Building event-driven Microservices with Kafka Ecosystem
Producer
Kafka Broker
Consumer
Schema Registry
Kafka Kafka
avroschema
avroavro
avroschema
Apache Kafka – scalable message processing and more!
Building event-driven Microservices with Kafka Ecosystem
Source Connector
trucking_driver
Kafka BrokerSink
Connector
Stream Processing
Schema Registry
Kafka Kafka
RESTful RESTful
Legacy Microservice
Change Data Capture (CDC)
Building event-driven Microservices with Kafka Ecosystem
RDBMS cdc-source trucking_driver
Customer Topicelasticsearc
h-sink NoSQL
Demo
Building event-driven Microservices with Kafka Ecosystem
Customer Microservice
{ }Customer API
CustomerCustomer Logic
Order Microservice
{ }Order API OrderOrder Logic
REST
REST
Kafka
Customer
Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL) log.retention.{ms | minutes | hours}
3. Size based log.retention.bytes
4. Log compaction based (entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181 \--create --topic customers \--replication-factor 1 \--partitions 1 \--config cleanup.policy=compact
Building event-driven Microservices with Kafka Ecosystem
Topic Viewed as Event Stream or State Stream (Change Log)
Event Stream State Stream (Change Log Stream)
2015-10-02 11,Peter,Muster,3010,Berne
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 11,Peter,Muster,3015,Berne2017-05-03 21,Lisa,Steffen,8001,Zurich2017-07-03 11,Peter,Muster,3052,Zollikofen
Building event-driven Microservices with Kafka Ecosystem
2015-10-02 11,Peter,Muster,3010,Berne
2016-10-04 12,Paul,Steffen,8001,Zurich
2016-12-02 21,Lisa,Meier,3043,Ittigen
2017-05-03 11,Peter,Muster,3015,Berne2017-05-03 21,Lisa,Steffen,8001,Zurich
2017-07-03 11,Peter,Muster,3052,Zollikofen
Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
OffsetKeyValue
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
OffsetKeyValue
Compaction
Building event-driven Microservices with Kafka Ecosystem
V1 V2
V3
V4 V5
V6
V7
V8
V9V10 V11
K1 K3 K4 K5K2 K6
Keep Topics both in Original and Compacted Form
Building event-driven Microservices with Kafka Ecosystem
Kafka BrokerA B C B A E K A E A C K
B E A C KProducer
Kafka BrokerA B C B A E K A E A C K
B E A C K
Producer
Consume and Produce
OR
TX
Enrich Stream with Static Data with Kafka Streams
Building event-driven Microservices with Kafka Ecosystem
Order Topic Consume and Produce
CustomerTable
Customer Topic Customer Handler
Join
Customer Local State
Order & Customer Topic
Hadoop ClusterdHadoop Cluster
Big Data
Summary
Billing &Ordering
CRM / Profile
MarketingCampaigns
SQL
Search
Service
BI Tools
Enterprise Data Warehouse
Search / Explore
Online & Mobile Apps
File Import / SQL Import
Event Hub
Data
Flow
Data
Flow
Change Data
Capture
Parallel Processing
Storage
Storage
Raw
Refined
Results
SQL Export
Microservice State
{ }
API
StreamProcessor
State
{ }
API
EventStream
EventStream
Search
Service
Location
Social
Clickstream
Sensor Data
MobileApps
WeatherData
Stream Processing
Microservices
Summary
Building event-driven Microservices with Kafka Ecosystem
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into• commands
• events
• queries
• Kafka is well suited as an event broker / event store• brings many more interesting features beyond just “message passing”
References
Building event-driven Microservices with Kafka Ecosystem
Microservices Blog Series, Ben Stopford, Confluent:
• https://www.confluent.io/blog/tag/microservices
Apache Kafka for Microservices: A Confluent Online Talk Series:
• https://www.confluent.io/landing-page/microservices-online-talk-series/
Turning the database inside-out with Apache Samza, Martin Kleppmann, Con
• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:
• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/
Immutability Changes Everything, Pat Helland, Salesforce:
• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:
• https://www.youtube.com/watch?v=B1-gS0oEtYc