building event-driven services with apache kafka
TRANSCRIPT
1
Building Event-Driven Services with Apache Kafka
Ben Stopford @benstopford
2
In this talk
1. How we traditionally build services with REST
2. How Event Driven Services are different 3. Building an Immutable, Shared Narrative. 4. Leveraging Materialized Views. 5. Relating to CQRS & Event Sourcing 6. Pulling it all together
3
Companies evolve in different ways
• Internet companies • Enterprise companies
4
Internet companies typically grow from front-facing websites.
pic
Growing fast
5
Carve pieces off
6
If stateful, they might share a DB
7
Or they might use their own
8
Over time => more complex
9
Over time => more async (async w.r.t. user)
10
Chained blocking commands
Buffering, handling failure, backpressure, scaling etc all get
pushed into service’s responsibilities
11
The synchronous world of request response protocols leads to tight, point-to-point
couplings.
12
Enterprises are typically different
13
pic
Enterprise companies form as heterogeneous application
conglomerates, often in sylos
14
The start life as independent islands
15
Linked together by files or via messaging
pic
16
Interactions are async from day-1
pic
17
Forwarding of data gets messy: Chinese Whispers / Telephone game
pic
18
Both these approaches often end up at the same point, but for different
reasons
19
Data is herded into specific services
Enterprise
20
Data is herded into specific services
Data Services
Stateless Services
Internet Co
21
Lots of Data Services => Distributed Join Problem
pic
This is a problem databases find hard, and they’re highly tuned for it!
Payments Service
Orders Service
Customers Service
Client Service performs join
22
Lots of Data Services => Distributed Join Problem
pic
TESTING BECOMES DIFFICULT!
Client Service performs join
Payments Service
Orders Service
Customers Service
23
In short: the high degree of “connectedness” makes it hard to evolve independently and to scale
24
LETS TRY
EVENT DRIVEN!
25
Interact through
Events,
Don't talk to
Services
26
This
Payments Service
Purchase Requested Events
Listen & React to Events
Payment Completed Events
27
Not this
Service Payment Service
http://payments/ID/42
Don’t Request/Command from individual services.
28
Let’s take an example...
29
UI Service
Payment Service
Stock Service
(1) Process Payment (2) Contact
Payment Provider
(3) Update stock
(4) Maybe order more
stock
Request Response Example
30
The Event Driven Way
UI Service
Payment Service
Stock Service
Payment Provider
Maybe order more stock
31
UI Service
Payment Service
Stock Service
Payment Provider
Kafka Topics: (1) Purchases (2) Payments
Maybe order more stock
The Event Driven Way
32
UI Service
Payment Service
Stock Service
Payment Provider
Maybe order more stock
New Order -> PurchaseRequested Event
Raise PurchaseRequested
Event
33
UI Service
Payment Service
Stock Service
Payment Provider
Maybe order more stock
Payment Service Listens In
Raise PurchaseRequested
Event
Listen to PurchaseRequested Events
34
UI Service
Payment Service
Stock Service
Payment Provider
Maybe order more stock
When Complete -> PaymentProcessed
Raise PurchaseRequested
Event
Listen to PurchaseRequested Events
Raise PaymentProcessed
Event
*Some might use the Command/Query Pattern
35
UI Service
Payment Service
Stock Service
Payment Provider
Maybe order more stock
Stock service hooks in too!
Raise PurchaseRequested
Event
Listen to PurchaseRequested Events
Raise PaymentProcessed
Event
Listen to PaymentProcessed
events
36
UI Service
Payment Service
Stock Service
Payment Provider
Maybe order more stock
Payment service listens
Raise PurchaseRequested
Event
Listen to PurchaseRequested Events
Raise PaymentProcessed
Event
Listen to PaymentProcessed
events
Listen to PaymentProcessed
Event
37
UI Service
Payment Service
Stock Service
Payment Provider
Stock service entirely owns its flow control
Maybe order more stock
Flow Control is Receiver Driven
38
UI Service
Payment Service
Stock Service
Quite different to Synchronous Model
Stock service flow control owned by payments service
39
1. Events are Immutable Facts (state changes taken from the real world and journalled)
UI Service
Purchases
Payments
Purchase Requested
Payment Completed
All the benefits of “Event Sourcing”
40
2. Services couple only to data flows
No knowledge of contributing services (RDFC)
UI Service
Purchases
Payments
Purchase Requested
Payment Completed
41
Because coupling is only to Events, not services
Purchase Requests
PurchaseRequest Topic UI Service
Payment Service
Stock Service
42
The architecture is “Pluggable”
Purchase Requests
PurchaseRequest Topic UI Service
Payment Service
Stock Service
Fraud Service
43
Communication & State Concepts Merge
• The concept of Data & Events become one.
• Kafka is both Event Store and Communication Channel
44
The Events form a Canonical, Shared Narrative
Payments
Orders
Shipments Customers
Evolving state of the system over time
45
Scaling is a concern of the broker, not “upstream” services
Purchase Requests
PurchaseRequest Topic UI Service
Payment Service
Stock Service
Fraud Service
Kafka • Linearly scalable • Fault Tolerant • Multi Tennant
46
But there is something missing!
How do we look things up? (i.e. the Q in CQRS) • What is the address for this customer? • Is this user allowed to view this stock? • What is the contents of this user’s shopping basket?
47
With REST Lookups are natural, if Remote
UI Service
Customer Service
http://customers/ID/42
Go to the relevant service and ask!
48
Which creates the dependency problem
Many services tightly coupled to one another
49
With Event Driven we create “views” or “projections” inside each bounded context
Projection of data from Outside
Context Boundary
Shipping Service
50
Pattern 1: Local KTables
Shipping Service
Customers Events
Customers By ID (saved by KStreams
using RocksDB)
Kafka
51
Pattern 1: Local KTables
Shipping Service
Customers Events
Customers By ID KTable customers = builder.table(
CustomerId, Customer, “customers-topic”, “customer-store”);
customers = streams.store("customers-store”…);
customer = customers.get(42)
52
As we’re using a streaming engine, we can translate into any Domain Model
(projection) we wish
Shipping Service
Customers Events
ShippingCustomers By CustomerName
53
Combine different streams from different services
Fraud Service
Payment Events
Purchase Events
PurchasePayments
payments.join(“purchases”,..).groupByKey().reduce(newValueReducer, “PurcahsePaymentsStore”)
54
Pattern 1: Local KTables
• Simple to create and query • Local to each service so fast • Powerful DSL for transformation • Inbuilt high availability • No external database • Controlled entirely within service’s bounded
context • KTable and GlobalKTable allow scale out
55
Pattern 2(a): Queryable Interface
Fraud Service
Kafka
Fraud Service (Creates streaming projection)
Query via Rest Interface
User Interface
Known as queryable state in Kafka
Context Boundary
56
Pattern 2(b): Query Service
Kafka
Bespoke View Service (Creates streaming
projection)
Rest Interface
Fraud Service
Context Boundary
57
Pattern 2: Exposing Materialised Projections
• Similar to Pattern 1 but with a “Query Interface” (using Kafka’s Queryable State feature)
• Commonly used with UI’s
58
Pattern 3: External DB
DB of choice
Kafka Connect
API Fraud Service
Context Boundary
59
Pattern 3: External DB
• Again similar, but heavier weight • Use when you cannot pre-compute
the view (i.e. to do ad hoc queries) • Trick: start with only the data
(fields) you need today. You can always go back for more.
60
Pattern 4: Hybrid
Fraud Service
Kafka
Rest Interface
Context Boundary
Booking Service
NB this breaks the async model, but can be a useful compromise in some cases
61
Pattern 4: Hybrid
• Quite common in practice • Allows you to break out of the
async world • Use sparingly, but do use when
appropriate (e.g. login service)
62
Benefits
• Forms a central, immutable narrative • Communication Protocol and Event Store
become one • Better decoupling (RDFC) • Excellent scalability • Several approaches for managing queries,
lightweight => heavy weight • Event sourcing & CQRS at its core
63
Interact through
Events,
Don't talk to
Services
64
But it's ok to break
the rules!
65
Pulling it all together
66
All order logic Orders-by-Customer view
(KStreams + Queryable State API)
UI Service Stream Maintenance
Fulfillment uses local state store, which is replicated to a
redundant instance
UI uses Orders-by-Customer View directly
via Queryable State
History Service pushes data to a local Elastic Search Instance
Orders Service
Derived View
UI joins data Tables & Streams
Fulfillment Service
History Service
Fully redundant instance
Orders Product Custo-mers KTables
KTables KTables
Schemas
67
Thank You! @benstopford
68
Discount code: kafcom17
Use the Apache Kafka community discount code to get $50 off www.kafka-summit.org Kafka Summit New York: May 8 Kafka Summit San Francisco: August 28
Presented by