building event-driven services with apache kafka

1

Building Event-Driven Services with Apache Kafka

Ben Stopford @benstopford

2

In this talk

1.  How we traditionally build services with REST

2.  How Event Driven Services are different 3.  Building an Immutable, Shared Narrative. 4.  Leveraging Materialized Views. 5.  Relating to CQRS & Event Sourcing 6.  Pulling it all together

3

Companies evolve in different ways

•  Internet companies •  Enterprise companies

4

Internet companies typically grow from front-facing websites.

pic

Growing fast

5

Carve pieces off

6

If stateful, they might share a DB

7

Or they might use their own

8

Over time => more complex

9

Over time => more async (async w.r.t. user)

10

Chained blocking commands

Buffering, handling failure, backpressure, scaling etc all get

pushed into service’s responsibilities

11

The synchronous world of request response protocols leads to tight, point-to-point

couplings.

12

Enterprises are typically different

13

pic

Enterprise companies form as heterogeneous application

conglomerates, often in sylos

14

The start life as independent islands

15

Linked together by files or via messaging

pic

16

Interactions are async from day-1

pic

17

Forwarding of data gets messy: Chinese Whispers / Telephone game

pic

18

Both these approaches often end up at the same point, but for different

reasons

19

Data is herded into specific services

Enterprise

20

Data is herded into specific services

Data Services

Stateless Services

Internet Co

21

Lots of Data Services => Distributed Join Problem

pic

This is a problem databases find hard, and they’re highly tuned for it!

Payments Service

Orders Service

Customers Service

Client Service performs join

22

Lots of Data Services => Distributed Join Problem

pic

TESTING BECOMES DIFFICULT!

Client Service performs join

Payments Service

Orders Service

Customers Service

23

In short: the high degree of “connectedness” makes it hard to evolve independently and to scale

24

LETS TRY

EVENT DRIVEN!

25

Interact through

Events,

Don't talk to

Services

26

This

Payments Service

Purchase Requested Events

Listen & React to Events

Payment Completed Events

27

Not this

Service Payment Service

http://payments/ID/42

Don’t Request/Command from individual services.

28

Let’s take an example...

29

UI Service

Payment Service

Stock Service

(1) Process Payment (2) Contact

Payment Provider

(3) Update stock

(4) Maybe order more

stock

Request Response Example

30

The Event Driven Way

UI Service

Payment Service

Stock Service

Payment Provider

Maybe order more stock

31

UI Service

Payment Service

Stock Service

Payment Provider

Kafka Topics: (1)  Purchases (2)  Payments

Maybe order more stock

The Event Driven Way

32

UI Service

Payment Service

Stock Service

Payment Provider

Maybe order more stock

New Order -> PurchaseRequested Event

Raise PurchaseRequested

Event

33

UI Service

Payment Service

Stock Service

Payment Provider

Maybe order more stock

Payment Service Listens In

Raise PurchaseRequested

Event

Listen to PurchaseRequested Events

34

UI Service

Payment Service

Stock Service

Payment Provider

Maybe order more stock

When Complete -> PaymentProcessed

Raise PurchaseRequested

Event

Listen to PurchaseRequested Events

Raise PaymentProcessed

Event

*Some might use the Command/Query Pattern

35

UI Service

Payment Service

Stock Service

Payment Provider

Maybe order more stock

Stock service hooks in too!

Raise PurchaseRequested

Event

Listen to PurchaseRequested Events

Raise PaymentProcessed

Event

Listen to PaymentProcessed

events

36

UI Service

Payment Service

Stock Service

Payment Provider

Maybe order more stock

Payment service listens

Raise PurchaseRequested

Event

Listen to PurchaseRequested Events

Raise PaymentProcessed

Event

Listen to PaymentProcessed

events

Listen to PaymentProcessed

Event

37

UI Service

Payment Service

Stock Service

Payment Provider

Stock service entirely owns its flow control

Maybe order more stock

Flow Control is Receiver Driven

38

UI Service

Payment Service

Stock Service

Quite different to Synchronous Model

Stock service flow control owned by payments service

39

1. Events are Immutable Facts (state changes taken from the real world and journalled)

UI Service

Purchases

Payments

Purchase Requested

Payment Completed

All the benefits of “Event Sourcing”

40

2. Services couple only to data flows

No knowledge of contributing services (RDFC)

UI Service

Purchases

Payments

Purchase Requested

Payment Completed

41

Because coupling is only to Events, not services

Purchase Requests

PurchaseRequest Topic UI Service

Payment Service

Stock Service

42

The architecture is “Pluggable”

Purchase Requests

PurchaseRequest Topic UI Service

Payment Service

Stock Service

Fraud Service

43

Communication & State Concepts Merge

•  The concept of Data & Events become one.

•  Kafka is both Event Store and Communication Channel

44

The Events form a Canonical, Shared Narrative

Payments

Orders

Shipments Customers

Evolving state of the system over time

45

Scaling is a concern of the broker, not “upstream” services

Purchase Requests

PurchaseRequest Topic UI Service

Payment Service

Stock Service

Fraud Service

Kafka •  Linearly scalable •  Fault Tolerant •  Multi Tennant

46

But there is something missing!

How do we look things up? (i.e. the Q in CQRS) •  What is the address for this customer? •  Is this user allowed to view this stock? •  What is the contents of this user’s shopping basket?

47

With REST Lookups are natural, if Remote

UI Service

Customer Service

http://customers/ID/42

Go to the relevant service and ask!

48

Which creates the dependency problem

Many services tightly coupled to one another

49

With Event Driven we create “views” or “projections” inside each bounded context

Projection of data from Outside

Context Boundary

Shipping Service

50

Pattern 1: Local KTables

Shipping Service

Customers Events

Customers By ID (saved by KStreams

using RocksDB)

Kafka

51

Pattern 1: Local KTables

Shipping Service

Customers Events

Customers By ID KTable customers = builder.table(

CustomerId, Customer, “customers-topic”, “customer-store”);

customers = streams.store("customers-store”…);

customer = customers.get(42)

52

As we’re using a streaming engine, we can translate into any Domain Model

(projection) we wish

Shipping Service

Customers Events

ShippingCustomers By CustomerName

53

Combine different streams from different services

Fraud Service

Payment Events

Purchase Events

PurchasePayments

payments.join(“purchases”,..).groupByKey().reduce(newValueReducer, “PurcahsePaymentsStore”)

54

Pattern 1: Local KTables

•  Simple to create and query •  Local to each service so fast •  Powerful DSL for transformation •  Inbuilt high availability •  No external database •  Controlled entirely within service’s bounded

context •  KTable and GlobalKTable allow scale out

55

Pattern 2(a): Queryable Interface

Fraud Service

Kafka

Fraud Service (Creates streaming projection)

Query via Rest Interface

User Interface

Known as queryable state in Kafka

Context Boundary

56

Pattern 2(b): Query Service

Kafka

Bespoke View Service (Creates streaming

projection)

Rest Interface

Fraud Service

Context Boundary

57

Pattern 2: Exposing Materialised Projections

•  Similar to Pattern 1 but with a “Query Interface” (using Kafka’s Queryable State feature)

•  Commonly used with UI’s

58

Pattern 3: External DB

DB of choice

Kafka Connect

API Fraud Service

Context Boundary

59

Pattern 3: External DB

•  Again similar, but heavier weight •  Use when you cannot pre-compute

the view (i.e. to do ad hoc queries) •  Trick: start with only the data

(fields) you need today. You can always go back for more.

60

Pattern 4: Hybrid

Fraud Service

Kafka

Rest Interface

Context Boundary

Booking Service

NB this breaks the async model, but can be a useful compromise in some cases

61

Pattern 4: Hybrid

•  Quite common in practice •  Allows you to break out of the

async world •  Use sparingly, but do use when

appropriate (e.g. login service)

62

Benefits

•  Forms a central, immutable narrative •  Communication Protocol and Event Store

become one •  Better decoupling (RDFC) •  Excellent scalability •  Several approaches for managing queries,

lightweight => heavy weight •  Event sourcing & CQRS at its core

63

Interact through

Events,

Don't talk to

Services

64

But it's ok to break

the rules!

65

Pulling it all together

66

All order logic Orders-by-Customer view

(KStreams + Queryable State API)

UI Service Stream Maintenance

Fulfillment uses local state store, which is replicated to a

redundant instance

UI uses Orders-by-Customer View directly

via Queryable State

History Service pushes data to a local Elastic Search Instance

Orders Service

Derived View

UI joins data Tables & Streams

Fulfillment Service

History Service

Fully redundant instance

Orders Product Custo-mers KTables

KTables KTables

Schemas

67

Thank You! @benstopford

68

Discount code: kafcom17

Use the Apache Kafka community discount code to get $50 off www.kafka-summit.org Kafka Summit New York: May 8 Kafka Summit San Francisco: August 28

Presented by