event driven architecture
DESCRIPTION
A whirlwind tour of Event Driven Architecture, extensibility, Domain Driven Design, Command and Query Responsibility Segregation (CQRS) and Complex Event ProcessingTRANSCRIPT
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Driven Architecture
Lourens Naudé – SAPO Codebits 2010
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
About
• Developer @ http://www.wildfireapp.com• Ruby / C• Well versed full stack• Active interest in Event Driven technologies
and related business processes
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Agenda
• Business and systems growth• Events• (Lack of) control• Event Driven Architecture
• Domain Driven Design primer• Command and Query Responsibility
Segregation• Complex Event Processing
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Maintenance cost is multiple times more than
development cost
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
http://en.wikipedia.org/wiki/Open/closed_principle
“Systems should ideally be open for extension, but
closed for modification”
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Driving forces for change
• Fulfillment of business requirements• Business expansion: rollout of new
processes to drive sales, market penetration etc• Mergers and acquisitions• Legislation
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Modification risks
• Introduces regression testing and a QA burden – never compromise a working system for a functional requirement
• Profiling overhead for (soft) realtime systems
• Excessive “refactoring” of existing stable interfaces
• May fracture a business domain
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
EDA imposes a system design that's geared for
extension, interoperability and unanticipated use.
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Interoperability VS Integration
• Two or more systems performing a task / process together
• Integration implies a formal link / bridge between two systems - high coupling
• Building with interoperability in mind is the easiest way for long-term integration
• Should be accounted for in system lifecycle as well: delays, cost, communication etc.
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Governance (lay down the law)• Technology standards, SLAs, development
processes• Cannot achieve interoperation without
governance• Poor governance: inappropriate policies,
people (habits) and business structures• Policies should be enforced at runtime
in an auditable manner• Otherwise like criminal law without
police and courts
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Business layers
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Business Layers
• Physical world: sensors and users as actors
• Transactional: physical world interactions with the business / domain layer
• Intelligence: reporting tier and insights, most notably sales and marketing performance
• This is where EDA really shines: visibility into business processes
• Transparent operation for easier compliance and lower support costs
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Events
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Definition
• Anything that happened (or didn't happen)
• A change in state• A condition that triggers a notification• An event is always named in the past tense
and is immutable
CustomerAddressChangedInventoryRelocatedOrderShipped
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Three levels of detail
• Basic fact that an event occurred• Event definition required in order to
recognize an event• Event detail / context• Always attempt to capture events at the point
of greatest information value (context)• This reduces downstream lookup and
correlation overhead
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Types
• Notification: informational• Alert: notification that should cause a
response• Event Object: record of an event - just data,
no behavior• Business Event: meaningful for conducting
trade activities• Complex: abstraction of one or more other
events that represents summary level insights
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Notification Types
• Observational• Reports an event
butdoes not changeanything
LowStockCartCleared
• Transactional• Reports an event
and causes change
• Reliable transport• Originates from
within the org.
OrderShippedCardProcessed
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
(Lack of) Control
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Driven Programming
• Control flow is driven by external events
• NO central controller (main) - contrary to what we're taught starting off as devs
• No well defined flow of data
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
EDA Components
• Event producers• Event consumers / listeners• Event processors• Event reactions• Messaging infrastructure
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Lack of control in EDA
• An EDA is driven by extensibility and not controllability
• Transaction state is managed by events, not some central controller
• Loosely coupled, asynchronous and stateless• Tell downstream components what
happened, but not what to do.• Business events enable a system to operate
under control, but without central control
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Reduced coupling
• There's always some coupling• We strive to reduce the degree of
coupling• Always coupled to data
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Stateless Order process
• Modifying processes managed by a central controller requires changes to a component as well as the controller
• Move state / data from the central controller to events
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Driven Architecture
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Use cases
• Government: security, border control, taxes etc.
• Health: stream processing + research• Compliance: guard against violations in
cross country laws• Track and trace: Fedex, UPS etc.• Service Level Agreements
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
When not to use an EDA
• If the potential for reuse is very little • Problem being solved is lower down the
application stack• Network bandwith and capacity is
limited – serialization overheads and tends to be chatty
• Security boundaries: immutability of events that carry sensitive data
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
EDA Workloads
• Simple event processing: narrow design• Stream processing: high message
volume, but only relevant events is filtered• Complex Event Processing (CEP): able
to correlate multiple streams
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Interaction Types
• Time-driven: actions initiated at specified times, eg. batch workflows etc.
• Request-driven: initiated by client and completes when the provider replies, eg. client server
• Event-driven: initiated by events and participants is open ended
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Hybrid sytems
Most event driven systems also include time and request
driven components.
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Components
• Event producers• Event consumers / listeners• Event processors• Event reactions• Messaging infrastructure
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Nervous System Analogy
• Hand is the event producer• Spinal cord is the ESB• Brain is the event listener• Brain is the event processor• Hand is the event reaction – we pull away
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Producers
• Generates event data OR transforms it into a format that the EDA can understand
• Data isn't always generated to feed an EDA, especially if the producer is a legacy component
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Consumers / Listeners
• Knows how to differentiate an event from data it receives
• Can only “hear” events it's supposed to
• Should be able to detect and interpret an event
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Processors
• Able to asses event importance and derives the next action to take
• No value in not handling a perceived event• "Do nothing" is a valid reaction as
well
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Reactions
• Automated responses• Notifications to participants or people• Human reactions / interventions for
offline processes
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Messaging Infrastructure
• Communication link between EDA participants
• Doesn't have to be a single piece of infrastructure - distinct backbones should be able to share information
• Embrace standards and be as universal as possible
• Promote decoupling between participants - pub/sub
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Data in Event Driven Systems
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Data
• Business event data held in an event object
• Context of what happened
ProductCategorized('dsaed32wed', 'Coffee')OrderCancelled('ko90323', 'No funds')
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
State Data
• Core system data that changes as a result of operational activity
Order('ko90323', 'Tom Jones', 13.45 'open')Product('dsaed32wed', 'Delta', 'Coffee', 5.8)
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Reference Data
• Stable / somewhat permanent - could change, but a lot less frequently than state data
• Reference data would be replaced whereas state data would change incrementally.
SupportedCurrency('USD')ProductCategory('Coffee')Address('Some Street', 'CA', 'USA')
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Lifecycle of Data
• New state data = state data + event data applied
• State data is the accumulation of event data over time
• Event data as deltas
Product('dsaed32wed', 'Delta', 'Coffee', 5.8)ProductCategorized('dsaed32wed', 'Tea')Product('dsaed32wed', 'Delta', 'Tea', 5.8)
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Domain Driven Design primer
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Domain Driven Design
• Suitable for behavior driven systems• Modeling is often data and not
behavior driven• A Domain Model is a representation of
relationships between Entities and Values• Try to have a business analyst review
processes before introducing new architecture
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Common Language
• Participants: Developers, Experts and Stakeholders
• Nouns: things with / without identity• Verbs: actions, hints at state changes• Domain models evolve through language• We often write different to how we speak and
think• Most failures of systems is communication
failures of people
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Entities
• Has an identity within the system• State can change
ProductCustomerOrder
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Value Objects (Values)
• No identity• Immutable• Represented by it's structure - structural
equivalence
MoneyOrderStatusTimezone
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Domain Services
• Actions, operations and activities• Business driven intent• Stateless• Should handle behavior that doesn't map
well to any particular entity
ShippingQuotesServicePaymentProcessingService
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Aggregates
• A group of Entities that belong together
• An instance of a process• A single unit of work – consistency /
transactional boundary• eg. Order and Order items• Order item is useless without an Order• Remove Order item: reconstitute the Order,
find item via Order, remove item• Usually identified by cascading delete
requirements
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Rental administration example
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Command and Query Responsibility Segregation
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Either perform an action, or return a result
Not both.
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Overview
• CAP theorem: only can have 2 of 3• Consistency steps aside for Availability and
Partitionability• Somewhat stale data is often acceptable• Business requirements define by how much• Split systems into distinct command and query
components (read and write)
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Conceptual view
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Benefits
• Split teams: high level + good communicators on the write side, junior devs on the read side
• Ability to scale command and reporting sides independently
• Auditable system (dependent on history)• Selectively disable functionality during
upgrades on the write side whilst the reporting side is still available
• Lends itself well to behavior driven extension• Easy to test
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
CQRS Components
• Commands: drives change in the system• Internal events: captures intent (event
store)• External events: republished to let other
components know• Queries: examines state changes
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Eventual Consistency
• All data is stale, unless the whole system uses pessimistic locking
• Most web apps request data *before* they render screens
• Add 250ms to 500ms on average for rendering and additional server side blocking operations
• Users don't react immediately to UI changes
• Strictly consistent on the write side, relaxed (eventual) consistency on the read side
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Rethinking UI / adaptors
• Users don't think in data, but in tasks and processes
• Cannot get benefits of CQRS without considering the UI first
• Focus on capturing user intent explicitly
• User experience should be an integral part of any business process
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Ambigious UI
UpdateComment / CommentUpdated
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Task Driven UI
ApproveComment / CommentApproved
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Commands
• A well defined task with parameters• Represents a process, not an entity• Distinct and transactional unit of work:
wholly accepted or rejected• The only way to change data in a system• Just the right amount of granularity and intent
to not cause concurrency conflicts of updates on large entities
• Rejectable with exception or fault event• Idempotent – should never be reprocessed
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Rethinking Validations
• There's a fundamental difference between a validation and a business rule
• Validation: well structured and formatted etc.• Business rule: uniqueness constraints etc.• Validate in client and in command handler -
not all clients are well behaved• Valid commands can fail, most often
through optimistic concurrency conflicts• Common for offline processes to reject requests
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Queries and Reports
• Read only view of data - examines state in the system
• Never goes through the domain model• Table per UI view (assuming relational
storage)• Single call to the reporting store returns
all (or most) data for a screen• Minimal / no transformation - reporting
tier is a cache
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Aggregates and Events
---------- begin consistency boundary ----------
=> Commands::CategorizeProduct("blends", "1517fd7efbffacdf928056656bbb61cf")
rebuild Aggregates::Product(1517fd7efbffacdf928056656bbb61cf) from Events::ProductCreated({:category=>"coffee", :product_id=>"1517fd7efbffacdf928056656bbb61cf", :sku=>"XYZ", :name=>"Delta"})
Aggregates::Product @ version 1
<= Events::ProductCategorized({:category=>:blends, :product_id=>"1517fd7efbffacdf928056656bbb61cf"})
Aggregates::Product @ new version 2
---------- end consistency boundary ----------
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Code example
class Aggregates::Product......def rename(name) apply Event(:ProductRenamed, uuid, name)endprivatedef apply_product_renamed(event) @name = event.nameend ......end
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Internal Events
• Domain / Event store is append only: no deletes, ad hoc reads.
• Compensating events to delete• Event store is a history of intent• Replay: events allow for reproducing error
conditions by rebuilding an aggregate via snapshot up to just before a problem occurred
• Republishes any handled events to other components
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
External Events
• Events handled by our domain that's been republished to the rest of our system
• Communicates any state changes to the reporting tier
• Integration / interoperation hook for business growth
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
ORM Issues
• Suitable for structural models only• Structural models tend to loose
information• Mapping objects to structed data is difficult
to optimize• CRUD – what changes on UPDATE ?
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Testing
• Command -> Domain -> [Event, Event …]• Able to assert behaviors in two ways• What happened ?• What didn't happen ?• Same pattern for testing the reporting side• Event -> View -> assert state from getters
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Assertion example
def test_relocate_inventory Domain << Command(:CreateInventory, uuid, 123, 'Delta', 'A', 'X',10) Domain << Command(:RelocateInventory, uuid, 'B', 'Y') assert_published :InventoryCreated, :InventoryRelocatedend
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Complex Event Processing
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Overview
• Connecting the dots• Summary level information gathered
from base events• Patterns identified by business experts, but
enforced at runtime by CEP software• Value is proportional to accuracy and
relevance• Accurate data can however still be
irrelevent• Allows for Management by Exception
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Consciousness / awareness
• Without consciousness can't react to situations without human intervention
• SOA's the nervous system• EDA brings awareness• The core of Military systems – situational
awareness on the battlefield derived from input streams
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Stream
• Linear (ordered by time) sequence of events• Produced by a single component / system• Little aggregation required• High throughput• Minimal insights into what happened• Example: market data feed
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Cloud
• Collection of Event Streams within a system• Aggregates streams into event sets• Much lower throughput• Very good situational awareness• Example: trading strategies that's
dependent on multiple market feeds, realtime news sources etc.
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Event Cloud layout
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
CEP in Retrospective
• Adds value to Event Driven systems• “Complex” is misleading – it simplifies input
and streams• Matches real-time events from an Event
Cloud with patterns from historical data and detect or predict situations
• Event -> correlate -> assess -> decide -> action
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
Summary
• Restist modification• Tasks and intent – behavior matters• Communicate• Reduce coupling and avoid central
control
Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929
@methodmissinghttp://www.github.com/methodmissing
CQRS Example
https://github.com/methodmissing/aftermath
Thanks !
Questions ?