wjax 2017: workflow and state machines at scale
TRANSCRIPT
Workflow and state machines at scale
WJAX, Munic, 08th of November 2017
[email protected] | @berndruecker
What is workflow?
The „hello world“ example
Human taskmanagement
Service orchestration
Events , e.g. timers
More realistic examples
Microservices &distributed systems
Message & Events
Timeout
Distributed transaction and compensation
Messages and events
And even „smaller“ use cases
Service(e.g. Go)
Kafka
RDMS
1. 3.
2.
And even „smaller“ use cases
Service(e.g. Go)
Kafka
RDMS
1. 3.
2.
And even „smaller“ use cases
Service(e.g. Go)
Kafka
RDMS
1. 3.
2.
Traditional workflow engines
Performance?
Status quo
Concrete numbersplease!*
*using my ThinkPad T460s consumer notebookin a totally not repeatable mannerwhile rendereing a talk recording ;-)
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Inst
an
ces
sta
rte
d /
se
c
Time
on average
181instances started/sec
Scaling
In general, we may conclude that Camunda performed better and more stable for all metrics when compared with WfMS A and WfMS B.Micro-Benchmarking BPMN 2.0 Workflow Management Systems with Workflow Patterns, June 2016, University of Stuttgart, Germany and University of Lugano, Switzerland
„
At Google, we have different ideas about scalability.Senior Architect at Google discussing BPMN.
„
„
Photo by Mimzy, available under Creative Commons CC0 1.0 license.
Telematics night-2420297_1920.jpg
Photo by jievoyage, available under Creative Commons CC0 1.0 license.
IOT
Photo by Mark Bradshaw, available under Creative Commons BY 2.0 license.
Logistics
Photo by Sam Churchill, available under Creative Commons BY 2.0 license.
Financial trading
Photo by 3844328, available under Creative Commons CC0 1.0 license.
Can you handle 200.000 trades/second?
„
New requirements -> new solutions
Remove RDMS
Distributed system
Design forperformance
ResilienceFlexible
persistence
Load
Zeebe in a nutshell
https://zeebe.io/
ZeebeBroker
Yourapplication
Binary (MsgPack) Support streaming & batching
Client
Horiziontallyscalable
Append only log / event sourcing
Peer-to-peercluster
Scale elastically
UP
DOWN
3 Nodes 30 Nodes
SCALE
Distributed system with topics & partitions
P1: FP2 : FP3 : L
P1: LP2 : FP3 : F
P1: FP2 : LP3 : F
Example:
- Topic with 3 partitionsand replication factor 3
- PX = Partition X- L = Leader- F = Follower
Replication uses the Raft protocol
Data flow (simplified)
Client
COMPLETEcommand
Broker - Partition Leader
Brokers -Partition Followers
1
2 append
replicate & commit
3
4 process
Stream Processor
State
Partition
5
decide & applyCOMPLETEDresponse
6
No external services needed
Publish / Subscribe
ZeebeBroker
Your codeZeebeClient
subscribe
Execute BPMNExecute Business Logic
Publish / Subscribe
ZeebeBroker
Service1
ZeebeClient
subscribe to A
Service2
ZeebeClient
subscribe to B
Live hacking
Performance?
on average
23.414events/sec
on average
4.682started instance/sec
The box said„RequiresWindows 7 or better“
so I installedLinux
on average
213.152events/sec
on average
35.525started instance/sec
Status & Roadmap
• Tech preview (0.4.0)
• Working towards production-ready state
• Focus on core, not (yet) on completeness of features
• We love to get your feedback and discuss!
Architecture choices (examples)
Zeebe
Service A
ZeebeClient
BPMN
Service B
ZeebeClient
*Isolation and autonomy can alsobe achieved in this architecture
Monolith
Zeebe
ZeebeClient
BPMN
Service A
ZeebeClient
BPMN
Zeebe
Service B
Kafka/AMQP/…
the monolith Microservices some hybrid*
CADENCE
https://github.com/samarabbas/cadence-samples/blob/master/cmd/samples/
CADENCE
X
No imageavailable
“[…]as the number of microservices grow and the complexity of the processes increases, getting visibility into these distributed workflows becomes difficult without a central orchestrator.”
https://netflix.github.io/conductor/metadata/kitchensink/
http://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html
Zeebe Uber Cadence Netflix Orcherstrator
AWS StepFunctions
Client language orAPI
Java, Go Go Java, REST AWS SDK, REST
Persistent storage Event log on file system
Cassandra Netflix Dynomite(k-v)
-
License and business model
OSS, backed byvendor (EE)
OSS project OSS project Cloud
Distribution Source, Binary Source Source -
Flow language BPMN, YAML, Java YAML, Go JSON JSON
Visibility BPMN None Proprietary Proprietary
BPMN
Executable*
Easy tounderstand**
Widespread
**(for Biz and Dev)* (and mature)
Should I throw away my traditional workflow engine?
A-F G-Z
Thank you!
Slides:https://bernd-ruecker.com
Blog:https://blog.bernd-ruecker.com
Feedback:https://bernd-ruecker.com/feedback
Images licensed from iStockno attribution required
Own photos
All icons licensed from Noun Projectno attribution required