camel desing patterns learned through blood, sweat, and tears
TRANSCRIPT
![Page 1: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/1.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears
June 2016
Bilgin Ibryam
@bibryam
![Page 2: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/2.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears2
Bilgin Ibryam
● Senior Middleware Architect at Red Hat UK
● Apache Camel Committer and PMC member
● Apache OFBiz Committer and PMC member
● Author of Camel Design Patterns (new)
● Author of Apache Camel Message Routing
● Twitter: @bibryam
● Email: [email protected]
● Blog: http://ofbizian.com
● LinkedIn: http://www.linkedin.com/in/bibryam
![Page 3: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/3.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears3
Apache Camel Project StatusIt has all necessary ingredients for a successful open source project.
● Community: 52 committers, 903 users
● Support by large vendors (Red Hat)
● Connectors (256), DataFormats (40)
● Enterprise Integration Patterns++
● Domain Specific Language
● Ecosystem: Karaf, ActiveMQ, CXF, Fabric,Hawtio, Spring and others
● Monolith, SOA, Microservices, Serverless
Source https://www.openhub.net/p/camel/
Stats Date: 06/06/2016
![Page 4: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/4.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears4
Application Integration with CamelWhat do you need to know to create great Camel applications?
![Page 5: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/5.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears5
Happy Path ScenariosHow Pipes and Filters Pattern looks like in Camel?
![Page 6: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/6.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears6
VETROWhat is a typical processing flow for a Camel route?
● Validate: validation, schematron, MSV, Jing, bean validation components
● Enrich: enrich and pollEnrich EIPs, custom beans
● Transform: Data formats, auto type conversion, templating components
● Route: Message routing EIPs
● Operate: this is the essence of the processing flow
![Page 7: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/7.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears7
Edge ComponentLet's start with a simple Camel route that consumes files
![Page 8: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/8.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears8
Edge ComponentHow to expose the same business functionality to multiple consumers?
![Page 9: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/9.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears9
Edge ComponentEncapsulate endpoint-specific details and prevent them from leaking intothe business logic of an integration flow.
![Page 10: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/10.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears10
Read vs Write OperationsHow to evolve Read and Write operations independently?
![Page 11: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/11.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears11
CQRSThis decouples read from write operations to allow them to evolveindependently.
![Page 12: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/12.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears12
Unhappy Path ScenariosHappy paths are the easy ones. More work is required for designing andimplementing the unhappy paths.
● Data Integrity Pattern ● Saga Pattern● Retry Pattern● Idempotent Filter Pattern ● Circuit Breaker Pattern● Error Channels Pattern● Throttling Pattern
![Page 13: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/13.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears13
Data IntegrityHow hard can it be to copy files from one location to another?
Download Data Integrity Chapter: http://bit.ly/came-design-patterns-sample
![Page 14: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/14.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears14
Data IntegrityTransactional systems
Local transaction manager
Global transaction manager
![Page 15: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/15.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears15
SagaHow to avoid distributed transactions and ensure data consistency?
![Page 16: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/16.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears16
SagaEnsures that each step of the business process has a compensating actionto undo the work completed in the case of partial failures.
![Page 17: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/17.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears17
RetryTo enable applications handle anticipated transient failures by transparentlyretrying a failed operation with expectation it to be successful.
● Which failures to retry? ● How often to retry? ● Idempotency ● Monitoring ● Timeouts and SLAs
![Page 18: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/18.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears18
RetryCamel RedeliveryPolicy
● The most well known retry mechanism in Camel● Retries only the failing endpoint ● Fully in-memory● Thread blocking behavior by default● Can be asynchronous● Good for small number of quick retries (in milliseconds)
![Page 19: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/19.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears19
RetryActiveMQ consumer RedeliveryPolicy
● Retries the message from the beginning of the Camel route● Not used very often, but enabled by default● Fully in-memory● Thread blocking by default● Good for small number of quick retries (in milliseconds)
![Page 20: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/20.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears20
RetryActiveMQ Broker Redelivery
● ActiveMQ specific and requires custom logic● It will consume the message again from a queue● Persisted at the broker rather than application memory● Can be clustered and use fail over, load balancing, etc● Good for long persisted retries (in minutes or hours)
![Page 21: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/21.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears21
Circuit BreakerHow to guard a system by cascading failures and slow responses fromother systems?
![Page 22: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/22.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears22
Circuit BreakerImproves the stability and the resilience of a system by guarding integrationpoints from cascading failures and slow responses.
Closed state
Open state
![Page 23: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/23.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears23
Circuit BreakerImproves the stability and the resilience of a system by guarding integrationpoints from cascading failures and slow responses.
![Page 24: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/24.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears24
Circuit BreakerTwo Circuit Breaker Implementations in Camel 2.18
![Page 25: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/25.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears25
BulkheadHow to enforce resource partitioning and damage containment in order topreserve partial functionality in the case of a failure?
![Page 26: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/26.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears26
BulkheadEnforces resource partitioning and damage containment in order topreserve partial functionality in the case of a failure.
● Multi-threaded EIPs: Delayer, Multicast, Recipient List, Splitter, Threads,Throttler, Wire Tap, Polling Consumer, ProducerTemplate, and OnCompletion.
● Async Error Handler ● Circuit Breaker EIP
Possible Camel bulkhead points:
![Page 27: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/27.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears27
Scalability ScenariosVertical scaling (performance tuning)
● Endpoints: messaging client buffers, DB client batching, template caching choices● Concurrent consumers option: Seda, VM, JMS, RabbitMQ, Disruptor, AWS-SQS● Data types choice: affects content based router, splitter, filter, aggregator● Multithreading: parallel processing EIPs, threads DSL construct, Seda component,
asynchronous redelivery/retry● Micro optimizations: log tuning, camel sampler EIP, disable JMX, disable message
history, disable original message record● Startup/Shutdown: Use lazyLoadTypeConverters for a faster application startup, or
configure the shutdownStrategy for a faster shutdown● Tune: JVM options, networking and operating system
Camel performance tuning blog post: http://bit.ly/camel-tuningCamel performance tuning blog post: http://bit.ly/camel-tuning
![Page 28: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/28.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears28
Horizontal ScalingService Instance Pattern for accommodating increasing workloads.
![Page 29: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/29.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears29
Service InstanceAreas to consider before horizontally scaling a Camel application.
● Service state: load balancer, circuit breaker, resequencer, sampler,throttler, idempotent consumer and aggregator are stateful EIPs!
● Request dispatcher: Messaging, HTTP, file (what about locking?)
● Message ordering: exclusive consumer, message groups, consumerpriority, message priority, virtual topics
● Singleton service requirements: for batch jobs, and concurrent polling
● Other resource contention and coupling considerations
![Page 30: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/30.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears30
What did we cover so far?
![Page 31: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/31.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears31
How patterns are changing?What is happening in the IT industry today?
● Canonical Data Model● Edge Component● Reusable Route● Runtime Reconfiguration● Singleton Service● Batch jobs in JVM
● Bounded Context● Standalone services● Favor code duplication● Less configuration, more redeployment● Container managed singleton● Container scheduling ● Circuit Breaker, Bulkhead, Health checks..
![Page 32: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/32.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears32
Win a print copy of Camel Design Patterns
When was the first commit to Apache Camel project done?
![Page 33: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/33.jpg)
Apache Camel Design Patterns Learned Through Blood, Sweat, and Tears33
Win a print copy of Camel Design Patterns
When was the first commit to Apache Camel project done?
![Page 34: Camel Desing Patterns Learned Through Blood, Sweat, and Tears](https://reader031.vdocuments.us/reader031/viewer/2022022203/58779d7a1a28ab826e8b4687/html5/thumbnails/34.jpg)
More Information
Learn more about Apache Camel: http://camel.apache.org
Check out Camel Design Patterns: https://leanpub.com/camel-design-patterns
Develop Apache Camel based integrations using Red Hat JBoss Fuse: red.ht/FuseDev