building a highly scalable file processing platform with nservicebus nsbcon by sam martindale
DESCRIPTION
Building a Highly Scalable File Processing Platform with NServiceBus How do we move this data into our systems?TRANSCRIPT
![Page 1: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/1.jpg)
Building a Highly Scalable File Processing Platform
with NServiceBus
Sam Martindale
![Page 2: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/2.jpg)
AI
2
Introduction
14+ Years Development Experience
Co-Founder & Managing Partner, AI
NServiceBus adopter since v1.x
Sam Martindale Architecting Innovation
![Page 3: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/3.jpg)
AI
3
The Problem
![Page 4: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/4.jpg)
AI
4
The ProblemHow do we move this data into our systems?
Internal SystemsExternal Data Files
![Page 5: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/5.jpg)
AI
5
The ProblemAutomated pickup of many file types
Fixed Width
![Page 6: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/6.jpg)
AI
6
The ProblemRecord Level Logic and Workflows
New Account
Loan Application
Account Payment
Overdue Credit Account
Account Payment
![Page 7: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/7.jpg)
AI
7
The ProblemMany Integration Points
![Page 8: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/8.jpg)
AI
8
The ProblemGuaranteed Delivery
![Page 9: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/9.jpg)
AI
9
The ProblemScalability
![Page 10: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/10.jpg)
AI
10
Why NServiceBus?
![Page 11: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/11.jpg)
AI
11
Evaluated Technologies!Sql Server Integration Services (SSIS) !
Workflow Integration Difficulty
Systems Integration Difficulty
Extensibility
Not just ETL
!!!!
![Page 12: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/12.jpg)
AI
12
Evaluated TechnologiesBizTalk/Sterling Commerce/Axway/Others !
Licensing Costs
Scalability Costs
Difficult to “extend” functionality easily
Requires development “specialists”
!
!!!!!
![Page 13: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/13.jpg)
AI
13
NServiceBus !Easy & inexpensive to scale !Easy to integrate with existing systems !Provides Message Durability
!Great API for experienced .NET developers
Evaluated Technologies
![Page 14: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/14.jpg)
AI
14
System Design
![Page 15: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/15.jpg)
AI
15
The System
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
![Page 16: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/16.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
16
ComponentsScheduler
Manages scheduled “jobs” for picking up client files from FTP, SFTP, FileShare, etc
Why we needed it: !1. Needed the ability to schedule file pickups dynamically
Scheduler Endpoint
![Page 17: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/17.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
17
Why we needed it: !1. Needed the ability to pickup files 2. Needed the ability to decrypt files
FileRetrieval Performs the actual file pickup and decryption at the request of the scheduler
Components
FileRetrieval Endpoint
![Page 18: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/18.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
18
FileParsing Performs the parsing of the file according to the configured client file format and layout. Divides the file into logical “record sets” for processing
Why we needed it: !Had to be able to dynamically parse any given file
Components
Parsing Endpoint
![Page 19: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/19.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
19
Why we needed it: !Needed the ability to persist data Serves as an extension point for current/new workflows
Processor Processes the record sets sent by the parser component by performing calculations and persisting records to the database
Components
Processing Endpoint
![Page 20: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/20.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
20
Why we needed it: !Had to keep track of the overall status for file processing Needed the ability to monitor for failures or partial failures
FileManagement Tracks the completion of the file processing in a saga by listening to various parsing events
Components
File Management Endpoint
![Page 21: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/21.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
21
Why we needed it: !Needed the ability to notify users of various events
Alerts Receives events from other components, sending notifications and emails to the end users for system monitoring and alerting
Components
Alerts Endpoint
![Page 22: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/22.jpg)
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
22
WebApi WebApi component with SignalR. Delivers real time events and updates to the web front end
Why we needed it: !Needed to notify users of created alerts in real-time Needed to allow users to quickly onboard clients !
Components
Api Endpoint
![Page 23: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/23.jpg)
AI
23
The System
Alerts Endpoint
Parsing Endpoint
Processing Endpoint
Scheduler Endpoint
File Management Endpoint
Api Endpoint
FileRetrieval Endpoint
![Page 24: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/24.jpg)
AI
24
What did we learn along the way?
![Page 25: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/25.jpg)
AI
25
Division of LaborProblem !
Monolithic Endpoints A naive implementation of our service placed far too many handlers within the same endpoint, making it impossible to prioritize messages and scale portions of the system
!
Alerts Scheduler
Api
Everything
![Page 26: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/26.jpg)
AI
26
Resolution !
A Service is NOT an endpoint In order to ensure our success, we had to split the system into many components, each with its own distinct responsibilities (and queues!)
!Granular Scaling
By splitting into several disparate components, we were able to meet our scalability needs more appropriately. (e.g. scaling only the parsing endpoint)
Division of Labor
![Page 27: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/27.jpg)
AI
27
Saga ResponsibilitiesProblem !
Scaling the File Processing Saga Our initial (read: naive) implementation was responsible for far too much business logic, making it difficult to scale
![Page 28: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/28.jpg)
AI
28
Resolution !
Keep it Simple! Sagas should be as clean and tight as possible !
Scaling Issues The more complicated the saga, the harder they are to scale
!“Observer" Implementation
In our case, we found that a “passive” saga responding to events and simply keeping track of state was far easier to maintain and scale
Saga Responsibilities
![Page 29: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/29.jpg)
AI
29
Saga MethodologyProblem !
Saga Contention Our initial (read: naive) implementation responded to too many events, creating a large degree of contention and a high number of rollbacks
Processing Endpoint
File Management Endpoint
33
9032
132
![Page 30: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/30.jpg)
AI
30
Resolution !
“Check” Pattern In order to reduce contention, we implemented a “check” pattern using timeouts within the saga
Processing Endpoint
File Management Endpoint
390
3213
2
Saga Methodology
![Page 31: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/31.jpg)
AI
31
Deployment AutomationProblem !
Deployment Challenges Hindering Proper Design The team (and client) were initially skeptical of breaking the system into many disparate deployable components, due to the increased difficulty of deployment
!!
Svc 1
Svc 2
Svc 4
Svc 3
Svc 4
Svc 5
Svc 1
Svc 4
Svc 5
Svc 1
Svc 2
Svc 4
Svc 1
Svc 2
Svc 3
Svc 1
Svc 3
Svc 5
![Page 32: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/32.jpg)
AI
32
Resolution !
Simple Automation By leveraging the NServiceBus.Powershell bits along with our own extended Powershell scripts, we were able to automate the deployment of all of our NServiceBus projects !
• Configuration Transformations • Distributed Deployments • Backup Strategy
Deployment Automation
![Page 33: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/33.jpg)
AI
33
Firewalls, Servers & Failure, Oh My!Problem !
Deploying Into New Environments is Hard! Misconfiguration, network issues, security issues… All these make moving into a new environment cumbersome and error prone!
![Page 34: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/34.jpg)
AI
34
Resolution !
ServicePulse, ServiceControl & Custom Checks !
• MSDTC Firewall Checks • RavenDB Firewall Checks • FTP Accessibility Checks • Databus File Share Accessibility Checks
Firewalls, Servers & Failure, Oh My!
![Page 35: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/35.jpg)
AI
35
The Result?
![Page 36: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/36.jpg)
AI
36
The ResultBusiness Impact
!Faster Processing Times
> 10K records/sec parsed and processed, per worker machine !
Happier Users Full system visibility and no lost data !
Happier Developers Clean, maintainable system !
Happier Executives Faster onboarding times for new clients
![Page 37: Building a Highly Scalable File Processing Platform with NServiceBus NSBCon by Sam Martindale](https://reader034.vdocuments.us/reader034/viewer/2022042521/5589e0ded8b42a802c8b459d/html5/thumbnails/37.jpg)
Thank You