workers and worker patterns at scale
TRANSCRIPT
Workers and Worker Patternsat Scale
Workers and Worker Patterns
● 10 team members and growing (all developers)● Funded by Baseline Ventures, Ignition Partners and
other leading cloud angels● Makers of IronMQ and IronWorker● Multi-cloud, multi-platform, multi-language
Who we are
Design Patterns: Recurring solutions to common problems in software design.
Worker Patterns
○ Small Independent Unit of Work○ Think Concurrently○ Optimal Worker Durations○ Scheduling & Quarterbacking○ Pass ID's not large Objects○ Log Everything
Small Independent Unit of Work
Pattern: Keep workers task specific and as light as possible with one purpose.
Much easier to maintain and scale.
Small Independent Unit of Work
Think ConcurrentlyPattern: Designed as collections of non-interacting workers that may be executed in parallel.
Independent and self-sufficient.
Loosely coupled, no shared state.
Think Concurrently
Optimal Task Durations
Pattern: Make task long enough to take advantage of initial setup (db connections, etc), but short enough to be able to quickly fix and retry on errors.
Too short doesn't make sense for worker setup/teardown time (db connections, etc). If too short, batch several tasks into one.
Too long increases opportunity for failure and complexity of retry.
Scheduling & Quarterbacking
Pattern: Create a low number of scheduled jobs and queue up lots of concurrent jobs.
Scheduling & Quarterbacking
Pass IDs not large Objects
Pattern: Pass the minimum amount of information necessary for your worker to run. Then get the full data while it's running.
● Better performance while queuing up tasks● Get latest data● Much easier to maintain state on that object.
Pass IDs not large Objects
Log EverythingPattern: Log everything using a cloud logger or other new global logger.
loggly.com, papertrail.com, IronWorker logging
Contact:
Chad [email protected]
skype: chadarimuratwitter: chadarimura
Iron.io Confidential (v1.1)
Contact Information