gobblin on-aws
TRANSCRIPT
Gobblin on AWS&
Gobblin as a ServiceRunning Gobblin on AWS and preview of upcoming Gobblin as a Service
Abhishek Tiwari
Gobblin Execution Model
Functional Mapping
Driver / Master: 1, 2, 3, 5*, 6, 7, 8Workers: 4, 5*
Gobblin on AWS Architecture
Workers Auto Scaling Group
Worker
Master ASG
Master
Network File System / EFS
W W W
Zookeeper
Helix
Task assignmentJob Submission
Master / Driver
Services: Helix Controller Job Conf Manager Job Scheduler
Worker
Services: Task Executor
Worker
Worker
LauncherServices: AWS Security ManagerAWS Coordinator
Provisions all resourcse By coordinating with AWS
Request ASGsfor Master andWorkers
Launch ASG and Instances
ASG = Auto Scaling Group
Demo
Next Steps
1 Multi master provisioning and HA
2 Replace NFS with EFS
3 Support for Spot Instances
4 Continuous Execution
5 Advanced hooks for ASG and exploit Helix
56 AMI coming up
Gobblin as a Service Preview Architecture
Workers Auto Scaling Group
Worker
Master ASG
Master
Network File System / EFS
W W W
Zookeeper
Helix
Task assignmentJob Submission
Master / Driver
Services: Helix Controller Job Conf Manager Job Scheduler
Worker
Services: Task Executor
Worker
Worker
Request ASGsfor Master andWorkers
Launch ASG and Instances
Service / LauncherServices: AWS Security ManagerAWS CoordinatorUI ServerMonitoring Service
Job Factory
Upload Job Configs To S3
Launch Zookeeper
ASG = Auto Scaling Group
Demo
Next Steps
1 Open up more Sources and Sinks
2 Add Graphs (Graphite / Graphana) and comprehensive metrics
3 More cluster types (Azure, Google cloud)
4 Continuous Execution
5 Open to public. Register for preview at: gobblin.io
8
Thank You
Register for Gobblin as a service preview at: gobblin.io
9