gobblin on-aws

Gobblin on AWS&

Gobblin as a ServiceRunning Gobblin on AWS and preview of upcoming Gobblin as a Service

Abhishek Tiwari

Gobblin Execution Model

Functional Mapping

Driver / Master: 1, 2, 3, 5*, 6, 7, 8Workers: 4, 5*

Gobblin on AWS Architecture

Workers Auto Scaling Group

Worker

Master ASG

Master

Network File System / EFS

W W W

Zookeeper

Helix

Task assignmentJob Submission

Master / Driver

Services: Helix Controller Job Conf Manager Job Scheduler

Worker

Services: Task Executor

Worker

Worker

LauncherServices: AWS Security ManagerAWS Coordinator

Provisions all resourcse By coordinating with AWS

Request ASGsfor Master andWorkers

Launch ASG and Instances

ASG = Auto Scaling Group

Demo

Next Steps

1 Multi master provisioning and HA

2 Replace NFS with EFS

3 Support for Spot Instances

4 Continuous Execution

5 Advanced hooks for ASG and exploit Helix

56 AMI coming up

Gobblin as a Service Preview Architecture

Workers Auto Scaling Group

Worker

Master ASG

Master

Network File System / EFS

W W W

Zookeeper

Helix

Task assignmentJob Submission

Master / Driver

Services: Helix Controller Job Conf Manager Job Scheduler

Worker

Services: Task Executor

Worker

Worker

Request ASGsfor Master andWorkers

Launch ASG and Instances

Service / LauncherServices: AWS Security ManagerAWS CoordinatorUI ServerMonitoring Service

Job Factory

Upload Job Configs To S3

Launch Zookeeper

ASG = Auto Scaling Group

Demo

Next Steps

1 Open up more Sources and Sinks

2 Add Graphs (Graphite / Graphana) and comprehensive metrics

3 More cluster types (Azure, Google cloud)

4 Continuous Execution

5 Open to public. Register for preview at: gobblin.io

8

Thank You

Register for Gobblin as a service preview at: gobblin.io

9