AHM 9-11 Nov 2009Bologna Engineering Ingegneria Informatica S.p.A.

INFSO-RI-223782

Andrea Turli e Daniele Rufini

9-11 nov 2009

AWS Submitter Analysis and Design

2

Bologna, 9-1 November 2009

ETICS SubmissionManager and AWS

Power of Cloud ArchitecturesFrom “GrepTheWeb-AWS white paper”• Processing Pipelines• Batch processing systems

o Nightly builds, automated unit/deployment testing • Websites

Amazon AWS • Amazon EC2 for running a large distributed processing

cluster on-demand• Amazon S3 for retrieving/storing persistent datasets• Amazon SQS for buffering requests - loosely coupled• Amazon SimpleDB for storing data - status, log

3

Bologna, 9-1 November 2009

Architectural Abstraction : Level 1

AWS as a Grid

Input System• sendJob : send ETICS jobs to Amazon Cloud • getStatus : receive information about the status of ETICS

job.Output System• GetOutput : result of processing ETICS jobs.

4

Bologna, 9-1 November 2009

Architectural Abstraction : Level 2

sendJob

getSatus getOutput

5

Bologna, 9-1 November 2009

Architectural Abstraction : Level 2

The ETICS Submitter• send a message in to the SQS queue for each request

The controller (EC2 instance):• takes a message from the SQS queue • use SimpleDB to store Job information• launches EC2 instances (worker nodes) to run the job• monitor the job status• Store the output on S3

6

Bologna, 9-1 November 2009

Architectural Abstraction : Level 3

7

Bologna, 9-1 November 2009

Architectural Abstraction : Level 3

On application start• SQS queue creation and start all the controller

Launch Controller• pick up message from launch queue, populated by ETICS• send the job request to EC2 cluster• update status in SimpleDB (ready)• delete the message in the launch queue• insert a message in monitor queue

Monitor Controller• pick up message from monitor queue• ping EC2 cluster to check the job status• take/update the status in SimpleDB (running)• insert a message in delete queueShutdown Controller• shutdown EC2 instances• pick up message from shutdown queue• update the status in SimpleDB• delete a message after processing