connecting workflow-oriented science gateways to multi-cloud systems zoltán farkas, péter kacsuk,...
Post on 26-Dec-2015
235 Views
Preview:
TRANSCRIPT
Connecting Workflow-Oriented Science Gateways to Multi-Cloud
Systems
Zoltán Farkas, Péter Kacsuk, Ákos HajnalMTA SZTAKI
2
Aim of the paper• Cloud technology has been there now for a while• Continuously deprecating grid middleware technologies• Science gateways enabling easy processing of scientific
computation should be able to make use of cloud technologies as well
• This transition should be completely transparent for the scientists
IWSG’15, 7th International Workshop on Science Gateways
3
Concept of cloud-enabling workflow-oriented science gateways
• Three areas: job submission, data management and security• Abstract architecture for discussing integration possibilities:
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission ComponentPluginPlugin
4
Job submission possibilities• Necessary actions to enable submission to a given cloud
middleware:o Workflow Design UI extension: new UI components should be created that enable the
cloud-specific configuration of the workflow node (ideally automatically generated based on a template)
o Workflow Storage extension: the Storage should store the cloud-specific properties of workflow nodes as well (ideally automatically serialized)
o Job Submission Component extension: this component should have a new plugin implemented which is able to arrange job submission to the target middleware based on the node’s description in the Storage
• All this performed as many times as many cloud services implementing different services we’d like to support
IWSG’15, 7th International Workshop on Science Gateways
5
Job submission to multiple cloud systems – Direct method
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission ComponentPlugin2Plugin1
UI1
UI2
Cloud1 Cloud2
Data1
Data2
Cloud1’
API1 API1 API2
6
Job submission to multiple cloud systems – Intermediate component
• Instead of interfacing with multiple cloud APIs, interface with a single Generic Cloud Access Service (GCAS) which hides the access details of different cloud technologies
• One single extension of the Workflow Design UI is necessary• Single extension of Workflow Storage• One additional plugin in the Job Submission Component
IWSG’15, 7th International Workshop on Science Gateways
7
Job submission to multiple cloud systems – Intermediate component
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission ComponentGCAS Plugin
GCAS UI
Cloud1 Cloud2
GCAS
Cloud1’
API1 API1 API2
GCASPlugin1 Plugin2
8
Data handling• Data sooner or later needs to be transferred between the
science gateway and the cloud infrastructure• Direct job submission:
o The data is transferred directly between the cloud storage and the science gateway
• GCAS-based job submission:o Option 1: data is transferred through GCAS (e.g. if no plain HTTP-based access is
available)o Option 2: data transfer is initiated through GCAS, but actual transfer happens between
the cloud storage and the science gateway (e.g. after redirecting HTTP GET requests from GCAS to the cloud storage)
IWSG’15, 7th International Workshop on Science Gateways
9
Data handling overview
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission ComponentGCAS Plugin
Cloud1 Cloud2Cloud1’
API1 API1 API2
GCASPlugin1
Plugin2
Opt1Opt2
10
Security considerations• Direct access to cloud services:
o The credentials need to be defined at the science gateway by the usero OR hidden, with the help of robot credentials
• GCAS-based access:o Option 1: GCAS can hide the access credentials of the cloud infrastructures it supports,
thus users need to authenticate only against the GCAS, so one single auth token needs to be provided by the user
o Option 2: GCAS can accept credentials from users, and forward them to the target cloud, so the user needs to define as many credentials as many cloud infrastructures she/he wants to access
o OR, robot credentials can be used to hide GCAS/cloud access credentials
IWSG’15, 7th International Workshop on Science Gateways
12
Gateway and cloud access use-cases• In an academic environment, the gateway and/or the cloud
infrastructure being used might not be publicly available• In such case applying a single GCAS service might be
problematic
IWSG’15, 7th International Workshop on Science Gateways
Portal Access Cloud Acccess GCAS used Solution
Public Public Yes or No No special setup needed
Public (or Private, but external)
Private Yes or No Open firewall, use proxy
Private Private Yes Open firewall, use proxy
Private Private No No special setup needed
13
Public gateway, public cloud access
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission Component GCAS Plugin
Cloud1
API1
GCAS
Plugin1
Plugin1
Services publicly accessibleNo special setup needed
14
Public (or private external) gateway, private cloud access
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission Component GCAS Plugin
Cloud1
API1
GCAS
Plugin1
Pro
xy
Plugin1
Cloud API on a private IPProxy or firewall setup needed
15
Private gateway, private cloud access
IWSG’15, 7th International Workshop on Science Gateways
Workflow Design
UIWorkfllowStorage
Workflow Enactor
Job Submission Component GCAS Plugin
Cloud1
API1
GCAS
Plugin1
Pro
xy
Plugin1
Portal and Cloud API in thesame networkDirect access: no setup neededGCAS: proxy/firewall needed
16
Cloud access implementetion based on GCAS
• Science gateway:WS-PGRADE/gUSE
• GCAS: CloudBroker Platform
IWSG’15, 7th International Workshop on Science Gateways
17
WS-PGRADE/gUSE Overview• Based on Liferay• General purpose• Workflow-oriented gateway framework• Supports the development and execution of
workflow-based applications• Supports the fast development of domain-
specific gateways by customization methodologies
• Most important design aspect is flexibility
IWSG’15, 7th International Workshop on Science Gateways
18
WS-PGRADE/gUSE Overview
IWSG’15, 7th International Workshop on Science Gateways
• DCI Bridge:o Job submission service of WS-PGRADE/gUSEo Accepting job submission requests from the workflow interpreter through a
standardized interfaceo Offering access to different types of middlewares
• Data Avenue:o Data management componento Offers convenient interfaces (web based and API) for managing data located on different
types of storageso Also used by DCI Bridge and WS-PGRADE to provide data for workflow nodes
The CloudBroker Platform• Easy, scalable, secure, integrable and pay-per-use access to scientific and
technical applications in the cloud• High performance computing application store and marketplace with accounting
and billing• Deployment and execution of applications in the cloud with the same
parameters and files as for local execution• Accessible through the internet via a web browser UI• Integration possibilities into third party tools via web service-based APIs• Public version available under https://platform.cloudbroker.com
• Using infrastructure as a service (IaaS) from cloud providers• Offering platform as a service (PaaS) for software vendors• Providing software as a service (SaaS) to end users• Available compute resources: Amazon, IBM, OpenStack, OpenNebula,
CloudSigma• Available storage resources: Amazon, Walrus, Rados S3
CloudBroker Platform Architecture
CloudBroker Platform
AmazonCloud
IBMCloud
…Cloud
ChemistryAppli-
cations
BiologyAppli-
cations
MedicineAppli-
cations
WebBrowser
UI
Engineering Appli-
cations
Web Service API
GenericWorkbenches CloudBroker Integration
Domain-Specific Gateways
R&D End Users and Software Vendors
CLI
…Appli-
cations
21
gUSE and CBP integration• DCI Bridge:
o Extended with a CloudBroker plugino Making use of the Java API of CBP
• WS-PGRADE:o Authentication portlet: allows users to enter their CBP credentials for communicating
with the CBP on behalf of the usero Workflow portlet: workflow node configuration extended with CBP-specific interfaces
• Workflow Interpreter:o Extended with generating JSDL for DCI Bridge based on CBP-specific node properties
IWSG’15, 7th International Workshop on Science Gateways
WS-PGRADE
gUSE
CloudBroker Platform
Seismology
Proteomics
Rendering
Simulation
Astrophysics
Workflow Portlet
Authentication Portlet
PortalCache
WFIDCI
Bridge
Repository
Rest API Java APIWeb
InterfaceGromac
sBlende
r Blast AutoDock
OpenStackAdapter
OpenNebulaAdapter
Amazon EC2Adapter
Amazon S3Adapter
Rados S3Adapter
OpenStack Cloud
OpenNebula Cloud
Amazon EC2 Rados S3
Amazon S3
Integration features I.• Support for accessing: Amazon EC2, OpenStack, OpenNebula,
CloudSigma• Data transfer happens between WS-PGRADE/gUSE and the
cloud storage – no need to flow data through the CloudBroker Platform
• PaaS-like mode: users can run their own application in the cloud inside a workflow node
• SaaS-like mode: users can select from a set of predeployed applications to be run in the cloud inside a workflow node
Integration features II.• Slight modification in the fully featured UI• Also available in every customization level, completely hiding
the fact that cloud is being used:o ASMo Remote APIo End-user viewo DCI Bridge
WS-PGRADEWF
DeveloperUI
gUSE DCI Bridge
DCI 1
DCI 2
DCI n
ApplicationSpecific
User Interface
ExistingApplicationSpecific UI
WS-PGRADEEnd-User
UI
Remote API
BES interface
ASM API
A
B
C
D
E
BES interface
Integration features III.• Support for commercial clouds with costs (prices configured in
CloudBroker Platform):o Estimated job cost before submissiono Actual job and workflow cost after execution
26
Projects making use of WS-PGRADE/gUSE and CBP integration
• SCI-BUS
• CloudSME
IWSG’15, 7th International Workshop on Science Gateways
Summary• Presented different methodologies of connecting science
gateways to cloud infrastructures• Focusing on an implementation connecting the
WS-PGRADE/gUSE science gateway framework to cloud infrastructures with the help of the CloudBroker Platform
• The presented implementation enables easy migration of already existing gateways onto the cloud platform – only workflow reconfiguration is necessary
top related