sharing, integrating and executing different workflows in heterogeneous multi-cloud systems
Post on 17-Jan-2016
36 Views
Preview:
DESCRIPTION
TRANSCRIPT
Sharing, integrating and executing different workflows in heterogeneous multi-cloud systems
Peter KacsukMTA SZTAKI
kacsuk@sztaki.hu
SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481 1
2
Motivations 1
• In many cases large simulations are organized as scientific workflows that run on Distributed Computing Infrastructures (DCIs)
• However, there are too many different• WF formalism• WF languages• WF engines
• If a community selected a WF system it is locked into this system:• They can not share their WFs with other
communities (even in the same scientific field)• They can not utilize WFs developed by other
communities
3
Motivations 2
• A WF system engine is typically connected to one particular DCI (Distributed Computing Infrastructure)
• As a result, if a community selected a WF system it is locked into this DCI• Porting the WF to another DCI requires extra effort• Parallel execution of the same WF in several DCIs is
usually not possible
Part of WF Ecosystem
4
er
What do we want to achieve?
5
XSEDE BOIN
C
Amazon
Bio1Bio2
BioN
CyberspaceWorkflows
InfrastructuresUsers should be able to access and use any WF and any infrastructure in an interoperable way no matter which is
their home WF system
Taverna Galaxy Kepler
WF systems
What does WF interoperability mean?
• If a user developed WF Y in WF system B she (or other users) should be able to1. Re-use WF Y as part of another WF
(e.g. WF X) developed in WF system A (Coarse-grained interoperability – CGI)
6
Coarse-grained interoperability CGI
Coarse-grained interoperability (CGI) = Nesting of different workflow systems to achieve interoperability of WF execution frameworks
DCI 1
DCI 2
DCI 3
7
A
Y
Features of CGI approach
• Advantages– No restrictions on the embedded WFs– You can run the embedded WFs in their native
DCI (even in parallel -> easy to achieve high degree of DCI parallelism)
– Easy to implement and connect a new WF system• Drawbacks
– Black-box approach: you cannot • modify the embedded workflow• control and observe the internal operation of
the embedded WF
8
What does WF interoperability mean?
• If a user developed WF Y in WF system B she (or other users) should be able to1. Run WF Y under another WF system
(e.g. WF system A) (Fine-grained interoperability – FGI)
2. Further develop WF Y under another WF system (e.g. WF system A) (FGI)
9
Transform WF Y into IWIR
WFY
Interoperable Workflow Intermediate Representation (IWIR)
Transform IWIR into WF X
WFX
Fine-grained interoperability (FGI) or white box WF interoperability:Enables to transform one WF to another WF system and further develop it in the new system
10
Features of FGI approach
• Advantages– White-box approach: you can
• modify the embedded workflow• control and observe the internal operation of the
embedded WF• Drawbacks
– There are some restrictions on the WFs that can be transformed
– You can run the embedded WFs only in the native DCIs of the target WF system
– Not easy to implement and connect a new WF system
11
What does infrastructure interoperability mean?
• If a user developed WF X in WF system A she (or other users) should be able to– Run WF X in any DCI without significant porting
effort– Run different nodes of WF X in different DCIs (if
these nodes are in parallel branches then they can simultaneously run in different DCIs)
12
Cloud1
Cloud N
EU Projects that develop solutions for these goals
• SHIWA– To solve WF and DCI interoperability
issues– Duration: 2 years (July 2010 – June
2012)
• SCI-BUS– To provide the required gateway
technology– Duration: 3 years (Oct 2011 – Sep 2014)
13
accessing a large set of various DCIs to make these WF applications run
Clouds
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systems
E-science infrastructure
What does a WF developer need?
WF App. Repository
Access to a large set of ready-to-run scientific WF applications
Portal
Using a portal/desktop to parameterize and run these applications, and to further develop them
Reference production infrastructure of SHIWA
SHIWA App.
Repository
Application developers
• Publish WF applications in a repository to be continued/used by other appl. Developers
SHIWA Portal
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)(EGEE, OSG, etc.)
Supercomputer based SGs
(DEISA, TeraGrid)
Grid systems
• Use the portal/desktop to develop complex applications (executable on various DCIs) based on WFs stored in the repository
facilitates publishing and sharing workflows
Supports:• Abstract workflows with multiple implementations of 10 workflow systems• Storing execution specific data
Available:• from the SHIWA Portal• standalone service at: repo.shiwa-workflow.eu
SHIWA Repository
16
SHIWA Bundle and SHIWA Desktop for WF interoperability
17
SHIWA Bundle
SHIWA App. Repository
WS-PGRADE
Triana
MOTEUR
ASKALON
SHIWA Desktop
SHIWA Desktop
SHIWA Desktop
SHIWA Desktop
SHIWA Bundle and SHIWA Desktop
• SHIWA Bundle: – object (stored as a zip file) containing everything needed
to expose a workflow for use– Provides a common language/format for workflow engines
• Workflows are stored as SHIWA bundle in the SHIWA Repository
• SHIWA Desktop connects a user’s desktop workflow environment to the SHIWA Repository
18
Extension of WF interoperability with DCI interoperability
19
SHIWA Bundle
SHIWA App. RepositorySHIWA
Desktop
WS-PGRADE
SHIWA Desktop
Triana
SHIWA Desktop
MOTEUR
SHIWA Desktop
ASKALON
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)
Supercomputer based SGs
Grid systems
DCI Bridge
BES interface
Production serviceJSDL Translator
Workflow Engine
DCI Bridge
J2
J1
J4
J3
jobs in non-JSDL
J2
J1
J4
J3
jobs in JSDL
DCI n
DCI 1
Accessing DCI Bridge
20
• BES requires JSDL for job submission• Therefore we need a JSDL generator to help WF engines
to create the JSDL for the jobs generated for WF nodes
Extension of WF interoperability with DCI interoperability (2)
21
SHIWA Bundle
SHIWA App. RepositorySHIWA
Desktop
WS-PGRADE
SHIWA Desktop
Triana
SHIWA Desktop
MOTEUR
SHIWA Desktop
ASKALON
Local clusters
Supercomputers
Desktop grids (DGs)(BOINC, Condor, etc.)
Cluster based service grids (SGs)
Supercomputer based SGs
Grid systems
DCI Bridge
BES interface
JSDL Translator
JSDL Translator
Where are we?
• Workflow interoperability done by– SHIWA Bundle– SHIWA Desktop– SHIWA Repository
• DCI interoperability done– DCI Bridge– JSDL Translator
• All of them are production services• What else do we need?
– A reference service through which anyone can try the technology
• The reference service is the SHIWA portal
22
SHIWA portal: WS-PGRADE/gUSE Generic-purpose gateway framework
23
• Based on Liferay• General purpose• Workflow-oriented portal framework• Supports the development and execution
of workflow-based applications• Enables the multi-cloud, multi-DCI
execution of any WF• Provides access to
• internal repository • external SHIWA Repository
Creating and running WS-PGRADE workflows
24
Step 1: Edit workflow
Step 2: Configuring the workflow
Cloud1
Cloud N
Step 3: Running workflow instance
26
Scalable architecture based on collaborating services
Seamless access to various types of DCIs
WFGraph editor
Liferay
WS-PGRADE
portal
Information System
WF Storage
File StorageWF Repository
WF Interpreter
GT5Grid
DC
I-B
ridge
Client machine
Portal Server machine
DCIs
BES interface
BES interface
BOINC
Grid
ARCGrid
CloudBroker
WFs in the clouds
• This issue is solved by the SCI-BUS project by integrating WS-PGRADE/gUSE with CloudBroker Platform
• Motivation:– Cloud resources are getting more and more popular– Clouds are more reliable than grids– WFs with cloud access are capable of satisfying
compute needs of complex scientific computations– Clouds can provide a vast amount of resources
• Aim: – Provide access to cloud resources in a transparent
way
29
WS-PGRADE/gUSE and SCI-BUS
30
Multi-cloudMulti-cloud
Integrated WS-PGRADE/CloudBroker Platform to access multi-clouds
CloudBroker
Platform
WS-PGRADE
n
IaaSCloud 1
IaaSCloud N
SEQ
SEQ
WS-PGRADE
1
• Supported clouds: Amazon, IBM, OpenStack, Eucalyptus, OpenNebula
• SaaS solution: • Preregistered services/jobs can run from WS-
PGRADE (Supported from gUSE 3.5.0)• IaaS solution:
• any services/jobs can be submitted from WS-PGRADE (Supported from gUSE 3.5.1) 31
CloudBroker Platform
• Web-based application repository for the deployment and execution of scientific and technical software in the cloud
• Offers these stored applications as SaaS service for end users
• On demand, pay per use, browser / programmatic / command-line access, cross-domain
• Uses infrastructure as a service (IaaS) from resource providers and offers these IaaS resources for users
32
User ToolsUser Tools
Java Client Library
CloudBroker Platform Architecture
20.09.2012 /
CloudBroker PlatformCloudBroker Platform
AmazonCloud
Open-StackCloud
…Cloud
ChemistryAppli-
cations
BiologyAppli-
cations
HealthAppli-
cations
WebBrowser
UI
…Appli-
cations
REST Web Service API
CloudB
roker IntegrationC
loudBroker Integration
End Users, Software Vendors, Resource Providers
CLI
EngineeringAppli-
cations
IBMCloud
Euca-lyptusCloud
33
Integrated architecture
Integration features
• Support for commercial clouds with costs (prices configured in CloudBroker Platform):– Estimated job cost before submission– Actual job and workflow cost after execution
Accessible Cloud Resources
• Access provided by the CloudBroker Platform• Commercial:
– Amazon EC2– IBM
• OpenSource/Free:– OpenStack– OpenNebula– Eucalyptus
• Currently, within SCI-BUS accessible:– MTA SZTAKI OpenNebula (400 cores)– BIFI OpenStack (50 cores)
Collaboration within and among communities based on gUSE
37
SHIWA Repository
gUSE Portal
Cloud 1OpenNebula
Cloud 2Amazon
Cloud nOpenStack
gUSE Portal
WF upload as SHIWA bundle
WF upload as SHIWA bundle
gUSE WF
Repo
gUSE WF
Repo
Success story: SHIWA solution for the LINGA experiment
Sub-WorkflowsManagement
38
Multi-Workflow
Maturity of implementation
• Production services:– SHIWA Repository– SHIWA Bundle and SHIWA Desktop– CGI approach - Connected WF systems:
• ASKALON, Galaxy, MOTEUR, Pegasus, Taverna, Triana, WS-PGRADE
– SHIWA portal based on gUSE– CloudBroker Platform - Connected clouds:
• Amazon, IBM, Eucalyptus, OpenNebula, OpenStack
• Prototype services– FGI approach - Connected WF systems:– ASKALON, MOTEUR, Triana, WS-PGRADE
• New EU project ER-Flow supports 6 user communities
39
Recent WS-PGRADE/gUSE releasesHistory since v3.4.0•Nov 2011: v3.4.0 (DCI Bridge)•Feb 2012: v3.4.1 (usage statistics portlet)•March 2012: v3.4.2 (support for new EMI release)•April 2012: v3.4.3 (support for Liferay 6.1)•…•Aug 2012: v3.5.0 (SaaS cloud access via CBP) •Sep 2012: v3.5.1 (IaaS cloud access via CBP) •Oct 2012: v3.5.2 (SHIWA workflow repository export/import)•March 2013: v3.5.3 (REST support, EMI-UI v1/v2 support, …)•April 2013: v3.5.4 (cloud cost estimation/reporting)•April 2013: v.3.5.5 (robot certificates)•May 2013: v.3.5.6 (Improved SHIWA workflow repository export/import)
gUSE sourceforge statistics
41
42
Where to find further information?• gUSE/WS-PGRADE:
– http://www.guse.hu/• gUSE on sourceforge
– http://sourceforge.net/projects/guse/– http://sourceforge.net/projects/guse/forums/forum/– http://sourceforge.net/projects/guse/develop
• SCI-BUS web page:– http://www.sci-bus.eu/
• SHIWA web page:– http://www.shiwa-workflow.eu/
• ER-Flow web page:– http://www.erflow.eu
43
Summary
• We have created a technology that enables to combine many different WFs, WF systems and DCIs in many different ways
• It is like a puzzle where you can put together the required pieces to create the final picture
44
gUSE WF system
OpenNebula
CloudKepler
WF
GalaxyWF
top related