shiwa is supported by the fp7 capacities programme under contract no. ri-261585
DESCRIPTION
SHIWA and Coarse-grained Workflow Interoperability Gabor Terstyanszky, University of Westminster Summer School Budapest 02-06 July 2012. SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585. Workflow Interoperability Challenge. SSP Objectives and Services. - PowerPoint PPT PresentationTRANSCRIPT
SHIWA andCoarse-grained Workflow
InteroperabilityGabor Terstyanszky, University of Westminster
Summer SchoolBudapest
02-06 July 2012
SHIWA is supported by the FP7 Capacities Programme under contract No. RI-261585
2
Workflow Interoperability Challenge
SSP Objectives and Services
Objectives:• To create an environment which enables seamless execution of
workflows of different workflow systems through workflow interoperability
Services:• To support the whole workflow lifecycle: editing, uploading, browsing
downloading and executing workflows
• To provide coarse- and fine-grained workflow interoperability solution
• To offer Distributed Computing Infrastructure interoperability solution
• To provide desktop computer and portal interfaces to manage workflows
Key actors:• researchers workflow engine developers
workflow developers
3
4
Workflow description abstract workflow•plain text which describes the inputs and outputs and explains what the workflow doesWorkflow implementations concrete workflow• It contains all implementation files or references to these files (via eg. URLs) and also holds other
data/metadata necessary to run the workflow on its associated workflow engine.Workflow configurations• It contains pre-defined input files and parameters or references to these files and parameters and other
data/metadata of a workflow.Workflow engines• It contains files and other data/metadata necessary to execute a workflow engine on a grid site or references
to them
SHIWA Data Structure
5
WFSubmission
clientSubmission
ServiceWorkflow Engine B
Workflow of Workflow Engine B
DCI
Workflow Engine A
Coarse-Grained Interoperability: Single Non-native Workflow
6
J2 WF4
J1
J3
Submission client
Submission Service
Workflow Engine B
Workflow of Workflow Engine B
Workflows of Workflow Engine A
Coarse-Grained Interoperability: Meta-Workflow
DCI
• native workflows: J1, J2, J3• non-native workflows: WF4
- black boxes which are managed as legacy code applications
Workflow Engine A
7
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
SHIWA Portal
GEMLCA admin
SHIWA Science Gateway
WE1 WEp
GEMLCA Repository
WF1 WFm
SHIWA Repository
WF1 WFn
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
GEMLCA Service
GEMLCA with GIB
SHIWA Proxy Server
Proxy Server
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
SSP: CGI Infrastructure
SHIWA Science Gateway Resourcesnative WE WS-PGRADE local resources: invocation of locally deployed WEs portal WS-PGRADE v3.4.1WE submission to local cluster repository GEMLCA + SHIWA repo remote resources: through remotely pre-deployedsubmitter GEMLCA with GIB WEs to ARC, gLite, Globus Unicore DCIsproxy management SHIWA Proxy Server
88
SHIWA Portal: Abstract Workflow
Jobs
Ports
9
SHIWA Portal: Configuring Workflow
10
SHIWA Portal: Executing Workflow
11
SHIWA Repository: Browse View
domain
search
12
inputsoutputs
data sets
implementation
SHIWA Repository: Browse View
1313
Browse Workflows
Browse Implementations
Search
Advanced Search
Sort
SHIWA Repository: Table View
1414
SHIWA Repository: Execution Support
Execution data
15
Taverna workflow
SHIWA Portal: Legacy Code Explorer
16
SHIWA Portal: Repository Portlet
Workflow Repository
17
CGI Developer Scenario: Specifying WE
18
CGI Developer Scenario: Specifying WF through portal
19
CGI Developer Scenario: Specifying WF through desktop
2020
CGI User Scenario: Native WE
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WE + WF
WF1 WFm
GEMLCA with GIB
WF list
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
user
SHIWA Proxy Server
Proxy Server
step 1search WF
step 2 edit WF
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
step 7 run WFstep 6
retrieve proxy
step 5retrieve WF
step 4submit WF
step 3retrieve WF data
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIB
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor WE + WF
user
GEMLCA Client
MOTEURWorkflow
Engine
MOTEUR Workflow
editor
GEMLCA UI
SHIWA Proxy Server
Proxy Server
CGI User Scenario: Non-native WE
step 1 search WF
step 2 edit WF
step 3 submit WFstep 6
retrieve proxy
step 5 retrieve WF
step 7 run WF
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
step 4 forward WF
22
SHIWA Repository
SHIWA Portal
WF1
SHIWA Science Gateway
GEMLCA Service
WFn
WE1 WEp
GEMLCA Repository
WF1 WFm
GEMLCA with GIB
WS-PGRADE Workflow
Engine
WS-PGRADE Workflow
Editor
SHIWA Proxy Server
Proxy Server
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
SHUWA
Desktop
WE
user
step 1 search WF
step 2 retrieve WF
step 3forward
WF bundle
step 4edit WF
step 6submit WF
step 5ask run WF
CGI User Scenario: WF Execution via Desktop
SHIWA RepositorySHIWA Portal
WF1
GEMLCA admin
SHIWA Science Gateway
WFn
WE1 WFp
GEMLCA Repository
WF1 WFm
GEMLCA submitter
WS-PGRADE Workflow
engine
WS-PGRADE Workflow
editor
SSP DCI & FGI Infrastructure
SHIWA Science Gateway Resourcesnative WE ASKALON, MOTEUR, WS-PGRADE
local WEs: Kepler, MOTEUR , Taverna & Triana WEsportal WS-PGRADE v3.4.1 submit to the local cluster repository GEMLCA + SHIWA remote WEs: ASKALON, GWES & MOTEUR WEsGEMLCA
ARC, gLite, Globus and Unicore DCIsproxy server
DCI Bridge
Meta Broker
Proxy Server
JSDL Translator
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCI
Kepler WEGWES WE
Taverna WE
Triana WE
MOTEUR WE
ASKALON WE
PGRADE WE
Galaxy WE
Pegasus WE
ProActive WE
Workflow Engines
2424
SHIWA RepositorySHIWA Portal
WF1
SHIWA Science Gateway
WFn
WE1 WFp
GEMLCA Repository
WF1 WFm
gLite DCI Globus DCI
DCIs
ARC DCI Unicore DCIWS-PGRADE
Workflow engine
DCI Interoperability Scenario
DCI Bridge
Meta Broker
Proxy Server
JSDL Translator
WS-PGRADE Workflow
editor
step 1 search WF
step 2 edit WF
step 3retrieve WF data
step 4translate WF
step 8submit WF
step 7retrieve
certificate
step 5forward WF
step 6select DCI
2525
SHIWA Simulation Platform:Access
SHIWA Science Gateway (SHIWA Portal + SHIWA Repository)
• joint portal and repository account given by the science gateway administrator
DCI resources
• certificate given by the relevant DCI administrator, for example the VO sysadmin
Access to the SHIWA Portal
ssp.shiwa-workflow.eu
Access to the SHIWA Repository
repo.shiwa-workflow.eu
26
Conclusions
• the simulation platform supports the whole lifecycle of workflows, i.e. creating, testing, uploading, browsing, downloading and running workflows
• users can use workflows of their own and of other workflow systems through the same user interface, i.e. they can re-cycle or share workflows created by other research teams or even other research communities
• workflow developers (or e-scientists) can create, run, upload workflows through the simulation platform
• researchers (astro physicists, earth scientists, bio scientists, etc.) can browse the workflow repository, select and run workflows through the simulation platform
• the simulation platform provides access to major European e-infrastructure, the European Grid Initiative (EGI) infrastructure which offers large pool of compute and storage resources