dominik stokłosa pozna ń supercomputing and networking center, supercomputing department ingrid...
TRANSCRIPT
![Page 1: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/1.jpg)
Dominik Stokłosa
Poznań Supercomputing and Networking Center, Supercomputing Department
INGRID 2008 Lacco Ameno, Island of Ischia , ITALY, April 9-11
Workflow management in Remote Instrumentation Infrastructure – based on e-VLBI experiences
![Page 2: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/2.jpg)
Introduction to the e-VLBIIntroduction to the e-VLBI
PSNC in EXPReS - FABRIC PSNC in EXPReS - FABRIC
Grid – VLBI design & technologyGrid – VLBI design & technology
SummarySummary
Outline
![Page 3: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/3.jpg)
Introduction to the e-VLBIIntroduction to the e-VLBI
VLBI is a technique, in which physically independent and widely
separated radio telescopes observe the same region of sky
simultaneously, in order to generate very high-resolution
continuum and spectral-line images of cosmic radio sources
Telescopes are usually separated by thousands of kilometres
Data from each telescope are digitally sampled and stored locally,
using high-capacity magnetic tape systems and magnetic disk-array
systems
Data are sent and correlated at the central point (JIVE)
The total flow of data into the central processor is approximately 10-
100 Terabytes per single observation, after processing this is reduced
to 10-100 Gbytes.
![Page 4: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/4.jpg)
Introduction to EXPReSIntroduction to EXPReS
EXPReS – the objective is to create a production-level
“electronic” VLBI (e-VLBI) service, in which the radio telescopes
are reliably connected to the central data processor at JIVE via a
high-speed optical-fibre communication network.
Project Details
Three years, started March 2006
International collaboration
Funded at 3.9 million EUR
FP6, Contract #026642
![Page 5: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/5.jpg)
PSNC in EXPReSPSNC in EXPReS
EXPReS – a Real-time e-VLBI Radio Telescope
- JRA1: Future Arrays of Broadband Radio-Telescopes
on Internet Computing (FABRIC)- Grid – VLBI collaboration- Grid Workflow management
- Grid Routing
Creating solution for incorporating Grid resources for
distributed correlation using existing infrastructure.
![Page 6: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/6.jpg)
Once upon a time (1)Everything was slow Telescopes collected data on tapes Sent via postal mail
Hard drive arrays slightly improved the situation
The entire cycle could easily require 6 months or more
![Page 7: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/7.jpg)
Once upon a time (2)
Hardware correlator; the EVN MkIV data correlator at JIVE dedicated, purpose designed/built hardware a super computer; ~100 T ops/sec
![Page 8: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/8.jpg)
Today / In the near future
Data can be transferred over the network
Each stage of the process can be speeded up
GRID resources
Software correlator
e-VLBI - electronic VLBI
![Page 9: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/9.jpg)
e-VLBI System - architecture
![Page 10: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/10.jpg)
e-VLBI System – control flow
![Page 11: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/11.jpg)
Workflow Manager module (WFM)
![Page 12: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/12.jpg)
CCF – correlator configuration file
![Page 13: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/13.jpg)
WFM – phase 1
Definition of radio telescopes – automatically based on theobservation schedule
![Page 14: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/14.jpg)
WFM – phase 2
Definition of file servers (each file sever is responsible for capturing data from RT)
![Page 15: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/15.jpg)
WFM – phase 3
Definition of correlation nodes and data flows between components
![Page 16: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/16.jpg)
WFM – properties
Definition of resource properties
![Page 17: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/17.jpg)
WFM (1)
The WFM application has several purposes:
•Workflow creation – a user is given a list of building blocks. Each block represents the available resource. It is the user’s responsibility to design and construct the experiment’s workflow from the available building blocks.
•Workflow submission – the scenario prepared in the previous step can be submitted to the system.
•Workflow management and monitoring – allows to manage the executed workflow.
![Page 18: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/18.jpg)
WFM (2)
Obrazek z WFM -> przeplywy danych
![Page 19: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/19.jpg)
Link definition
![Page 20: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/20.jpg)
Type definition
Radio telescope
File server
Correlator
![Page 21: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/21.jpg)
Properties definition
Dynamic forms
Parameters stored in Json
format
![Page 22: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/22.jpg)
PSNC in EXPReSPSNC in EXPReS
EXPReS – a Real-time e-VLBI Radio Telescope
- JRA1: Future Arrays of Broadband Radio-Telescopes
on Internet Computing (FABRIC)- Grid – VLBI collaboration- Grid Workflow management
- Grid Routing
Creating solution for incorporating Grid resources for
distributed correlation using existing infrastructure.
![Page 23: Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow](https://reader034.vdocuments.us/reader034/viewer/2022051819/5514ca80550346b0338b4d8b/html5/thumbnails/23.jpg)
Thank you for your attention
http://www.expres-eu.org/[email protected]
EXPReS is made possible through the support of the EC, 6th FP, Contract #026642