:: idc 2009 :: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: 06/10/2009 :::: 1 ::...
TRANSCRIPT
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 1 ::
Workflows and HPC?:: The relation between workflows and High Performance Computing ::
Contact: Lutz Schubert [[email protected]], Stefan Wesner [[email protected]]
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 2 ::
The relation between workflows and High Performance Computing
No workflows in HPC!
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 3 ::
The relation between workflows and High Performance Computing
No workflows in HPC!
• Classically, a workflow is – A predefined sequence of tasks / actions– That are exposed in a standard fashion – And communicate via a centralized point
• Or:“Any program is a workflow”
[thanks Mr. Turing]
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 4 ::
The relation between workflows and High Performance Computing
No workflows in HPC!
• At the same time: it’s the least efficient way to execute a program:– Centralized communication– Communication overhead through
standards– Next steps need to be identified in a
central point– …
ÞJust the opposite of what HPC programs try to achieve
1
23
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 5 ::
The relation between workflows and High Performance Computing
?• But why should workflows act directly on the process
level?
Þ Can also act as a means to control higher-order processesI. Initiate a HPC job as part of a larger processII. Trigger jobs upon specific events (that provide data)III. Coupling jobs according to outcome / status
No workflows in HPC?
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 6 ::
The relation between workflows and High Performance Computing
I. Higher-Order Processes with HPC Tasks
The “Grid” concept:• In particular in engineering cases, extensive
calculations need to take place at various stages• In large scale, collaborative engineering examples,
workflows can reduce the complexity of providing data / jobs
• A sub-workflow can take over recurring setup and configuration tasks
• Examples: – Collaborative Engineering (BAE )– Virtual Engineering (ANSYS)
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 7 ::
The relation between workflows and High Performance Computing
Uses
Uses analysis services from
Delivers sub-system design to
Analysis VO
Analysis S/W HouseAnalysis S/W
Provider
Accesses product database, sends reports to
UsesAnalysis Team
Analysis Provider
HPC Provider
Buys storage fro m
CE3 VO
Production FacilitiesProduction Consultants
Resource Planning
Populates design database
Sub-system providerSub-system
Design Team
Stores results to
Storage Provider
Product design database
Aircraft Maintenance
Aircraft
Contracts maintenance to Maintains
Owns
Airline VO
Airline
Product Config Storage Provider
Updates
Uses
Uses analysis services from
Delivers sub-system design to
Analysis VO
Analysis S/W HouseAnalysis S/W
Provider
Analysis S/W HouseAnalysis S/W HouseAnalysis S/W HouseAnalysis S/W
ProviderAnalysis S/W ProviderAnalysis S/W Provider
Accesses product database, sends reports to
UsesAnalysis TeamAnalysis TeamAnalysis Team
Analysis Provider
HPC ProviderHPC ProviderHPC Provider
Buys storage fro m
CE3 VO
Production FacilitiesProduction Consultants
Production FacilitiesProduction FacilitiesProduction FacilitiesProduction ConsultantsProduction ConsultantsProduction Consultants
Resource PlanningResource Planning
Populates design database
Sub-system providerSub-system
Design Team
Sub-system providerSub-system providerSub-system
Design TeamSub-system Design Team
Stores results to
Storage ProviderStorage ProviderStorage Provider
Product design databaseProduct design database
Aircraft Maintenance
Aircraft
Contracts maintenance to Maintains
Owns
Airline VO
Airline
Product Config Storage Provider
Updates
Aircraft MaintenanceAircraft Maintenance
AircraftAircraft
Contracts maintenance to Maintains
Owns
Airline VO
AirlineAirline
Product Config Storage ProviderProduct Config Storage Provider
Updates
I. Basic Collaborative Structure
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 8 ::
The relation between workflows and High Performance Computing
I. High-Level Workflow
NegotiateRequirementsWith Customer Extract
Engineeringrequirements
Generate newdesigns
Analyse Designs
ActivateAntennaAnalysis
Send ModelData
Retrieve Outputdata
Write AnalysisReport
Read ReportAnd
update design
Customer Negotiator
DesignTeam
AnalysisTeam
HPCService
StorageProviderService
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 9 ::
The relation between workflows and High Performance Computing
II. Event-Based Job Control
The “Cloud” Principle:• Some recurring jobs need to be executed e.g. with updates in data,
under certain environmental conditions etc.• These events may trigger recurring configuration workflows (cf. I)• Potentially same job offered for multiple users on-demand (with
personal data)• Not a workflow as such, but easier defined and adapted by making
use of workflow descriptions• Examples:
– Microsoft Financial Computing– Google Market Evaluations
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 10 ::
The relation between workflows and High Performance Computing
Data pro- visioning
Job execution
Data sub- mission
Job sub- mission
II. Financial Calculation Principle
specific request
changes on the global stock market
“personal” results
triggers
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 11 ::
The relation between workflows and High Performance Computing
III. Coupled Applications
• Multiple HPC jobs that depend on each other• Next job is triggered with the conclusion of the
previous one (passing the data)• The selection of the next job may depend on the
results of the previous ones• Intermediary results may trigger evaluation jobs in
parallel• Examples:
– Material Stress Test– Virtual Physiological Human
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 12 ::
The relation between workflows and High Performance Computing
Critical material characteristics
Critical points
III. Material Stress Test Principle
whole model individual elementsin-depth
material test
Result feedbackImpact on whole model
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 13 ::
The relation between workflows and High Performance Computing
III. A Virtual Physiological Human Model
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 14 ::
The relation between workflows and High Performance Computing
III. A Virtual Physiological Human Model
• Same base model, but different detailed elements• No direct coupling between elements• New knowledge about one element can impact on other behaviour• Simulation of specific diseases may lead to coupling, depending on
goal, e.g.– how medicine spreads if the heart muscle is affected– how a muscular disease spreads to and affects the heart– etc.
Þ Data is only exchanged between elements under certain conditionsÞ The workflow must model these events and conditions
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 15 ::
The relation between workflows and High Performance Computing
To Workflow or Not To Workflow
Good For• Describing relationships
between actors and tasks• Recurring configuration and
submission processes • Result-dependent
relationships between jobs• Modelling event-based
triggers
Bad For• Tightly coupled process
control & execution• Distribution of large data
between jobs• Fast interactions with
external processes
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 16 ::
The relation between workflows and High Performance Computing
Summary
Þ Workflows can be used for higher-order controlÞ Tasks in a workflow could be considered as “triggers”Þ (Large) datasets should be hosted in a well-known place
rather than provided via the workflowÞ Recurring tasks that are not time-critical can be supported
by workflows (e.g. configuration steps)Þ They are more comprehensible than batch jobs and hence
easier to model and adapt
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 17 ::
The relation between workflows and High Performance Computing
THANK YOU FOR YOUR ATTENTION-
ANY QUESTIONS?
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 18 ::
The relation between workflows and High Performance Computing
Old slides
OLD SLIDES
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 19 ::
The relation between workflows and High Performance Computing
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 20 ::
The relation between workflows and High Performance Computing
To Workflow or Not To Workflow
• Workflows can particularly support recurring tasks and typical relationships between jobs / tasks– configuration and submission steps– decision logic– event based triggers
• They are not sensible to control the distributed nature of a HPC job
• The stronger the coupling between tasks / elements, the less sensible is workflow support
Þ Higher-order support only
:: IDC 2009 ::
::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :::::
:: 06/10/2009 :: :: 21 ::
The relation between workflows and High Performance Computing
III. A Virtual Physiological Human Model