icarus: a paraview plug-in for steering and in-situ ...icarus.pdf · “icarus” framework is a...

ICARUS: a ParaView plug-infor steering and in-situ visualization

(Use case: PLUTO)

Luigi Calori – CINECA [email protected] Muscianisi – CINECA [email protected]

INTRODUCTION

� In depth description of usage of the framework “ICARUS” to couple simulation and visualization codes to do on-the-fly visualization and steering.steering.

� “ICARUS” Framework is a set of open source components developed by Jerome Soumagne and John Biddiscombe at CSCS for interfacing an arbitrary HPC simulation code with an interactive ParaView session using the HDF5 parallel IO library as the API.

� “ICARUS” Framework has been used to instrument PLUTO code, an astrophisical simulation code for plasma jets.

2

INTRODUCTION

� Framework for interfacing an arbitrary HPC simulation code with an interactive ParaView session using the HDF5 parallel IO library as the API. The implementation allows a flexible combination of parallel the API. The implementation allows a flexible combination of parallel simulation, concurrent parallel analysis and GUI client, all of which may be on the same or separate machines.

� Data transfer between the simulation and the ParaView server takes place using a virtual file driver for HDF5 that bypasses the disk entirely by directly communicationg between the coupled applications in parallel (e.g. simulation and visualization tool).

The simulation and ParaView tasks run as separate MPI jobs and

3

� The simulation and ParaView tasks run as separate MPI jobs and may therefore use different core counts and/or hardware configurations/platforms, making it possible to carefully tailor the amount of resources dedicated to each part of the workload.

INTRODUCTION

� The coupled applications write and read datasets to the shared virtual HDF5 file layer, which allows the user to read data representing any aspect of the simulation and modify it using representing any aspect of the simulation and modify it using ParaView pipelines, then write it back, to be reread by the simulation (or vice versa).

� This allows not only simple parameter changes, but complete remeshing of grids, or operations involving regeneration of field values over the entire domain, to be carried out.

� To avoid the problem of manually customizing the GUI for each application that is to be steered, the framework make use of XML

4

application that is to be steered, the framework make use of XML templates describing outputs from the simulation, inputs back to it, and what user interactions are permitted on the controlled elements. This XML is used to generate GUI and 3D controls for manipulation of the simulation without requiring explicit knowledge of the underlying model.

Agenda

� Description of a technique for parallel in-situ post-processing and computational steering based onprocessing and computational steering based on

− HDF5 DSM Virtual File Driver

− ICARUS: a ParaView plug-in

− XML template

� How to use the ParaView plug-in “ICARUS” at CINECA cluster PLX

Use case: PLUTO

5

� Use case: PLUTO

Agenda

� Description of a technique for parallel in-situ post-processing and computational steering based on:processing and computational steering based on:



− XML template


Use case: PLUTO

6

� Use case: PLUTO

MOTIVATIONS

� The modern simulation codes:

become more powerful, � become more powerful,

� use more and more resources,

� produce larger and larger data.

� Monitoring and post-processing simulation data in-situ:

being able to get data as fast as possible,

7

� being able to get data as fast as possible,

� reducing the IO bottleneck bypassing the file system.

A SOLUTION

� HDF5 DSM Virtual File Driver: Parallel virtual file driver for the HDF5 library whichParallel virtual file driver for the HDF5 library whichallows the simulation to write in parallel to remotely locatedistributed shared memory (DSM) buffer instead of writingto disk. Allows for live visualiztion and steering.

� ICARUS:ParaView plug-in for interfacing an arbitray HPC simulationcode with an interactive ParaView session.

8

code with an interactive ParaView session.

Developed in the contest of NextMuSE project (http://nextmuse.cscs.ch).

Different codes/ different application domains: all need to be interfaced through the same interface.

HDF5 – Virtual File Drivers

� HDF5: widely known data format, used by several codes

http://www.hdfgroup.org/HDF5http://www.hdfgroup.org/HDF5

� HDF5 based on a Virtual File Driver (VFD) architecture:

� Modular low-level structure called “drivers” (H5FDxxx)

� Mapping between HDF5 format address space and storage

� HDF5 already provides users with different IO methods, commonly called “drivers”, such as

H5FDsec2 (serial posix IO to file), H5FDmpio (parallel IO to file

9

� H5FDsec2 (serial posix IO to file), H5FDmpio (parallel IO to file using MPI-IO)

� H5FDcore (serial IO to memory)� H5FDstream (serial IO to socket)

H5FDdsm driver

� Based on the distributed shared memory (DSM) model:

operates such that by using the common HDF5 API, data is trasparently sent operates such that by using the common HDF5 API, data is trasparently sent (across the network) to the DSM which is distributed among several nodes and is seen by the simulation writing data as a uniform memory space.

10

H5FDdsm driver

� H5FDdsm driver:

derived from core and mpio standard HDF5 drivers. derived from core and mpio standard HDF5 drivers. The core driver allows files to be written in memory instead of disk,

whilst the mpio driver uses MPI-IO to write file in parallel and gives support to the user for collective/independent IO operations.

� Advantage of such driver:

allows very high speed parallel transfer of data directly between coupled simulations, or a simulation and a post-processing application such

11

simulations, or a simulation and a post-processing application such as ParaView, for which a custom plug-in, ICARUS, has been created which allows full control of the visualization of live data.

From H5FDmpio @

12

@ to H5FDdsm

13

HDF5dsm –Configuration of DSM

1. DSM allocated on the post-processing nodes.

When data is written, it is transferred across the network and the read process When data is written, it is transferred across the network and the read process on the visualization machine operates directly on the local DSM network. (default)

1. DSM allocated on the computation nodes.

Data is written immediately into memory and transfers across the network are only initiated when the remote machine makes read requests.

1. Both simulation and DSM may reside on the same machine.

This means that only a single network is traversed and the DSM is used as a

14

This means that only a single network is traversed and the DSM is used as a convenient means of interfacing the partner codes. In this configuration, the DSM is symmetric with data transfer into the DSM initiated from the simulation and read by the coupled process.

HDF5dsm –Modular Communication

� Inter-communicator: handles the transmission of data over the network between the coupled application. over the network between the coupled application. Tipology: MPI/Socket .

� Intra-communicator: handles data locally between processes defining the DSM. Tipology: always MPI.

15

HDF5dsm –Modular Communication

� Inter-communication modes available: MPI and Sockets

� Both use the same communication model with a dedicated � Both use the same communication model with a dedicated “service” thread on the nodes hosting the DSM

� Best communication mode depends on the machine/architecture available

16

HDF5dsm – Synchronization

� After the simulation or the client has finished writing, need to switch communicator for reading, client waits for a “ready for writing” flag

� Message order guarantee that all the data sent is received in the DSM when we start reading

17

DSM when we start reading

� Do not switch to intra-communicator until all remote processes have sent a completion message

� If no one writes, no completion message is sent

In-situ Visualization –How to use the HDF5 DSM VFD

Create a file access property list

fapl_id = H5Pcreate(H5P_FILE_ACCESS);fapl_id = H5Pcreate(H5P_FILE_ACCESS);

Instead of using the MPI-IO driver

H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL);

Insert the DSM driver

H5Pset_fapl_dsm(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL);

Then carry on as usual

18

Then carry on as usual

file_id = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT,

acc_plist_id);

H5Pclose(acc_plist_id);

H5Fclose(file_id);

NOTE: Comment out any MPI_COLLECTIVE dataset transfer properties.

In-situ Visualization –How to use the HDF5 DSM VFD

(C) herr_t H5Pset_fapl_dsm(hid_t fapl_id, MPI_Comm comm,

void *dsmBuffer);

(F90)h5pset_fapl_dsm_f(prp_id, comm, hdferr)

INTEGER(HID_T) prp_id

INTEGER comm, hdferr

� dsmBuffer argument (C/C++ interface only) is optional:

� DSM buffer object can be passed to further use DSM buffer class methods directly, necessary for tweaking the DSM behavior

19

directly, necessary for tweaking the DSM behavior

� If NULL is passed, a DSM buffer is automatically created and configuration is taken from a .dsm_client_config file (H5FD_DSM_CONFIG_PATH)

[Comm]

DSM_COMM_SYSTEM = mpi

DSM_BASE_HOST = test.staff.cscs.ch

DSM_BASE_PORT = 22000

Integration of the DSM interface within ParaView – ICARUS

� DSM host has been integrated into a plug-in (ICARUS) for ParaView package which enables automatic display and analysis of data from any simulation making use of the HDF5 driver. any simulation making use of the HDF5 driver.

� To better support visualization of diverse simulation data, several readers have been integrated in such plug-in which accept HDF5 data, including a reader based on Xdmf.

� The generation of XML descriptions of the data (required by Xdmf to create topological connectivity from the raw HDF5 arrays) from a simple template can be supplied by the user.

20

� The plug-in checks for data ready synchronization messages and automatically updates the active pipelines in the ParaView GUI so that the visualization and any user generated analyses automatically track the ongoing simulation.

Integration of the DSM interface within ParaView

� Only one ParaView instance (built-in mode)

� One Pvserver on a remote machine – One Pvclient on a local � One Pvserver on a remote machine – One Pvclient on a local machine

� Multiple Pvserver distributed among different machines or nodes – One PVclient

21

In-situ Visualization –Creation of the XML template

� Built on top of Xdmf model and format

� Generate Xdmf files to drive Xdmf reader plug-in� Generate Xdmf files to drive Xdmf reader plug-in

� Uses in-memory H5dump to get the missing (metadata) information

� Much simpler to write

� Allows variable number of elements

22

In-situ Visualization –Creation of the XML template

The template allows one or more Grids to be defined which are mapped to

<Domain>...<Grid>be defined which are mapped to

datasets in PV/VTK parlance. If the dataset written to the DSM are multi-block, as many grids as the number of blocks must be defined.

Each Grid contains at least a Topologyfield with the topology type, a Geometryfield with the geometry type and the HDF5 path to access the data representing the geometry. Several

<Grid><Topology TopologyType=”2DSMESH”></Topology><Geometry GeometryType=”X_Y”><DataItem Name=”x”>/node_coords/X</DataItem><DataItem Name=”y”>/node_coords/Y</DataItem>

</Geometry><Attribute Name=”Pressure”

AttributeType=”Scalar”>

23

representing the geometry. Several Attributes can then be added specifying for each the HDF5 path to access the data.

Note: specific XDMF operations such as the JOIN can still be provided.

AttributeType=”Scalar”><DataItem>/vars/pr</DataItem>

</Attribute><Attribute Name=”Density”

AttributeType=”Scalar”><DataItem>/vars/rho</DataItem>

</Attribute></Grid>

...</Domain>

In-situ Viasualization –ParaView GUI with ICARUS

24

Computational Steering Interface

� Extension of HDF5 DSM VFD to provide users with convenient methods for computational steering, i.e.:

� Synchronize post-processing application and simulation

� Test presence of arrays

� Write back arrays

25

Operation modes for timing of

steering interactions – Wait mode

Computation/Analysis not overlapped: The simulation writes data periodically and waits for the analysis

before continuingbefore continuing

26

Operation modes for timing of steering interactions – Free mode

Computation/Analysis overlapped:The simulation loops iterations without waiting, the user interacts

via GUI controls and new data is picked up whenever the via GUI controls and new data is picked up whenever the simulation check for it.

27

Steering API (built on top of HDF5 + DSM)

� Initialize the steering extension (buffer is optional)

H5FD_dsm_steering_init(MPI_Comm comm, void *buffer);H5FD_dsm_steering_init(MPI_Comm comm, void *buffer);

� Get steering commands and disabled objects

H5FD_dsm_steering_update();

� Test if “name” is a disabled object

H5FD_dsm_steering_is_enabled(const char *name);

� Get/Set user defined scalar parameter

28

� Get/Set user defined scalar parameter

H5FD_dsm_steering_scalar_get(const char *name, hid_t mem_type, void *data);

� Get/Set user defined vector parameter

H5FD_dsm_steering_vector_get(const char *name, hid_t mem_type, hsize_t

number_of_elements, void *data);

Steering API

� Test if a user defined parameter is set

H5FD_dsm_steering_is_set(const char *name, int *set);H5FD_dsm_steering_is_set(const char *name, int *set);

� Wait for an incoming “play” command

H5FD_dsm_steering_wait();

� Begin/End query and avoid multiple metadata accesses

H5FD_dsm_steering_begin_query();

� Get/Free dataset handle for data specific parallel read

29

� Get/Free dataset handle for data specific parallel read

H5FD_dsm_Steering_get_handle(const char *name, hid_t *handle);

H5FD_dsm_steering_free_handle(hid_t handle);

XML template

� To use ICARUS, the user has to modify a Xml template which describe the outputs from the template which describe the outputs from the simulation, the parameters which may be controlled, and the inputs back to it.

� The Xml description templates are divided in two distinct parts:

� “Domain”, describing the data for visualization only

30

only

� “Interactions”, defining the list of steering parameters and commands one can control.

XML template – “Domain”

The same of the in-situ visualization:

<Domain>...<Grid><Topology TopologyType=”2DSMESH”></Topology><Geometry GeometryType=”X_Y”><DataItem Name=”x”>/node_coords/X </DataItem><DataItem Name=”y”>/node_coords/Y </DataItem>

</Geometry><Attribute Name=”Pressure”

AttributeType=”Scalar”><DataItem>/vars/pr</DataItem>

31

<DataItem>/vars/pr</DataItem></Attribute><Attribute Name=”Density”

AttributeType=”Scalar”><DataItem>/vars/rho</DataItem>

</Attribute></Grid>

...</Domain>

XML template – “Interactions”

Definition of Steering Properties:

� Int/Double/String VectorProperty:� Int/Double/String VectorProperty:allow scalar, vector and string parameteres to e defined and

generated in the GUI and are the same as existing PV properties.

� CommandProperty:represented in the GUI as a button, but without any state, when it is

clicked, a flag of the defined name is set in the Interactions group and can be checked and cleared by the simulation.

� DataExportProperty:

32

� DataExportProperty:

defines an input back to the simulation, it allows a whole PV dataset or a single data array to be written in a file.

Code and Xml template

33

Code and Xml template

Description of the previous figure:

Example of usage between a simulation code and ParaView. Example of usage between a simulation code and ParaView. The user defines in a description template the interactions that the simulation will be able to access, GUI controls are automatically generated and modified parameteres are passed to the H5FDdsm library. The simulation gets the parameteres and the commands by reading them from the DSM using the same name as specified in the template.

34

Performances

H5FDdsm:

� Not all post-processing apps scale as well as the simulation –� Not all post-processing apps scale as well as the simulation –Run PV on fat nodes and simulation on a very large number of nodes

� Can perform really well over high speed network

� Take advantages of the MPI communication layer

� Improvements can be made by enabling non-linear memory address for file or having dynamic resizing of the DSM

Improvements on memory copies using one-sided

35

� Improvements on memory copies using one-sided communications

Performances

Steering overhead on the H5FDdsm:

� Very low overhead when steering requests performed – Only � Very low overhead when steering requests performed – Only small transaction packets

� Global overhead depends on what one wants to actually send back

36

More details on the articles in bibliography.

Remarks

� Parallel real-time post-processing of the simulation

Easy to use if simulation code uses HDF5 already� Easy to use if simulation code uses HDF5 already

� Do not need to have a big amount of memory on compute nodes since everything is sent to a remote DSM (default mode)

� Live visualization requires only the inclusion of the H5FDdsm library and the changing of only one line in

37

H5FDdsm library and the changing of only one line in the code

� Computational steering can be performed, but with a deeper intrusion in the code

References

HDF5 dsm Virtual File Driver and ICARUS:

� J. Soumagne, J. Biddiscombe, J. Clarke, An HDF5 MPI Virtual File Driver for Parallel In-situ Post-processing. In: Recent Advances in the Message Passing Interface, R. Keller, E. situ Post-processing. In: Recent Advances in the Message Passing Interface, R. Keller, E. Gabriel, M. Resch, J. Dongarra, (eds.), Springer Berlin / Heidelberg, 2010, vol. 6305 of Lecture Notes in Computer Science (2010) 62-71.

� J. Soumagne, J. Biddiscombe, Computational Steering and Parallel Online Monitoring Using RMA through the HDF5 DSM Virtual File Driver. Procedia Computer Science 4 (2011) 479-488.

� J. Biddiscombe, J. Soumagne, G. Oger, D. Guibert, J-G. Piccinali, Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver. EGPGV 2011, 91-100.

38

Agenda



− In-situ visualization

− Computational steering


− XML template

� How to use the ParaView plug-in “ICARUS” at CINECA

39


� Use case: PLUTO

Pre-requirement

� The software interface ICARUS (simulation + visualization system) consists of the following list of pre-requirements:

− simulation side: simulation application + hdf5-vfd package + h5fddsm (please refer to the H5FDdsm project page / Mediawiki section for more details)

− visualization side: hdf5-vfd package + h5fddsm + ParaView-icarus (ParaView standard package to be hopefully soon supported) + icarus plugin

40

Links to framework components

Package: Icarus e Paraview Icarus

https://hpcforge.org/projects/icarus/https://hpcforge.org/projects/icarus/

Package: H5fddsm

https://hpcforge.org/projects/h5fddsm/

41

Package: Xdmfgenerator

https://hpcforge.org/projects/xdmfgenerator/

Agenda



− In-situ visualization

− Computational steering


− XML template for describing the output of the simulation

� How to use the ParaView plug-in “ICARUS” at CINECA

42


� Use case: PLUTO

PLUTO

PLUTO is an astrophysical code developed at the University of Turin.

It is entirely written in C and can run on either single processor machines It is entirely written in C and can run on either single processor machines or large parallel clusters through the MPI library. This application provides a simple user-interface based on the Python scripting language to setup a physical problem in a quick and self-explanatory way.

Computations may be carried on either static or adaptive (structured) grids. The code embeds different hydrodynamic modules and multiple algorithms to solve the equations describing Newtonian, relativistic, MHD, or relativistic MHD fluids in cartesian or curvilinear coordinates.

43

MHD, or relativistic MHD fluids in cartesian or curvilinear coordinates.

PLUTO: Jets from young stellar objects

The numerical computations aim at investigating the formation of radiative shock waves in supersonic, axisymmetric MHD jets from Young Stellar Objects (YSOs). It is indeed widely accepted that these jets derive the bulk of their emission from post-shocked regions of gas where the large temperature and compression are responsible for the emission line spectrum commonly revealed by observations. compression are responsible for the emission line spectrum commonly revealed by observations.

The model investigated consists of a stationary jet model with a superimposed time-dependent injection velocity that produces a chain of perturbations eventually steepening into shock waves. This model is based on a comprehensive set of parameters, among which the perturbation period (PRD) and amplitude (AMP) play a fundamental role. A third parameter (BEG) controls the onset of perturbations since a pre-existing equilibrium state must have been reached.

The equations of MHD are solved describing the evolution of density, momentum, energy density, magnetic field and number of chemical elements including neutral hydrogen, helium and the first three ionization stages of C, O, N, Ne and S. Species are coupled through a comprehensive chemical network describing ionization and recmbination processes. The computational domain is measured in AU and it is defined by 0 < r < 400 and 0 < z < 1600 with axis-symmetric boundary condition at r = 0 and injection

44

is defined by 0 < r < 400 and 0 < z < 1600 with axis-symmetric boundary condition at r = 0 and injection condition at the bottom boundary (z = 0). Zero gradient conditions are applied on the remaining sides. The resolution amounts to 128 x 512 grid zones in the radial and vertical direction, respectively.

H5FDdsm in PLUTO code

To perform only in-situ visualization, the HDF5 mpio driver was replaced with the new DSM driver in the simulation code.

To perform steering in the free mode, the procedures for steering and for the manual signaling of To perform steering in the free mode, the procedures for steering and for the manual signaling of the writing/reading of the HDF5 output file has been added, according to the following scheme:

. . .

H5FD_dsm_steering_init;

. . .

main loop:

H5FD_dsm_steering_update;

. . .

if(writing on the HDF5 file):

H5FD_dsm_set_mode(manual signaling);

45

H5FD_dsm_set_mode(manual signaling);

H5FD_dsm_steering_scalar_get(“pertamplit”);

H5FD_dsm_steering_scalar_get(“pertperiod”);

if(one or both these two parameters has been modified):

update the corresponding variable of the simulation.

endif

H5FD_dsm_server_update;

endif

. . .

end loop;

Xml template for PLUTO

<Interaction>

<DoubleVectorProperty name="pertamplit"

<Domain>

<Grid Name="Ysojet-grid"> name="pertamplit" command="SetSteeringValueDouble" label="PERT AMP" number_of_elements="1" default_values="70" > <DoubleRangeDomain name="range" min="10" max="100" />

</DoubleVectorProperty>

<DoubleVectorPropertyname="pertperiod" command="SetSteeringValueDouble" label="PERT PRD" number_of_elements="1" default_values="10.0"> <DoubleRangeDomain name="range" min="1.0" max="70.0" />


<Topology TopologyType="2DSMESH">

</Topology>

<Geometry GeometryType="X_Y">

<DataItem Name="x">/node_coords/X</DataItem>

<DataItem Name="y">/node_coords/Y</DataItem>

46


</Interaction>

</Geometry>

<Attribute Name="Pressure" AttributeType="Scalar"> <DataItem>/vars/pr</DataItem>

</Attribute>

. . .

</Grid>

</Domain>

Results for PLUTO

Comparison of two simulations starting from the same input file.

The comparison shows the results at the same time step but with different values of the steering parameters (AMP and PRD), modified from the GUI of ParaView.

Jet produced at the time

47

Jet produced at the time t = 25.00. Steering parameters are still

equal to the initial ones: AMP = 70.00, PRD = 10.00.

48

Jet at the time t = 31.50. The cooling is started: chain of perturbations are produced from below. (a) Parameters are not changed. (b) The perturbation period PRD (3rd cursor of the GUI, moved by the user at t = 30.00) is decreased from 10 to 5. In fact, you can see that in the right image more perturbations are injected.

49

Jet at the time t = 35.00 . (a) The perturbation amplitude AMP (2rd cursor of the GUI, moved by the user at t = 32.00) is decreased from 70 to 10, PRD = 10. (b) Parameters are not changed: AMP = 70, PRD = 5.

50

Jet at the time t = 41.50 . (a) PRD is increased from 10 to 60, AMP = 10. (b) AMP is decreased from 70 to 20, PRD = 5.

References

In-situ visualization in CINECA:

M.Rivi, G.Muscianisi, L.Calori, V.Slavnic. State-of-the-art and Some Use Cases.M.Rivi, G.Muscianisi, L.Calori, V.Slavnic. State-of-the-art and Some Use Cases.PRACE 1IP, whitepaper (2011)

http://www.prace-ri.eu/IMG/pdf/In-situ_Visualization_State-of-the-art_and_Some_Use_Cases.pdf

PLUTO code:

http://plutocode.ph.unito.it/

A. Mignone, G. Bodo, S. Massaglia, T. Matsakos, O. Tesileanu, C. Zanni, A. Ferrari, A numerical code for computational astrophysics. Astrophysical J. Suppl., 170 (2007) 228-242.

51

(2007) 228-242.

QUESTIONS ???

52

icarus: a paraview plug-in for steering and in-situ ...icarus.pdf · “icarus” framework is a...

Documents