scientific visualization at jsc · 30.11.-02.12. 2016 eocoe-f3f meeting - rome folie 1 jens henrik...
TRANSCRIPT
Folie 1EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Jens Henrik Göbbert1
1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany
Cross-Sectional-Team „Visualization“
30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome
Scientific Visualization at JSC
Mitglie
d d
er
Helm
holtz-G
em
ein
schaft
Folie 2EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Visualization at JSCCross-Sectional Team “Visualization”
Domain-specific User Support and Research at JSC
Scientific Visualization– R&D + support
for visualization of scientific data
Virtual Reality– VR systems
for the analysis and presentation
Multimedia– multimedia productions
for websites, presentations
or on TV
http://www.fz-juelich.de/ias/jsc/EN/Expertise/Support/Visualization/_node.html
Folie 3EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
12x Visualization Nodes
2 GPUs Nvidia Tesla K40 per node
12 GB RAM on each GPU
2x Vis. Login Nodes
jurecavis.fz-juelich.de
10x Vis. Batch Nodes
8 nodes with 512 GB RAM
2 nodes with 1024 GB RAM
Keep in mind:
Visualization is NOT limited to
Visualization nodes (incl. GPUs) only.
(software rendering is possible on any node)
Visualization at JSCGeneral Hardware Setup
http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JURECA/UserInfo/VisNodes.html?nn=1538466
Data
GPFS
vis login
nodes
vis batch
nodes
compute
nodes
JURECA
10x
2x
InfiniB
and
1872x
12xlogin
nodes
Folie 4EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Visualization at JSC
– Cross-Sectional Team “Visualization”
– Setup - Software & Hardware
Simplify your life: Remote 3d Visualization
In Situ Visualization
– Introduction
– Coupling Strategies
– Example
Outline
Folie 5EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Mitglie
d d
er
Helm
holtz-G
em
ein
schaft
Simplify your life:
Remote 3D Visualization
on JURECA with VNC
Folie 6EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Why remote visualization …
… makes sense for the user.
Do not spend time installing & configuring visualization software.
Do not move your data.
Do not be satisfied with your local hardware.
… makes sense for CS-Team „Visualization“.
Install & configure visualization software once for all users.
Any new feature is available to all users.
Fix bugs once for all users.
Motivationlowering the barriers to visualization
More time left for the interesting things.
Folie 7EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
nice blue JSC background clock
counting
up/down
MOTD windowCPU, memory
utilization
GPU utilization
VNC utilization
desktop
symbels for
vis apps,
LLview, …
visualization application
Motivationlowering the barriers to visualization
Folie 8EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
User’s Work-
station
vis login node:
- direct user access
- no accounting
- shared with other users
- no parallel jobs (no srun)
vis batch node:
- access via batch system
- accounting
- exclusive usage
- parallel jobs possible
Data
GPFS
vis login
nodes
vis batch
nodes
compute
nodes
JURECA
10x
2x
InfiniB
and
1872x
12xlogin
nodes
Remote 3D VisualizationGeneral Setup
Firewall
Folie 9EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Remote 3D Visualizationwith X forwarding + Indirect Rendering
Traditional Approach (X forwarding + Indirect Rendering)
ssh –X <USERID>@<SERVER>
uses GLX extension to X Window System
X display runs on user workstation
OpenGL command are encaplusated inside X11 protocol stream
OpenGL commands are executed on user workstation
disadvantages User´s workstation requires a running X server.
User´s workstation requires a graphic card capable of the requied OpenGL.
User´s workstation defines the quality and speed of the visualization.
User´s workstation requires all data needed to visualize the 3d scene.
Try to AVOID for 3D visualization.
Folie 10EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
State-of-the-Art Approach (VNC with VirtualGL)
vncserver, vncviewer
platform independent
application independent
session sharing possible
advantages No X is required on user´s workstation (X display on server, one per session).
No OpenGL is required on user´s workstation (only images are send).
Quality of visualization does not depend on user´s workstation.
Data size send is independent from data of 3d scene.
http://www.virtualgl.org
Remote 3D Visualizationwith VNC (Virtual Network Computing) + GLX forking
Try to USE for 3D visualization.
Folie 11EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
GLX forking
vglrun (VirtualGL)
OpenGL applications send both GLX and
X11 commands to the same X display.
Once VirtualGL is preloaded into an
OpenGL application, it intercepts the
GLX function calls from the application
and rewrites them.
The corresponding GLX commands are
then sent to the X display of the 3d X
server, which has a 3D hardware
accelerator attached.
Disadvantages:
⁻ Multiple users on a login node share the same graphic cards.
⁻ No resource management
for memory/compute of graphic cards.
⁻ CPU encodes the VNC stream.
⁻ Currently no hardware accelerated image encoding.
Remote 3D Visualizationwith VNC (Virtual Network Computing) + GLX forking
http://www.virtualgl.org
Folie 12EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
… manually startup VNC connection1. ssh to HPC system
2. authenticate via SSH key pair
3. look for an existing VNC server
(you may want to reconnect)
4. submit job via slurm
5. wait for job to start
start VNC server on node
6. establish SSH tunnel
7. setting up SSH agent forwarding
8. start TurboVNC client and connect
Remote 3D VisualizationSimplify the access
Data
GPFS
vis login
nodes
vis batch
nodes
compute
nodes
JURECA
10x
2x
InfiniB
and
1872x
12xlogin
nodes
Firewall
https://trac.version.fz-juelich.de/vis/wiki/vnc3d/manual
This is far too time consuming.
Folie 13EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
… automatically startup VNC
ScienTific Remote Desktop Launcher (Strudel)
– Windows, OS X, Linux
– written in Python
– developed by
• Monash University, Melbourne, Australia
Remote 3D VisualizationSimplify the access
Data
GPFS
vis login
nodes
vis batch
nodes
compute
nodes
JURECA
10x
2x
InfiniB
and
1872x
12xlogin
nodes
https://trac.version.fz-juelich.de/vis/wiki/vnc3d/strudel
One-Click solution.
Firewall
Folie 14EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Jens Henrik Göbbert1
1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany
Cross-Sectional-Team „Visualization“
30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome
Mitglie
d d
er
Helm
holtz-G
em
ein
schaft
Invitation:
Try JURECA Visualization Nodes
Folie 15EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
InvitationTry JURECA Visualization Nodes - 30.11.16 – 09.12.16
VNC with VirtualGL (using Strudel)
https://trac.version.fz-juelich.de/vis/wiki/vnc3d/strudel
download & install
• SSH (Putty for Windows)
• TurboVNC
• Strudel
configure
• download SSH keys
• https://fz-juelich.sciebo.de/index.php/s/qec7SKVOEBDenpH• get passphrase for one of the SSH keys
• load SSH key to your SSH agent
run & play
• start Strudel and connectPlease follow the instructions from
https://trac.version.fz-juelich.de/vis/wiki/vnc3d/strudel
Folie 16EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
InvitationTry JURECA Visualization Nodes - 30.11.16 – 09.12.16
VNC with VirtualGL (using Strudel)
https://trac.version.fz-juelich.de/vis/wiki/vnc3d/strudel
Keep in mind:
• SSH key agent must be running on your system.
• SSH key for JURECA must be loaded into the SSH key agent.
Please send us your Feedback.
Folie 17EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Try JURECA Visualization NodesOSPRay
http://www.ospray.org/
http://www.sdvis.org/ospray/download/talks/IEEEVis2016_OSPRay_talk.pdf
CPU ray tracing framework
for scientific vis. rendering
efficient rendering on CPUs
ray tracing / high fidelity rendering
made for scientific visualization
Built on top of
Embree (Intel ray tracing kernels)
Intel SIMD Program Compiler
Integrated into
ParaView, VMD, VisIt, VL3,
EasternGraphics,...
ParaView
OSPRay
“FIU” Ground Water Simulation
Texas Advanced Computing Center (TACC) and Florida International University
Folie 18EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Try JURECA Visualization NodesOSPRay with ParaView
http://www.ospray.org/
http://www.sdvis.org/ospray/download/talks/IEEEVis2016_OSPRay_talk.pdf
Ray tracing within ParaView
Build option in ParaView 5.2 by default
Why ray tracing?
gives more realistic results
adds “depth” to your image
can be faster on large data
Requirement:
CPUs: Anything SSE4 and newer
(in part, including Intel©
Xeon Phi™ Knights Landing)
Cooperation with Electrochemical Process Engineering (IEK-3)
Jülich Forschungszentrum GmbH, Germany
ParaView
(standard renderer)
ParaView
(OSPRay)
Folie 19EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Try JURECA Visualization NodesParallel ParaView
Folie 20EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Jens Henrik Göbbert1
1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany
Cross-Sectional-Team „Visualization“
30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome
Mitglie
d d
er
Helm
holtz-G
em
ein
schaft
In Situ Visualization
Folie 21EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Why do we need In Situ Visualization ?Visualization “the old way”
JUQUEEN, @ FZ Jülich Data Archive, @ FZ
Jülich
Jülich Storage Cluster
(JUST), @ FZ Jülich
@ AIA, RWTH Aachen University
Why changing a running System …. ?
JURECA, @ FZ Jülich
wikipedia.org, Chevron
Folie 22EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Why do we need In Situ Visualization ?Bottlenecks for Visualization
Example:
AIA, RWTH Aachen University
Large Eddy Simulation
Analysis of Chevron Geometries
123 TB (= 30 days to download with 50 MB/s)
@ AIA, RWTH Aachen University
X
Moving data between computing centers
becomes a major bottleneck.
Folie 23EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Why do we need In Situ Visualization ?Bottlenecks for Visualization
XX
Moving data to the storage devices
will become a major bottleneck.
Folie 24EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
What is In Situ Visualization ?Basics
Data compression
Reduce I/O traffic and volume.
Access to more data.
Post-processing
In situ processingDump Times:
Enable real-time simulation
exploration.
Reduce latency to first result.
Avoid large-scale post-processing.
…..
…..
Visualizing results of a running simulations …
… without the need to copy the data to a storage device.
Folie 25EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
What is In Situ Visualization ?Basics
But ….
Simulation- and Visualization code …
… might have different internal data structures.
… might scale best with different parallelization strategies.
Simulation- and Visualization developer …
… might have different priorities.
… might not work in the same team.
Visualizing results of a running simulations …
… without the need to copy the data to a storage device.
Coupling simulation- and visualization codes is a challenge.
Folie 26EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
What is In Situ Visualization ?Staged vs. On-Node
„Staged“ In Situ Visualization
(or in transit visualization)
„On-Node“ In Situ Visualization
Folie 27EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
What is In Situ Visualization ?Staged vs. On-Node
„Staged“ In-Situ Visualization „On-Node“ In-Situ Visualization
Advantages:
No need to copy data between nodes
or clusters.
Zero-copy possible.
Simpler to implement.
Disadvantages:
Hardware resources (cpu / memory)
need to be shared between simulation
and visualization.
Advantages:
No need to share hardware resources
(cpu / memory) between simulation and
visualization.
Disadvantages:
Data needs to be copied between
nodes or even clusters.
No zero-copy possible.
More difficult to implement.
Folie 28EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
What is In Situ Visualization ?Loose vs. Tight
„Loose“ Coupling „Tight“ Coupling
Advantages:
Simpler to implement.
Single executable.
Zero-copy possible.
Disadvantages:
Bug in visualization code might affect
simulation.
„On-Node“ in situ visualization by
design.
Advantages:
Less dependencies at compile/link time
Two separate applications.
Disadvantages:
Data transfer between Simulation and
Visualization more difficult to
implement.
Zero-copy difficult.
Folie 29EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Major barriers to in situ visualization are …
… first,
the individual implementation-, optimization- and coupling-costs
to integrate the needed functionality to each simulation code and setup
can often not be justified.
… second,
the usage of in situ visualization requires much training for scientists
who's research work in general does not focus on visualization in the first
place.
Lowering the barriers to in situ visualizationWhat are the major barriers?
Folie 30EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Direct Approchcoupling simulation code to in-situ visualization
sim. developer
simulation
VisIt/
Libsim
ParaView/
Catalyst
vis. developer
ParaView/Catalyst
VisIt/Libsim
Folie 31EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Damaris - http://damaris.gforge.inria.fr
SENSEI - http://www.sensei-insitu.org
Strawman -
GLEAN -
JUSITU -
Adapter Approachcoupling simulation code to in situ visualization
sim. developer vis. developer
simulation
VisIt/
Libsim
ParaView/
Catalyst
Folie 32EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Adapter Approachcoupling simulation code to in situ visualization
simulation
VisIt/
Libsim
ParaView/
Catalyst
data adapter bridge analyze adapter
Folie 33EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Adapter Approachcoupling simulation code to in situ visualization
data adapter bridge analyze adapter
Folie 34EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Jens Henrik Göbbert1
1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany
Cross-Sectional-Team „Visualization“
30.11.-02.12. 2016 EoCoE-F3F Meeting - Rome
Mitglie
d d
er
Helm
holtz-G
em
ein
schaft
In Situ Visualization
Example
Folie 35EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
… psOpen
JUSITU
flow solver (LES, DNS)
multiphysics, multiscale
finite volume method
lattice-Boltzmann method
discontinuous Galerkin method
level-set for surface tracking
Lagrangian particle solver
Institute of Aerodynamics Aachen
RWTH Aachen University, Germany
C++11 + MPI + OpenMP + GPU
… CIAO … ZFS
flow solver (LES, DNS)
multiphysics, multiscale
structured finite difference
method
level-set for surface tracking
level-set/volume-of-fluid interface
Lagrange particle solver
tabulated/finite rate chemistry
overset mesh refinement
moving meshes
Institute for Combustion
Technology
RWTH Aachen University,
Germany
Fortran90 + MPI
JUQUEEN BigWeek participant
flow solver (DNS)
highly resolved turbulence
pseudo-spectral approach
Institute for Combustion Technology
RWTH Aachen University, Germany
Chair of Num. Thermo-Fluid Dyn.
TU Freiberg, Germany
Fortran90 + MPI + OpenMP
JUQUEEN BigWeek participant
Applications lately coupled with VisIt/Libsim
Folie 36EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Why In Situ Visualization – An example.spray injection & droplet formation
Folie 37EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Baumgarten, C., Mixture formation in internal combustion engines, Springer, 2006.
Aye, M.M. et al., Studying the influence of k-factor of different nozzles on spray and
combustion under diesel engine-like conditions in a high pressure spray chamber (submitted).
Geometry of injection systems
increasingly complex
Emission and pollutant formation
strongly depend on injection systems
Experimental investigation of geometry
impact very challenging smaller
k-factorlarger
k-factor
Ra
tio o
f so
ot
tra
ns.
inte
nsity
High-fidelity spray simulations required!
Why In Situ Visualization – An example.spray injection & droplet formation
Folie 38EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Experimental investigation of
injection systems very challenging
due to small length (~100 μm) and
high velocity scales (~600 m/s).
Highly resolved simulations result in
large amount of data which can be
reduced with in-situ visualization.
Predictive simulations using in-situ
visualization and studying injection
systems and resulting cavitation
give more insight.
Le Chenadec, V. and Pitsch, H., “A 3D Unsplit Forward/Backward Volume-of-Fluid Approach and Coupling to the Level Set Method,” Journal of Computational Physics 233:10-33, 2013
Le Chenadec, V. and Pitsch, H., “A monotonicity preserving conservative sharp interface flow solver for high density ratio two-phase flows,” Journal of Computational Physics 249:185-203, 2013
Why In Situ Visualization – An example.spray injection & droplet formation
Folie 39EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Why In Situ Visualization – An example.spray injection & droplet formation
Dump data with high temporal and spatial resolution needed.
Folie 40EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Application CIAO on JUQUEEN (specific setup)
■ excellent scaling of solver
■ file output is bottleneck at scaledump data with high temporal resolution not possible
=> In situ visualization helps!
Why In Situ Visualization – An example.spray injection & droplet formation
Folie 41EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Summary & Conclusion
Try the visualization nodes at JSC
Send me your feedback.
In Situ Visualization
Size of simulation data is rising constantly
and I/O becomes a major bottleneck.
A huge number of simulation codes and scientists will
benefit from in situ visualization (in situ processing).
But, decisions have to be made
how to coupling simulation and analysis tools in future.
Folie 42EoCoE-F3F Meeting - Rome30.11.-02.12. 2016
Questions … ?
rendered with Blender from a DNS of a diesel injection spray of ITV, RWTH Aachen University