August 19, 2014

ParaView Coprocessing Visualization

of Differential Equations

PIs Benson Muite (Jared Whitehead)

ESRT Mark Van Moer

Asymptotic Scaling of Heat Transport in

Infinite Prandtl Number Fluids at Very

High Resolution – TG-CTS110010

Introduction to Parallel Numerical

Methods – TG-DMS110033

• Research and Teaching allocations with significant overlap.

• ESRT support Q2-Q4 2012

2

Research Overview

• Rayleigh-Benard convection

• Grid up to 12,2882

• Aim to improve computational methods for partial diff eqs with singular perturbations

3

Teaching Overview

• Solve ODEs and PDEs with separation of variables

• Numerical time-stepping routines

• Pseudo-spectral methods with DFT and FFT

• Serial and parallel coding in Matlab, Fortran, and Python

• Visualization

• Course Wikibook

4

Motivation for Visualization

• Data too large to store

• Often want visualization anyway

• Quick way to get rough idea of regions of interest

5

1st approach: VisIt in situ with libsim2

• Application instrumented to attach to VisItrender engine.

• Emphasis on steering

• Callback architecture

• Tested 2D Navier-Stokes sim, VisIt 2.5.2

6

VisIt in situ, cont’d.

Decided against because:

• Required attaching VisIt to running job, no batch ability in 2.5.2

• Significant refactoring

– Maintenance burden

– Obscured teaching code

7

2nd approach: ParaView Coprocessing

• Sim calls ParaView server which executes vispipeline

• Save images or data

• Pipeline exported from client (Python script)

– Create with low res data

– Can also hand-code pipeline

8

PV Copro cont’d

• Code requires very few additions

– 4 lines, can pull rest to module

– Also removed existing Fortran I/O

• Must write code to format data ParaViewunderstands, essentially a VTK DataSet

– Trivial for regular grids

• Must write code to activate coprocessor

9

2D Navier Stokes CUDA Fortran

• Ran on NCSA Forge

– Single node, 10242 grid

– PGI CUDA Fortran, revealed some ParaView bugs

• Github repo

10

2D Navier Stokes same sim, diff pipelines

11

3D Navier Stokes with MPI

• (serial 3D code essentially same as 2D)

• Add handling for domain decomposition

– Code node based

– Must send halo nodes to coprocessor to avoid gaps in images

– Interior, boundary halos differ, requires additional logic

12

3D Navier Stokes w MPI, cont’d

• If mesh cell < pixel, can justify interpreting simulation nodes as vis cells.

• Send just pencils to coprocessor.

• May require Cell Data to Point Data filter in pipeline

13

3D Navier Stokes w MPI, cont’d

14

Other 3D work

• Instrumented Klein-Gordon and Cubic Non-linear Schrodinger codes

• Ported to Nautilus with help from Pragnesh Patel, NICS

• Started Kraken port, stalled on cross-compiling

15

Then the project ended.

Afterwards

• Successfully ported to

– Titan by Hong Yi, RENCI

– Vienna Scientific Cluster 2 by Michael Quell, TU Wien

– Beacon by Mark Van Moer, NCSA

– Neser by Shuabi Arshad, Samar Aseeri, MadhuSrinivasan, KAUST and Utkarsh Ayarchit, Kitware

– Vedur by Oleg Batrashev, Tartu

In progress: Mira, Juqueen, Shaheen

16

Afterwards, cont’d

• Cross-compiling ParaView was non-trivial, Thank You! to Kitware for extra assistance

– Utkarsh Ayarchit, Andy Bauer, David DeMarle, Ken Moreland

• Kitware released Catalyst--lightweight builds, slight API changes

• Michael Quell’s very interesting YouTube channel

17

Afterwards, cont’d

• 2 papers in preparation, one on scaling w/co-processing, one on general scaling.

• Poster: Quell, M. “Performance of a distributed three dimensional Coprocessing code for the Klein Gordon equation”

• Srinivasan, M., Muite, B.K. “KVL Project Focus: Shockwave Propagation Through Elastic Media”

• Parallel Spectral Numerical Methods wikibook

– Visualization Chapter

• PSNM github repo

18

Links

• ParaView Co Processing guide for 3.14.1, 3.98

• ParaView Catalyst for 4.0+

• VisIt Users in-situ tutorial

19