PI: Scott C. Noble (U. Tulsa)

co-PI: M. Campanelli (RIT)co-PI: J. Krolik (JHU)

Investigators:M. Avara (PD, RIT)D. Bowen (GR, RIT)R. Cheng (PD, JHU —> LANL)S. d’Ascoli (GR, RIT)B. Dell (UG, RIT)J. Healy (PD, RIT)S. James (GR, U. Tulsa)C. Lousto (Prof, RIT)V. Mewes (PD, RIT)L. Moon (UG, RIT)H. Shiokawa (PD, CfA)Y. Zlochower (Prof, RIT)

NCSA POC: Jing Li

Image Credit: Mark Vanmoer (NCSA)

Predicting the Transient Signals from Galactic Centers: Circumbinary Disks and Tidal Disruptions Around

Black HolesBW IDs: PRAC_gk5, DD_gku

Blue Waters Symposium, Wednesday May 17th, 2017

Thanks to NSF PRAC OCI-0725070, NSF CDI AST-1028087, NSF PRAC ACI-1515969, NSF AST-1515982

Based on:• Bowen et. al, ApJ, 838, 42 (2017). • Healy & Lousto, PRD, 95, 024037 (2017) • Healy, Lousto, Campanelli, arXiv, 1703.03423 • Shiokawa et al., arXiv, 1701.05610 (in review)

Why It Matters: Mysteries of Supermassive Black Holes

Tidal Disruption EventsAccreting Supermassive Black Hole Binaries (SMBBHs)

• Principal source of low-frequency GWs (LISA). • Multi-messenger astronomy offers:

• Simul. EM/GW to probe strong-field gravity; • New measurement of cosmological expansion;

• Rare opportunity to learn how BHs become active and grow.

• Means by which to understand feedback and stellar populations in galactic centers.

• Both provide significant insight into galaxy evolution and black hole growth. • Both lack resolved, long time-scale MHD simulations thereof in the relativistic regime. • Both are being investigated using current high-cadence data (PAN-STARRS, Catalina RTS),

and future ones like the Large Synoptic Survey Telescope (LSST).

• Demonstrated break down of Newtonian prediction (Paczynski 1977) of truncation radius, at separations < 20-30M for equal-mass binaries;

• Sloshing seen in previous runs using Newtonian gravity, though has not been studied closely.

• Closer binaries: —>Shallower potential —> More mass in sloshing region

Bowen et. al, ApJ, 838, 42 (2017)

Accomplishments:

2-d Hydrodynamic SMBBH Mini-disks

Key Challenges: • Our novel near-equilibrium initial data

allows us to reach steady-state quicker and mitigate initial data transients.

• Resolve the vast dynamic range of the problem.

Bowen et. al, ApJ, 838, 42 (2017) 2-d Hydrodynamic SMBBH Mini-disks

Accomplishments:

• Frequency of mass modulation not trivially connected to orbital frequency —> simulations are key!

• Modulation can help identify binaries and characterize their parameters.

Why It Matters:

Modulation Power of Mass in Sloshing Region

• Closer binaries: —>Shallower potential —> More mass in sloshing region

• Dissipation of sloshing streams will emit significant luminosity, which can modulate the signal from the binary.

—> Signature of strong-field gravitational dynamics.

(in progress) 3-d MHD SMBBH Mini-disks• First of a kind simulation: consistent

GRMHD in inspiral regime.

• Will 2-d hydro. results carry over to more realistic scenario?

• 3-d also allows us to maintain consistent thermodynamics with radiated power, useful exploring relativistic beaming, gravitational redshift effects on observables.

• Discover the significance of relativistic outflows (jets).

Top-down, large scale view

• The run is an “intermediate”-sized job requiring more than a month of sustained run time: • 600x160x640 (6e7) cells. • 600 nodes for 40 days.

• Unaffordable on other systems. • Difficult to push through the queue

quickly because of its size and BW’s queue’s priorities.

Why Blue Waters?

• David King and Jing Li (and others) both helped setup reservations for the 3-d runs to accelerate the jobs through the queue.

BW Team Contributions:

(in progress) 3-d MHD SMBBH Mini-disksTop-down, large scale view • First of a kind simulation: consistent

GRMHD in inspiral regime.

• Will 2-d hydro. results carry over to more realistic scenario?

• 3-d also allows us to maintain consistent thermodynamics with radiated power, useful exploring relativistic beaming, gravitational redshift effects on observables.

• Discover the significance of relativistic outflows (jets).

BW Team Contributions:• David King and Jing Li (and others) both

helped setup reservations for the 3-d runs to accelerate the jobs through the queue.

• The run is an “intermediate”-sized job requiring more than a month of sustained run time: • 600x160x640 (6e7) cells. • 600 nodes for 40 days.

• Unaffordable on other systems. • Difficult to push through the queue

quickly because of its size and BW’s queue’s priorities.

Why Blue Waters?

Numerical Relativity (vacuum) Binary Black Simulations

Healy & Lousto, PRD, 95, 024037 (2017)Healy et al., arXiv, 1703.03423

• RIT’s public catalog of NumRel gravitational waveforms: • 16 of 126 waveforms were done on BW for first public catalog. • 60 more in progress, for a total of 76 out of 300 waveforms in the next catalog. • Used in LIGO’s Algorithm Library for use in matched filtering GW signals. • Used to improve fitting formula for final BH mass, spin, kick velocity, GW luminosity, etc.

• Additional simulations include those of GW151226 and LVT151012 LIGO events.

Products & Broader Impacts:

Accomplishments:• Validated multi-patch method using

patches with different: • Coordinate systems; • Resolutions; • Local gravity models; • Time step sizes;

• Various tests; • Shock tube; • Spherical Sedov-Taylor blast wave;• Linear waves;

Shiokawa et al., arXiv, 1701.05610 (in review)

Patchwork: a Multipatch Infrastructure for Multiphysics/Multiscale/Multiframe Fluid Simulations

12

0.0e+00

2.0e-03

4.0e-03

6.0e-03

�

4 ⇥ 104

Exact Monopatch Global Local

0.0e+00

2.0e-09

4.0e-09

6.0e-09

p

250 3000.0e+00

1.0e-03

2.0e-03

v r

5 ⇥ 104

250 300 350

r

2 ⇥ 105

500 550

Figure 5. Sedov-Taylor 3D spherical blast wave at three di↵erent times (t = 4 ⇥ 10

4

, 5 ⇥ 10

4

, 2 ⇥ 10

5

). Monopatch data (red

X’s) are contrasted with multipatch data (green squares for the global patch, cyan circles for the local patch) and with the

analytical solution (black line). Each column of three panels shows 1D radial cuts in density ⇢, pressure p, and radial velocity

v

r

.

believe is the principal issue, we study an idealized prob-lem, one in which matter flows from a patch in whichit has acquired order-unity amplitude fine-scale struc-ture into a more coarsely-resolved patch. The parame-ter that appears to a↵ect conservation errors the mostis the ratio between the lengthscale of the structure andthe resolution scale of the coarser patch.

To illustrate this dependence, we construct a 3D sys-tem in which the problem volume extends from x =�200 to x = +600 in Cartesian coordinates, but in thetransverse directions (y and z) spans only the range[�20, +20]. The local patch is stationary and occu-pies the region �200 x 0 in global coordinates.

Both patches have uniform grids that are parallel toeach other, but the x-direction cell-sizes of the globalpatch (5) are 10⇥ that of the local patch (0.5), whilethe y- and z-direction ratio is 11.4 (5.7 as opposed to0.5).

The initial condition for the test is shown in Figure 6.At t = 0, all of the fluid is traveling at V

x

= 0.1 in thex-direction, but its density and pressure di↵er sharplyacross the line x = +10, located a short distance into theglobal patch from the local patch boundary. To the leftof that line, ⇢

L

= 1 + sin2(!n

y) and p

L

= 10�12

⇢

L

(asin the previous tests, c = 1 in our units), while on theright ⇢

R

= 10�16 and p

R

= 10�28. This sharp pressure

Sedov-Taylor blast waveMultipatch Monopatch

vs.

Accomplishments:• Scaling Performance:

• Communication load insignificant up to ~103 cores per patch;

• Improves with smaller surface/volume ratio;

• Moving patches add overhead as inter-patch connections need to be updated more often.

• Still using naive, unoptimized communication strategy;

• Heterogenous time stepping yields 2-3x speedup!

Shiokawa et al., arXiv, 1701.05610 (in review)

Patchwork: a Multipatch Infrastructure for Multiphysics/Multiscale/Multiframe Fluid Simulations

16

104

105

N·c

ycl

es/(

s·P

)

Stationary

monopatch, 203 monopatch, 403 multipatch, 203 multipatch, 403

Moving

101 102 103

P: # of processors per patch

10�2

10�1

100

t com

/ttot

101 102 103

Figure 8. Computational e�ciency as a function of numbers of processors per patch and for di↵erent numbers of cells per

processor. Monopatch method data are plotted with open symbols, multipatch with filled symbols. Runs with 20

3

cells per

processor are shown with black circles, runs with 40

3

cells per processor with red squares. Left (right) panels show multipatch

simulations with a stationary (moving) patch. Top panels: Processing speed in zone-cycles per processor per second. Bottom

panels: Fraction of total wall-clock time spent on communication.

structed a solution to this problem—a client-router-server framework—that updates these connections ef-ficiently. When the patches are stationary relative toone another, the connections need to be identified onlyonce, so the overhead due to multipatch operations isfairly small, especially for larger numbers of cells perprocessor. When they move, the overhead is more sig-nificant and scales with the number of processors perpatch, producing a reduction in cell-update rate of abouta factor of ⇠ 1.4 for 512 processors per patch or afactor of ⇠ 2 for 1728 processors per patch when us-ing 403 cells per processor. We note, however, thatthese comparisons assume that monopatch and multi-patch approaches use the same total number of cells;because multipatch operation permits tuning the gridto match local requirements, in practise multipatch sim-ulations may use a much smaller total number of cellsthan would be required for a monopatch simulation ofthe same problem—if a monopatch simulation could deal

with the problem at all.Many extant fluid codes are automatically consistent

with this infrastructure. Its sole substantive stipulationis that the dependent variables involved in boundarydata exchange should be consistent in all patches. Al-though we were motivated to build this system by rel-ativistic problems, and our transformation methods arefamiliar because of their frequent application to rela-tivistic dynamics, in fact they really stem from moregeneral considerations of di↵erential geometry; theytherefore apply to any context in which scalars, vectors,and tensors can be defined.PATCHWORK may be refined and extended, both

in terms of its computational e�ciency and the spanof physical problems on which it can be used. Theamount of time spent on interpolation and inter-patchdata transmission can be reduced by minimizing thenumber of arrays transferred or by eliminating unnec-essary steps in the coordinate transformations. Mov-

Weak & Strong Scaling Performance (Stampede scaling shown, BW scaling in progress)

Products: • Intend to publicly release

Patchwork code eventually as soon as possible.

Blue Waters Team Contributions: Visualizing Accreting SMBBHs

All Images by M. Vanmoer

M. Van Moer, S. Noble, and R. Sisneros. "Orbiting Black Holes Magnetohydrodynamics." Presented at the Visualization Showcase. XSEDE16 Conference on Diversity, Big Data, and Science at Scale. Miami, USA. July 17-21, 2016.

Broader Impacts & Blue Waters Team Contributions:Educating the Next-generation of Scientific Visualizers

• With Mark Van Moer and Roberto Sisneros (NCSA), Manuela Campanelli (RIT), S. Noble (Tulsa);

• Undergraduate Students: • Laura Moon (Chemical Eng., RIT) • Brennan Dell (Software Eng., RIT)

• Goals: • Provide advanced scientific visualization

education to undergraduate students, using Visit and Python.

• Discover the particular distribution, topology and evolution of magnetic field lines near accreting binary black holes.

• Advance current practices on animating the same set of field lines continuously in time.

• Student Results: • Learned how to use Python scripts to

interface with Visit and control camera position/movement, and field line seed point locations.

• Have begun exploring how to advect the seed points in time to maintain field line continuity.

All Images by L. Moon and B. Dell (RIT)

PAID Project: Load Balancing with HARM3dTeam Leads: Sanjay Kale, S. Noble

Team: Spencer James (Tulsa), Sam White (UIUC), Juan Galvez (UIUC), BW ID: DD_gku

(see my PAID talk Thursday afternoon)

Blue Waters Team Contributions:

Key Challenges: • Nonuniform FLOP-count over domain due to harder spacetime metric evaluation near BHs

(SMBBHs), or multi-patch evolutions. • Include topology awareness. • Use Charm++/AMPI tools to provide solution within a year’s time.

• Thread Local Storage solution proved incompatible with migrating HARM3d; • Implemented “work around” using AMPI’s “Pack-UnPack” framework to explicitly instruct

how data is migrated, allocated, and deallocated during migrating virtual processes. • Required moving global variables into a dynamically allocated structure which is passed in

function arguments. • Rewrite resulted in insignificant performance impact. • Validated (load balanced) single black hole disk configuration. • Verified success of AMPI’s load balancer with artificially imbalanced single BH disk

configuration, leading to ~50% SU savings. • Currently testing MHD binary black hole configuration.

Accomplishments:

Summary & ConclusionsKey Science Achievements:• First of a kind GRMHD mini-disk simulations of inspiral SMBBHs;

• BW is essential for this project; • Have reached critical mass of person-power —> great things to come as long

as the SUs are there: • Dynamic GR radiative transfer; • Spinning/precessing/merging binaries; • Relaxation of circumbinary disk, long-term signature of merger;

• Developed Patchwork code for future public use; • Contributing gravitational waveforms and improved parameter estimation formulae

to the LIGO Scientific Community;

Benefits of Blue Waters:• Necessary SUs and reliability for the job; • Helpful staff and resources (e.g., queue througput, PAID); • Expert and enthusiastic visualization staff for creating innovative scientific imagery and

animations; • PAID program provides a rare resource for focusing effort on code improvements;

Optical Depth, Side View in Orbital Plane

Time-dependent Post-process GR Radiative Transfer of SMBBH

Teaser trailer for next year:

by Stephane d’Ascoli (RIT)