Overview of Broadband Platform Software as used in SWUS Project

Philip MaechlingBBP Modelers Meeting12 June 2013

SCEC Software Activities on BBPSCEC software responsibilities on project include:

1. Integration of scientific methods into Broadband Platform

• Added Irikura, Composite Source Model, EXSIM in 2013

• Added multiple Part A and Part B reports

• Added GMPE codes and utilities (e.g. rotD50, pynga)

2. Running BBP simulations using SCEC computing resources and posting simulation results

• Ran full Part A and Part B (BBP v13.5) for most methods

• Ran portions of Part A and Part B (BBP v13.6) for most methods

Define the BBP PlatformThe SCEC BBP Platform (BBP) integrates:

• scientific software (e.g. C and Fortran)

• formatting scripts (e.g. python)

• seismic software utilities (e.g. rotd50, pynga)

• post-processing plotting routines

• observational data (e.g. seismograms)

• expected results (e.g. seismograms, Peak SA values)

Standard BBP Inputs and OutputsStandard Inputs:• Earthquake Source File (.src file)• Station List (.stl file)• Velocity model specification (region) • Computational Method Each method has region specific input files (e.g. Greens functions)

Standard outputs: • Velocity seismograms• Acceleration seismograms• Standard Rupture Format (4 methods), • Metadata list of programs and input files used • Metadata compute environment information

Running the BBP Platform

'/home/epicenter/fsilva/bbp/13.6.0/comps/run_bbp.py', '-x', '/home/epicenter-03/fsilva/june13/gp-nps-50r/Xml/northpalmsprings-gp-0000.xml', '-s', '10000000', '-l', '/home/epicenter-03/fsilva/june13/gp-nps-50r/Sims/logs/10000000/10000000_northpalmsprings-gp.log'

Sim Specification FileName: Simulation_ID.xml (e.g. 10000049.xml)

Example Simulation Spec for Part A Validation Simulation:

1. Genslip $BBP_INSTALL_GF/Mojave/gp/mojave_generic1d-urs01.vmod /home/epicenter-03/fsilva/june13/gp-nps-50r/Src/northpalmsprings-gp-0049.src mojave_generic1d-urs01.vmod northpalmsprings-gp-0049.src metadata.txt northpalmsprings-gp-0049.srf Mojave

2. Jbsim $BBP_INSTALL_GF/Mojave/gp/mojave_generic1d-urs01.vmod /home/epicenter-03/

3. Hfsims $BBP_INSTALL_GF/Mojave/gp/mojave_generic1d-urs01.vmod /home/epicenter-03

4. Plot_Map /home/epicenter-03/fsilva/june13/gp-nps-50r/Src/northpalmsprings-gp-0049.src

5. PlotSeis $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl northps_v13_3_1.stl True

6. RotD50 $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl northps_v13_3_1.stl GMPEComparison $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl /home/

7. GenPlots $BBP_INSTALL_VAL/NORTHPS/gp/northps_v13_3_1.stl northps_v13_3_1.stl

8. AccPEER3 acc 6.12 NORTHPS 90 False GenHTML $BBP_INSTALL_VAL/NORTHPS/gp/

BBP Software Version Strategy

• As a general principle, the newest version of the broadband platform is better than any previous version and users should upgrade at first opportunity.

• Older versions of the code will be archived to support reproducibility of previously obtained results.

• Any use of an older version of the platform should be examined carefully to understand the scientific goals of using an older version.

Versioning BBP ResultsBBP results should be attributed to specific BBP software and configuration versions because results may change as methods improve.

For a Part A Validation Result, changes to any of these three versions that may impact the results:

Broadband Version 13.6.0-30-May-2013- This is the source code version

Velocity model version LABasin - 29-May-2013-1.07- This is the region specific input parameters used in the simulation

Validation package version NR - 29-May-2013-1.05- This is the set of observational data and associated corrections used in the GOF plots for this validation simulation

These are found in the index.html file in the outdata directory for each simulation.

Planned BBP DistributionsPlans to create 2 BBP Distributions SWUS (Complete) Standard (no obs data):

SWUS Distribution1. Broadband Version 13.6.0-30-May-20132. Velocity model version LABasin - 29-May-2013-1.073. Validation package versionNR - 29-May-2013-1.05

Standard Distribution:.4. Broadband Version 13.6.0-30-May-20135. Velocity model version LABasin - 29-May-2013-1.07

Comparing Results Between BBP Software Versions

When comparing BBP results, all results should be produced with the same version of the broadband platform. Standard problems should be re-run when new versions of the platform are released.

We assume that:• Post processing changes affect results for all methods.• Changes to any method’s source code, including python wrappers, may affect

results from other methods• Changes to one method parameters do not affect results from another method

We make this assumption to simplify our analysis of results:• We do not have the resources to detect all possible sources of coupling

between scientific programs in broadband platform• This assumption can be relaxed, if additional analysis is done to confirm results

are equivalent between BBP software releases.

Results From Two Software Versions

We have posted results. These should be considered results from BBP v13.6.0

Results are posted under two studies:• http://bbpvault.usc.edu/bbp/BBP_June2013_Study/

• http://bbpvault.usc.edu/bbp/BBP_May2013_Study/

When accessing these results:• Use v13.6 results for any simulation if the results exist.• If no v13.6 results exist, use v13.5 results.• If no v13.5 results exist, simulation was not completed.

BBP Simulation ResultsPosted Results are Organized by Release, Method, Sim Type

• http://bbpvault.usc.edu/bbp/BBP_June2013_Study/

• http://bbpvault.usc.edu/bbp/BBP_May2013_Study/

List of Methods: Within Method (GP):

csm/ gmpe/

exsim/ landers/

gp/lomap/

irikura/ niigata/

sdsu/nps/

ucsb/nr/

whittier/

tottori/

BBP Results DirectoryResults Organized in two Base Directory:

• http//bbpvault.usc.edu/bbp/BBP_June_2013/

• http//bbpvault.usc.edu/bbp/BBP_June_2013/

Part A Results:

• Method Name: [csm,exsim,gp,irikura,sdsu,ucsb]

• Event Name: [landers,lomap,niigata,nps,nr,whittier,tottori]

• Simulation ID: [10000000, 10000049]

http://bbpvault.usc.edu/bbp/BBP_June2013_Study/gp/landers/outdata/10000049/

Absolute pathname carries information about what method was used and what event was used to produce the simulation results contained in a directory. In each simulation directory, a simulation specification file (sim_id.xml) provides detailed information about the programs run to produce the results in the directory.

Part A Results Directory Listing:

http://bbpvault.usc.edu/bbp/BBP_June2013_Study/gp/landers/outdata/10000049/index-10000049.html

BBP Output (Results) DirectoryResults Organized in two Base Directories:

• http//bbpvault.usc.edu/bbp/BBP_June_2013/

• http//bbpvault.usc.edu/bbp/BBP_June_2013/

Part B Results:

• Base File location: Study Name/Method Name/gmpe/

• http://bbpvault.usc.edu/bbp/BBP_June2013_Study/gp/gmpe

• Filename formed from: [method, mag, mechanism, region, dist]

• bp-gmpe-ucsb-62-ss-nocal-20km.pdf

• bp-gmpe-ucsb-66-ss-socal-50km_w_gmpe.pdf

BBP Metadata FilesSimulation Directories Metadata Files:• index-10000049.html• 10000049.xml• system_status-10000049.txt• software_status-10000049.txt

Example: software_status-10000049.txt• Python version: 2.7 (r27:82500, Sep 16 2010, 18:02:00) • GCC Version: gcc (GCC) 4.5.1 20100924 (Red Hat 4.5.1-4)• ICC Version: icc (ICC) 12.0.4 20110427• IFORT Version: ifort (IFORT) 12.0.4 20110427• G77 Version: GNU Fortran (GCC) 3.4.6 20060404 (Red Hat 3.4.6-20.fc14)• GFORTRAN Version: GNU Fortran (GCC) 4.5.1 20100924 (Red Hat 4.5.1-4)• NumPy Version: 1.4.1• SciPy Version: 0.7.2• Matplotlib Version: 1.0.1

BBP Seismogram FilesEXSIM:BBP file with 2 header lines. dt: 0.005000 (200 sps) npts: 32768 (163.84 secs) CSM:BBP file with 2 header lines: dt: 0.020000 (50 sps) npts: 4096 (81.92 secs) GP:BBP file with 1 header lines. dt: 0.025000 (40 sps) npts: 4096 (102.4 secs) Irikura:BBP file with 1 header lines. dt: 0.008333 (120.004 sps) npts: 14400 (119.9952 secs) UCSB:BBP file with 1 header lines. dt: 0.020000 (50 sps) npts: 8192 (163.84 secs) SDSU:BBP file with 1 header lines. dt: 0.003124 (320 sps) npts: 32770 (102.373 secs)

BBP Compute and Storage EstimatesEstimates for Part A and Part B Simulations Method

Compute Hours Outputs (GB) Outdata (GB)

GP 50 620 19

SDSU 150 620 43EXSIM 36 500 40CSM 71 390 9Irikura 52 472 20UCSB 150 620 7

609 * 32 CPUs19,488 CPU Hrs 3222 GB

137 GB

Estimate per Part A and Part B runs 20k SU (3.2 TB (Total) , 0.2 TB (Outdata))

BBP Compute and Storage EstimatesEstimates for Fling Study Method

Compute Hours Outputs (GB) Outdata (GB) GP 1416

??? 1601416* 16 CPUs = 22656

Assume all 6 methods run Fling

22656 * 6 = 135936 160* 6 = 960 (GB)

Cumulative Totals (Part A & Part B + Fling) For 6 Methods (Outdata Only): (20k + 135k) = 155k CPU hours (200 + 960) =

1.2 TB Storage

BPP Software PersonnelBBP Developers by Method:

• GP – Robert Graves, Arbin Pitarka• Composite Source Model – John Anderson• SDSU – Kim Olsen, Rumi Takedatsu• UCSB – Jorge Crempien, Ralph Archuleta• EXSIM – Karen Assatourians , Gail Atkinson• Irikura – Jeff Bayless, Yajie (Jerry) J. Lee, Paul Somerville

BBP Software Developers At SCEC:

• Fabio Silva• Philip Maechling• Scott Callaghan• David Gill

End