CyberShake Study 2.3 Science Readiness Review

Study re-versioning

SCEC software uses year.month versioning Suggest renaming this study to 13.4

Study 13.4 Scientific Goals

Compare Los Angeles-area hazard maps― RWG V3.0.3 vs AWP-ODC-SGT (CPU)― CVM-S4 vs CVM-H

● Different version of CVM-H than previous runs● Adds San Bernardino, Santa Maria basins

286 sites (10 km mesh + points of interest)

Proposed Study sites

Study 13.4 Data Products

2 CVM-S4 Los Angeles-area hazard maps 2 CVM-H 11.9 Los Angeles-area hazard maps Hazard curves for 286 sites – 10s, 5s, 3s

― Calculated with OpenSHA v13.0.0 1144 sets of 2-component SGTs Seismograms for all ruptures (about 470M) Peak amplitudes in DB for 10, 5, 3s Access via CyberShake Data Product Site (in

development)

Study 13.4 Notables

First AWP-ODC-SGT hazard maps First CVM-H 11.9 hazard maps First CyberShake use of Blue Waters (SGTs) First CyberShake use of Stampede (post-

processing) Largest CyberShake calculation by 4x

Study 13.4 Parameters

0.5 Hz, deterministic post-processing― 200 m spacing

CVMs― Vs min = 500 m/s― GTLs for both velocity models

UCERF 2 Latest rupture variation generator

Verification work

4 sites (WNGC, USC, PAS, SBSM)― RWG V3.0.3, CVM-S― RWG V3.0.3, CVM-H― AWP, CVM-S― AWP, CVM-H

Plotted with previously calculated RWG V3 Expect RWG V3 slightly higher than the others

WNGCCVM-S CVM-H

RWG V3.0.3 - GreenAWP - Purple

RWG V3 - Orange

USCCVM-S CVM-H

RWG V3.0.3 - GreenAWP - Purple

RWG V3 - Orange

PASCVM-S CVM-H

RWG V3.0.3 - GreenAWP - Purple

RWG V3 - Orange

SBSMCVM-S CVM-H

RWG V3.0.3 - GreenAWP - Purple

RWG V3 - Orange

SBSM Velocity Profile

SGT Computational Requirements

SGTs on Blue Waters Computational time: 8.4 M SUs

― RWG: 16k SUs/site x 286 sites = 4.6 M SUs ― AWP: 13.5k Sus/site x 286 sites = 3.8 M SUs― 22.35 M SU allocation, 22 M SUs remaining

Storage: 44.7 TB― 160 GB/site x 286 sites = 44.7 TB

PP computational requirements

Post-processing on Stampede Computational time:

― 4000 SUs/site x 286 sites = 1.1 M SUs― 4.1 M SU allocation, 3.9 M remaining

Storage: 44.7 TB input, 13 TB output― 44.7 TB of SGT inputs; will need to rotate out― Seismograms: 46 GB/site x 286 sites = 12.8 TB― PSA files: 0.8 GB/site x 286 sites = 0.2 TB

Long-term storage

44.7 TB SGTs:― To be archived to tape (NCSA? TACC?

Somewhere else?) 13 TB Seismograms, PSA data

― Have been using SCEC storage - scec-04? 5.5 TB workflow logs

― Can compress after mining for stats CyberShake database

― 1.4 B entries, 330 GB data (scaling issues?)

Estimated Duration

Limiting factors:― Blue Waters queue time

● Uncertain how many sites in parallel― Blue Waters → Stampede transfer

● 100 MB/sec seems sustainable from tests, but could get much worse

● 50 sites/day; unlikely to reach

Estimated completion by end of June

Personnel Support

Scientists― Tom Jordan, Kim Olsen, Rob Graves

Technical Lead― Scott Callaghan

Job Submission / Run Monitoring― Scott Callaghan, David Gill, Phil Maechling

Data Management― David Gill

Data Users― Feng Wang, Maren Boese, Jessica Donovan

Risks

Stampede becomes busier― Post-processing still probably shorter than SGTs

CyberShake database unable to handle data― Would need to create other DBs, distributed DB,

change technologies Stampede changes software stack

― Last time, necessitated change to MPI library― Can use Kraken as backup PP site while resolving

issues

Thanks for your time!