1 project achievementshypocenter.usc.edu/research/incite/incite_2016/scec... · 2015-07-31 · 1...

High Frequency Ground Motion Simulations for Seismic Hazard Analysis Lead PI: Jordan

1

1 PROJECT ACHIEVEMENTS

The Southern California Earthquake Center (SCEC) conducts a broad program of earthquake system science that seeks to develop a predictive understanding of earthquake processes with a practical mission aimed at providing society with improved understanding of seismic hazards. The SCEC Community Model Environment (SCEC/CME) research collaboration develops physics-based computational models of earthquake processes, and performs large-scale earthquake system science research (Jordan et al., 2003) calculations. The SCEC/CME earthquake system science computational research integrates new scientific advances and new technical capabilities into broad impact seismic hazard data products. We develop physics-based predictive models of earthquake rupture and ground motion processes and improve existing seismic hazard computational methods by integrating new, computationally intensive, methods. The SCEC/CME research collaboration involves seismologists, computer scientists, structural geologists, and engineers working to transform seismic hazard analysis into a physics-based science through HPC implementations of computational pathways (Figure 1). Each of these computational pathways represents a research activity that can contribute to improved earthquake ground motion forecasts. The pathways are interdependent, and we are iteratively improving our capabilities in each area, and then integrating changes together into improved system models.

Our 2015-2016 INCITE proposal defined the following four objectives: O1: Improve the resolution of dynamic rupture simulations by an order of magnitude and investigate the effects of realistic friction laws, geologic heterogeneity, and near-fault stress states on seismic radiation. O2: Extend deterministic simulations of strong ground motions to 10 Hz for investigating the upper frequency limit of deterministic ground-motion prediction. O3: Compute physics-based Probabilistic Seismic Hazard Attenuation (PSHA) maps and validate those using seismic and paleoseismic data. O4: Improve 3D earth structure models through full 3D tomography using observed seismicity and ambient noise, and through more complete representations of earth material properties including energy losses. During the Project Year 1 in our 2-year INCITE allocation, we made substantial progress towards these objectives. The highlights of the significant achievements are as follows:


2

a) We defined and validated a new central California 3D velocity model, that we call Central California

Area (CCA), and used full 3D tomography computational methods to improve the 3D seismic velocity model based on both observed moderate earthquakes and ambient seismic noise observations. As of July 2015, we have completed five iterative refinements to the starting 3D velocity model, substantially improving our ability to model seismic wave propagation across the region.

b) We performed high-frequency simulations (out to 5 Hz) on the OLCF Titan supercomputer using

GPU-optimized finite-difference and finite-element codes that include frequency-dependent attenuation, small-scale near-surface heterogeneities, tomography, and a nonlinear dissipation in the near-fault and near-surface regions. These simulations set the stage for the ground motion prediction modeling at frequencies beyond 1 Hz.

c) We completed a first 1-Hz urban seismic hazard model for the Los Angeles region. The new model,

which comprises more than 300 million synthetic seismograms sampling the Uniform California Earthquake Rupture Forecast, was computed from a new high-resolution image of crustal structure derived using full-3D tomography (CVM-S4.26) on ALCF Mira. It will be registered into the USGS Urban Seismic Hazard Mapping Project, and the results will be submitted for use in the 2020 update of the Recommended Seismic Provisions of the National Earthquake Hazards Reduction Program.

d) The earthquake wave propagation GPU software development of AWP-ODC led by co-PI Dr. Yifeng

Cui was recognized with NVIDIA’s 2015 Global Impact Award that are benefiting our production CyberShake Study 15.4 run on Titan.

1.1 Significance of Accomplishments to Date The San Andreas fault system is prone to major earthquakes, yet Los Angeles has not experienced a major quake since its urbanization in the early twentieth century. Data for the region are available from smaller quakes, but such information doesn’t give emergency officials and structural engineers the information they need to prepare for an earthquake of magnitude 7.5 or bigger. CyberShake Study 15.4 represents a major milestone in physics-based hazard analysis for Southern California. The performance of the code and improved workflow management, combined with the new physics it models (e.g., fault roughness, small-scale heterogeneities, frequency-dependent attenuation, near-surface nonlinearities), take physics-based seismic hazard analysis to a new level and pioneer the use of Petascale heterogeneous computing resources for ground motion simulations used in building engineering design and evaluation. The reduction of peak velocities in our models caused by mostly shallow, near-fault nonlinear effects may have important implications for the scaling of ground motion intensities between surface-rupturing and buried earthquakes. Our nonlinear simulation results show that nonlinearity in the fault zone is important even for conservative values of cohesion, suggesting that current simulations based on a linear behavior of rocks are over-predicting the level of ground motion in the Los Angles sedimentary basins during future large earthquakes on the southern San Andreas Fault, and possibly for other large earthquake scenarios. This will have far-reaching implications on earthquake emergency planning scenarios that are based on ground motions predictions, such as the damage scenario of the 2008 Great California ShakeOut. The addition of statistical models of near-surface small-scale heterogeneities has enabled us to capture the “within-event” variability of earthquakes more accurately, providing models that can be used to improve physics-based seismic hazard analysis. a) We inverted ambient-field correlagrams to obtain a new central California 3D velocity model


3

The SCEC research team applied our full 3D tomographic (F3DT) method using ALCF Mira to produce a 3D seismic velocity model for the Central California Area (CCA). Realistic ground-motion simulations require highly accurate crustal structural models. The purpose of the F3DT for central California is to improve the crustal 3D seismic velocity model for central California. An accurate 3D velocity model is essential input needed for accurate deterministic earthquake wave propagation simulations. We established our F3DT methodology under previous INCITE awards, using ALCF Mira to construct full-3D, high-resolution crustal seismic velocity models in southern California (Lee et al. 2014ab). F3DT represents the latest development in seismic tomography techniques. Its application to seismic data recorded in Southern California has yielded a new community velocity model for the region, CVM-S4.26, which has unprecedented resolution of crustal structure. CVM-s4.26 is the 3D structural model used in the CyberShake 15.4 study. During the Project Year 1 of this INCITE award, we have further improved the computational efficiency of our F3DT workflow on Mira and we are now able to apply our F3DT method to Central California and statewide. As of July 2015, we have carried out 5 F3DT iterations for Central California and 3 F3DT iterations for the statewide California. Our improved Central California velocity model provides substantially better fit to over 12,000 seismic waveforms at frequencies up to 0.2 Hz and shows interesting small-scale structures in the upper to mid crust that agree with local geology and other independent geophysical evidence. Our latest statewide velocity model significantly improves the fit to over 27,000 waveforms at frequencies up to 0.1 Hz, and it has revealed new structural features in the mid to lower crust that are consistent with our understanding of the geotectonic development in California. More F3DT iterations will be carried out for both Central California and statewide. Gradual improvements in our velocity models have allowed us to incorporate an increasing volume of observed

seismograms into our F3DT workflow, which is allowing us to resolve finer structural details with higher accuracy. The map in Figure 2 shows the bounding box of this new velocity model. The purpose of the F3DT for central California is to improve the crustal velocity model in central California for more accurate ground motion predictions. Our initial model is based on the updated Community Velocity Model for Southern California, CVM-S4.26 (Lee et al., 2014), and other existing velocity models for northern California (Xu et al., 2013). Our study area is located on the edges of the two models, where the data coverage is poor for the two models. In the following sections, we provide details about our advances in full 3D tomography on Mira during the first year of our current allocation. The 5th full-3D tomography (F3DT) iteration for Central California crust. During the first six months of 2015, we have completed the 5th iteration for the Central California crustal F3DT. In this iterative process, we used the available ambient noise Green’s functions (ANGFs) among the broadband and short-period stations to invert the velocity model. We applied two bandpass filters to the ANGF waveforms to separate


4

the high (0.1~0.18Hz) and low (0.03~0.1Hz) frequencies sources. In our 5th iteration, more than 12,000 waveform windows have been picked and more than 59,000 frequency-dependent measurements have been made for the Central California tomographic inversion. To evaluate the waveform improvement, we measured the difference between observed µk (t) and its corresponding synthetic µk (t) waveforms within the time window [tk, tk'] by the relative waveform misfit (RWM) statistic, defined by the integral

RWM =[µk (t)−µk (t)]

2 dttk

tk1

∫

µk (t)2 dt µk (t)

2 dttk

tk1

∫tk

tk1

∫

Here, the time [tk, tk'] for the kth waveform window runs from the starting and end time of the window. After our 5 tomographic inversions, the sum of RWM has reduced more than 28% when compare to that of the initial model (Figure 3a (left)). In addition, the iteratively inversions have reduced the variance of frequency dependent group delay measurements (dtg) by over 35% relative to the starting model (Figure

3b (right)). The perturbations in velocities have begun to heal the velocity artifacts inherited from the starting model (Figure 4). Many features revealed in the model are consistent with independent geophysical observations in Central California, including controlled-source tomography, gravity anomalies, and the locations of active faults. Full-wave centroid moment tensor (CMT) inversion in an updated 3D velocity model for earthquakes in Central California. To include the earthquake recordings in the next iteration, we have applied a full-wave Central Moment Tensor (CMT) inversion algorithm to more than 200 earthquakes recorded in Central California (Figure 5). The procedure relies on the use of receiver-side Green tensors (RGTs), which comprise the spatial-temporal displacements produced by the three orthogonal unit impulsive point forces acting at the receiver. We have constructed a RGT database for more than 180 broadband stations in Central California using the updated Central California F3DT to reduce the potential errors in velocity structures. In our CMT inversion method, we implement the Bayesian inference on our measurements. An important advantage of the Bayesian approach is that, instead of a single best solution, the complete posterior probability density on the sample space is obtained, which allows formal estimation of the uncertainties associated with the derived source parameters. More robust earthquake source parameter estimates are critical for both geologic interpretation of active faults and seismic hazard analysis.


5

Validation of Central California F3DT using earthquake recordings not used in previous inversions. To validate the updated Central California F3DT, we tested the waveform predictions of the model using the earthquake recordings not used in previous tomographic inversions. More than 11,500 three-component broadband seismograms with signal-to-noise ratio (SNR) larger than 4 has been used in this validation (Figure 6). All synthetic and observed seismograms were band-pass filtered using a Butterworth filter with corners at 0.02 Hz and 0.2 Hz. We measured RWM between an individual observed seismogram µk (t) and its corresponding synthetics µk (t) of the initial model (CCA00) and the updated model (CCA05). The time window for the waveforms are from the first arrival to the end of the main surface wave group, so that RWM measures the net waveform difference across all of the main phases on the seismograms. In those comparisons, the synthetics computed using the updated velocity model (CCA05) provide better fit to observed seismograms at frequencies below 0.2 Hz than those computed using the initial model (CCA00) (Figure 6).


6

b) We performed high-frequency simulations (out to 5 Hz) on the OLCF Titan. The SCEC finite-element wave propagation solver, Hercules, which integrates an efficient octree-based hexahedral mesh generator with an explicit FE formulation, has been optimized on Titan this year achieving near perfect strong and weak scaling. Its GPU capabilities are currently being used in verification and validation studies for the 2014 Mw 5.1 La Habra earthquake on Titan, to test the accuracy of the code compared to other codes, and to examine how close the predicted ground motions are to observations. We have implemented non-associated Drucker-Prager nonlinear rheology following the return map algorithm in the scalable AWP-ODC code, and we have used this code to model ground motions from the M7.8 ShakeOut scenario source description (Figure 7). This work accounts for the limited strength of crustal rocks; i.e., to simulate the absorption of rupture energy by permanent rock deformation. Our results suggest that this nonlinear behavior could reduce previous simulation-based predictions of expected ground motion velocity in the Los Angeles basin during a large-magnitude event on the southern San Andreas Fault by 30 to 70 percent. Nonlinear material response occurs in soft soils near the surface, typically reducing high-frequency (> 1 Hz) shaking that controls damage to low- and mid-rise buildings. Our simulations show that nonlinear response in crustal rocks may also reduce the amplitudes of long-period surface waves that pose a hazard to high-rise buildings, implying less destruction than previously anticipated. Although more research will be needed to quantify the impact of these findings on damage and casualty estimates for future large-magnitude earthquakes on the San Andreas Fault, the study pioneers more accurate earthquake scenarios based on better representations of the nonlinearity in the Earth's crust.

We have implemented realistic attenuation structure (frequency-dependent Q, or Q(f)) in the GPU-based AWP-ODC code (Withers et al., 2015). Tests using the 2008 Mw 5.4 Chino Hills earthquake indicate that Q(f) generally fits the strong motion data better than for constant Q models for frequencies over 1-Hz, which becomes more and more important as the distance increases from the fault. We also found that media heterogeneity reduces the within-event variability to that for observations and is thus important to characterize the ground motion.


7

c) We completed a 1-Hz urban seismic hazard model for the Los Angeles region. SCEC's research team used the OLCF Titan and NCSA Blue Waters supercomputers to perform CyberShake Study 15.4 initiated in April, 2015 and completed within 38 days, before end of May, 2015. This computation produced a Los Angeles region seismic hazard model (Figure 8a-8b) that doubled the maximum seismic frequency represented in the Los Angeles urban seismic hazard model, from 0.5 Hz to 1 Hz. Seismic hazard curves were derived from large ensembles of seismograms at frequencies below this maximum for 336 surface sites distributed across the Los Angeles region. This new probabilistic model uses refined earthquake rupture descriptions through revisions to the conditional hypocenter distributions and the conditional slip distributions. This seismic hazard calculation used the CVM-S4.26 3D velocity model, which was validated and improved using ALCF Mira, as the best available southern California 3D velocity model.

In order to complete our first 1Hz CyberShake simulations within weeks, rather than months, we divided the computational work between OLCF Titan and NCSA Blue Waters. We defined the distributed calculation using scientific workflows that automatically executed our parallel GPU intensive calculations on OLCF Titan, executed GPU parallel jobs and CPU-based post-processing on Blue Waters, and transferred scientific data products back to SCEC systems for visualization and archiving. Combined uses of both systems enable us to complete a CyberShake hazard model within the practical operational limits of our research group. We ran a previous CyberShake study 14.3 using only Blue Waters. Adding OLCF Titan resources enabled us to complete a 1Hz CyberShake hazard model for the first time by proving timely access to large number of GPUs, and support for automated, distributed end-to-end scientific production simulation projects. The CS15.4 model produced this year provides new seismic hazard information of interest to broad impact customers of CyberShake, including seismologists, utility companies, and civil engineers


8

responsible for California building codes. The new model, which samples the complete Uniform California Earthquake Rupture Forecast, will be registered into the USGS Urban Seismic Hazard Mapping Project. The GPU-based anelastic wave propagation AWP-ODC software was used to run CPU-based post-processing calculations that synthesized over 300 million seismograms. In Study 15.4, SCEC utilized approximately 440 pilot jobs of up to 8000 nodes to run CyberShake tasks on Titan resources. We integrated Titan into our workflow management system to automatically submit requests for pilot jobs based on the demand of CyberShake workflows. This enabled us to provision Titan resources for our workflow tasks to execute on. In this way, we could execute CyberShake on Titan while utilizing our workflow tools to manage job execution, manage data, and provide fault tolerance. Over 80% of the node-hours burned on Titan were from jobs that ran on 25% or more of the machine. Approximately 200 TB of SGT data was transferred from Titan to Blue Waters automatically as part of the workflow. On Titan, the accelerated calculations of the GPU Strain Green Tensor (SGT) implementation are 6.3 times more efficient than the CPU implementation, which saved us 2 million node-hours over the course of the study. d) Our GPU development was recognized with NVIDIA’s 2015 Global Impact Award. The GPU-accelerated code Cui and his team developed, known as AWP-ODC, ran an estimated five times faster on Titan’s GPUs than it would have on traditional architecture. The code was originally developed by Kim Olsen and Steven Day at SDSU and is used by SCEC to improve its CyberShake platform, an ongoing sweep of millions of earthquake simulations that model many rupture sites across California. In the press coverage for this award, SCEC Director and INCITE project PI Tom Jordan discussed the impact of the GPU code as follows “The full three-dimensional treatment of seismic-wave propagation has the potential to improve seismic hazard analysis models considerably, and that is where the accelerating technology is particularly helpful at this moment. With GPU computing power we’re gaining insight as to how the ground will move in high-risk areas, and how we can better plan for the aftermath of a major event.”

1.2 Allocation Use

The following usage numbers were collected from INCITE resources on July 27, 2015. At ALCF, we have used only the Mira system so far. Year 1 Allocation: Granted (48.0M) Used (55.6M) Remaining (-7.6M). With 57% of the Year 1 Allocation time completed, we have used 116% of our awarded computing time on Mira. This is a result of the active computational process used by our Mira research team, which aims to compute one complete tomographic inversion iteration per month. Although we have used our requested computing hours, we expect to refine our existing 0.2Hz model with one, or two, additional full 3D inversion iterations using Mira back-fill queues during the remainder of Year 1. At OLCF, we have used only the Titan system so far. Year 1 Allocation: Granted (119.0M) Used (43.9M) Remaining (75.1M). With 57% of Year 1 Allocation time completed, we have used 37% of our awarded computing time on Titan. A significant portion of our unused computing time can be attributed to the efficiency and low computing cost on Titan required to calculate a 1Hz CyberShake hazard model. In our Year 1 computing estimates, we overestimated the computation cost of our first 1Hz CyberShake hazard map. After completing this computational goal earlier this year, we were able to achieve our CyberShake 1Hz research results at lower computational cost due to several reasons including good GPU code efficiency on Titan, and the ability to use Blue Waters for a significant share of the CyberShake CPU and GPU calculations. Depending on personnel status, we would like to perform a second 1Hz CyberShake hazard model calculation using Titan before end of our year 1 allocation. For this second CyberShake


9

hazard model, our technical goal is to reduce our computational time by running the full calculation on Titan. We will work with our OLCF catalyst to evaluate the technical possibilities and potential benefits for a full CyberShake on OLCF Titan. In addition, we have a suite of high-F simulations for the Northridge earthquake up to 8 Hz planned for the 2nd half of the year, expected to consume a significant amount of the remaining SUs on Titan.

1.3 Application Parallel Performance

In 2010, the CPU-based AWP-ODC code has demonstrated super-linear strong scaling speedup on the full Jaguar system. This highly optimized code was used to conduct a full dynamic simulation of a magnitude-8 earthquake on the southern San Andreas fault up to 2-Hz with 220 Teraflop/s sustained performance using 223K Jaguar XT5 cores at the time (Cui et al., 2010). This code is recently measured with 653 Teraflop/s on Cray XE6 Blue Waters. AWP-ODC also demonstrated outstanding performance on ALCF Mira, and achieved 104 Teraflop/s on 32,768 Mira cores. The benchmarks were based on pure MPI code without hybrid or QPX implementation. AWP-ODC-GPU has achieved perfect weak scalability on Titan, achieving a sustained 2.3 Petaflop/s and 100% parallel efficiency up to 8,192 GPUs. The GPU-powered SGT code attained a performance improvement of a factor of 3.7 on XK7 compared to the CPU code running on XE6 at a node-to-node level. Hercules has historically shown very good strong and weak scalability on various HPC systems such as NICS’s Kraken, NCSA’s Blue Waters, and ALCF Mira. As part of the previous 2014 INCITE allocation we implemented GPU capabilities on Hercules and tested it on Titan, where the code exhibited the same level of excellent performance as in other systems. In the first part of 2015 we completed the implementation of a module in Hercules to measure performance on Titan and identify development candidate issues. The GPU solver demonstrated very good scalability on Titan with a speedup of 2.5X over the CPU solver, indicating that it is a cost effective code for this system. In year 2 we expect to used Hercules to produce various 8-Hz simulations with unstructured meshes of up to ~70 billion finite-elements. We made use of our Pegasus-WMS workflow tools on Titan for the first time. In the process of running our CyberShake 15.4 study, our workflow tools automatically provisioned an average of 660 nodes, with a maximum of 14874 nodes, 80% of Titan. To reduce our queue times, we bundled multiple workflow tasks into a single pilot job, so each pilot job executed up to 4 tasks from each of 3 workflows. On average, the pilot jobs supported execution of 6 workflow tasks during the duration of the study.

1.4 Data Storage

CyberShake Study 15.4 calculations on Titan produced approximately 215 TB of temporary files. Another 200 TB of output SGT data was generated that was then transferred to NCSA Blue Waters for CPU post-processing. To reduce our scratch usage, we used a cleanup feature of the workflow tools to delete our temporary and output data once the transfer was successfully completed, which enabled us to stay under our quota of 200 TB. We have chosen not to archive our output SGT data, and will regenerate it instead should we need it in the future. The large storage capabilities of Mira have been essential to our progress on California full 3D tomography. To support our push to higher resolution, higher frequency tomography, we have prototyped a compression algorithm that will reduce our storage requirements in the future. We will work with our ALCF catalyst to integrate this into our Mira processing.

1 project achievementshypocenter.usc.edu/research/incite/incite_2016/scec... · 2015-07-31 · 1...

Documents