bluefire accomplishment report826...the low-resolution model with no other load. the dell takes 31.2...
Post on 04-Jan-2020
1 Views
Preview:
TRANSCRIPT
bluefire Accomplishment Report
Copyright 2010 by Carl Drews
Distributed under the Creative Commons Attribution License 3.0.
Author: Carl Drews
Atmospheric Chemistry Division
National Center for Atmospheric Research
Boulder, Colorado USA
Phone: 303-497-1429 E-mail: drews@ucar.edu
CISL Project Number: 35071377
CISL Project End Date: May 2010
Lead User: Carl Drews
Institution: University of Colorado at Boulder
Project Title: Master's Thesis: Application of Storm Surge Modeling to Moses'
Crossing of the Red Sea; and to Manila Bay, the Philippines.
GAUs Allocated: 1000 GAUs Used: 939
1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
AbstractDuring 2009-2010 NCAR's Computational and Information Systems Laboratory (CISL)
granted to Carl Drews an account on the "bluefire" supercomputer, with an allocation of 1000
GAUs. Carl Drews was a Master's candidate in Atmospheric and Oceanic Sciences at the
University of Colorado in Boulder with professor Weiqing Han, and currently works as a
Software Engineer in NCAR/NESL's Atmospheric Chemistry Division. This report describes
the scientific results that were achieved through the use of CISL computer resources. Bluefire
supports higher-resolution modeling than a workstation can support, and this increased
resolution reveals important features of wind setdown and storm surge in coastal areas. Wind
setdown is the drop in water level that occurs when strong winds blow offshore for an
extended period of time.
IntroductionCoastal ocean modeling requires a transition from deep ocean to shallow and intricate
shorelines. While open-ocean dynamics can be represented using grid scales greater than 1
kilometer, harbor and coastal features often require grid resolution on the order of 100 meters.
This range of scale presents a modeling and computational challenge. Unless the ocean
model supports variation in the grid resolution (by nested grids or an unstructured grid), the
modeler must increase the number of grid points to match the smallest feature in the domain.
Following this strategy can easily increase the computational burden by a factor of 100,
requiring a different class of computer on which to solve the problem. The computational
needs of this research project required the author to migrate from a Dell workstation to the
bluefire supercomputer; this report describes that migration and the corresponding
2
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
improvement in scientific results.
Previous ResearchThe biblical book of Exodus contains in the 14th chapter a description of Moses and the
Israelites making a dramatic escape from Pharaoh's chariot army through the parted waters
of the Red Sea. This narrative has been identified as a wind setdown event since the late
1800s [Bartlett 1879][Tulloch 1896], but it is only in the past two decades that analytical
techniques and computational resources have been available to study the reported event in
detail. Nof and Paldor found analytical solutions to the governing equations for an idealized 1-
dimensional model of the Gulf of Suez [Nof and Paldor 1992]. Voltzinger and Androsov used a
3-D model to analyze a possible crossing along an underwater reef at 29.88° N, about 10 km
south of Suez. The author studied a configuration of Nile channels and coastal lagoons in the
eastern Nile delta, at the northern end of the isthmus of Suez.
MethodsThe Regional Ocean Modeling System (ROMS) is a modern ocean model that implements a
free surface and a scheme for wetting and drying. The Shuttle Radar Topography Mission
(SRTM) provides topography and bathymetry data at resolutions of 30 and 3 arc-seconds
(860 meters and 86 meters) worldwide. The author constructed two ROMS domains at low
and high grid resolution, then applied wind forcing from the east at 28 m/s. The low-resolution
grid covers the eastern Nile delta with 240x240 grid points (30° - 32°N, 31° - 33°E). The high-
resolution grid spans a somewhat smaller geographic region (30.5°N - 31.5°N, 31.5°E -
33°E); there are 1800 grid points from west to east, and 1200 grid points from south to north.
3
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
In general, the Dell Optiplex GX280 ran the low-resolution model, and bluefire ran the high-
resolution model. The Dell workstation executes ROMS with a single thread. The bluefire
supercomputer executes ROMS on a single node with 32 processors, using 8x4 tiling for the
domain grid and OpenMP parallel execution.
The SRTM data must be modified to reflect the most likely topography of the eastern Nile
delta in 1250 BC. Geological and archaeological sources suggest that there once was a large
coastal lagoon known as the Lake of Tanis, into which the Pelusiac branch of the Nile flowed.
The original intent was to reconstruct this topography using the same modifications in both
grid resolutions. However, certain river channels are 200 meters wide, and cannot be
resolved with the 860-meter grid. Consequently, the low-resolution domain had to implement
an idealized version of the ancient topography, using straight lines and connected grid cells
instead of the sinuous curves that a natural river channel would follow.
Research ResultsThe author ran 13 simulation experiments with the Tanis model, exercising various
configurations of wind and topography. The Dell workstation takes 0.8 wall-clock hours to run
the low-resolution model with no other load. The Dell takes 31.2 hours to run the high-
resolution domain (39 times longer). The bluefire supercomputer takes 2.8 wall-clock hours to
run the high-resolution model (1/11 as long), at a cost of about 62 GAUs on the economy
queue.
The following figures compare model output at low and high resolutions.
4
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
Figure 1. Crossing site at the Kedua Gap at 12:00 hours.
Both models show a well-developed Pelusiac jet, which is the high-velocity stream of water
extending westward from Pelusium / Baal-Zephon. However, the low-resolution model (top)
merely shows a strip of water 2-3 grid cells wide. The high-resolution model (bottom) reveals
a parabolic profile on both sides of the jet, with turbulence in the shallow water near its edges.
5
88
89
90
91
92
93
94
Figure 2. Currents within the Kedua Gap. Left: low-resolution model. Right: high-resolution.
The numerical results are very similar, despite the 10x difference in horizontal grid resolution.
Note that the magnitudes of the surface difference reflect the change in grid size. The only
significant discrepancy is in the V surface difference at 5-6 hours, and this may be explained
by noting that the site location for the high-resolution model was 0.8 km farther south, and
therefore more directly centered within the Kedua Gap.
6
95
96
97
98
99
100
101
102
Figure 3. Flow vectors at Kedua. For clarity, only 1/10 of the vectors are shown for the high-
resolution model.
Although the snapshots are taken at different times, both the low-resolution (left) and high-
resolution models (right) display in part A (top) a three-way convergence of return surges from
the west, south, and east onto the upper center of the figure. The magnitudes of the currents
are similar. In part B (bottom) both models show a convergence of flow near Pelusium.
7
103
104
105
106
107
108
109
110
Scientific Publications
Drews, Carl and Weiqing Han (2010). Dynamics of Wind Setdown at Suez and the Eastern
Nile Delta. Submitted to PLoS ONE on April 5, 2010.
Graduate Students
The following graduate students used CISL resources during this project:
Carl Drews
Department of Atmospheric and Oceanic Sciences
University of Colorado at Boulder
(Now working at the NCAR Earth Systems Laboratory.)
Master's Thesis
Drews, Carl (2009). Application of Storm Surge Modeling to Moses' Crossing of the Red Sea;
and to Manila Bay, the Philippines. Master's thesis, Department of Atmospheric and Oceanic
Sciences, University of Colorado at Boulder.
Accessible on-line at ProQuest: http://gradworks.umi.com/14/68/1468999.html
8
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
Discussion
The author was pleasantly surprised by the close agreement of numerical results between the
low-resolution and the high-resolution ROMS domains. This result indicates that the ROMS
ocean model scales well; the model results are not significantly affected by a 10x change in
grid resolution. Even river channels one or two grid cells wide behave realistically. Certain
hydrodynamic features cannot be resolved by the larger grid resolution of 860 meters, and
these detailed features require the high-resolution model. A practical approach would be to
run both models in concert, depending on the computing resources available to the project.
The low-resolution model is suitable for a large-scale view, and for generating boundary
conditions for a smaller grid nested within the larger domain. The high-resolution grid is best
for viewing smaller features of scientific and operational interest.
Conclusions
The bluefire supercomputer at NCAR provides a valuable resource for high-resolution
modeling. Under normal work hours, the difference between a 3-hour model run and a 31-
hour model run can determine whether model results are available for analysis twice a day or
once every two days. Thus bluefire can significantly accelerate scientific progress.
The ROMS ocean model produces numerically consistent results across different scales of
grid resolution (860 meters to 86 meters). If the ocean modeler needs large-scale results
quickly (gross transport, average current, sea level), then a low-resolution simulation may be
sufficient. If smaller features are of interest (channels, jetties, eddies, turbulence), then a high-
9
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
resolution model should be used to resolve these features. In practice a workstation can be
used to run the low-resolution domain, while bluefire runs the corresponding high-resolution
domain. The two sets of results should be numerically similar, but bluefire will produce a
richer and more detailed picture of the system under study.
AcknowledgmentsThe author is grateful to the University Corporation for Atmospheric Research (UCAR/NCAR)
for tuition and computing support, and for travel support by the Office of Naval Research
(ONR) N00014-07-1-0413. Weiqing Han is also supported by NASA Ocean Vector Wind
Science Team 1283568 and NSF OCE 0452917. The National Center for Atmospheric
Research is sponsored by the National Science Foundation. Any opinions, findings and
conclusions or recommendations expressed in the publication are those of the author(s) and
do not necessarily reflect the views of the National Science Foundation.
ReferencesBartlett, S.C., From Egypt to Palestine, Harper, New York, NY, 1879.
Nof, D. and N. Paldor, "Are There Oceanographic Explanations for the Israelites' Crossing of
the Red Sea?" Bulletin of the American Meteorological Society, 73(3): 305-314, March 1992.
Tulloch, A.B., "Passage of the Red Sea by the Israelites," Journal of the Transactions of the
Victoria Institute (now Faith and Thought), 28: pp. 267-280, 1896.
10
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
Voltzinger, N.E. and A.A. Androsov, "Modeling the Hydrodynamic Situation of the Exodus,"
Izvestiya, Atmospheric and Oceanic Physics, 39(4): 482-496, translated from Russian to
English by E. Kadyshevich, 2003.
11
173
174
175
176
top related