bluefire Accomplishment Report

Copyright 2010 by Carl Drews

Distributed under the Creative Commons Attribution License 3.0.

Author: Carl Drews

Atmospheric Chemistry Division

National Center for Atmospheric Research

Boulder, Colorado USA

Phone: 303-497-1429 E-mail: drews@ucar.edu

CISL Project Number: 35071377

CISL Project End Date: May 2010

Lead User: Carl Drews

Institution: University of Colorado at Boulder

Project Title: Master's Thesis: Application of Storm Surge Modeling to Moses'

Crossing of the Red Sea; and to Manila Bay, the Philippines.

GAUs Allocated: 1000 GAUs Used: 939

1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

AbstractDuring 2009-2010 NCAR's Computational and Information Systems Laboratory (CISL)

granted to Carl Drews an account on the "bluefire" supercomputer, with an allocation of 1000

GAUs. Carl Drews was a Master's candidate in Atmospheric and Oceanic Sciences at the

University of Colorado in Boulder with professor Weiqing Han, and currently works as a

Software Engineer in NCAR/NESL's Atmospheric Chemistry Division. This report describes

the scientific results that were achieved through the use of CISL computer resources. Bluefire

supports higher-resolution modeling than a workstation can support, and this increased

resolution reveals important features of wind setdown and storm surge in coastal areas. Wind

setdown is the drop in water level that occurs when strong winds blow offshore for an

extended period of time.

IntroductionCoastal ocean modeling requires a transition from deep ocean to shallow and intricate

shorelines. While open-ocean dynamics can be represented using grid scales greater than 1

kilometer, harbor and coastal features often require grid resolution on the order of 100 meters.

This range of scale presents a modeling and computational challenge. Unless the ocean

model supports variation in the grid resolution (by nested grids or an unstructured grid), the

modeler must increase the number of grid points to match the smallest feature in the domain.

Following this strategy can easily increase the computational burden by a factor of 100,

requiring a different class of computer on which to solve the problem. The computational

needs of this research project required the author to migrate from a Dell workstation to the

bluefire supercomputer; this report describes that migration and the corresponding

2

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

improvement in scientific results.

Previous ResearchThe biblical book of Exodus contains in the 14th chapter a description of Moses and the

Israelites making a dramatic escape from Pharaoh's chariot army through the parted waters

of the Red Sea. This narrative has been identified as a wind setdown event since the late

1800s [Bartlett 1879][Tulloch 1896], but it is only in the past two decades that analytical

techniques and computational resources have been available to study the reported event in

detail. Nof and Paldor found analytical solutions to the governing equations for an idealized 1-

dimensional model of the Gulf of Suez [Nof and Paldor 1992]. Voltzinger and Androsov used a

3-D model to analyze a possible crossing along an underwater reef at 29.88° N, about 10 km

south of Suez. The author studied a configuration of Nile channels and coastal lagoons in the

eastern Nile delta, at the northern end of the isthmus of Suez.

MethodsThe Regional Ocean Modeling System (ROMS) is a modern ocean model that implements a

free surface and a scheme for wetting and drying. The Shuttle Radar Topography Mission

(SRTM) provides topography and bathymetry data at resolutions of 30 and 3 arc-seconds

(860 meters and 86 meters) worldwide. The author constructed two ROMS domains at low

and high grid resolution, then applied wind forcing from the east at 28 m/s. The low-resolution

grid covers the eastern Nile delta with 240x240 grid points (30° - 32°N, 31° - 33°E). The high-

resolution grid spans a somewhat smaller geographic region (30.5°N - 31.5°N, 31.5°E -

33°E); there are 1800 grid points from west to east, and 1200 grid points from south to north.

3

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

In general, the Dell Optiplex GX280 ran the low-resolution model, and bluefire ran the high-

resolution model. The Dell workstation executes ROMS with a single thread. The bluefire

supercomputer executes ROMS on a single node with 32 processors, using 8x4 tiling for the

domain grid and OpenMP parallel execution.

The SRTM data must be modified to reflect the most likely topography of the eastern Nile

delta in 1250 BC. Geological and archaeological sources suggest that there once was a large

coastal lagoon known as the Lake of Tanis, into which the Pelusiac branch of the Nile flowed.

The original intent was to reconstruct this topography using the same modifications in both

grid resolutions. However, certain river channels are 200 meters wide, and cannot be

resolved with the 860-meter grid. Consequently, the low-resolution domain had to implement

an idealized version of the ancient topography, using straight lines and connected grid cells

instead of the sinuous curves that a natural river channel would follow.

Research ResultsThe author ran 13 simulation experiments with the Tanis model, exercising various

configurations of wind and topography. The Dell workstation takes 0.8 wall-clock hours to run

the low-resolution model with no other load. The Dell takes 31.2 hours to run the high-

resolution domain (39 times longer). The bluefire supercomputer takes 2.8 wall-clock hours to

run the high-resolution model (1/11 as long), at a cost of about 62 GAUs on the economy

queue.

The following figures compare model output at low and high resolutions.

4

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

Figure 1. Crossing site at the Kedua Gap at 12:00 hours.

Both models show a well-developed Pelusiac jet, which is the high-velocity stream of water

extending westward from Pelusium / Baal-Zephon. However, the low-resolution model (top)

merely shows a strip of water 2-3 grid cells wide. The high-resolution model (bottom) reveals

a parabolic profile on both sides of the jet, with turbulence in the shallow water near its edges.

5

88

89

90

91

92

93

94

Figure 2. Currents within the Kedua Gap. Left: low-resolution model. Right: high-resolution.

The numerical results are very similar, despite the 10x difference in horizontal grid resolution.

Note that the magnitudes of the surface difference reflect the change in grid size. The only

significant discrepancy is in the V surface difference at 5-6 hours, and this may be explained

by noting that the site location for the high-resolution model was 0.8 km farther south, and

therefore more directly centered within the Kedua Gap.

6

95

96

97

98

99

100

101

102

Figure 3. Flow vectors at Kedua. For clarity, only 1/10 of the vectors are shown for the high-

resolution model.

Although the snapshots are taken at different times, both the low-resolution (left) and high-

resolution models (right) display in part A (top) a three-way convergence of return surges from

the west, south, and east onto the upper center of the figure. The magnitudes of the currents

are similar. In part B (bottom) both models show a convergence of flow near Pelusium.

7

103

104

105

106

107

108

109

110

Scientific Publications

Drews, Carl and Weiqing Han (2010). Dynamics of Wind Setdown at Suez and the Eastern

Nile Delta. Submitted to PLoS ONE on April 5, 2010.

Graduate Students

The following graduate students used CISL resources during this project:

Carl Drews

Department of Atmospheric and Oceanic Sciences

University of Colorado at Boulder

(Now working at the NCAR Earth Systems Laboratory.)

Master's Thesis

Drews, Carl (2009). Application of Storm Surge Modeling to Moses' Crossing of the Red Sea;

and to Manila Bay, the Philippines. Master's thesis, Department of Atmospheric and Oceanic

Sciences, University of Colorado at Boulder.

Accessible on-line at ProQuest: http://gradworks.umi.com/14/68/1468999.html

8

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

Discussion

The author was pleasantly surprised by the close agreement of numerical results between the

low-resolution and the high-resolution ROMS domains. This result indicates that the ROMS

ocean model scales well; the model results are not significantly affected by a 10x change in

grid resolution. Even river channels one or two grid cells wide behave realistically. Certain

hydrodynamic features cannot be resolved by the larger grid resolution of 860 meters, and

these detailed features require the high-resolution model. A practical approach would be to

run both models in concert, depending on the computing resources available to the project.

The low-resolution model is suitable for a large-scale view, and for generating boundary

conditions for a smaller grid nested within the larger domain. The high-resolution grid is best

for viewing smaller features of scientific and operational interest.

Conclusions

The bluefire supercomputer at NCAR provides a valuable resource for high-resolution

modeling. Under normal work hours, the difference between a 3-hour model run and a 31-

hour model run can determine whether model results are available for analysis twice a day or

once every two days. Thus bluefire can significantly accelerate scientific progress.

The ROMS ocean model produces numerically consistent results across different scales of

grid resolution (860 meters to 86 meters). If the ocean modeler needs large-scale results

quickly (gross transport, average current, sea level), then a low-resolution simulation may be

sufficient. If smaller features are of interest (channels, jetties, eddies, turbulence), then a high-

9

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

resolution model should be used to resolve these features. In practice a workstation can be

used to run the low-resolution domain, while bluefire runs the corresponding high-resolution

domain. The two sets of results should be numerically similar, but bluefire will produce a

richer and more detailed picture of the system under study.

AcknowledgmentsThe author is grateful to the University Corporation for Atmospheric Research (UCAR/NCAR)

for tuition and computing support, and for travel support by the Office of Naval Research

(ONR) N00014-07-1-0413. Weiqing Han is also supported by NASA Ocean Vector Wind

Science Team 1283568 and NSF OCE 0452917. The National Center for Atmospheric

Research is sponsored by the National Science Foundation. Any opinions, findings and

conclusions or recommendations expressed in the publication are those of the author(s) and

do not necessarily reflect the views of the National Science Foundation.

ReferencesBartlett, S.C., From Egypt to Palestine, Harper, New York, NY, 1879.

Nof, D. and N. Paldor, "Are There Oceanographic Explanations for the Israelites' Crossing of

the Red Sea?" Bulletin of the American Meteorological Society, 73(3): 305-314, March 1992.

Tulloch, A.B., "Passage of the Red Sea by the Israelites," Journal of the Transactions of the

Victoria Institute (now Faith and Thought), 28: pp. 267-280, 1896.

10

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

Voltzinger, N.E. and A.A. Androsov, "Modeling the Hydrodynamic Situation of the Exodus,"

Izvestiya, Atmospheric and Oceanic Physics, 39(4): 482-496, translated from Russian to

English by E. Kadyshevich, 2003.

11

173

174

175

176