current emc global model dynamics and beyond...2016/05/24 · 3 june 2016 twpgfs2016, h. juang 1...

3 June 2016 TWPGFS2016, H. Juang 1

Current EMC global model dynamics and beyond

Hann-Ming Henry Juang

Environment Modeling Center, NOAA/NWS/NCEP, Washington, DC

Concerns

• Current EMC global model dynamics is a spectral model.

• Two dynamics versions in the code

– Operational hybrid coordinate SL

– Generalized coordinate enthalpy NDSL

• Beyond (in-house GSM and NGGPS)

– Nonhydrostatic system

– Deep atmosphere

Contents

• Advantages of traditional global spectral model

• Developments has been done – Improve grid structure

– Improve thermodynamics

– Improve semi-Lagrangian

– Improve spectral transform

• Under testing and further plan– enhance parallelization

– different spectral truncation

– nonhydrostatic system

– deep atmosphere system

Traditional GSM

• The problem on sphere => Spherical coordinate

• Spherical harmonic (spectral transform)

– > leads to no pole problem

• Highest accuracy in horizontal discretization

• No bias due to computing direction

• Linear computation has no phase error

• easy to do semi-implicit time scheme

• Only vertical grid discretization needed

Contemporary GSM

• Reduced Spherical Gaussian grid

• Generalized vertical coordinates

• Considering gases into thermodynamics

• Semi-Lagrangian advection with semi-implicit

• Refine Legendre coefficient for high resolution

• Expand full dimensional MPP

• Exam cubic truncation with less diffusion

• Consider deep atmospheric dynamics

Juang 2004

Based accuracy of associated Legendrepolynomial function, higher latitudes havefew significant waves leads to smaller gridnumber for spectral transform

Table 2: Details of model configurations.

NH-GFS (Baseline) * FV3 MPAS NIM NMMB-UJ NEPTUNE IFS (RAPS13) *

Resolution 13 km (TL1534) ~12 km (C768)* 12km * 13.4 * 13 km 12.71 km * 12 5 km (Tc799)

Grid Points3072x1536 (unreduced)

3,126,128 (reduced)

6x768x768

3,538,9444,096,002 ** 3,317,762

6x768x768

3,538,944 *3,110,402 **

3,336,946

(reduced)

Vertical Layers * 128 127 ** 127 *** 128 128 127 *** 137

Time Step TBD

600s (slow phys)

150s (vertical, fast

150/10 (horiz.

acoustic)

72 s (RK3 dynamics)

12 s (acoustic)

72 s (RK3 scalar

transport)

72 s 24 s **75 s (advective),

15 s (sound) ****450

Resolution 3 km (TL6718) ~3 km (C3072) * 3km 3.3 km ** 3 km 3.13 km * 3.125 km (Tc3199)

Grid Points13440x6720 (unred.)

59,609,088 (reduced)

6x3072x3072

56,623,10465,536,002 53,084,162

6x3072x3072

56,623,104 *61,440,000 **

51,572,436

(reduced)

Vertical Layers * 128 127 ** 127 *** 128 128 128 137

Time Step TBD

150 s (slow phys)

37 5 s (vertical,

fast phys)

37.5/10 s (horiz.

acoustic)

18 s (RK3 dynamics)

3 s (acoustic)

18 s (RK3 scalar

transport)

18 s 6 s **15 s (slow RK3 dyn.)

2 5 s (fast dyn.)120

* Unl es s noted,

la yers refers to the

number of la yers ,

not the number of

i nterfa ces between

la yers + top +

bottom

* Baseline configuration is

tentative, pending test

evaluation.

** Rough estimate for

reduced Gaussian grid

based on reduction factor

(0.66) of 13 km grid. This will

likely be revised after

further testing of accuracy

of spectral transform at

TL6718.

* True resolution is

average over equator

and/or from south to

north pole. For 13km,

max cell size (edge of

finite volume): 14.44

km, min: 10.21 km,

global avg: 12.05 km.

For 3.25 km, divide by

** Favorable OpenMP

Performance

* Resolution refers to

mean cell-center

spacing on the mesh

** Subdivision of 60 km

mesh by factor of 5.

*** Following the FV3

configuration, we will

use 127 levels where

density,

theta and horizontal

momentum are defined

(on our Lorenz-grid

vertical

discretization) and 128

levels for w (that

includes both the lower

boundary and the

model top "lid").

* Generated by 6

bisections followed

by 2 trisections.

Distances between

neighbors: 13.367

average, 12.245

min., 14.397 max..

Maximum ratio of

neighboring grid

point distances:

1.17577

** Generated by 8

bisections followed

by 2 trisections.

Distances between

neighbors: 3.3417

average, 3.060 min.,

3.601 max..

Maximum ratio of

neighboring grid

point distances:

1.1765.

* B-grid mass points

** For fast modes and

advection of basic

model variables. Time

step for tracers is

longer by 2x.

* Resolution refers to the

representative nodal

spacing in the element

measured as the

midpoint between the

minimum and mean

nodal spacing and

averaged over the globe.

** Horizontal grid points

is six faces of cube times

number of elements per

face times polynomial

order squared.

* Hydrostatic

The Tc799 cubic grid has

the same number of grid

columns as a TL1599

linear grid.

While the Tc3199 cubic

grid has the same

number of grid columns

as a TL6399 linear grid.

AVEC report: NGGPS Level-1 Benchmark and software evaluation

Generalized vertical coordinate

• First version of GSM used sigma coordinate, then move to hybrid sigma-pressure coordinate

• A dynamics system can be used for sigma, sigma-pressure, and sigma-theta, sigma-theta-pressure etc (Juang 2011), instead of only sigma coordinate or hybrid sigma-p coordinates

• The same model dynamics to be easily compared for different coordinates

June 3, 2016 Hann-Ming Henry Juang 13

From the ideal-gas law for individual gas as

p = rRT

pi = riRiT

p = riRii=1

å T = rrirRi

å T = rRT we havethrough

R = qiRii=1

qi =rir

dt= rQ

Then put them into internal equations, we have

we let

h =CPT as a prognostic variable, enthalpy.

we have

CP = qiCP ii=1

14Black s: operational GFS Red t: sigma-theta GFS

Case Results - AC

PROW: gc s-pPROEW: gc s-p enthalpy

Root Mean Square error

Opr GFS vs WAM

64 layers150 layers

3 June 2016 TWPGFS2016, H. Juang

Example of T profileof 150 layers

WAM uses generalized hybrid coordinatewith enthalpy CpT as thermodynamicsvariables , where Cp is summation ofeach gases.

O 519.674 1299.18

O2 259.837 918.096

O3 173.225 820.239

Dry air 296.803 1039.64

H2O 461.50 1846.00

3 June 2016 H. Juang

in do_dynamics_two_loop for spdmx at kdt= 40825spdmx(001:010)= 19. 20. 21. 23. 25. 26. 27. 28. 28. 28.spdmx(011:020)= 28. 27. 27. 27. 27. 28. 28. 28. 29. 30.spdmx(021:030)= 31. 33. 35. 37. 40. 42. 44. 46. 49. 53.spdmx(031:040)= 58. 61. 63. 63. 62. 60. 55. 47. 45. 44.spdmx(041:050)= 45. 45. 47. 49. 52. 55. 59. 62. 65. 68.spdmx(051:060)= 72. 76. 80. 84. 87. 90. 93. 95. 97. 98.spdmx(061:070)= 102. 110. 118. 127. 135. 143. 149. 153. 155. 152.spdmx(071:080)= 147. 145. 142. 138. 135. 132. 130. 126. 121. 119.spdmx(081:090)= 114. 112. 110. 106. 100. 95. 94. 90. 89. 89.spdmx(091:100)= 87. 82. 91. 95. 99. 97. 104. 100. 111. 120.spdmx(101:110)= 125. 133. 148. 167. 172. 164. 159. 160. 147. 124.spdmx(111:120)= 117. 125. 133. 138. 137. 157. 183. 202. 220. 243.spdmx(121:130)= 269. 297. 319. 338. 355. 368. 378. 386. 392. 396.spdmx(131:140)= 399. 402. 404. 405. 406. 407. 408. 409. 410. 410.spdmx(141:150)= 411. 412. 412. 413. 413. 414. 414. 415. 415. 418.

Maxima wind (m/s) at NCEP GFS 150 layers WAM

WAM Euleriandt=180sec

T62L150

WAM NDSLdt=400sec

Non-iteration and Dimensional-split Semi-Lagrangian advection method

• No need to iteratively find departure points • Dimensional split with MPI transpose, no need of

halo, so avoid the decision of halo side • Positive defined in interpolation, so no q<0• No polar grid needed • Option to have mass conservation• Based on generalized vertical coordinate enthalpy

system

June 3, 2016 Henry Juang 27

72h fcst specific humidity

at model layer 40

control

dt=600

dt=1800

24hr fcst cloud water

at model layer 30

control

ndslmcpd

T574 GC Eulerian

T574 Opr EulerianTL878 NDSL

TL878 NDSL

dt=120dt=600

Opr T574 EulerianTL878 NDSL

dt=120dt=600

Opr T574 EulerianNDSL TL878

dt=120dt=600

Anomaly correlation- Day 5 HGT

Leads to higher and higher resolutionfor semi-Lagrangian vs Eulerian comparison

Examine Legendre coefficient

• We found Legendre polynomial function in sp lib and model have problem of spherical harmonic transform while using high truncation, say T1000 and above

• X-number technique fixes the problem• Furthermore, increase precision only in

computing Gaussian weight helps accuracy by order of 3

• These steps provide necessities to implement TL1534 with enhanced and corrected spectral transform

underflow overflow

2-960 2960

Machine representable real number

X number takes care of under and over flow by xB**I with B=2**960, soxB**0 is machine representable real number but xB**(-1) represents underflowAnd xB**(+1) represents overflow. The details in Fukushima 2011

Implement X-number into SP lib and GSM model dynamics

xB**ix time yB**iy, should be equal to x*yB**(ix+iy), but we should take care underflow by x*y, to do so, consider normalize of any X-number into the rangeof 2**(-480) and 2**(480). If x and y are normalized, then x*y will not be over-or under-flow.

Normalized range

AVEC Report:NGGPS Level-1 Benchmarks and Software Evaluation

Full Dimensional MPP

• Current NCEP GSM has multi-threading by OpenMP and one dimensional MPI

• The limitation of OpenMP may be 12 and 1DMPI based on half of spectral truncation

• For 2D MPI, it is limited to spectral truncation and vertical layer number.

• Extend to 3DMPI with MPI-FFT and MPI-Legendre transform

• Invest halo or transpose for dimensional dependency computation

1D MPIGlobal communication

2D MPIGroup communication

June 3, 2016 13th on the use of hpc in meteorology 47

Halo Exchange for opr SL

Extra memory is required,which may be as huge

as computing grid while number of MPP cpu increases.

June 3, 2016 13th on the use of hpc in meteorology 48

0 . . . . 180

Transpose for NDSL

No halo is requiredNo increasing memory with

Increasing number of PE(cpu)

Black grouped with red ones forMPI FFT without transpose

Black grouped with yellow ones forMPI Legendre w/o transpose

Black grouped with blue ones forregrouping into vertical formodel physics and vertical dependentmodel dynamics computations

So spectral transform will be no transpose, only vertical needs….One side MPI ....

3D MPI

Comparison

• Using TL1534 L64 as example

• 1DMPI+12 threads

=> We have 767 MPI 12 thread = 9,204 cpus

• 2DMPI + 12 threads

=> We have 767*64 MPI 12 thread = 589,056 cpus

• 3DMPI + 12 threads

=> 767*64*1534 MPI 12 threads= 903,611,904 cpus

Experimental cubic truncation

• Based on ECMWF IFS recent experiments, they have implemented cubic truncation octahedral reduced Gaussian grid for IFS Tco1279 ( 9 km )

• For semi-Lagrangian, we used to have linear truncation, for Eulerian, we have quadratic truncation to avoid aliasing

• Linear truncation resolves up to 2dx wave, quadratic to 3dx, and cubic to 4dx.

• Use cubic truncation with much less diffusion and without de-aliasing filter

• Octahedral grid is for future spectral/grid hybrid system and requires different FFT package (FFTW)

ECMWF Newsletter#147 spring2016

Implement Cycle 41r2Move from linear truncation reduced gridTo cubic truncation octahedral reduced grid

IFS octahedral reduced grid, more regular than spherical reduced grid, they arepreparing for next generation of hybrid spectral/grid-point configuration.

More resolving small scales

More efficient in computing time

IFS experimental results

ECMWF Newsletter #146 winter2015/2016

More small scale waves in TcBest mass conservation in Tc

Deep Atmos vs non-Hydro• From Deep atmosphere, we require r changes with time,

thus we need dw/dt equation• And we need full curvature and Coriolis force terms to

satisfy conservation• Thus, based on conservation requirement, a deep

atmospheric dynamic is a non-hydrostatic dynamic. A non-hydrostatic dynamics can be shallow or deep atmospheric dynamics.

• Both r and vertical components of curvature and Coriolisforce should be considered in deep atmosphere; and should not be considered in shallow atmosphere. (see Juang 2014 NCEP ON#477)

• Derive deep atmospheric system as the same form as a shallow atmospheric system with options for easy implementation

dt+u*w

r - fsv

* + fc*w +

ø÷ = Fu

dt +v*w

r + fsu

* +m2 s*2

rsinf +

ø÷ = Fv

dt -m2 s

r -m2 fc

*u* +kh

¶r +g = Fw

dt= Fh

¶t+m2

¶r* u*

¶r* v*

+¶r* z

¶z= Fr

dt= Fqi

p = rkh

¶t+ l

· ¶()

¶t+m2u* ¶()

r¶l+m2v* ¶()

r¶j+z

· ¶()

¶z ; r* = r

fs = 2Wsinf ; fc* = 2Wcos2 f ; g = g(r) ; k =

CP ; g =

CV ; s*2

Deep Atmos Dyn

in spherical mapping

& generalized coordinates

Staniforth and Wood (2003)Juang (2014) NCEP Office Note

To derive deep-atmosphere continuity equation into the same

form as shallow atmosphere

We define a coordinate pressure as hydrostatic one based on

the coordinate density as

Put it into deep-atmosphere continuity equation

We have

¶t+m2

¶r* u*

¶r* v*

+¶r* z

¶z= 0

where g0 is constant

In deep atmosphere, horizontal wind with Gaussian weighting

Modify continuity equation

We have

u* = rcos2 fdl

dt= ucosf

We can define, height-weighted horizontal wind as

v* = rcosfdf

dt= vcosf

Since and r* > 0

Thus is monotone with vertical coordinate

We can use it for coordinate definition

For opr compatibility, we use

and relation with height as

because r* = rr2

so rk2wk =

k+1wk+1 +1

we have wk =1

2wk+1 +

k+1 +1

From angular momentum principle, we have relationbetween model layer and model level as

so vertical momentum eq can be written as

with g =a2

= rg0a2

Then the hydrostatic means

In summary, we have following

So prognostic variables are

dt= 0 and

where e =r

Summary• NCEP GSM may be an “old” model, but we are using

GSM2010s not GSM1980s.

• Current operational GSM is TL1534 (13km) semi-Lagrangian two-time level scheme with reduced spherical Gaussian grid and refined spectral transform.

• Current experimental GSM has options to be as Whole Atmospheric Model (WAM) with enthalpy and NDSL.

• Further speed up in computational wall time by multi-threading and full dimension MPI, maybe 2D MPI is more than enough.

• Experimental cubic and/or quadratic truncations with spherical reduced Gaussian grid instead of linear truncation for semi-Lagrangian, and with less diffusion.

• Developing GSM Deep-Atmospheric model beyond nonhydrostatic system with shallowness options for NGGPS.

current emc global model dynamics and beyond...2016/05/24 · 3 june 2016 twpgfs2016, h. juang 1...

Documents

hmm tutorial by rabiner and juang

analisis nilai juang mahasiswa itb angkatan 2003 2

resume charng hsein juang personal data...

angela juang, bo brown, daniel so. background & issues...

jia-fong fan and hann-ming henry juang environmental m...

emc business continuity for physical security - · pdf...

use of eye movement gestures for web browsing kevin juang...

linda p. juang, ph.d. - uni-potsdam.de€¦ ·...

emc documentum emc documentum content servertm v5.3 and emc...

rabiner & juang - fundamentals of speech recognition

william c. mcginnis · 2017. 5. 31. · company profile ....

emc corporation (emc)

hidden markov models for speech recognition b. h. juang; l....

hubungan daya juang dengan kecemasan ...kecerdasan...

emc basics of emc en

emc(1) theenhanced machine controller emc(1) emc − emc...

deploying virtualized microsoft dynamics ax 2012 · pdf...

juang jaya breeding marketing center english

deploying virtualized microsoft dynamics ax 2012 r2 - dell...

jer-nan juang astrodynamics...