Developments in the dynamical core of the global semi-Lagrangian SL-AV modelMikhail Tolstykh, Vladimir Shashkin

Institute of Numerical Mathematics, Russian Academy of Sciences

andHydrometcentre of Russia

With contribution by Vassili Mizyak

Outline

• SL-AV model• Mass-conservative shallow-water version on

the reduced lat-lon grid - results of standard SW tests• New solvers for U-V reconstruction and

Helmholtz problem

SL-AV model. General features

• Global finite-difference semi-Lagrangian semi-implicit dynamical core of own development + parameterizations of subgrid-scale processes from French model ARPEGE/ALADIN.

• Main Russian global operational model for medium-range and seasonal forecasts.

• So far, regular lat-lon grid is used. Reduced lat-lon grid in the shallow water version

SL-AV model.

Distinct features of dynamical core

• Vorticity-divergence formulation on the unstaggered grid, wide use of 4th order finite differences, usual and compact (Tolstykh, JCP 2002).

• Semi-implicit solver, reconstruction of U and V, horizontal diffusion are carried out in Fourier space in longitude.

• Version with variable resolution in latitude (quasioperational)

• Scales up to 100-300 cores, MPI+OpenMP

Reduced lat-lon grid

Reduced grid in the SL-AV model

• A part of calculations is carried out in space of Fourier coefficients in longitude.

• The semi-Lagrangian advection is used (no nonlinear advective terms)

=> Necessary latitudinal derivatives (i.e. geopotential gradient) can

be calculated in Fourier space. Number of gridpoints at each latitudinal circle should be suitable

for FFT.

Currently, implemented in the shallow-water version (standard and mass-conservative versions).

Motivation for a new version of dynamical core

• Future use for climate simulations• Future horizontal resolution of O(1-10km)=>• Mass conservation• More scalability => new algorithms

Next version of the SL-AV dynamical core

• Mass conservative. More uniform resolution on the sphere – reduced lat-lon grid.

• More scalable- abandon data transpositions- new parallel algorithms to solve block-

tridiagonal linear systems.

Mass-conservative shallow-water version of SL-AV

• Mass-conservative semi-Lagrangian advection scheme on the sphere on the reduced lat-lon grid.

• Mass-conservative semi-implicit time integration scheme. This implies a conservative Helmholtz problem solver on the sphere.

Mass-conservative semi-Lagrangian advection scheme on the sphere on the reduced grid

• Global conservative cascade scheme (CCS) was implemented for SL advection following “Efficient Conservative Global Transport Schemes for Climate and Atmospheric Chemistry Models” R.D. Nair et al. MWR 2002. (Meridional CFL number ≤ 1)

• Some changes were introduced for using CCS on the reduced grid.

Example of the reduced grid (region)Stars – grid pointsRegions bounded by dashed lines – grid cells

Departure volume calculation

Red lines – Lagrangian (departure) latitudes; Red stars – corners of the departure volume; Green pentagons – points where Lagrangian latitudes cross longitudes of the arrival grid (points of the intermediate grid); green dotted lines – midpoint values for latitudes of the intermediate grid defining north and south boundaries of the intermediate grid cells (grey rectangles). Orange lines – approximation for west (east) boundaries of departure volume

CCS on the reduced grid (1)

MM - mass of the advected quantity bounded in the cell of arrival (computational) grid. M * - mass bounded in the cell of departure grid.

Direct conservative cascade remapping of the mass from the reduced grid to any kind of departure grid is impossible due to the absence of well defined grid longitudes matching poles of the sphere in the reduced grid. So we implemented slightly different scheme:

Mn Mn* = Mn+1

Conservative cascade remapping

Mn+1*

Conservative cascade remapping

Mn

Conservative remapping

Mnreg Mn

* = Mn+1

Conservative cascade remapping

TThe reduced grid is used as computational and arrival grid. Mnreg - masses of

the advected quantity bounded in the cells of regular grid.

The structure of the departure grid corresponding to the reduced arrival grid is suitable for CCS since such departure grid forms well-defined intermediate cells (see Nair et al. for details)

CCS on the reduced grid (2)

We have chosen regular grid such that latitude grid lines of regular and reduced grid coincide and the number of grid points in latitude line of regular grid is equal to the maximum number of grid points in the latitude lines of the reduced grid.

1D remapping technique we use for remapping between reduced and regular grids is the same as the one used in 2D conservative cascade remapping

The first 1D remapping in the CCS is in latitudinal direction. The application of conservative scheme using first remapping in longitudinal direction (ex. SLICE by Zerroukat et al.) on the reduced grid is much more complicated.

Mass-conservative Helmholtz problem solver Semi-implicit time discretization of the SW equations leads to the

Helmholtz problem to be solved at each time step:

Φ – height, Φref – reference height, F – some known function, D – divergence of the wind field, A – some known vector, μ is a constant.

If Si, j – the volume of i, j-th cell of computational grid then the mass conservation is expressed by

The mass-conservation can be achieved if discrete Laplacian and divergence operators satisfy

refref

nnnnn AdivDF

;

),,(1212

jiji

n

jiji

n SS,

,,

,1

0;0,

,,,

,,2

jijiji

jijiji SAdivS

These operators are constructed with a finite-volume approach

(in Zerroukat et al 2009, FDs are used)

Results of standard tests - preliminaries

Tests were carried out using the regular grids of 1.5, 1, 0.75, 0.5 degrees resolution in latitude and longitude and the reduced grid of 1.5 and 0.75 degrees resolution in latitude and the same maximum resolution in longitude. Test cases ## 1, 7 were carried out only on the regular and reduced grids of 1.5 degree resolution.

Reduced grid for the tests was constructed by R. Fadeev (Fadeev R.Yu., Construction of a reduced latitude-longitude grid for a global numerical weather prediction problem, Russian Meteorology and Hydrology, 2006, N 9). Reduction rate was 10%.

For comparison, the same tests were carried out with the original non-conservative SW model.

In the tables, the abbreviation SL denotes the original non-conservative model and CM denotes its mass conservative modification

Resolution 1.50 1.00 0.750 0.50

Timestep(sec) 2700 1800 1350 900

Test case 1 - advection over the poleResolution in latitude and longitude (maximum resolution for reduced grid) - 1,50.

Meridional Courant number 0.5

Resolution l1 x10-2 l2x10-2 l∞x10-2 MAX x 10-2

CM regular grid 1.50 1.97 1.34 1.34 -0.4

CM reduced grid 1.50 1.97 1.34 1.34 -0.4

SL regular grid 1.50 6.03 3.78 3.10 -1.8

Height error measures after 1 revolution

The initial distribution (cosine bell) (A). Error fields after one revolution – CM on regular grid (B), CM on reduced grid (C), SL on regular grid (D)

CA B DC

Test case 2 – Global steady state geostrophic flow Angle α = π/4.

Resolution 1.50 1.50 1.00 .750 .750

Grid reduced regular regular reduced regular

CM 0.50E-04 0.18E-04 0.86E-05 0.11E-04 0.45E-05

SL 0.29E-04 0.22E-04 0.90E-05 0.67E-05 0.50E-05

l2 height error after 5 days of modeling

l2 height error for CM on the regular grid with resolution 1.50 (red line), 1.00 (green line), .750 (blue line)

Test case 6 – Rossby-Haurwitz wave #4

l2 height error after 14 days of modeling

Height field after 14 days of modeling with CM (regular grid 1.50 of resolution). Reference solution (A), CM error (B), SL error (C) (all on regular grid).

Resolution 1.50 1.50 1.00 .750 .750 .50

Grid reduced regular regular reduced regular regular

CM 0.47E-04 0.47E-04 0.30E-04 0.23E-04 0.22E-04 0.18E-04

SL 0.48E-04 0.48E-04 0.30E-04 0.23E-04 0.23E-04 0.18E-04

A B C

Test cases 7 – Real data

Tests were carried out with the fixed resolution of 1.50 .

l2 height error after 5 days of modeling

Test case 7a 7a 7b 7b 7c 7c

Grid reduced regular reduced regular reduced regular

CM 0.17E-02 0.17E-02 0.21E-02 0.21E-02 0.22E-02 0.22E-02

SL 0.17E-02 0.17E-02 0.22E-02 0.22E-02 0.23E-02 0.23E-02

A B C

Galewsky case4th order diffusion was applied to divergence field only. CM solution on the regular grid with

the resolution of 0.3250 is a reference.

Vorticity field after 144h. CM (regular grid): 1.0 – A, .750 – B, .50 – C, .3250 – D. Panel E - vorticity errors with respect to the reference solution

A B C

A B

C

D

E

New solver for U-V reconstruction (1)

• Old solver: solution of 2 Poisson equations, + differentiation using

Helmholtz relations

In a parallel environment, requires given Fourier coefficient at all latitudinal points. So the need for data transposition that limits scalability.

D 2

cos

11

aau

aa

v1

cos

1

2

New solver for U-V reconstruction (2)

New solver: direct solution of the system

=>

• Possibility to apply a parallel solver. No need for data transposition.

cos

cos

1

cos

cos

1

u

a

Du

a

kre

k

rek

im

kim

k

imk

re

u

u

aak

Daa

k

cos

cos1

cos

cos

cos1

cos

New solver for U-V reconstruction (3)

k

iki

ki

ki

ki

ki

i

k

k

i

k

k

i

k

k

aaa

Da

Da

Da

uC

uB

uA

11

11

11

63

2

6

63

2

6

62

cos2

cos

61

1

k

h

h

k

Ai

i

,

3

20

03

2

k

k

B,

62

cos2

cos

61

1

k

h

h

k

Ci

i

.

0,00E+00

2,00E-04

4,00E-04

6,00E-04

8,00E-04

1,00E-03

1,20E-03

1,40E-03

1,60E-03

1.5 2.0 2.5

U, original solver

V, original solver

U, new solver

V, new solver

Normalized L2-error for U in 3D steady-state test 1 by Jablonowski et al: old solver (left), new

solver (right)

0,000000000000000E+00

5,000000000000000E-03

1,000000000000000E-02

1,500000000000000E-02

2,000000000000000E-02

2,500000000000000E-02

3,000000000000000E-02

3,500000000000000E-02

4,000000000000000E-02

4,500000000000000E-02

0 100 200 300 400 500 600 700

Ряд1

0,000000000000000E+00

1,000000000000000E-02

2,000000000000000E-02

3,000000000000000E-02

4,000000000000000E-02

5,000000000000000E-02

6,000000000000000E-02

0 500 1000 1500 2000 2500

New solver also can provide better parallelization using parallel dichotomy algorithm (Terekhov, Par. Computing 2010)

Conclusions• Mass-conservative shallow-water version of SL-AV model

was created. • The error measures in standard SW tests are the same for

conservative and non-conservative versions, except for test #1 where conservative scheme gives smaller errors, and test #2 where conservative scheme gives larger errors.

• The impact of reduced grid on errors is negligible except for test #2.

• The new solver implemented for U-V reconstruction is cheaper, more accurate and more scalable.

• Further work – to implement mass-conservative version of the 3D model.

• There is also ongoing work on NH dynamical core development

Thank you for attention!

Mass-conservative Helmholtz problem solver (2) To obtain discrete divergence operator, the cell-averaged divergence is

represented as a flux of vector field through the cell-boundaries. If Ci,j is an arbitrary non-polar grid cell, and (u,v) is an vector field then:

I east , I west, I north, I south designate flux of (u,v) through one of the cell boundaries. Since cell boundaries are parallel to the coordinate axes, flux operators I are defined by straightforward expressions.

)(),(cos),(1

),( ,,,,2,

,,

jisouth

jinorth

jiwest

jieast

ijCijCijji IIII

S

andvu

S

addavudiv

Svudiv

jiji

If C0 is the south polar cap, and Ci,1 , i=1,M are the cells around, then polar cap averaged divergence is defined as follows

M

i

isouthI

S

avudiv

1

1,

00),(

Flux operators and so cell-averaged divergence operator are discretized with fourth-order accuracy. Laplacian is discretized as cell-averaged divergence of gradient with second order.

Divergence and Laplacian discretized in this way satisfy the last equality from the previous slide and produce conservative Helmholtz problem solver implying tridiagonal matrix inversion.

Another way to achieve mass-conservation is the use of conservative finite-differences for discretization of Helmholtz equation (as in Zerroukat et al. 2009)

Non-conservative model: relative total mass errors

Regular grid with the resolution 1.50 – red lines, 10 – green lines, 0.750 – blue lines; tests #1 (A), #2 (B), #6 (C), #7a (D)

A B

C

D

B

C

A

D