manual cfd

Contents

1 Overview of MITgcm 91.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.2 Illustrations of the model in action . . . . . . . . . . . . . . . . . 13

1.2.1 Global atmosphere: ‘Held-Suarez’ benchmark . . . . . . . 131.2.2 Ocean gyres . . . . . . . . . . . . . . . . . . . . . . . . . . 151.2.3 Global ocean circulation . . . . . . . . . . . . . . . . . . . 151.2.4 Convection and mixing over topography . . . . . . . . . . 151.2.5 Boundary forced internal waves . . . . . . . . . . . . . . . 191.2.6 Parameter sensitivity using the adjoint of MITgcm . . . . 211.2.7 Global state estimation of the ocean . . . . . . . . . . . . 211.2.8 Ocean biogeochemical cycles . . . . . . . . . . . . . . . . 221.2.9 Simulations of laboratory experiments . . . . . . . . . . . 22

1.3 Continuous equations in ‘r’ coordinates . . . . . . . . . . . . . . . 221.3.1 Kinematic Boundary conditions . . . . . . . . . . . . . . . 291.3.2 Atmosphere . . . . . . . . . . . . . . . . . . . . . . . . . . 291.3.3 Ocean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301.3.4 Hydrostatic, Quasi-hydrostatic, Quasi-nonhydrostatic and

Non-hydrostatic forms . . . . . . . . . . . . . . . . . . . . 311.3.5 Solution strategy . . . . . . . . . . . . . . . . . . . . . . . 351.3.6 Finding the pressure field . . . . . . . . . . . . . . . . . . 361.3.7 Forcing/dissipation . . . . . . . . . . . . . . . . . . . . . . 381.3.8 Vector invariant form . . . . . . . . . . . . . . . . . . . . 391.3.9 Adjoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

1.4 Appendix ATMOSPHERE . . . . . . . . . . . . . . . . . . . . . . 391.4.1 Hydrostatic Primitive Equations for the Atmosphere in

pressure coordinates . . . . . . . . . . . . . . . . . . . . . 391.5 Appendix OCEAN . . . . . . . . . . . . . . . . . . . . . . . . . . 41

1.5.1 Equations of motion for the ocean . . . . . . . . . . . . . 411.6 Appendix:OPERATORS . . . . . . . . . . . . . . . . . . . . . . . 46

1.6.1 Coordinate systems . . . . . . . . . . . . . . . . . . . . . 46

1

2 CONTENTS

2 Discretization and Algorithm 472.1 Time-stepping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.2 Pressure method with rigid-lid . . . . . . . . . . . . . . . . . . . 482.3 Pressure method with implicit linear free-surface . . . . . . . . . 512.4 Explicit time-stepping: Adams-Bashforth . . . . . . . . . . . . . 522.5 Implicit time-stepping: backward method . . . . . . . . . . . . . 532.6 Synchronous time-stepping: variables co-located in time . . . . . 532.7 Staggered baroclinic time-stepping . . . . . . . . . . . . . . . . . 562.8 Non-hydrostatic formulation . . . . . . . . . . . . . . . . . . . . . 582.9 Variants on the Free Surface . . . . . . . . . . . . . . . . . . . . . 61

2.9.1 Crank-Nickelson barotropic time stepping . . . . . . . . . 622.9.2 Non-linear free-surface . . . . . . . . . . . . . . . . . . . . 64

2.10 Spatial discretization of the dynamical equations . . . . . . . . . 702.10.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 702.10.2 The finite volume method: finite volumes versus finite

difference . . . . . . . . . . . . . . . . . . . . . . . . . . . 702.10.3 C grid staggering of variables . . . . . . . . . . . . . . . . 712.10.4 Grid initialization and data . . . . . . . . . . . . . . . . . 722.10.5 Horizontal grid . . . . . . . . . . . . . . . . . . . . . . . . 722.10.6 Vertical grid . . . . . . . . . . . . . . . . . . . . . . . . . 752.10.7 Topography: partially filled cells . . . . . . . . . . . . . . 77

2.11 Continuity and horizontal pressure gradient terms . . . . . . . . 792.12 Hydrostatic balance . . . . . . . . . . . . . . . . . . . . . . . . . 802.13 Flux-form momentum equations . . . . . . . . . . . . . . . . . . 80

2.13.1 Advection of momentum . . . . . . . . . . . . . . . . . . . 812.13.2 Coriolis terms . . . . . . . . . . . . . . . . . . . . . . . . . 822.13.3 Curvature metric terms . . . . . . . . . . . . . . . . . . . 822.13.4 Non-hydrostatic metric terms . . . . . . . . . . . . . . . . 832.13.5 Lateral dissipation . . . . . . . . . . . . . . . . . . . . . . 842.13.6 Vertical dissipation . . . . . . . . . . . . . . . . . . . . . . 852.13.7 Derivation of discrete energy conservation . . . . . . . . . 86

2.14 Vector invariant momentum equations . . . . . . . . . . . . . . . 862.14.1 Relative vorticity . . . . . . . . . . . . . . . . . . . . . . . 872.14.2 Kinetic energy . . . . . . . . . . . . . . . . . . . . . . . . 872.14.3 Coriolis terms . . . . . . . . . . . . . . . . . . . . . . . . . 882.14.4 Shear terms . . . . . . . . . . . . . . . . . . . . . . . . . . 882.14.5 Gradient of Bernoulli function . . . . . . . . . . . . . . . 892.14.6 Horizontal dissipation . . . . . . . . . . . . . . . . . . . . 892.14.7 Horizontal dissipation . . . . . . . . . . . . . . . . . . . . 892.14.8 Vertical dissipation . . . . . . . . . . . . . . . . . . . . . . 90

2.15 Tracer equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 902.15.1 Time-stepping of tracers: ABII . . . . . . . . . . . . . . . 91

2.16 Linear advection schemes . . . . . . . . . . . . . . . . . . . . . . 922.16.1 Centered second order advection-diffusion . . . . . . . . . 922.16.2 Third order upwind bias advection . . . . . . . . . . . . . 952.16.3 Centered fourth order advection . . . . . . . . . . . . . . 96

CONTENTS 3

2.16.4 First order upwind advection . . . . . . . . . . . . . . . . 962.17 Non-linear advection schemes . . . . . . . . . . . . . . . . . . . . 97

2.17.1 Second order flux limiters . . . . . . . . . . . . . . . . . . 972.17.2 Third order direct space time . . . . . . . . . . . . . . . . 982.17.3 Third order direct space time with flux limiting . . . . . . 992.17.4 Multi-dimensional advection . . . . . . . . . . . . . . . . . 103

2.18 Comparison of advection schemes . . . . . . . . . . . . . . . . . . 1032.19 Shapiro Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

3 Getting Started with MITgcm 1073.1 Where to find information . . . . . . . . . . . . . . . . . . . . . . 1073.2 Obtaining the code . . . . . . . . . . . . . . . . . . . . . . . . . . 108

3.2.1 Method 1 - Checkout from CVS . . . . . . . . . . . . . . 1083.2.2 Method 2 - Tar file download . . . . . . . . . . . . . . . . 109

3.3 Model and directory structure . . . . . . . . . . . . . . . . . . . . 1113.4 MITgcm Example Experiments . . . . . . . . . . . . . . . . . . . 112

3.4.1 Full list of model examples . . . . . . . . . . . . . . . . . 1123.4.2 Directory structure of model examples . . . . . . . . . . . 113

3.5 Building MITgcm . . . . . . . . . . . . . . . . . . . . . . . . . . . 1143.6 Running MITgcm . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

3.6.1 Output files . . . . . . . . . . . . . . . . . . . . . . . . . . 1163.6.2 Looking at the output . . . . . . . . . . . . . . . . . . . . 117

3.7 Tutorials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1193.8 Barotropic Gyre MITgcm Example . . . . . . . . . . . . . . . . . 119

3.8.1 Equations Solved . . . . . . . . . . . . . . . . . . . . . . . 1193.8.2 Discrete Numerical Configuration . . . . . . . . . . . . . . 1213.8.3 Code Configuration . . . . . . . . . . . . . . . . . . . . . 122

3.9 Baroclinic Gyre MITgcm Example . . . . . . . . . . . . . . . . . 1293.9.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 1293.9.2 Equations solved . . . . . . . . . . . . . . . . . . . . . . . 1303.9.3 Discrete Numerical Configuration . . . . . . . . . . . . . . 1313.9.4 Code Configuration . . . . . . . . . . . . . . . . . . . . . 1333.9.5 Running The Example . . . . . . . . . . . . . . . . . . . . 141

3.10 Global Ocean MITgcm Exmaple . . . . . . . . . . . . . . . . . . 1433.10.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 1433.10.2 Discrete Numerical Configuration . . . . . . . . . . . . . . 1443.10.3 Experiment Configuration . . . . . . . . . . . . . . . . . . 146

3.11 P coordinate Global Ocean MITgcm Example . . . . . . . . . . . 1573.11.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 1573.11.2 Discrete Numerical Configuration . . . . . . . . . . . . . . 1583.11.3 Experiment Configuration . . . . . . . . . . . . . . . . . . 159

3.12 Held-Suarez Atmosphere MITgcm Example . . . . . . . . . . . . 1763.12.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 1763.12.2 Discrete Numerical Configuration . . . . . . . . . . . . . . 1773.12.3 Experiment Configuration . . . . . . . . . . . . . . . . . . 180

3.13 Surface Driven Convection . . . . . . . . . . . . . . . . . . . . . . 190

4 CONTENTS

3.13.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 1903.13.2 Equations solved . . . . . . . . . . . . . . . . . . . . . . . 1933.13.3 Discrete numerical configuration . . . . . . . . . . . . . . 1933.13.4 Numerical stability criteria and other considerations . . . 1933.13.5 Experiment configuration . . . . . . . . . . . . . . . . . . 1943.13.6 Running the example . . . . . . . . . . . . . . . . . . . . 203

3.14 Gravity Plume On a Continental Slope . . . . . . . . . . . . . . . 2053.14.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 2063.14.2 Binary input data . . . . . . . . . . . . . . . . . . . . . . 2063.14.3 Code configuration . . . . . . . . . . . . . . . . . . . . . . 2093.14.4 Model parameters . . . . . . . . . . . . . . . . . . . . . . 2103.14.5 Build and run the model . . . . . . . . . . . . . . . . . . . 210

3.15 Centennial Time Scale Tracer Injection . . . . . . . . . . . . . . . 2123.15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2123.15.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 2123.15.3 Discrete Numerical Configuration . . . . . . . . . . . . . . 2133.15.4 Code Configuration . . . . . . . . . . . . . . . . . . . . . 215

3.16 Customizing MITgcm . . . . . . . . . . . . . . . . . . . . . . . . 2203.16.1 Building/compiling the code elsewhere . . . . . . . . . . . 2203.16.2 Using genmake2 . . . . . . . . . . . . . . . . . . . . . . . 2223.16.3 Building with MPI . . . . . . . . . . . . . . . . . . . . . . 2253.16.4 Configuration and setup . . . . . . . . . . . . . . . . . . . 2263.16.5 Computational domain, geometry and time-discretization 2263.16.6 Equation of state . . . . . . . . . . . . . . . . . . . . . . . 2283.16.7 Momentum equations . . . . . . . . . . . . . . . . . . . . 2293.16.8 Tracer equations . . . . . . . . . . . . . . . . . . . . . . . 2313.16.9 Simulation controls . . . . . . . . . . . . . . . . . . . . . . 232

3.17 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2343.17.1 Using testreport . . . . . . . . . . . . . . . . . . . . . . 2343.17.2 Automated testing . . . . . . . . . . . . . . . . . . . . . . 234

4 Software Architecture 2354.1 Overall architectural goals . . . . . . . . . . . . . . . . . . . . . . 2354.2 WRAPPER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

4.2.1 Target hardware . . . . . . . . . . . . . . . . . . . . . . . 2374.2.2 Supporting hardware neutrality . . . . . . . . . . . . . . . 2374.2.3 WRAPPER machine model . . . . . . . . . . . . . . . . . 2394.2.4 Machine model parallelism . . . . . . . . . . . . . . . . . 2394.2.5 Communication mechanisms . . . . . . . . . . . . . . . . 2404.2.6 Shared memory communication . . . . . . . . . . . . . . . 2414.2.7 Distributed memory communication . . . . . . . . . . . . 2444.2.8 Communication primitives . . . . . . . . . . . . . . . . . . 2444.2.9 Memory architecture . . . . . . . . . . . . . . . . . . . . . 2464.2.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 248

4.3 Using the WRAPPER . . . . . . . . . . . . . . . . . . . . . . . . 2484.3.1 Specifying a domain decomposition . . . . . . . . . . . . . 249

CONTENTS 5

4.3.2 Starting the code . . . . . . . . . . . . . . . . . . . . . . . 2544.3.3 Controlling communication . . . . . . . . . . . . . . . . . 259

4.4 MITgcm execution under WRAPPER . . . . . . . . . . . . . . . 2654.4.1 Annotated call tree for MITgcm and WRAPPER . . . . . 2654.4.2 Measuring and Characterizing Performance . . . . . . . . 2724.4.3 Estimating Resource Requirements . . . . . . . . . . . . . 272

5 Automatic Differentiation 2735.1 Some basic algebra . . . . . . . . . . . . . . . . . . . . . . . . . . 273

5.1.1 Forward or direct sensitivity . . . . . . . . . . . . . . . . . 2745.1.2 Reverse or adjoint sensitivity . . . . . . . . . . . . . . . . 2745.1.3 Storing vs. recomputation in reverse mode . . . . . . . . 278

5.2 TLM and ADM generation in general . . . . . . . . . . . . . . . 2805.2.1 General setup . . . . . . . . . . . . . . . . . . . . . . . . 2815.2.2 Building the AD code . . . . . . . . . . . . . . . . . . . . 2825.2.3 The AD build process in detail . . . . . . . . . . . . . . . 2835.2.4 The cost function (dependent variable) . . . . . . . . . . 2855.2.5 The control variables (independent variables) . . . . . . . 287

5.3 Sensitivity of Air-Sea Exchange to Tracer Injection Site . . . . . 2945.3.1 Overview of the experiment . . . . . . . . . . . . . . . . . 2945.3.2 Code configuration . . . . . . . . . . . . . . . . . . . . . . 2955.3.3 Compiling the model and its adjoint . . . . . . . . . . . . 301

5.4 The gradient check package . . . . . . . . . . . . . . . . . . . . . 3045.4.1 Code description . . . . . . . . . . . . . . . . . . . . . . . 3045.4.2 Code configuration . . . . . . . . . . . . . . . . . . . . . . 304

5.5 Adjoint dump & restart – divided adjoint (DIVA) . . . . . . . . 3055.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3055.5.2 Recipe 1: single processor . . . . . . . . . . . . . . . . . . 3075.5.3 Recipe 2: multi processor (MPI) . . . . . . . . . . . . . . 308

6 Physical Parameterization and Packages 3096.1 Using MITgcm Packages . . . . . . . . . . . . . . . . . . . . . . . 310

6.1.1 Package Inclusion/Exclusion . . . . . . . . . . . . . . . . 3106.1.2 Package Activation . . . . . . . . . . . . . . . . . . . . . . 311

6.2 Package Coding Standards . . . . . . . . . . . . . . . . . . . . . . 3116.2.1 Packages are Not Libraries . . . . . . . . . . . . . . . . . 311

6.3 Gent/McWiliams/Redi SGS Eddy parameterization . . . . . . . 3156.3.1 Redi scheme: Isopycnal diffusion . . . . . . . . . . . . . . 3156.3.2 GM parameterization . . . . . . . . . . . . . . . . . . . . 3166.3.3 Griffies Skew Flux . . . . . . . . . . . . . . . . . . . . . . 3166.3.4 Variable κGM . . . . . . . . . . . . . . . . . . . . . . . . . 3176.3.5 Tapering and stability . . . . . . . . . . . . . . . . . . . . 3186.3.6 Tapering: Danabasoglu and McWilliams, J. Clim. 1995 . 3206.3.7 Tapering: Large, Danabasoglu and Doney, JPO 1997 . . . 3216.3.8 Package Reference . . . . . . . . . . . . . . . . . . . . . . 321

6.4 DIC Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

6 CONTENTS

6.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3226.4.2 Key subroutines and parameters . . . . . . . . . . . . . . 3226.4.3 Do’s and Don’ts . . . . . . . . . . . . . . . . . . . . . . . 3236.4.4 Reference Material . . . . . . . . . . . . . . . . . . . . . . 323

6.5 Ocean vertical mixing – the nonlocal K-profile parameterizationscheme KPP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324

6.6 Thermodynamic Sea Ice Package: “thsice” . . . . . . . . . . . . . 3256.7 Sea Ice Package: “seaice” . . . . . . . . . . . . . . . . . . . . . . 3326.8 Bulk Formula Package . . . . . . . . . . . . . . . . . . . . . . . . 3336.9 Generic Advection/Diffusion . . . . . . . . . . . . . . . . . . . . . 338

6.9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3386.9.2 Key subroutines, parameters and files . . . . . . . . . . . 338

6.10 Atmospheric Intermediate Physics: AIM . . . . . . . . . . . . . . 3396.10.1 Key subroutines, parameters and files . . . . . . . . . . . 339

6.11 Land package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3406.12 Coupling interface for Atmospheric Intermediate code . . . . . . 342

6.12.1 Key subroutines, parameters and files . . . . . . . . . . . 3426.13 Coupler for mapping between AIM and ocean . . . . . . . . . . 343

6.13.1 Key subroutines, parameters and files . . . . . . . . . . . 3436.14 Toolkit for building couplers . . . . . . . . . . . . . . . . . . . . . 344

6.14.1 Key subroutines, parameters and files . . . . . . . . . . . 3446.15 NetCDF I/O Integration: MNC . . . . . . . . . . . . . . . . . . . 345

6.15.1 Using MNC . . . . . . . . . . . . . . . . . . . . . . . . . . 3456.15.2 MNC Internals . . . . . . . . . . . . . . . . . . . . . . . . 347

6.16 MDSIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3516.17 Simulation state monitoring toolkit . . . . . . . . . . . . . . . . . 352

6.17.1 Key subroutines, parameters and files . . . . . . . . . . . 3526.18 exch2: Extended Cubed Sphere Topology . . . . . . . . . . . . . 353

6.18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3536.18.2 Invoking exch2 . . . . . . . . . . . . . . . . . . . . . . . . 3536.18.3 Generating Topology Files for exch2 . . . . . . . . . . . . 3546.18.4 exch2, SIZE.h, and Multiprocessing . . . . . . . . . . . . 3566.18.5 Key Variables . . . . . . . . . . . . . . . . . . . . . . . . . 3596.18.6 Key Routines . . . . . . . . . . . . . . . . . . . . . . . . . 362

6.19 Diagnostics–A Flexible Infrastructure . . . . . . . . . . . . . . . 3646.19.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3646.19.2 Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 3646.19.3 Key Subroutines and Parameters . . . . . . . . . . . . . . 3646.19.4 Usage Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 3676.19.5 Dos and Donts . . . . . . . . . . . . . . . . . . . . . . . . 4046.19.6 Diagnostics Reference . . . . . . . . . . . . . . . . . . . . 404

6.20 RW Basic binary I/O utilities . . . . . . . . . . . . . . . . . . . . 4056.20.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 405

6.21 FFT Filtering Code . . . . . . . . . . . . . . . . . . . . . . . . . 4066.21.1 Key subroutines, parameters and files . . . . . . . . . . . 406

6.22 GCHEM Package . . . . . . . . . . . . . . . . . . . . . . . . . . . 407

CONTENTS 7

6.22.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4076.22.2 Key subroutines and parameters . . . . . . . . . . . . . . 4076.22.3 Do’s and Don’ts . . . . . . . . . . . . . . . . . . . . . . . 4086.22.4 Reference Material . . . . . . . . . . . . . . . . . . . . . . 409

7 Diagnostics and tools 4117.1 Utilities supplied with the model . . . . . . . . . . . . . . . . . . 411

7.1.1 utils/scripts . . . . . . . . . . . . . . . . . . . . . . . . . . 4117.1.2 utils/matlab . . . . . . . . . . . . . . . . . . . . . . . . . 411

7.2 Pre-processing software . . . . . . . . . . . . . . . . . . . . . . . 412

8 Interface with ECCO 4138.1 The ECCO state estimation cost function DRAFT!!! . . . . . . 413

8.1.1 Sea surface height from TOPEX/Poseidon and ERS-1/2altimetry . . . . . . . . . . . . . . . . . . . . . . . . . . . 413

8.1.2 Hydrographic constraints . . . . . . . . . . . . . . . . . . 4168.2 The external forcing package exf . . . . . . . . . . . . . . . . . . 420

8.2.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 4208.3 The calendar package cal . . . . . . . . . . . . . . . . . . . . . . 423

8.3.1 Basic assumptions for the calendar tool . . . . . . . . . . 4238.3.2 Format of calendar dates . . . . . . . . . . . . . . . . . . 4238.3.3 Calendar dates and time intervals . . . . . . . . . . . . . 4248.3.4 Using the calendar together with MITgcm . . . . . . . . . 4248.3.5 The individual calendars . . . . . . . . . . . . . . . . . . . 4258.3.6 Short routine description . . . . . . . . . . . . . . . . . . 425

8.4 The line search optimisation algorithm . . . . . . . . . . . . . . 4288.4.1 General features . . . . . . . . . . . . . . . . . . . . . . . 4288.4.2 The online vs. offline version . . . . . . . . . . . . . . . . 4288.4.3 Number of iterations vs. number of simulations . . . . . . 428

9 Model Uses 435

BIBLIOGRAPHY 439

8 CONTENTS

Chapter 1

Overview of MITgcm

This document provides the reader with the information necessary to carry outnumerical experiments using MITgcm. It gives a comprehensive description ofthe continuous equations on which the model is based, the numerical algorithmsthe model employs and a description of the associated program code. Along withthe hydrodynamical kernel, physical and biogeochemical parameterizations ofkey atmospheric and oceanic processes are available. A number of examplesillustrating the use of the model in both process and general circulation studiesof the atmosphere and ocean are also presented.

1.1 Introduction

MITgcm has a number of novel aspects:

• it can be used to study both atmospheric and oceanic phenomena; onehydrodynamical kernel is used to drive forward both atmospheric andoceanic models - see fig 1.1

• it has a non-hydrostatic capability and so can be used to study both small-scale and large scale processes - see fig 1.2

• finite volume techniques are employed yielding an intuitive discretizationand support for the treatment of irregular geometries using orthogonalcurvilinear grids and shaved cells - see fig 1.3

• tangent linear and adjoint counterparts are automatically maintained alongwith the forward model, permitting sensitivity and optimization studies.

• the model is developed to perform efficiently on a wide variety of compu-tational platforms.

Key publications reporting on and charting the development of the modelare [30, 39, 38, 5, 40, 6, 9, 37]:

9

10 CHAPTER 1. OVERVIEW OF MITGCM

Atmospheric Ocean

Dynamical Kernel

AtmosphericPhysics

ModelModel

OceanPhysics

Figure 1.1: MITgcm has a single dynamical kernel that can drive forward eitheroceanic or atmospheric simulations.

Hill, C. and J. Marshall, (1995)Application of a Parallel Navier-Stokes Model to Ocean Circulation inParallel Computational Fluid DynamicsIn Proceedings of Parallel Computational Fluid Dynamics: Implementationsand Results Using Parallel Computers, 545-552.Elsevier Science B.V.: New York

Marshall, J., C. Hill, L. Perelman, and A. Adcroft, (1997)Hydrostatic, quasi-hydrostatic, and nonhydrostatic ocean modelingJ. Geophysical Res., 102(C3), 5733-5752.

Marshall, J., A. Adcroft, C. Hill, L. Perelman, and C. Heisey, (1997)A finite-volume, incompressible Navier Stokes model for studies of the oceanon parallel computers,J. Geophysical Res., 102(C3), 5753-5766.

Adcroft, A.J., Hill, C.N. and J. Marshall, (1997)Representation of topography by shaved cells in a height coordinate oceanmodelMon Wea Rev, vol 125, 2293-2315

Marshall, J., Jones, H. and C. Hill, (1998)Efficient ocean modeling using non-hydrostatic algorithmsJournal of Marine Systems, 18, 115-134

Adcroft, A., Hill C. and J. Marshall: (1999)

1.1. INTRODUCTION 11

~10 000km~1 000km~100km

~1km~10km

~100m

Figure 1.2: MITgcm has non-hydrostatic capabilities, allowing the model toaddress a wide range of phenomenon - from convection on the left, all the waythrough to global circulation patterns on the right.


te Vo u e: S aved ce s

Figure 1.3: Finite volume techniques (bottom panel) are user, permitting atreatment of topography that rivals σ (terrain following) coordinates.

1.2. ILLUSTRATIONS OF THE MODEL IN ACTION 13

A new treatment of the Coriolis terms in C-grid models at both high and lowresolutions,Mon. Wea. Rev. Vol 127, pages 1928-1936

Hill, C, Adcroft,A., Jamous,D., and J. Marshall, (1999)A Strategy for Terascale Climate Modeling.In Proceedings of the Eighth ECMWF Workshop on the Use of Parallel Processorsin Meteorology, pages 406-425World Scientific Publishing Co: UK

Marotzke, J, Giering,R., Zhang, K.Q., Stammer,D., Hill,C., and T.Lee, (1999)Construction of the adjoint MIT ocean general circulation model andapplication to Atlantic heat transport variabilityJ. Geophysical Res., 104(C12), 29,529-29,547.

We begin by briefly showing some of the results of the model in action togive a feel for the wide range of problems that can be addressed using it.

1.2 Illustrations of the model in action

The MITgcm has been designed and used to model a wide range of phenomena,from convection on the scale of meters in the ocean to the global pattern ofatmospheric winds - see figure 1.2. To give a flavor of the kinds of problemsthe model has been used to study, we briefly describe some of them here. Amore detailed description of the underlying formulation, numerical algorithmand implementation that lie behind these calculations is given later. Indeedmany of the illustrative examples shown below can be easily reproduced: simplydownload the model (the minimum you need is a PC running Linux, togetherwith a FORTRAN 77 compiler) and follow the examples described in detail inthe documentation.

1.2.1 Global atmosphere: ‘Held-Suarez’ benchmark

A novel feature of MITgcm is its ability to simulate, using one basic algorithm,both atmospheric and oceanographic flows at both small and large scales.

Figure 1.4 shows an instantaneous plot of the 500mb temperature field ob-tained using the atmospheric isomorph of MITgcm run at 2.8◦ resolution on thecubed sphere. We see cold air over the pole (blue) and warm air along an equa-torial band (red). Fully developed baroclinic eddies spawned in the northernhemisphere storm track are evident. There are no mountains or land-sea con-trast in this calculation, but you can easily put them in. The model is driven byrelaxation to a radiative-convective equilibrium profile, following the descriptionset out in Held and Suarez; 1994 designed to test atmospheric hydrodynamicalcores - there are no mountains or land-sea contrast.


Figure 1.4: Instantaneous plot of the temerature field at 500mb obtained usingthe atmospheric isomorph of MITgcm


As described in Adcroft (2001), a ‘cubed sphere’ is used to discretize theglobe permitting a uniform griding and obviated the need to Fourier filter. The‘vector-invariant’ form of MITgcm supports any orthogonal curvilinear grid, ofwhich the cubed sphere is just one of many choices.

Figure 1.5 shows the 5-year mean, zonally averaged zonal wind from a 20-level configuration of the model. It compares favorable with more conventionalspatial discretization approaches. The two plots show the field calculated usingthe cube-sphere grid and the flow calculated using a regular, spherical polarlatitude-longitude grid. Both grids are supported within the model.

1.2.2 Ocean gyres

Baroclinic instability is a ubiquitous process in the ocean, as well as the at-mosphere. Ocean eddies play an important role in modifying the hydrographicstructure and current systems of the oceans. Coarse resolution models of theoceans cannot resolve the eddy field and yield rather broad, diffusive patternsof ocean currents. But if the resolution of our models is increased until thebaroclinic instability process is resolved, numerical solutions of a different andmuch more realistic kind, can be obtained.

Figure 1.6 shows the surface temperature and velocity field obtained fromMITgcm run at 1

6

◦ horizontal resolution on a lat−lon grid in which the pole hasbeen rotated by 90◦ on to the equator (to avoid the converging of meridian innorthern latitudes). 21 vertical levels are used in the vertical with a ‘lopped cell’representation of topography. The development and propagation of anomalouslywarm and cold eddies can be clearly seen in the Gulf Stream region. Thetransport of warm water northward by the mean flow of the Gulf Stream is alsoclearly visible.

1.2.3 Global ocean circulation

Figure 1.7 (top) shows the pattern of ocean currents at the surface of a 4◦ globalocean model run with 15 vertical levels. Lopped cells are used to representtopography on a regular lat − lon grid extending from 70◦N to 70◦S. Themodel is driven using monthly-mean winds with mixed boundary conditions ontemperature and salinity at the surface. The transfer properties of ocean eddies,convection and mixing is parameterized in this model.

Figure 1.7 (bottom) shows the meridional overturning circulation of theglobal ocean in Sverdrups.

1.2.4 Convection and mixing over topography

Dense plumes generated by localized cooling on the continental shelf of theocean may be influenced by rotation when the deformation radius is smaller thanthe width of the cooling region. Rather than gravity plumes, the mechanismfor moving dense fluid down the shelf is then through geostrophic eddies. Thesimulation shown in the figure 1.8 (blue is cold dense fluid, red is warmer, lighter


Figure 1.5: Five year mean, zonally averaged zonal flow for latitude-longitudesimulation (bottom) and cube-sphere simulation(top) using Held-Suarez forcing.Note the difference in the solutions over the pole - the cubed sphere is superior.


Figure 1.6: Instantaneous temperature map from a 16

◦ simulation of the NorthAtlantic. The figure shows the temperature in the second layer (37.5m deep).


180W 150W 120W 90W 60W 30W 0 30E 60E 90E 120E 150E 90S

60S

30S

0

30N

60N

90NCurrents at 25M, t=1000 years

0.5 m/s

OCEAN

Figure 1.7: Pattern of surface ocean currents (top) and meridional overturningstream function (in Sverdrups) from a global integration of the model at 4◦

horizontal resolution and with 15 vertical levels.


Figure 1.8: MITgcm run in a non-hydrostatic configuration to study convectionover a slope.

fluid) employs the non-hydrostatic capability of MITgcm to trigger convectionby surface cooling. The cold, dense water falls down the slope but is deflectedalong the slope by rotation. It is found that entrainment in the vertical plane isreduced when rotational control is strong, and replaced by lateral entrainmentdue to the baroclinic instability of the along-slope current.

1.2.5 Boundary forced internal waves

The unique ability of MITgcm to treat non-hydrostatic dynamics in the presenceof complex geometry makes it an ideal tool to study internal wave dynamics andmixing in oceanic canyons and ridges driven by large amplitude barotropic tidalcurrents imposed through open boundary conditions.

Fig. 1.9 shows the influence of cross-slope topographic variations on internalwave breaking - the cross-slope velocity is in color, the density contoured. Theinternal waves are excited by application of open boundary conditions on the left.They propagate to the sloping boundary (represented using MITgcm’s finitevolume spatial discretization) where they break under nonhydrostatic dynamics.


Figure 1.9: Simulation of internal waves forced at an open boundary (on the left)impacting a sloping shelf. The along slope velocity is shown colored, contourlines show density surfaces. The slope is represented with high-fidelity usinglopped cells.


180W 150W 120W 90W 60W 30W 0 30E 60E 90E 120E 150E 180E 90S

60S

30S

0

30N

60N

90NHeat Flux (Min = −7.7 10−4 Sv W−1 m2; Max = 42.9 10−4 Sv W−1 m2)

−10 −5 0 5 10 15 20 25 30 35 40 45 50

10−4 Sv W−1 m2

Figure 1.10: Sensitivity of meridional overturning strength to surface heat fluxchanges. Contours show the magnitude of the response (in Sv × 10−4) that apersistent +1Wm−2 heat flux anomaly at a given gid point would produce.

1.2.6 Parameter sensitivity using the adjoint of MITgcm

Forward and tangent linear counterparts of MITgcm are supported using an‘automatic adjoint compiler’. These can be used in parameter sensitivity anddata assimilation studies.

As one example of application of the MITgcm adjoint, Figure 1.10 maps thegradient ∂J

∂Hwhere J is the magnitude of the overturning stream-function shownin figure 1.7 at 60◦N and H(λ, ϕ) is the mean, local air-sea heat flux over a 100year period. We see that J is sensitive to heat fluxes over the Labrador Sea, oneof the important sources of deep water for the thermohaline circulations. Thiscalculation also yields sensitivities to all other model parameters.

1.2.7 Global state estimation of the ocean

An important application of MITgcm is in state estimation of the global oceancirculation. An appropriately defined ‘cost function’, which measures the depar-ture of the model from observations (both remotely sensed and in-situ) over aninterval of time, is minimized by adjusting ‘control parameters’ such as air-seafluxes, the wind field, the initial conditions etc. Figure 1.11 shows the large scaleplanetary circulation and a Hopf-Muller plot of Equatorial sea-surface height.Both are obtained from assimilation bringing the model in to consistency with


altimetric and in-situ observations over the period 1992-1997.

1.2.8 Ocean biogeochemical cycles

MITgcm is being used to study global biogeochemical cycles in the ocean. Forexample one can study the effects of interannual changes in meteorological forc-ing and upper ocean circulation on the fluxes of carbon dioxide and oxygenbetween the ocean and atmosphere. Figure 1.12 shows the annual air-sea fluxof oxygen and its relation to density outcrops in the southern oceans from asingle year of a global, interannually varying simulation. The simulation is runat 1◦ × 1◦ resolution telescoping to 1

3

◦ × 13

◦ in the tropics (not shown).

1.2.9 Simulations of laboratory experiments

Figure 1.13 shows MITgcm being used to simulate a laboratory experiment in-quiring into the dynamics of the Antarctic Circumpolar Current (ACC). Aninitially homogeneous tank of water (1m in diameter) is driven from its freesurface by a rotating heated disk. The combined action of mechanical and ther-mal forcing creates a lens of fluid which becomes baroclinically unstable. Thestratification and depth of penetration of the lens is arrested by its instabilityin a process analogous to that which sets the stratification of the ACC.

1.3 Continuous equations in ‘r’ coordinates

To render atmosphere and ocean models from one dynamical core we exploit‘isomorphisms’ between equation sets that govern the evolution of the respec-tive fluids - see figure 1.14. One system of hydrodynamical equations is writtendown and encoded. The model variables have different interpretations depend-ing on whether the atmosphere or ocean is being studied. Thus, for example,the vertical coordinate ‘r’ is interpreted as pressure, p, if we are modeling theatmosphere (right hand side of figure 1.14) and height, z, if we are modelingthe ocean (left hand side of figure 1.14).

The state of the fluid at any time is characterized by the distribution ofvelocity �v, active tracers θ and S, a ‘geopotential’ φ and density ρ = ρ(θ, S, p)which may depend on θ, S, and p. The equations that govern the evolution ofthese fields, obtained by applying the laws of classical mechanics and thermo-dynamics to a Boussinesq, Navier-Stokes fluid are, written in terms of a genericvertical coordinate, r, so that the appropriate kinematic boundary conditionscan be applied isomorphically see figure 1.15.

D �vh

Dt+(2�Ω × �v

)h

+ ∇hφ = F �vhhorizontal mtm (1.1)

Dr

Dt+ k ·(2�Ω × �v

)+∂φ

∂r+ b = Fr vertical mtm (1.2)

1.3. CONTINUOUS EQUATIONS IN ‘R’ COORDINATES 23

50 100 150 200 2500

100

200

300

400

500

600

700

(a) Simulation

Longitude

Day

(fro

m 1

/1/1

997)

50 100 150 200 250

(b) Assimilation

Longitude

−30

−20

−10

0

10

20

30

50 100 150 200 250Longitude

(c) Observation (T/P) cm

Figure 1.11: Top panel shows circulation patterns from a multi-year, globalcirculation simulation constrained by Topex altimeter data and WOCE cruiseobservations. Bottom panel shows the equatorial sea-surface height in uncon-strained (left), constrained (middle) simulations and in observations. This out-put is from a higher resolution, shorter duration experiment with equatoriallyenhanced grid spacing.


26.8

26.2

25.5

26.5

27

26.8

26.2

26

25.5

25.5

25.8

25

26

26.5

26.2

26.8

26.5

MITgcm air−sea O2 flux (mol/m2/yr) with contoured potential density

150 oW

120 oW

60

o W

30o W

0o

30 oE

60 oE

90oE

120

o E

150o E

180oW

70oS

60oS

50oS

40oS

−15 −10 −5 0 5 10 15

Figure 1.12: Annual air-sea flux of oxygen (shaded) plotted along with potentialdensity outcrops of the surface of the southern ocean from a global 1◦ × 1◦

integation with a telescoping grid (to 13

◦) at the equator.


20

22

24

26

28

30

10 20 30 40 50 60 70 80 90 100

Figure 1.13: A numerical simulation (left) of a 1m diameter laboratory experi-ment (right) using MITgcm.


Figure 1.14: Isomorphic equation sets used for atmosphere (right) and ocean(left).


z

p

w=0

ω=0

z = 0 mz = -50 m

z = -100 m

z = -175 m

p = 1000 mbp = 900 mb

p = 650 mb

p = 350 mb

p = 150 mb

p = 0 mb

ω

w

Figure 1.15: Vertical coordinates and kinematic boundary conditions for atmo-sphere (top) and ocean (bottom).


∇h · �vh +∂r

∂r= 0 continuity (1.3)

b = b(θ, S, r) equation of state (1.4)

Dθ

Dt= Qθ potential temperature (1.5)

DS

Dt= QS humidity/salinity (1.6)

Here:

r is the vertical coordinate

D

Dt=

∂

∂t+ �v · ∇ is the total derivative

∇ = ∇h + k∂

∂ris the ‘grad’ operator

with ∇h operating in the horizontal and k ∂∂r operating in the vertical, where k

is a unit vector in the vertical

t is time

�v = (u, v, r) = (�vh, r) is the velocity

φ is the ‘pressure’/‘geopotential’

�Ω is the Earth’s rotation

b is the ‘buoyancy’

θ is potential temperature

S is specific humidity in the atmosphere; salinity in the ocean

F�v are forcing and dissipation of �v

Qθ are forcing and dissipation of θ

QS are forcing and dissipation of S

The F ′s and Q′s are provided by ‘physics’ and forcing packages for atmo-sphere and ocean. These are described in later chapters.


1.3.1 Kinematic Boundary conditions

1.3.1.1 vertical

at fixed and moving r surfaces we set (see figure 1.15):

r = 0 at r = Rfixed(x, y) (ocean bottom, top of the atmosphere) (1.7)

r =Dr

Dtat r = Rmoving (ocean surface,bottom of the atmosphere) (1.8)

Here

Rmoving = Ro + η

where Ro(x, y) is the ‘r−value’ (height or pressure, depending on whether weare in the atmosphere or ocean) of the ‘moving surface’ in the resting fluid andη is the departure from Ro(x, y) in the presence of motion.

1.3.1.2 horizontal

�v · �n = 0 (1.9)

where �n is the normal to a solid boundary.

1.3.2 Atmosphere

In the atmosphere, (see figure 1.15), we interpret:

r = p is the pressure (1.10)

r =Dp

Dt= ω is the vertical velocity in p coordinates (1.11)

φ = g z is the geopotential height (1.12)

b =∂Π∂p

θ is the buoyancy (1.13)

θ = T (pc

p)κ is potential temperature (1.14)

S = q, is the specific humidity (1.15)

where

T is absolute temperature


p is the pressure

z is the height of the pressure surfaceg is the acceleration due to gravity

In the above the ideal gas law, p = ρRT , has been expressed in terms of theExner function Π(p) given by (see Appendix Atmosphere)

Π(p) = cp(p

pc)κ (1.16)

where pc is a reference pressure and κ = R/cp with R the gas constant and cpthe specific heat of air at constant pressure.

At the top of the atmosphere (which is ‘fixed’ in our r coordinate):

Rfixed = ptop = 0

In a resting atmosphere the elevation of the mountains at the bottom is givenby

Rmoving = Ro(x, y) = po(x, y)

i.e. the (hydrostatic) pressure at the top of the mountains in a resting atmo-sphere.

The boundary conditions at top and bottom are given by:

ω = 0 at r = Rfixed (top of the atmosphere) (1.17)

ω =Dps

Dt; at r = Rmoving (bottom of the atmosphere) (1.18)

Then the (hydrostatic form of) equations (1.1-1.6) yields a consistent set ofatmospheric equations which, for convenience, are written out in p coordinatesin Appendix Atmosphere - see eqs(1.59).

1.3.3 Ocean

In the ocean we interpret:

r = z is the height (1.19)

r =Dz

Dt= w is the vertical velocity (1.20)

φ =p

ρcis the pressure (1.21)

b(θ, S, r) =g

ρc(ρ(θ, S, r) − ρc) is the buoyancy (1.22)

where ρc is a fixed reference density of water and g is the acceleration due togravity.


In the aboveAt the bottom of the ocean: Rfixed(x, y) = −H(x, y).The surface of the ocean is given by: Rmoving = ηThe position of the resting free surface of the ocean is given by Ro = Zo = 0.Boundary conditions are:

w = 0 at r = Rfixed (ocean bottom) (1.23)

w =Dη

Dtat r = Rmoving = η (ocean surface) (1.24)

where η is the elevation of the free surface.Then equations (1.1-1.6) yield a consistent set of oceanic equations which, for

convenience, are written out in z coordinates in Appendix Ocean - see eqs(1.99)to (1.104).

1.3.4 Hydrostatic, Quasi-hydrostatic, Quasi-nonhydrostaticand Non-hydrostatic forms

Let us separate φ in to surface, hydrostatic and non-hydrostatic terms:

φ(x, y, r) = φs(x, y) + φhyd(x, y, r) + φnh(x, y, r) (1.25)

and write eq( 1.1) in the form:

∂ �vh

∂t+ ∇hφs + ∇hφhyd + εnh∇hφnh = �G�vh

(1.26)

∂φhyd

∂r= −b (1.27)

εnh∂r

∂t+∂φnh

∂r= Gr (1.28)

Here εnh is a non-hydrostatic parameter.The(�G�v, Gr

)in eq(1.26) and (1.28) represent advective, metric and Coriolis

terms in the momentum equations. In spherical coordinates they take the form1 - see Marshall et al 1997a for a full discussion:

Gu = −�v.∇u−{

urr − uv tan ϕ

r

}−{−2Ωv sinϕ+ 2Ωr cosϕ

}+Fu

⎫⎪⎪⎪⎬⎪⎪⎪⎭⎧⎪⎪⎨⎪⎪⎩

advectionmetricCoriolisForcing/Dissipation

(1.29)

1 In the hydrostatic primitive equations (HPE) all underlined terms in (1.29), (1.30) and(1.31) are omitted; the singly-underlined terms are included in the quasi-hydrostatic model(QH). The fully non-hydrostatic model ( NH) includes all terms.


Gv = −�v.∇v−{

vrr − u2 tan ϕ

r

}−{−2Ωu sinϕ}+Fv

⎫⎪⎪⎪⎬⎪⎪⎪⎭⎧⎪⎪⎨⎪⎪⎩


(1.30)

Gr = −�v.∇r+{

u2 +v2

r

}+2Ωu cosϕFr

⎫⎪⎪⎪⎬⎪⎪⎪⎭⎧⎪⎪⎨⎪⎪⎩


(1.31)

In the above ‘r’ is the distance from the center of the earth and ‘ϕ ’ islatitude.

Grad and div operators in spherical coordinates are defined in appendixOPERATORS.

1.3.4.1 Shallow atmosphere approximation

Most models are based on the ‘hydrostatic primitive equations’ (HPE’s) in whichthe vertical momentum equation is reduced to a statement of hydrostatic bal-ance and the ‘traditional approximation’ is made in which the Coriolis force istreated approximately and the shallow atmosphere approximation is made. TheMITgcm need not make the ‘traditional approximation’. To be able to supportconsistent non-hydrostatic forms the shallow atmosphere approximation can berelaxed - when dividing through by r in, for example, (1.29), we do not replacer by a, the radius of the earth.

1.3.4.2 Hydrostatic and quasi-hydrostatic forms

These are discussed at length in Marshall et al (1997a).In the ‘hydrostatic primitive equations’ (HPE) all the underlined terms in

Eqs. (1.29 → 1.31) are neglected and ‘r’ is replaced by ‘a’, the mean radiusof the earth. Once the pressure is found at one level - e.g. by inverting a 2-dElliptic equation for φs at r = Rmoving - the pressure can be computed at allother levels by integration of the hydrostatic relation, eq( 1.27).

In the ‘quasi-hydrostatic’ equations (QH) strict balance between gravity andvertical pressure gradients is not imposed. The 2Ωu cosϕ Coriolis term are notneglected and are balanced by a non-hydrostatic contribution to the pressurefield: only the terms underlined twice in Eqs. ( 1.29→ 1.31) are set to zero and,simultaneously, the shallow atmosphere approximation is relaxed. In QH allthe metric terms are retained and the full variation of the radial position of aparticle monitored. The QH vertical momentum equation (1.28) becomes:

∂φnh

∂r= 2Ωu cosϕ


Figure 1.16: Spherical polar coordinates: longitude λ, latitude ϕ and r thedistance from the center.


making a small correction to the hydrostatic pressure.QH has good energetic credentials - they are the same as for HPE. Im-

portantly, however, it has the same angular momentum principle as the fullnon-hydrostatic model (NH) - see Marshall et.al., 1997a. As in HPE only a2-d elliptic problem need be solved.

1.3.4.3 Non-hydrostatic and quasi-nonhydrostatic forms

The MIT model presently supports a full non-hydrostatic ocean isomorph, butonly a quasi-non-hydrostatic atmospheric isomorph.

Non-hydrostatic Ocean In the non-hydrostatic ocean model all terms inequations Eqs.(1.29 → 1.31) are retained. A three dimensional elliptic equationmust be solved subject to Neumann boundary conditions (see below). It isimportant to note that use of the full NH does not admit any new ‘fast’ wavesin to the system - the incompressible condition eq(1.3) has already filtered outacoustic modes. It does, however, ensure that the gravity waves are treatedaccurately with an exact dispersion relation. The NH set has a complete angularmomentum principle and consistent energetics - see White and Bromley, 1995;Marshall et.al. 1997a.

Quasi-nonhydrostatic Atmosphere In the non-hydrostatic version of ouratmospheric model we approximate r in the vertical momentum eqs(1.28) and(1.30) (but only here) by:

r =Dp

Dt=

1g

Dφ

Dt(1.32)

where phy is the hydrostatic pressure.

1.3.4.4 Summary of equation sets supported by model

Atmosphere Hydrostatic, and quasi-hydrostatic and quasi non-hydrostaticforms of the compressible non-Boussinesq equations in p−coordinates are sup-ported.

Hydrostatic and quasi-hydrostatic The hydrostatic set is written outin p−coordinates in appendix Atmosphere - see eq(1.59).

Quasi-nonhydrostatic A quasi-nonhydrostatic form is also supported.

Ocean

Hydrostatic and quasi-hydrostatic Hydrostatic, and quasi-hydrostaticforms of the incompressible Boussinesq equations in z−coordinates are sup-ported.


Figure 1.17: Basic solution strategy in MITgcm. HPE and QH forms diagnosethe vertical velocity, in NH a prognostic equation for the vertical velocity isintegrated.

Non-hydrostatic Non-hydrostatic forms of the incompressible Boussi-nesq equations in z− coordinates are supported - see eqs(1.99) to (1.104).

1.3.5 Solution strategy

The method of solution employed in the HPE, QH and NH models is sum-marized in Figure 1.17. Under all dynamics, a 2-d elliptic equation is firstsolved to find the surface pressure and the hydrostatic pressure at any levelcomputed from the weight of fluid above. Under HPE and QH dynamics, thehorizontal momentum equations are then stepped forward and r found fromcontinuity. Under NH dynamics a 3-d elliptic equation must be solved forthe non-hydrostatic pressure before stepping forward the horizontal momentumequations; r is found by stepping forward the vertical momentum equation.

There is no penalty in implementing QH over HPE except, of course, somecomplication that goes with the inclusion of cosϕ Coriolis terms and the re-laxation of the shallow atmosphere approximation. But this leads to negligibleincrease in computation. In NH, in contrast, one additional elliptic equation- a three-dimensional one - must be inverted for pnh. However the ‘overhead’of the NH model is essentially negligible in the hydrostatic limit (see detaileddiscussion in Marshall et al, 1997) resulting in a non-hydrostatic algorithm that,in the hydrostatic limit, is as computationally economic as the HPEs.


1.3.6 Finding the pressure field

Unlike the prognostic variables u, v, w, θ and S, the pressure field must beobtained diagnostically. We proceed, as before, by dividing the total (pres-sure/geo) potential in to three parts, a surface part, φs(x, y), a hydrostatic partφhyd(x, y, r) and a non-hydrostatic part φnh(x, y, r), as in (1.25), and writingthe momentum equation as in (1.26).

1.3.6.1 Hydrostatic pressure

Hydrostatic pressure is obtained by integrating (1.27) vertically from r = Ro

where φhyd(r = Ro) = 0, to yield:∫ Ro

r

∂φhyd

∂rdr = [φhyd]

Ro

r =∫ Ro

r

−bdr

and so

φhyd(x, y, r) =∫ Ro

r

bdr (1.33)

The model can be easily modified to accommodate a loading term (e.g at-mospheric pressure pushing down on the ocean’s surface) by setting:

φhyd(r = Ro) = loading (1.34)

1.3.6.2 Surface pressure

The surface pressure equation can be obtained by integrating continuity, (1.3),vertically from r = Rfixed to r = Rmoving∫ Rmoving

Rfixed

(∇h · �vh + ∂r r) dr = 0

Thus:

∂η

∂t+ �v.∇η +

∫ Rmoving

Rfixed

∇h · �vhdr = 0

where η = Rmoving −Ro is the free-surface r-anomaly in units of r. The abovecan be rearranged to yield, using Leibnitz’s theorem:

∂η

∂t+ ∇h ·

∫ Rmoving

Rfixed

�vhdr = source (1.35)

where we have incorporated a source term.Whether φ is pressure (ocean model, p/ρc) or geopotential (atmospheric

model), in (1.26), the horizontal gradient term can be written

∇hφs = ∇h (bsη) (1.36)


where bs is the buoyancy at the surface.In the hydrostatic limit (εnh = 0), equations (1.26), (1.35) and (1.36) can

be solved by inverting a 2-d elliptic equation for φs as described in Chapter 2.Both ‘free surface’ and ‘rigid lid’ approaches are available.

1.3.6.3 Non-hydrostatic pressure

Taking the horizontal divergence of (1.26) and adding ∂∂r of (1.28), invoking the

continuity equation (1.3), we deduce that:

∇23φnh = ∇. �G�v − (∇2

hφs + ∇2φhyd

)= ∇.�F (1.37)

For a given rhs this 3-d elliptic equation must be inverted for φnh subject toappropriate choice of boundary conditions. This method is usually called ThePressure Method [Harlow and Welch, 1965; Williams, 1969; Potter, 1976]. Inthe hydrostatic primitive equations case (HPE), the 3-d problem does not needto be solved.

Boundary Conditions We apply the condition of no normal flow throughall solid boundaries - the coasts (in the ocean) and the bottom:

�v.n = 0 (1.38)

where n is a vector of unit length normal to the boundary. The kinematiccondition (1.38) is also applied to the vertical velocity at r = Rmoving. No-slip (vT = 0) or slip (∂vT /∂n = 0) conditions are employed on the tangentialcomponent of velocity, vT , at all solid boundaries, depending on the form chosenfor the dissipative terms in the momentum equations - see below.

Eq.(1.38) implies, making use of (1.26), that:

n.∇φnh = n.�F (1.39)

where

�F = �G�v − (∇hφs + ∇φhyd)

presenting inhomogeneous Neumann boundary conditions to the Elliptic prob-lem (1.37). As shown, for example, by Williams (1969), one can exploit classi-cal 3D potential theory and, by introducing an appropriately chosen δ-functionsheet of ‘source-charge’, replace the inhomogeneous boundary condition on pres-sure by a homogeneous one. The source term rhs in (1.37) is the divergenceof the vector �F. By simultaneously setting n.�F = 0 and n.∇φnh = 0 on theboundary the following self-consistent but simpler homogenized Elliptic problemis obtained:

∇2φnh = ∇.�F


where �F is a modified �F such that �F.n = 0. As is implied by (1.39) the modifiedboundary condition becomes:

n.∇φnh = 0 (1.40)

If the flow is ‘close’ to hydrostatic balance then the 3-d inversion convergesrapidly because φnh is then only a small correction to the hydrostatic pressurefield (see the discussion in Marshall et al, a,b).

The solution φnh to (1.37) and (1.39) does not vanish at r = Rmoving, andso refines the pressure there.

1.3.7 Forcing/dissipation

1.3.7.1 Forcing

The forcing terms F on the rhs of the equations are provided by ‘physics pack-ages’ and forcing packages. These are described later on.

1.3.7.2 Dissipation

Momentum Many forms of momentum dissipation are available in the model.Laplacian and biharmonic frictions are commonly used:

DV = Ah∇2hv +Av

∂2v

∂z2+A4∇4

hv (1.41)

where Ah and Av are (constant) horizontal and vertical viscosity coefficientsand A4 is the horizontal coefficient for biharmonic friction. These coefficientsare the same for all velocity components.

Tracers The mixing terms for the temperature and salinity equations havea similar form to that of momentum except that the diffusion tensor can benon-diagonal and have varying coefficients.

DT,S = ∇.[K∇(T, S)] +K4∇4h(T, S) (1.42)

where K is the diffusion tensor and the K4 horizontal coefficient for biharmonicdiffusion. In the simplest case where the subgrid-scale fluxes of heat and saltare parameterized with constant horizontal and vertical diffusion coefficients,K, reduces to a diagonal matrix with constant coefficients:

K =

⎛⎝ Kh 0 00 Kh 00 0 Kv

⎞⎠ (1.43)

where Kh and Kv are the horizontal and vertical diffusion coefficients. Thesecoefficients are the same for all tracers (temperature, salinity ... ).

1.4. APPENDIX ATMOSPHERE 39

1.3.8 Vector invariant form

For some purposes it is advantageous to write momentum advection in eq(1.1)and (1.2) in the (so-called) ‘vector invariant’ form:

D�vDt

=∂�v∂t

+ (∇× �v) × �v + ∇[12(�v · �v)]

(1.44)

This permits alternative numerical treatments of the non-linear terms based ontheir representation as a vorticity flux. Because gradients of coordinate vectorsno longer appear on the rhs of (1.44), explicit representation of the metric termsin (1.29), (1.30) and (1.31), can be avoided: information about the geometry iscontained in the areas and lengths of the volumes used to discretize the model.

1.3.9 Adjoint

Tangent linear and adjoint counterparts of the forward model are described inChapter 5.

1.4 Appendix ATMOSPHERE

1.4.1 Hydrostatic Primitive Equations for the Atmospherein pressure coordinates

The hydrostatic primitive equations (HPEs) in p-coordinates are:

D�vh

Dt+ f k× �vh + ∇pφ = �F (1.45)

∂φ

∂p+ α = 0 (1.46)

∇p · �vh +∂ω

∂p= 0 (1.47)

pα = RT (1.48)

cvDT

Dt+ p

Dα

Dt= Q (1.49)

where �vh = (u, v, 0) is the ‘horizontal’ (on pressure surfaces) component ofvelocity, D

Dt = �vh · ∇p + ω ∂∂p is the total derivative, f = 2Ω sinϕ is the Coriolis

parameter, φ = gz is the geopotential, α = 1/ρ is the specific volume, ω = DpDt

is the vertical velocity in the p−coordinate. Equation(1.49) is the first law ofthermodynamics where internal energy e = cvT , T is temperature, Q is the rateof heating per unit mass and pDα

Dt is the work done by the fluid in compressing.It is convenient to cast the heat equation in terms of potential temperature

θ so that it looks more like a generic conservation law. Differentiating (1.48) weget:

pDα

Dt+ α

Dp

Dt= R

DT

Dt


which, when added to the heat equation (1.49) and using cp = cv +R, gives:

cpDT

Dt− α

Dp

Dt= Q (1.50)

Potential temperature is defined:

θ = T (pc

p)κ (1.51)

where pc is a reference pressure and κ = R/cp. For convenience we will makeuse of the Exner function Π(p) which defined by:

Π(p) = cp(p

pc)κ (1.52)

The following relations will be useful and are easily expressed in terms of theExner function:

cpT = Πθ ;∂Π∂p

=κΠp

; α =κΠθp

=∂ Π∂p

θ ;DΠDt

=∂Π∂p

Dp

Dt

where b = ∂ Π∂p θ is the buoyancy.

The heat equation is obtained by noting that

cpDT

Dt=D(Πθ)Dt

= ΠDθ

Dt+ θ

DΠDt

= ΠDθ

Dt+ α

Dp

Dt

and on substituting into (1.50) gives:

ΠDθ

Dt= Q (1.53)

which is in conservative form.For convenience in the model we prefer to step forward (1.53) rather than

(1.49).

1.4.1.1 Boundary conditions

The upper and lower boundary conditions are :

at the top: p = 0 , ω =Dp

Dt= 0 (1.54)

at the surface: p = ps , φ = φtopo = g Ztopo (1.55)

In p-coordinates, the upper boundary acts like a solid boundary (ω = 0); in z-coordinates and the lower boundary is analogous to a free surface (φ is imposedand ω �= 0).

1.5. APPENDIX OCEAN 41

1.4.1.2 Splitting the geo-potential

For the purposes of initialization and reducing round-off errors, the model dealswith perturbations from reference (or “standard”) profiles. For example, thehydrostatic geopotential associated with the resting atmosphere is not dynami-cally relevant and can therefore be subtracted from the equations. The equationswritten in terms of perturbations are obtained by substituting the following def-initions into the previous model equations:

θ = θo + θ′ (1.56)α = αo + α′ (1.57)φ = φo + φ′ (1.58)

The reference state (indicated by subscript “0”) corresponds to horizontallyhomogeneous atmosphere at rest (θo, αo, φo) with surface pressure po(x, y) thatsatisfies φo(po) = g Ztopo, defined:

θo(p) = fn(p)αo(p) = Πpθo

φo(p) = φtopo −∫ p

p0

αodp

The final form of the HPE’s in p coordinates is then:

D�vh

Dt+ f k × �vh + ∇pφ

′ = �F (1.59)

∂φ′

∂p+ α′ = 0 (1.60)

∇p · �vh +∂ω

∂p= 0 (1.61)

∂Π∂p

θ′ = α′ (1.62)

Dθ

Dt=

QΠ

(1.63)

1.5 Appendix OCEAN

1.5.1 Equations of motion for the ocean

We review here the method by which the standard (Boussinesq, incompressible)HPE’s for the ocean written in z-coordinates are obtained. The non-Boussinesq


equations for oceanic motion are:

D�vh

Dt+ f k× �vh +

1ρ∇zp = �F (1.64)

εnhDw

Dt+ g +

1ρ

∂p

∂z= εnhFw (1.65)

1ρ

Dρ

Dt+ ∇z · �vh +

∂w

∂z= 0 (1.66)

ρ = ρ(θ, S, p) (1.67)Dθ

Dt= Qθ (1.68)

DS

Dt= Qs (1.69)

These equations permit acoustics modes, inertia-gravity waves, non-hydrostaticmotions, a geostrophic (Rossby) mode and a thermohaline mode. As written,they cannot be integrated forward consistently - if we step ρ forward in (1.66),the answer will not be consistent with that obtained by stepping (1.68) and(1.69) and then using (1.67) to yield ρ. It is therefore necessary to manipulatethe system as follows. Differentiating the EOS (equation of state) gives:

Dρ

Dt=∂ρ

∂θ

∣∣∣∣S,p

Dθ

Dt+

∂ρ

∂S

∣∣∣∣θ,p

DS

Dt+∂ρ

∂p

∣∣∣∣θ,S

Dp

Dt(1.70)

Note that ∂ρ∂p = 1

c2s

is the reciprocal of the sound speed (cs) squared. Substi-tuting into 1.66 gives:

1ρc2s

Dp

Dt+ ∇z · �v + ∂zw ≈ 0 (1.71)

where we have used an approximation sign to indicate that we have assumedadiabatic motion, dropping the Dθ

Dt and DSDt . Replacing 1.66 with 1.71 yields a

system that can be explicitly integrated forward:

D�vh

Dt+ f k× �vh +

1ρ∇zp = �F (1.72)

εnhDw

Dt+ g +

1ρ

∂p

∂z= εnhFw (1.73)

1ρc2s

Dp

Dt+ ∇z · �vh +

∂w

∂z= 0 (1.74)

ρ = ρ(θ, S, p) (1.75)Dθ

Dt= Qθ (1.76)

DS

Dt= Qs (1.77)


1.5.1.1 Compressible z-coordinate equations

Here we linearize the acoustic modes by replacing ρ with ρo(z) wherever itappears in a product (ie. non-linear term) - this is the ‘Boussinesq assumption’.The only term that then retains the full variation in ρ is the gravitationalacceleration:

D�vh

Dt+ f k× �vh +

1ρo

∇zp = �F (1.78)

εnhDw

Dt+gρ

ρo+

1ρo

∂p

∂z= εnhFw (1.79)

1ρoc2s

Dp

Dt+ ∇z · �vh +

∂w

∂z= 0 (1.80)

ρ = ρ(θ, S, p) (1.81)Dθ

Dt= Qθ (1.82)

DS

Dt= Qs (1.83)

These equations still retain acoustic modes. But, because the “compressible”terms are linearized, the pressure equation 1.80 can be integrated implicitlywith ease (the time-dependent term appears as a Helmholtz term in the non-hydrostatic pressure equation). These are the truly compressible Boussinesqequations. Note that the EOS must have the same pressure dependency as thelinearized pressure term, ie. ∂ρ

∂p

∣∣∣θ,S

= 1c2

s, for consistency.

1.5.1.2 ‘Anelastic’ z-coordinate equations

The anelastic approximation filters the acoustic mode by removing the time-dependency in the continuity (now pressure-) equation (1.80 ). This could bedone simply by noting that Dp

Dt ≈ −gρoDzDt = −gρow, but this leads to an

inconsistency between continuity and EOS. A better solution is to change thedependency on pressure in the EOS by splitting the pressure into a referencefunction of height and a perturbation:

ρ = ρ(θ, S, po(z) + εsp′)

Remembering that the term DpDt in continuity comes from differentiating the

EOS, the continuity equation then becomes:

1ρoc2s

(Dpo

Dt+ εs

Dp′

Dt

)+ ∇z · �vh +

∂w

∂z= 0

If the time- and space-scales of the motions of interest are longer than thoseof acoustic modes, then Dp′

Dt << (Dpo

Dt ,∇ · �vh) in the continuity equations and∂ρ∂p

∣∣∣θ,S

Dp′

Dt << ∂ρ∂p

∣∣∣θ,S

Dpo

Dt in the EOS (1.70). Thus we set εs = 0, removing


the dependency on p′ in the continuity equation and EOS. Expanding Dpo(z)Dt =

−gρow then leads to the anelastic continuity equation:

∇z · �vh +∂w

∂z− g

c2sw = 0 (1.84)

A slightly different route leads to the quasi-Boussinesq continuity equationwhere we use the scaling ∂ρ′

∂t + ∇3 · ρ′�v << ∇3 · ρo�v yielding:

∇z · �vh +1ρo

∂ (ρow)∂z

= 0 (1.85)

Equations 1.84 and 1.85 are in fact the same equation if:

1ρo

∂ρo

∂z=

−gc2s

(1.86)

Again, note that if ρo is evaluated from prescribed θo and So profiles, then theEOS dependency on po and the term g

c2s

in continuity should be referred to thosesame profiles. The full set of ‘quasi-Boussinesq’ or ‘anelastic’ equations for theocean are then:

D�vh

Dt+ f k × �vh +

1ρo

∇zp = �F (1.87)

εnhDw

Dt+gρ

ρo+

1ρo

∂p

∂z= εnhFw (1.88)

∇z · �vh +1ρo

∂ (ρow)∂z

= 0 (1.89)

ρ = ρ(θ, S, po(z)) (1.90)Dθ

Dt= Qθ (1.91)

DS

Dt= Qs (1.92)

1.5.1.3 Incompressible z-coordinate equations

Here, the objective is to drop the depth dependence of ρo and so, technically,to also remove the dependence of ρ on po. This would yield the “truly” incom-


pressible Boussinesq equations:

D�vh

Dt+ f k× �vh +

1ρc

∇zp = �F (1.93)

εnhDw

Dt+gρ

ρc+

1ρc

∂p

∂z= εnhFw (1.94)

∇z · �vh +∂w

∂z= 0 (1.95)

ρ = ρ(θ, S) (1.96)Dθ

Dt= Qθ (1.97)

DS

Dt= Qs (1.98)

where ρc is a constant reference density of water.

1.5.1.4 Compressible non-divergent equations

The above “incompressible” equations are incompressible in both the flow andthe density. In many oceanic applications, however, it is important to retaincompressibility effects in the density. To do this we must split the density thus:

ρ = ρo + ρ′

We then assert that variations with depth of ρo are unimportant while thecompressible effects in ρ′ are:

ρo = ρc

ρ′ = ρ(θ, S, po(z)) − ρo

This then yields what we can call the semi-compressible Boussinesq equations:

D�vh

Dt+ f k× �vh +

1ρc

∇zp′ = �F (1.99)

εnhDw

Dt+gρ′

ρc+

1ρc

∂p′

∂z= εnhFw (1.100)

∇z · �vh +∂w

∂z= 0 (1.101)

ρ′ = ρ(θ, S, po(z)) − ρc (1.102)Dθ

Dt= Qθ (1.103)

DS

Dt= Qs (1.104)

Note that the hydrostatic pressure of the resting fluid, including that associatedwith ρc, is subtracted out since it has no effect on the dynamics.

Though necessary, the assumptions that go into these equations are messysince we essentially assume a different EOS for the reference density and the


perturbation density. Nevertheless, it is the hydrostatic (εnh = 0 form of theseequations that are used throughout the ocean modeling community and referredto as the primitive equations (HPE).

1.6 Appendix:OPERATORS

1.6.1 Coordinate systems

1.6.1.1 Spherical coordinates

In spherical coordinates, the velocity components in the zonal, meridional andvertical direction respectively, are given by (see Fig.2) :

u = r cosϕDλ

Dt

v = rDϕ

Dt

r =Dr

Dt

Here ϕ is the latitude, λ the longitude, r the radial distance of the particlefrom the center of the earth, Ω is the angular speed of rotation of the Earth andD/Dt is the total derivative.

The ‘grad’ (∇) and ‘div’ (∇.) operators are defined by, in spherical coordi-nates:

∇ ≡(

1r cosϕ

∂

∂λ,1r

∂

∂ϕ,∂

∂r

)

∇.v ≡ 1r cosϕ

{∂u

∂λ+

∂

∂ϕ(v cosϕ)

}+

1r2∂(r2r)

∂r

Chapter 2

Discretization andAlgorithm

This chapter lays out the numerical schemes that are employed in the coreMITgcm algorithm. Whenever possible links are made to actual program codein the MITgcm implementation. The chapter begins with a discussion of thetemporal discretization used in MITgcm. This discussion is followed by sectionsthat describe the spatial discretization. The schemes employed for momentumterms are described first, afterwards the schemes that apply to passive anddynamically active tracers are described.

2.1 Time-stepping

The equations of motion integrated by the model involve four prognostic equa-tions for flow, u and v, temperature, θ, and salt/moisture, S, and three diag-nostic equations for vertical flow, w, density/buoyancy, ρ/b, and pressure/geo-potential, φhyd. In addition, the surface pressure or height may by describedby either a prognostic or diagnostic equation and if non-hydrostatics terms areincluded then a diagnostic equation for non-hydrostatic pressure is also solved.The combination of prognostic and diagnostic equations requires a model algo-rithm that can march forward prognostic variables while satisfying constraintsimposed by diagnostic equations.

Since the model comes in several flavors and formulation, it would be con-fusing to present the model algorithm exactly as written into code along withall the switches and optional terms. Instead, we present the algorithm for eachof the basic formulations which are:

1. the semi-implicit pressure method for hydrostatic equations with a rigid-lid, variables co-located in time and with Adams-Bashforth time-stepping,

2. as 1. but with an implicit linear free-surface,

47

48 CHAPTER 2. DISCRETIZATION AND ALGORITHM

3. as 1. or 2. but with variables staggered in time,

4. as 1. or 2. but with non-hydrostatic terms included,

5. as 2. or 3. but with non-linear free-surface.

In all the above configurations it is also possible to substitute the Adams-Bashforth with an alternative time-stepping scheme for terms evaluated explic-itly in time. Since the over-arching algorithm is independent of the particulartime-stepping scheme chosen we will describe first the over-arching algorithm,known as the pressure method, with a rigid-lid model in section 2.2. This al-gorithm is essentially unchanged, apart for some coefficients, when the rigid lidassumption is replaced with a linearized implicit free-surface, described in sec-tion 2.3. These two flavors of the pressure-method encompass all formulationsof the model as it exists today. The integration of explicit in time terms is out-lined in section 2.4 and put into the context of the overall algorithm in sections2.6 and 2.7. Inclusion of non-hydrostatic terms requires applying the pressuremethod in three dimensions instead of two and this algorithm modification isdescribed in section 2.8. Finally, the free-surface equation may be treated moreexactly, including non-linear terms, and this is described in section 2.9.2.

2.2 Pressure method with rigid-lid

The horizontal momentum and continuity equations for the ocean (1.99 and1.101), or for the atmosphere (1.45 and 1.47), can be summarized by:

∂tu+ g∂xη = Gu (2.1)∂tv + g∂yη = Gv (2.2)

∂xu+ ∂yv + ∂zw = 0 (2.3)

where we are adopting the oceanic notation for brevity. All terms in the mo-mentum equations, except for surface pressure gradient, are encapsulated in theG vector. The continuity equation, when integrated over the fluid depth, H ,and with the rigid-lid/no normal flow boundary conditions applied, becomes:

∂xHu+ ∂yHv = 0 (2.4)

Here, Hu =∫

H udz is the depth integral of u, similarly for Hv. The rigid-lidapproximation sets w = 0 at the lid so that it does not move but allows a pressureto be exerted on the fluid by the lid. The horizontal momentum equations andvertically integrated continuity equation are be discretized in time and space asfollows:

un+1 + Δtg∂xηn+1 = un + ΔtG(n+1/2)

u (2.5)vn+1 + Δtg∂yη

n+1 = vn + ΔtG(n+1/2)v (2.6)

∂xHun+1 + ∂yHvn+1 = 0 (2.7)

2.2. PRESSURE METHOD WITH RIGID-LID 49

( )

u,vG(n+½)

u,v n+1

Δ2η = Δ. uv

**

Δ(n+1) tΔtime

n t

n *u,v u,v

Figure 2.1: A schematic of the evolution in time of the pressure method algo-rithm. A prediction for the flow variables at time level n+1 is made based onlyon the explicit terms, G(n+1/2), and denoted u∗, v∗. Next, a pressure field isfound such that un+1, vn+1 will be non-divergent. Conceptually, the ∗ quantitiesexist at time level n+ 1 but they are intermediate and only temporary.

FORWARD STEPDYNAMICS

TIMESTEP u∗,v∗ (2.8,2.9)SOLVE FOR PRESSURE

CALC DIV GHAT Hu∗,Hv∗ (2.10)CG2D ηn+1 (2.10)

MOMENTUM CORRECTION STEPCALC GRAD PHI SURF ∇ηn+1

CORRECTION STEP un+1,vn+1 (2.11,2.12)

Figure 2.2: Calling tree for the pressure method algorithm (FORWARD STEP)


As written here, terms on the LHS all involve time level n+ 1 and are referredto as implicit; the implicit backward time stepping scheme is being used. Allother terms in the RHS are explicit in time. The thermodynamic quantities areintegrated forward in time in parallel with the flow and will be discussed later.For the purposes of describing the pressure method it suffices to say that thehydrostatic pressure gradient is explicit and so can be included in the vector G.

Substituting the two momentum equations into the depth integrated conti-nuity equation eliminates un+1 and vn+1 yielding an elliptic equation for ηn+1.Equations 2.5, 2.6 and 2.7 can then be re-arranged as follows:

u∗ = un + ΔtG(n+1/2)u (2.8)

v∗ = vn + ΔtG(n+1/2)v (2.9)

∂xΔtgH∂xηn+1 + ∂yΔtgH∂yη

n+1 = ∂xHu∗ + ∂yHv∗ (2.10)un+1 = u∗ − Δtg∂xη

n+1 (2.11)vn+1 = v∗ − Δtg∂yη

n+1 (2.12)

Equations 2.8 to 2.12, solved sequentially, represent the pressure method algo-rithm used in the model. The essence of the pressure method lies in the factthat any explicit prediction for the flow would lead to a divergence flow fieldso a pressure field must be found that keeps the flow non-divergent over eachstep of the integration. The particular location in time of the pressure field issomewhat ambiguous; in Fig. 2.1 we depicted as co-located with the future flowfield (time level n + 1) but it could equally have been drawn as staggered intime with the flow.

The correspondence to the code is as follows:

• the prognostic phase, equations 2.8 and 2.9, stepping forward un and vn

to u∗ and v∗ is coded in TIMESTEP()

• the vertical integration, Hu∗ and Hv∗, divergence and inversion of theelliptic operator in equation 2.10 is coded in SOLVE FOR PRESSURE()

• finally, the new flow field at time level n+ 1 given by equations 2.11 and2.12 is calculated in CORRECTION STEP().

The calling tree for these routines is given in Fig. 2.2.

Need to discuss implicit viscosity somewhere:

1Δt

un+1 − ∂zAv∂zun+1 + g∂xη

n+1 =1

Δtun +G(n+1/2)

u (2.13)

1Δt

vn+1 − ∂zAv∂zvn+1 + g∂yη

n+1 =1

Δtvn +G(n+1/2)

v (2.14)

2.3. PRESSURE METHOD WITH IMPLICIT LINEAR FREE-SURFACE 51

2.3 Pressure method with implicit linear free-

surface

The rigid-lid approximation filters out external gravity waves subsequently mod-ifying the dispersion relation of barotropic Rossby waves. The discrete form ofthe elliptic equation has some zero eigen-values which makes it a potentiallytricky or inefficient problem to solve.

The rigid-lid approximation can be easily replaced by a linearization of thefree-surface equation which can be written:

∂tη + ∂xHu+ ∂yHv = P − E +R (2.15)

which differs from the depth integrated continuity equation with rigid-lid (2.4)by the time-dependent term and fresh-water source term.

Equation 2.7 in the rigid-lid pressure method is then replaced by the timediscretization of 2.15 which is:

ηn+1 + Δt∂xHun+1 + Δt∂yHvn+1 = ηn + Δt(P − E) (2.16)

where the use of flow at time level n+1 makes the method implicit and backwardin time. The is the preferred scheme since it still filters the fast unresolved wavemotions by damping them. A centered scheme, such as Crank-Nicholson, wouldalias the energy of the fast modes onto slower modes of motion.

As for the rigid-lid pressure method, equations 2.5, 2.6 and 2.16 can bere-arranged as follows:

u∗ = un + ΔtG(n+1/2)u (2.17)

v∗ = vn + ΔtG(n+1/2)v (2.18)

η∗ = εfs (ηn + Δt(P − E)) − Δt∂xHu∗ + ∂yHv∗ (2.19)

∂xgH∂xηn+1 + ∂ygH∂yη

n+1 − εfsηn+1

Δt2= − η∗

Δt2(2.20)

un+1 = u∗ − Δtg∂xηn+1 (2.21)

vn+1 = v∗ − Δtg∂yηn+1 (2.22)

Equations 2.17 to 2.22, solved sequentially, represent the pressure method al-gorithm with a backward implicit, linearized free surface. The method is stillformerly a pressure method because in the limit of large Δt the rigid-lid methodis recovered. However, the implicit treatment of the free-surface allows the flowto be divergent and for the surface pressure/elevation to respond on a finitetime-scale (as opposed to instantly). To recover the rigid-lid formulation, weintroduced a switch-like parameter, εfs, which selects between the free-surfaceand rigid-lid; εfs = 1 allows the free-surface to evolve; εfs = 0 imposes therigid-lid. The evolution in time and location of variables is exactly as it wasfor the rigid-lid model so that Fig. 2.1 is still applicable. Similarly, the callingsequence, given in Fig. 2.2, is as for the pressure-method.


FORWARD STEPTHERMODYNAMICS

CALC GTGAD CALC RHS Gn

θ = Gθ(u, θn)either EXTERNAL FORCING Gn

θ = Gnθ + Q

ADAMS BASHFORTH2 G(n+1/2)θ (2.24)

or EXTERNAL FORCING G(n+1/2)θ = G

(n+1/2)θ + Q

TIMESTEP TRACER τ∗ (2.23)IMPLDIFF τ (n+1) (2.27)

Figure 2.3: Calling tree for the Adams-Bashforth time-stepping of temperaturewith implicit diffusion.

2.4 Explicit time-stepping: Adams-Bashforth

In describing the the pressure method above we deferred describing the timediscretization of the explicit terms. We have historically used the quasi-secondorder Adams-Bashforth method for all explicit terms in both the momentumand tracer equations. This is still the default mode of operation but it is nowpossible to use alternate schemes for tracers (see section 2.16).

In the previous sections, we summarized an explicit scheme as:

τ∗ = τn + ΔtG(n+1/2)τ (2.23)

where τ could be any prognostic variable (u, v, θ or S) and τ∗ is an explicitestimate of τn+1 and would be exact if not for implicit-in-time terms. Theparenthesis about n + 1/2 indicates that the term is explicit and extrapolatedforward in time and for this we use the quasi-second order Adams-Bashforthmethod:

G(n+1/2)τ = (3/2 + εAB)Gn

τ − (1/2 + εAB)Gn−1τ (2.24)

This is a linear extrapolation, forward in time, to t = (n + 1/2 + εAB)Δt.An extrapolation to the mid-point in time, t = (n + 1/2)Δt, corresponding toεAB = 0, would be second order accurate but is weakly unstable for oscillatoryterms. A small but finite value for εAB stabilizes the method. Strictly speaking,damping terms such as diffusion and dissipation, and fixed terms (forcing), donot need to be inside the Adams-Bashforth extrapolation. However, in thecurrent code, it is simpler to include these terms and this can be justified if theflow and forcing evolves smoothly. Problems can, and do, arise when forcing ormotions are high frequency and this corresponds to a reduced stability comparedto a simple forward time-stepping of such terms. The model offers the possibilityto leave the forcing term outside the Adams-Bashforth extrapolation, by turningoff the logical flag forcing In AB (parameter file data, namelist PARM01,default value = True).

2.5. IMPLICIT TIME-STEPPING: BACKWARD METHOD 53

A stability analysis for an oscillation equation should be given at this point.AJA needs to find his noteson this...A stability analysis for a relaxation equation should be given at this point....and for this too.

2.5 Implicit time-stepping: backward method

Vertical diffusion and viscosity can be treated implicitly in time using the back-ward method which is an intrinsic scheme. Recently, the option to treat thevertical advection implicitly has been added, but not yet tested; therefore, thedescription hereafter is limited to diffusion and viscosity. For tracers, the timediscretized equation is:

τn+1 − Δt∂rκv∂rτn+1 = τn + ΔtG(n+1/2)

τ (2.25)

where G(n+1/2)τ is the remaining explicit terms extrapolated using the Adams-

Bashforth method as described above. Equation 2.25 can be split split into:

τ∗ = τn + ΔtG(n+1/2)τ (2.26)

τn+1 = L−1τ (τ∗) (2.27)

where L−1τ is the inverse of the operator

L = [1 + Δt∂rκv∂r] (2.28)

Equation 2.26 looks exactly as 2.23 while 2.27 involves an operator or matrixinversion. By re-arranging 2.25 in this way we have cast the method as anexplicit prediction step and an implicit step allowing the latter to be insertedinto the over all algorithm with minimal interference.

Fig. 2.3 shows the calling sequence for stepping forward a tracer variablesuch as temperature.

In order to fit within the pressure method, the implicit viscosity must notalter the barotropic flow. In other words, it can only redistribute momentumin the vertical. The upshot of this is that although vertical viscosity may bebackward implicit and unconditionally stable, no-slip boundary conditions maynot be made implicit and are thus cast as a an explicit drag term.

2.6 Synchronous time-stepping: variables co-located

in time

The Adams-Bashforth extrapolation of explicit tendencies fits neatly into thepressure method algorithm when all state variables are co-located in time.Fig. 2.4 illustrates the location of variables in time and the evolution of the


Δ(n+1) tΔ(n−1) t Δtime

n t

,Sθ ,Sθ ,Sθ

,Sθ n+1

L−1

θ,S θ,S θ,S

θ,SΦh

L−1

u,v n+1

n−1 n *Gn−1 GGn (n+½)

n−1 n *u,v u,v u,vu,vGn−1

u,v

u,v u,vG(n+½)Gn

Figure 2.4: A schematic of the explicit Adams-Bashforth and implicit time-stepping phases of the algorithm. All prognostic variables are co-located in time.Explicit tendencies are evaluated at time level n as a function of the state atthat time level (dotted arrow). The explicit tendency from the previous timelevel, n− 1, is used to extrapolate tendencies to n+ 1/2 (dashed arrow). Thisextrapolated tendency allows variables to be stably integrated forward-in-timeto render an estimate (∗-variables) at the n + 1 time level (solid arc-arrow).The operator L formed from implicit-in-time terms is solved to yield the statevariables at time level n+ 1.

2.6. SYNCHRONOUS TIME-STEPPING: VARIABLES CO-LOCATED IN TIME55

FORWARD STEPEXTERNAL FIELDS LOADDO ATMOSPHERIC PHYSDO OCEANIC PHYS

THERMODYNAMICSCALC GT

GAD CALC RHS Gnθ = Gθ(u, θn) (2.29)

EXTERNAL FORCING Gnθ = Gn

θ + QADAMS BASHFORTH2 G

(n+1/2)θ (2.30)

TIMESTEP TRACER θ∗ (2.31)IMPLDIFF θ(n+1) (2.32)

DYNAMICSCALC PHI HYD φn

hyd (2.33)MOM FLUXFORM or MOM VECINV Gn

�v (2.34)TIMESTEP �v∗ (2.35, 2.36)IMPLDIFF �v∗∗ (2.37)

UPDATE R STAR or UPDATE SURF DR (NonLin-FS only)SOLVE FOR PRESSURE

CALC DIV GHAT η∗ (2.38)CG2D ηn+1 (2.39)

MOMENTUM CORRECTION STEPCALC GRAD PHI SURF ∇ηn+1

CORRECTION STEP un+1,vn+1 (2.40)TRACERS CORRECTION STEP

CYCLE TRACER θn+1

FILTER Shapiro Filter, Zonal Filter (FFT)CONVECTIVE ADJUSTMENT

Figure 2.5: Calling tree for the overall synchronous algorithm using Adams-Bashforth time-stepping. The place where the model geometry (hFac factors)is updated is added here but is only relevant for the non-linear free-surface al-gorithm. For completeness, the external forcing, ocean and atmospheric physicshave been added, although they are mainly optional


algorithm with time. The algorithm can be represented by the sequential solu-tion of the follow equations:

Gnθ,S = Gθ,S(un, θn, Sn) (2.29)

G(n+1/2)θ,S = (3/2 + εAB)Gn

θ,S − (1/2 + εAB)Gn−1θ,S (2.30)

(θ∗, S∗) = (θn, Sn) + ΔtG(n+1/2)θ,S (2.31)

(θn+1, Sn+1) = L−1θ,S(θ∗, S∗) (2.32)

φnhyd =

∫b(θn, Sn)dr (2.33)

�Gn�v = �G�v(�vn, φn

hyd) (2.34)

�G(n+1/2)�v = (3/2 + εAB)�Gn

�v − (1/2 + εAB)�Gn−1�v (2.35)

�v∗ = �vn + Δt�G(n+1/2)�v (2.36)

�v∗∗ = L−1�v (�v∗) (2.37)

η∗ = εfs (ηn + Δt(P − E)) − Δt∇ ·H�v∗∗ (2.38)

∇ · gH∇ηn+1 − εfsηn+1

Δt2= − η∗

Δt2(2.39)

�vn+1 = �v∗ − Δtg∇ηn+1 (2.40)

Fig. 2.4 illustrates the location of variables in time and evolution of the al-gorithm with time. The Adams-Bashforth extrapolation of the tracer ten-dencies is illustrated by the dashed arrow, the prediction at n + 1 is indi-cated by the solid arc. Inversion of the implicit terms, L−1

θ,S, then yields thenew tracer fields at n + 1. All these operations are carried out in subroutineTHERMODYNAMICS an subsidiaries, which correspond to equations 2.29 to2.32. Similarly illustrated is the Adams-Bashforth extrapolation of accelera-tions, stepping forward and solving of implicit viscosity and surface pressuregradient terms, corresponding to equations 2.34 to 2.40. These operations arecarried out in subroutines DYNAMCIS, SOLVE FOR PRESSURE and MO-MENTUM CORRECTION STEP. This, then, represents an entire algorithmfor stepping forward the model one time-step. The corresponding calling tree isgiven in 2.5.

2.7 Staggered baroclinic time-stepping

For well stratified problems, internal gravity waves may be the limiting processfor determining a stable time-step. In the circumstance, it is more efficient tostagger in time the thermodynamic variables with the flow variables. Fig. 2.6illustrates the staggering and algorithm. The key difference between this andFig. 2.4 is that the thermodynamic variables are solved after the dynamics,using the recently updated flow field. This essentially allows the gravity waveterms to leap-frog in time giving second order accuracy and more stability.

2.7. STAGGERED BAROCLINIC TIME-STEPPING 57

L−1

,Sθ ,Sθ ,Sθ

L−1

θ,S θ,S

θ,S

,SθΦh

θ,S

Δ (n+½) tΔ(n−½) t

Δ(n+1) tΔ(n−1) t Δtime

n t

(n−1½) tΔ

*u,v u,v u,vu,vG u,vGu,v

G

*G GG

(n)

(n+½)

n−½

n

n−1½

n−1

n−1½ n−½

u,v

u,v(n+½)

n+1

n−1 n

Figure 2.6: A schematic of the explicit Adams-Bashforth and implicit time-stepping phases of the algorithm but with staggering in time of thermodynamicvariables with the flow. Explicit momentum tendencies are evaluated at timelevel n − 1/2 as a function of the flow field at that time level n − 1/2. Theexplicit tendency from the previous time level, n − 3/2, is used to extrapolatetendencies to n (dashed arrow). The hydrostatic pressure/geo-potential φhyd isevaluated directly at time level n (vertical arrows) and used with the extrapo-lated tendencies to step forward the flow variables from n−1/2 to n+1/2 (solidarc-arrow). The implicit-in-time operator Lu,v (vertical arrows) is then appliedto the previous estimation of the the flow field (∗-variables) and yields to thetwo velocity components u, v at time level n+1/2. These are then used to calcu-late the advection term (dashed arc-arrow) of the thermo-dynamics tendenciesat time step n. The extrapolated thermodynamics tendency, from time leveln− 1 and n to n+ 1/2, allows thermodynamic variables to be stably integratedforward-in-time (solid arc-arrow) up to time level n+ 1.


The essential change in the staggered algorithm is that the thermodynamicssolver is delayed from half a time step, allowing the use of the most recentvelocities to compute the advection terms. Once the thermodynamics fields areupdated, the hydrostatic pressure is computed to step forwrad the dynamics.Note that the pressure gradient must also be taken out of the Adams-Bashforthextrapolation. Also, retaining the integer time-levels, n and n+1, does not givea user the sense of where variables are located in time. Instead, we re-writethe entire algorithm, 2.29 to 2.40, annotating the position in time of variablesappropriately:

φnhyd =

∫b(θn, Sn)dr (2.41)

�Gn−1/2�v = �G�v(�vn−1/2) (2.42)�G(n)

�v = (3/2 + εAB)�Gn−1/2�v − (1/2 + εAB)�Gn−3/2

�v (2.43)

�v∗ = �vn−1/2 + Δt(�G(n)

�v −∇φnhyd

)(2.44)

�v∗∗ = L−1�v (�v∗) (2.45)

η∗ = εfs

(ηn−1/2 + Δt(P − E)n

)− Δt∇ ·H�v∗∗ (2.46)

∇ · gH∇ηn+1/2 − εfsηn+1/2

Δt2= − η∗

Δt2(2.47)

�vn+1/2 = �v∗ − Δtg∇ηn+1/2 (2.48)Gn

θ,S = Gθ,S(un+1/2, θn, Sn) (2.49)

G(n+1/2)θ,S = (3/2 + εAB)Gn

θ,S − (1/2 + εAB)Gn−1θ,S (2.50)

(θ∗, S∗) = (θn, Sn) + ΔtG(n+1/2)θ,S (2.51)

(θn+1, Sn+1) = L−1θ,S(θ∗, S∗) (2.52)

(2.53)

The corresponding calling tree is given in 2.7. The staggered algorithm is ac-tivated with the run-time flag staggerTimeStep=.TRUE. in parameter filedata, namelist PARM01.

The only difficulty with this approach is apparent in equation 2.49 and il-lustrated by the dotted arrow connecting u, vn+1/2 with Gn

θ . The flow used toadvect tracers around is not naturally located in time. This could be avoidedby applying the Adams-Bashforth extrapolation to the tracer field itself and ad-vecting that around but this approach is not yet available. We’re not aware ofany detrimental effect of this feature. The difficulty lies mainly in interpretationof what time-level variables and terms correspond to.

2.8 Non-hydrostatic formulation

The non-hydrostatic formulation re-introduces the full vertical momentum equa-tion and requires the solution of a 3-D elliptic equations for non-hydrostatic

2.8. NON-HYDROSTATIC FORMULATION 59

FORWARD STEPEXTERNAL FIELDS LOADDO ATMOSPHERIC PHYSDO OCEANIC PHYS

DYNAMICSCALC PHI HYD φn

hyd (2.41)MOM FLUXFORM or MOM VECINV G

n−1/2�v (2.42)

TIMESTEP �v∗ (2.43, 2.44)IMPLDIFF �v∗∗ (2.45)

UPDATE R STAR or UPDATE SURF DR (NonLin-FS only)SOLVE FOR PRESSURE

CALC DIV GHAT η∗ (2.46)CG2D ηn+1/2 (2.47)

MOMENTUM CORRECTION STEPCALC GRAD PHI SURF ∇ηn+1/2

CORRECTION STEP un+1/2,vn+1/2 (2.48)THERMODYNAMICS

CALC GTGAD CALC RHS Gn

θ = Gθ(u, θn) (2.49)EXTERNAL FORCING Gn

θ = Gnθ + Q

ADAMS BASHFORTH2 G(n+1/2)θ (2.50)

TIMESTEP TRACER θ∗ (2.51)IMPLDIFF θ(n+1) (2.52)

TRACERS CORRECTION STEPCYCLE TRACER θn+1

FILTER Shapiro Filter, Zonal Filter (FFT)CONVECTIVE ADJUSTMENT

Figure 2.7: Calling tree for the overall staggered algorithm using Adams-Bashforth time-stepping. The place where the model geometry (hFac factors)is updated is added here but is only relevant for the non-linear free-surfacealgorithm.


pressure perturbation. We still intergrate vertically for the hydrostatic pres-sure and solve a 2-D elliptic equation for the surface pressure/elevation for thisreduces the amount of work needed to solve for the non-hydrostatic pressure.

The momentum equations are discretized in time as follows:

1Δt

un+1 + g∂xηn+1 + ∂xφ

n+1nh =

1Δt

un +G(n+1/2)u (2.54)

1Δt

vn+1 + g∂yηn+1 + ∂yφ

n+1nh =

1Δt

vn +G(n+1/2)v (2.55)

1Δt

wn+1 + ∂rφn+1nh =

1Δt

wn +G(n+1/2)w (2.56)

(2.57)

which must satisfy the discrete-in-time depth integrated continuity, equation 2.16and the local continuity equation

∂xun+1 + ∂yv

n+1 + ∂rwn+1 = 0 (2.58)

As before, the explicit predictions for momentum are consolidated as:

u∗ = un + ΔtG(n+1/2)u

v∗ = vn + ΔtG(n+1/2)v

w∗ = wn + ΔtG(n+1/2)w

but this time we introduce an intermediate step by splitting the tendancy of theflow as follows:

un+1 = u∗∗ − Δt∂xφn+1nh u∗∗ = u∗ − Δtg∂xη

n+1 (2.59)vn+1 = v∗∗ − Δt∂yφ

n+1nh v∗∗ = v∗ − Δtg∂yη

n+1 (2.60)

Substituting into the depth integrated continuity (equation 2.16) gives

∂xH∂x

(gηn+1 + φn+1

nh

)+ ∂yH∂y

(gηn+1 + φn+1

nh

)− εfsη

∗

Δt2= − η∗

Δt2(2.61)

which is approximated by equation 2.20 on the basis that i) φn+1nh is not yet

known and ii) ∇φnh << g∇η. If 2.20 is solved accurately then the implica-tion is that φnh ≈ 0 so that thet non-hydrostatic pressure field does not drivebarotropic motion.

The flow must satisfy non-divergence (equation 2.58) locally, as well as depthintegrated, and this constraint is used to form a 3-D elliptic equations for φn+1

nh :

∂xxφn+1nh + ∂yyφ

n+1nh + ∂rrφ

n+1nh = ∂xu

∗∗ + ∂yv∗∗ + ∂rw

∗ (2.62)

The entire algorithm can be summarized as the sequential solution of the

2.9. VARIANTS ON THE FREE SURFACE 61

following equations:

u∗ = un + ΔtG(n+1/2)u (2.63)

v∗ = vn + ΔtG(n+1/2)v (2.64)

w∗ = wn + ΔtG(n+1/2)w (2.65)

η∗ = εfs (ηn + Δt(P − E)) − Δt∂xHu∗ + ∂yHv∗ (2.66)

∂xgH∂xηn+1 + ∂ygH∂yη

n+1 − εfsηn+1

Δt2= − η∗

Δt2(2.67)

u∗∗ = u∗ − Δtg∂xηn+1 (2.68)

v∗∗ = v∗ − Δtg∂yηn+1 (2.69)

∂xxφn+1nh + ∂yyφ

n+1nh + ∂rrφ

n+1nh = ∂xu

∗∗ + ∂yv∗∗ + ∂rw

∗ (2.70)un+1 = u∗∗ − Δt∂xφ

n+1nh (2.71)

vn+1 = v∗∗ − Δt∂yφn+1nh (2.72)

∂rwn+1 = −∂xu

n+1 − ∂yvn+1 (2.73)

where the last equation is solved by vertically integrating for wn+1.

2.9 Variants on the Free Surface

We now describe the various formulations of the free-surface that include non-linear forms, implicit in time using Crank-Nicholson, explicit and [one day]split-explicit. First, we’ll reiterate the underlying algorithm but this time usingthe notation consistent with the more general vertical coordinate r. The ellipticequation for free-surface coordinate (units of r), corresponding to 2.16, andassuming no non-hydrostatic effects (εnh = 0) is:

εfsηn+1 −∇h · Δt2(Ro −Rfixed)∇hbsη

n+1 = η∗ (2.74)

where

η∗ = εfs ηn − Δt∇h ·

∫ Ro

Rfixed

�v∗dr + εfwΔt(P − E)n (2.75)

S/R SOLVE FOR PRESSURE (solve for pressure.F)u∗: GuNm1 (DYNVARS.h)v∗: GvNm1 (DYNVARS.h)η∗: cg2d b (SOLVE FOR PRESSURE.h)ηn+1: etaN (DYNVARS.h)

Once ηn+1 has been found, substituting into ?? yields �vn+1 if the model ishydrostatic (εnh = 0):

�vn+1 = �v∗ − Δt∇hbsηn+1

This is known as the correction step. However, when the model is non-hydrostatic (εnh = 1) we need an additional step and an additional equation for


φ′nh. This is obtained by substituting 2.54, 2.55 and 2.56 into continuity:

[∇2h + ∂rr

]φ′nh

n+1 =1

Δt(∇h · �v∗∗ + ∂r r

∗) (2.76)

where�v∗∗ = �v∗ − Δt∇hbsη

n+1

Note that ηn+1 is also used to update the second RHS term ∂r r∗ since the

vertical velocity at the surface (rsurf ) is evaluated as (ηn+1 − ηn)/Δt.Finally, the horizontal velocities at the new time level are found by:

�vn+1 = �v∗∗ − εnhΔt∇hφ′nh

n+1 (2.77)

and the vertical velocity is found by integrating the continuity equation ver-tically. Note that, for the convenience of the restart procedure, the verticalintegration of the continuity equation has been moved to the beginning of thetime step (instead of at the end), without any consequence on the solution.

S/R CORRECTION STEP (correction step.F)ηn+1: etaN (DYNVARS.h)φn+1

nh : phi nh (DYNVARS.h)u∗: GuNm1 (DYNVARS.h)v∗: GvNm1 (DYNVARS.h)un+1: uVel (DYNVARS.h)vn+1: vVel (DYNVARS.h)

Regarding the implementation of the surface pressure solver, all computa-tion are done within the routine SOLVE FOR PRESSURE and its dependentcalls. The standard method to solve the 2D elliptic problem (2.74) uses theconjugate gradient method (routine CG2D); the solver matrix and conjugategradient operator are only function of the discretized domain and are there-fore evaluated separately, before the time iteration loop, within INI CG2D.The computation of the RHS η∗ is partly done in CALC DIV GHAT and inSOLVE FOR PRESSURE.

The same method is applied for the non hydrostatic part, using a conjugategradient 3D solver (CG3D) that is initialized in INI CG3D. The RHS terms of2D and 3D problems are computed together at the same point in the code.

2.9.1 Crank-Nickelson barotropic time stepping

The full implicit time stepping described previously is unconditionally stablebut damps the fast gravity waves, resulting in a loss of potential energy. Themodification presented now allows one to combine an implicit part (β, γ) andan explicit part (1 − β, 1 − γ) for the surface pressure gradient (β) and for thebarotropic flow divergence (γ).For instance, β = γ = 1 is the previous fully implicit scheme; β = γ = 1/2 isthe non damping (energy conserving), unconditionally stable, Crank-Nickelsonscheme; (β, γ) = (1, 0) or = (0, 1) corresponds to the forward - backward scheme


that conserves energy but is only stable for small time steps.In the code, β, γ are defined as parameters, respectively implicSurfPress,implicDiv2DFlow. They are read from the main parameter file ”data” andare set by default to 1,1.

Equations 2.17 – 2.22 are modified as follows:

�vn+1

Δt+ ∇hbs[βηn+1 + (1 − β)ηn] + εnh∇hφ

′nh

n+1 =�v∗

Δt

εfsηn+1 − ηn

Δt+ ∇h ·

∫ Ro

Rfixed

[γ�vn+1 + (1 − γ)�vn]dr = εfw(P − E) (2.78)

where:

�v∗ = �vn + Δt�G(n+1/2)�v + (β − 1)Δt∇hbsη

n + Δt∇hφ′hyd

(n+1/2)

η∗ = εfsηn + εfwΔt(P − E) − Δt∇h ·

∫ Ro

Rfixed

[γ�v∗ + (1 − γ)�vn]dr

In the hydrostatic case (εnh = 0), allowing us to find ηn+1, thus:

εfsηn+1 −∇h · βγΔt2bs(Ro −Rfixed)∇hη

n+1 = η∗

and then to compute (CORRECTION STEP):

�vn+1 = �v∗ − βΔt∇hbsηn+1

Notes:

1. The RHS term of equation 2.78 corresponds the contribution of fresh wa-ter flux (P-E) to the free-surface variations (εfw = 1, useRealFreshWa-ter=TRUE in parameter file data). In order to remain consistent withthe tracer equation, specially in the non-linear free-surface formulation,this term is also affected by the Crank-Nickelson time stepping. The RHSreads: εfw(γ(P − E)n+1/2 + (1 − γ)(P − E)n−1/2)

2. The non-hydrostatic part of the code has not yet been updated, and there-fore cannot be used with (β, γ) �= (1, 1).

3. The stability criteria with Crank-Nickelson time stepping for the purelinear gravity wave problem in cartesian coordinates is:

• β + γ < 1 : unstable• β ≥ 1/2 and γ ≥ 1/2 : stable• β + γ ≥ 1 : stable if

c2max(β − 1/2)(γ − 1/2) + 1 ≥ 0

with cmax = 2Δt√gH

√1

Δx2+

1Δy2


2.9.2 Non-linear free-surface

Recently, new options have been added to the model that concern the freesurface formulation.

2.9.2.1 pressure/geo-potential and free surface

For the atmosphere, since φ = φtopo − ∫ p

psαdp, subtracting the reference state

defined in section 1.4.1.2 :

φo = φtopo −∫ p

po

αodp with φo(po) = φtopo

we get:

φ′ = φ− φo =∫ ps

p

αdp−∫ po

p

αodp

For the ocean, the reference state is simpler since ρc does not dependent on z(bo = g) and the surface reference position is uniformly z = 0 (Ro = 0), and thesame subtraction leads to a similar relation. For both fluid, using the isomorphicnotations, we can write:

φ′ =∫ rsurf

r

b dr −∫ Ro

r

bodr

and re write : φ′ =∫ rsurf

Ro

b dr +∫ Ro

r

(b− bo)dr (2.79)

or : φ′ =∫ rsurf

Ro

bodr +∫ rsurf

r

(b− bo)dr (2.80)

In section 1.3.6, following eq.2.79, the pressure/geo-potential φ′ has beenseparated into surface (φs), and hydrostatic anomaly (φ′hyd). In this section,the split between φs and φ′hyd is made according to equation 2.80. This slightlydifferent definition reflects the actual implementation in the code and is validfor both linear and non-linear free-surface formulation, in both r-coordinate andr*-coordinate.

Because the linear free-surface approximation ignore the tracer content ofthe fluid parcel between Ro and rsurf = Ro + η, for consistency reasons, thispart is also neglected in φ′hyd :

φ′hyd =∫ rsurf

r

(b− bo)dr ∫ Ro

r

(b − bo)dr

Note that in this case, the two definitions of φs and φ′hyd from equation 2.79 and2.80 converge toward the same (approximated) expressions: φs =

∫ rsurf

Robodr

and φ′hyd =∫ Ro

r b′dr.On the contrary, the unapproximated formulation (”non-linear free-surface”, see


the next section) retains the full expression: φ′hyd =∫ rsurf

r(b − bo)dr . This is

obtained by selecting nonlinFreeSurf=4 in parameter file data.

Regarding the surface potential:

φs =∫ Ro+η

Ro

bodr = bsη with bs =1η

∫ Ro+η

Ro

bodr

bs bo(Ro) is an excellent approximation (better than the usual numericaltruncation, since generally |η| is smaller than the vertical grid increment).

For the ocean, φs = gη and bs = g is uniform. For the atmosphere, how-ever, because of topographic effects, the reference surface pressure Ro = po haslarge spatial variations that are responsible for significant bs variations (from0.8 to 1.2 [m3/kg]). For this reason, when uniformLin PhiSurf =.FALSE.(parameter file data, namelist PARAM01) a non-uniform linear coefficient bs isused and computed (S/R INI LINEAR PHISURF) according to the referencesurface pressure po: bs = bo(Ro) = cpκ(po/P

oSL)(κ−1)θref (po). with P o

SL themean sea-level pressure.

2.9.2.2 Free surface effect on column total thickness (Non-linearfree-surface)

The total thickness of the fluid column is rsurf − Rfixed = η + Ro − Rfixed.In most applications, the free surface displacements are small compared to thetotal thickness η Ho = Ro − Rfixed. In the previous sections and in olderversion of the model, the linearized free-surface approximation was made, as-suming rsurf −Rfixed Ho when computing horizontal transports, either in thecontinuity equation or in tracer and momentum advection terms. This approx-imation is dropped when using the non-linear free-surface formulation and thetotal thickness, including the time varying part η, is considered when computinghorizontal transports. Implications for the barotropic part are presented here-after. In section 2.9.2.3 consequences for tracer conservation is briefly discussed(more details can be found in [8]) ; the general time-stepping is presented insection 2.9.2.4 with some limitations regarding the vertical resolution in section2.9.2.5.

In the non-linear formulation, the continuous form of the model equationsremains unchanged, except for the 2D continuity equation (2.16) which is nowintegrated from Rfixed(x, y) up to rsurf = Ro + η :

εfs∂tη = r|r=rsurf+ εfw(P − E) = −∇h ·

∫ Ro+η

Rfixed

�vdr + εfw(P − E)

Since η has a direct effect on the horizontal velocity (through ∇hΦsurf ), thisadds a non-linear term to the free surface equation. Several options for the timediscretization of this non-linear part can be considered, as detailed below.


If the column thickness is evaluated at time step n, and with implicit treat-ment of the surface potential gradient, equations (2.74 and 2.75) becomes:

εfsηn+1 −∇h · Δt2(ηn +Ro −Rfixed)∇hbsη

n+1 = η∗

where

η∗ = εfs ηn − Δt∇h ·

∫ Ro+ηn

Rfixed

�v∗dr + εfwΔt(P − E)n

This method requires us to update the solver matrix at each time step.Alternatively, the non-linear contribution can be evaluated fully explicitly:

εfsηn+1 −∇h · Δt2(Ro −Rfixed)∇hbsη

n+1 = η∗ + ∇h · Δt2(ηn)∇hbsηn

This formulation allows one to keep the initial solver matrix unchanged thoughthroughout the integration, since the non-linear free surface only affects theRHS.

Finally, another option is a ”linearized” formulation where the total columnthickness appears only in the integral term of the RHS (2.75) but not directlyin the equation (2.74).

Those different options (see Table 2.1) have been tested and show littledifferences. However, we recommend the use of the most precise method (the1rst one) since the computation cost involved in the solver matrix update isnegligible.

parameter value description-1 linear free-surface, restart from a pickup file

produced with #undef EXACT CONSERV code0 Linear free-surface

nonlinFreeSurf 4 Non-linear free-surface3 same as 4 but neglecting

∫ Ro+η

Rob′dr in Φ′hyd

2 same as 3 but do not update cg2d solver matrix1 same as 2 but treat momentum as in Linear FS0 do not use r∗ vertical coordinate (= default)

select rStar 2 use r∗ vertical coordinate1 same as 2 but without the contribution of the

slope of the coordinate in ∇Φ

Table 2.1: Non-linear free-surface flags

2.9.2.3 Tracer conservation with non-linear free-surface

To ensure global tracer conservation (i.e., the total amount) as well as localconservation, the change in the surface level thickness must be consistent withthe way the continuity equation is integrated, both in the barotropic part (tofind η) and baroclinic part (to find w = r).


To illustrate this, consider the shallow water model, with a source of freshwater (P):

∂th+ ∇ · h�v = P

where h is the total thickness of the water column. To conserve the tracer θ wehave to discretize:

∂t(hθ) + ∇ · (hθ�v) = Pθrain

Using the implicit (non-linear) free surface described above (section 2.3) wehave:

hn+1 = hn − Δt∇ · (hn �vn+1) + ΔtP

The discretized form of the tracer equation must adopt the same “form” inthe computation of tracer fluxes, that is, the same value of h, as used in thecontinuity equation:

hn+1 θn+1 = hn θn − Δt∇ · (hn θn �vn+1) + ΔtPθrain

The use of a 3 time-levels time-stepping scheme such as the Adams-Bashforthmake the conservation sightly tricky. The current implementation with theAdams-Bashforth time-stepping provides an exact local conservation and pre-vents any drift in the global tracer content ([8]). Compared to the linear free-surface method, an additional step is required: the variation of the water columnthickness (from hn to hn+1) is not incorporated directly into the tracer equation.Instead, the model uses the Gθ terms (first step) as in the linear free surfaceformulation (with the ”surface correction” turned ”on”, see tracer section):

Gnθ =(−∇ · (hn θn �vn+1) − rn+1

surfθn)/hn

Then, in a second step, the thickness variation (expansion/reduction) is takeninto account:

θn+1 = θn + Δthn

hn+1

(G

(n+1/2)θ + P (θrain − θn)/hn

)Note that with a simple forward time step (no Adams-Bashforth), these twoformulations are equivalent, since (hn+1−hn)/Δt = P−∇·(hn �vn+1) = P+rn+1

surf

2.9.2.4 Time stepping implementation of the non-linear free-surface

The grid cell thickness was hold constant with the linear free-surface ; with thenon-linear free-surface, it is now varying in time, at least at the surface level.This implies some modifications of the general algorithm described earlier insections 2.6 and 2.7.


A simplified version of the staggered in time, non-linear free-surface algo-rithm is detailed hereafter, and can be compared to the equivalent linear free-surface case (eq. 2.42 to 2.52) and can also be easily transposed to the syn-chronous time-stepping case. Among the simplifications, salinity equation, im-plicit operator and detailed elliptic equation are omitted. Surface forcing is ex-plicitly written as fluxes of temperature, fresh water and momentum, Qn+1/2, Pn+1/2, Fn

v

respectively. hn and dhn are the column and grid box thickness in r-coordinate.

φnhyd =

∫b(θn, Sn, r)dr (2.81)

�Gn−1/2�v = �G�v(dhn−1, �vn−1/2) ; �G(n)

�v =32�Gn−1/2

�v − 12�Gn−3/2

�v (2.82)

�v∗ = �vn−1/2 + Δtdhn−1

dhn

(�G(n)

�v + Fn�v /dh

n−1)− Δt∇φn

hyd (2.83)

update model geometry : hFac(dhn)

ηn+1/2 = ηn−1/2 + ΔtPn+1/2 − Δt∇ ·∫�vn+1/2dhn

= ηn−1/2 + ΔtPn+1/2 − Δt∇ ·∫(�v∗ − gΔt∇ηn+1/2

)dhn (2.84)

�vn+1/2 = �v∗ − gΔt∇ηn+1/2 (2.85)

hn+1 = hn + ΔtPn+1/2 − Δt∇ ·∫�vn+1/2dhn (2.86)

Gnθ = Gθ(dhn, un+1/2, θn) ; G

(n+1/2)θ =

32Gn

θ − 12Gn−1

θ (2.87)

θn+1 = θn + Δtdhn

dhn+1

(G

(n+1/2)θ + (Pn+1/2(θrain − θn) +Qn+1/2)/dhn

)(2.88)

Two steps have been added to linear free-surface algorithm (eq. 2.42 to 2.52):Firstly, the model “geometry” (here the hFacC,W,S) is updated just beforeentering SOLVE FOR PRESSURE, using the current dhn field. Secondly, thevertically integrated continuity equation (eq.2.86) has been added (exactCon-serv=TRUE, in parameter file data, namelist PARM01) just before computingthe vertical velocity, in subroutine INTEGR CONTINUITY. This ensures thattracer and continuity equation discretization a Although this equation mightappear redundant with eq.2.84, the integrated column thickness hn+1 can bedifferent from ηn+1/2 +H :

• when Crank-Nickelson time-stepping is used (see section 2.9.1).

• when filters are applied to the flow field, after (2.85) and alter the diver-gence of the flow.

• when the solver does not iterate until convergence ; for example, becausea too large residual target was set (cg2dTargetResidual, parameter filedata, namelist PARM02).

2.10. SPATIAL DISCRETIZATION OF THE DYNAMICAL EQUATIONS69

In this staggered time-stepping algorithm, the momentum tendencies are com-puted using dhn−1 geometry factors. (eq.2.82) and then rescaled in subroutineTIMESTEP, (eq.2.83), similarly to tracer tendencies (see section 2.9.2.3). Thetracers are stepped forward later, using the recently updated flow field vn+1/2

and the corresponding model geometry dhn to compute the tendencies (eq.2.87);Then the tendencies are rescaled by dhn/dhn+1 to derive the new tracers values(θ, S)n+1 (eq.2.88, in subroutine CALC GT, CALC GS).

Note that the fresh-water input is added in a consistent way in the conti-nuity equation and in the tracer equation, taking into account the fresh-watertemperature θrain.

Regarding the restart procedure, two 2.D fields hn−1 and (hn −hn−1)/Δt inaddition to the standard state variables and tendencies (ηn−1/2, vn−1/2, θn, Sn,Gn−3/2

v , Gn−1θ,S ) are stored in a ”pickup” file. The model restarts reading this

”pickup” file, then update the model geometry according to hn−1, and computehn and the vertical velocity before starting the main calling sequence (eq.2.81to 2.88, S/R FORWARD STEP).

INTEGR CONTINUITY (integr continuity.F)hn+1 −Ho: etaH (DYNVARS.h)hn −Ho: etaHnm1 (SURFACE.h)hn+1 − hn/Δt: dEtaHdt (SURFACE.h)

2.9.2.5 Non-linear free-surface and vertical resolution

When the amplitude of the free-surface variations becomes as large as the verti-cal resolution near the surface, the surface layer thickness can decrease to nearlyzero or can even vanish completely. This later possibility has not been imple-mented, and a minimum relative thickness is imposed (hFacInf, parameter filedata, namelist PARM01) to prevent numerical instabilities caused by very thinsurface level.

A better alternative to the vanishing level problem has been found and im-plemented recently, and rely on a different vertical coordinate r∗ : The timevariation ot the total column thickness becomes part of the r* coordinate mo-tion, as in a σz , σp model, but the fixed part related to topography is treated asin a height or pressure coordinate model. A complete description is given in [4].

The time-stepping implementation of the r∗ coordinate is identical to thenon-linear free-surface in r coordinate, and differences appear only in the spacialdiscretization. needs a subsection ref. here

2.10 Spatial discretization of the dynamical equa-

tions

Spatial discretization is carried out using the finite volume method. Thisamounts to a grid-point method (namely second-order centered finite difference)


in the fluid interior but allows boundaries to intersect a regular grid allowinga more accurate representation of the position of the boundary. We treat thehorizontal and vertical directions as separable and differently.

2.10.1 Notation

The notations we use to describe the discrete formulation of the model are sum-marized hereafter:general notation:Δx,Δy,Δr grid spacing in X,Y,R directions.Ao : Area of the face orthogonal to ”o” direction (o=u,v,w ...).Vu,Vv,Vv,Vθ : Volume of the grid box surrounding u, v, w, θ point;i, j, k : current index relative to X,Y,R directions;basic operator:δi : δiΦ = Φi+1/2 − Φi−1/2

i : Φi= (Φi+1/2 + Φi−1/2)/2

δx : δxΦ = 1ΔxδiΦ

∇ = gradient operator : ∇Φ = {δxΦ, δyΦ}∇· = divergence operator : ∇ ·�f = 1

A{δiΔyfx + δjΔxfy}∇2

= Laplacian operator : ∇2Φ = ∇ · ∇Φ

2.10.2 The finite volume method: finite volumes versusfinite difference

The finite volume method is used to discretize the equations in space. Theexpression “finite volume” actually has two meanings; one is the method ofembedded or intersecting boundaries (shaved or lopped cells in our terminology)and the other is non-linear interpolation methods that can deal with non-smoothsolutions such as shocks (i.e. flux limiters for advection). Both make use of theintegral form of the conservation laws to which the weak solution is a solution oneach finite volume of (sub-domain). The weak solution can be constructed outof piece-wise constant elements or be differentiable. The differentiable equationscan not be satisfied by piece-wise constant functions.

As an example, the 1-D constant coefficient advection-diffusion equation:

∂tθ + ∂x(uθ − κ∂xθ) = 0

can be discretized by integrating over finite sub-domains, i.e. the lengths Δxi:

Δx∂tθ + δi(F ) = 0

is exact if θ(x) is piece-wise constant over the interval Δxi or more generally ifθi is defined as the average over the interval Δxi.

The flux, Fi−1/2, must be approximated:

F = uθ − κ

Δxc∂iθ


uv

v u

w

w

Figure 2.8: Three dimensional staggering of velocity components. This facili-tates the natural discretization of the continuity and tracer equations.

and this is where truncation errors can enter the solution. The method forobtaining θ is unspecified and a wide range of possibilities exist including cen-tered and upwind interpolation, polynomial fits based on the the volume averagedefinitions of quantities and non-linear interpolation such as flux-limiters.

Choosing simple centered second-order interpolation and differencing recov-ers the same ODE’s resulting from finite differencing for the interior of a fluid.Differences arise at boundaries where a boundary is not positioned on a regularor smoothly varying grid. This method is used to represent the topographyusing lopped cell, see [5]. Subtle difference also appear in more than one di-mension away from boundaries. This happens because the each direction isdiscretized independently in the finite difference method while the integratingover finite volume implicitly treats all directions simultaneously. Illustration ofthis is given in [2].

2.10.3 C grid staggering of variables

The basic algorithm employed for stepping forward the momentum equations isbased on retaining non-divergence of the flow at all times. This is most naturallydone if the components of flow are staggered in space in the form of an ArakawaC grid [7].

Fig. 2.8 shows the components of flow (u,v,w) staggered in space such thatthe zonal component falls on the interface between continuity cells in the zonaldirection. Similarly for the meridional and vertical directions. The continuitycell is synonymous with tracer cells (they are one and the same).


2.10.4 Grid initialization and data

Initialization of grid data is controlled by subroutine INI GRID which in callsINI VERTICAL GRID to initialize the vertical grid, and then either of INI CARTESIAN GRID,INI SPHERICAL POLAR GRID or INI CURVILINEAR GRID to initialize thehorizontal grid for cartesian, spherical-polar or curvilinear coordinates respec-tively.

The reciprocals of all grid quantities are pre-calculated and this is donein subroutine INI MASKS ETC which is called later by subroutine INITIAL-IZE FIXED.

All grid descriptors are global arrays and stored in common blocks in GRID.hand a generally declared as RS.

S/R INI GRID (model/src/ini grid.F)S/R INI MASKS ETC (model/src/ini masks etc.F)grid data: (model/inc/GRID.h)

2.10.5 Horizontal grid

The model domain is decomposed into tiles and within each tile a quasi-regulargrid is used. A tile is the basic unit of domain decomposition for parallelizationbut may be used whether parallelized or not; see section ?? for more details.Although the tiles may be patched together in an unstructured manner (i.e.irregular or non-tessilating pattern), the interior of tiles is a structured grid ofquadrilateral cells. The horizontal coordinate system is orthogonal curvilinearmeaning we can not necessarily treat the two horizontal directions as separable.Instead, each cell in the horizontal grid is described by the length of it’s sidesand it’s area.

The grid information is quite general and describes any of the available coor-dinates systems, cartesian, spherical-polar or curvilinear. All that is necessaryto distinguish between the coordinate systems is to initialize the grid data (de-scriptors) appropriately.

In the following, we refer to the orientation of quantities on the computa-tional grid using geographic terminology such as points of the compass. This isCaution!purely for convenience but should note be confused with the actual geographicorientation of model quantities.

Fig. 2.9a shows the tracer cell (synonymous with the continuity cell). Thelength of the southern edge, Δxg, western edge, Δyg and surface area, Ac,presented in the vertical are stored in arrays DXg, DYg and rAc. The “g”Ac: rAc

Δxg: DXgΔyg: DYg

suffix indicates that the lengths are along the defining grid boundaries. The “c”suffix associates the quantity with the cell centers. The quantities are staggeredin space and the indexing is such that DXg(i,j) is positioned to the south ofrAc(i,j) and DYg(i,j) positioned to the west.

Fig. 2.9b shows the vorticity cell. The length of the southern edge, Δxc,western edge, Δyc and surface area, Aζ , presented in the vertical are stored inarrays DXg, DYg and rAz. The “z” suffix indicates that the lengths areAζ : rAz

Δxc: DXcΔyc: DYc

measured between the cell centers and the “ζ” suffix associates points with the


a) Δy

Δx

g

g

u

vv

v

uu

v

v

u u

u

v

Ac

b)

Δyc

Δxc

u

vv

v

uu

v v

v

u

u

u

Aζ

c)Δyf

Δxv

u

v

v

uu

v

v

v

v

u

u

Au

d) Δxf

Δyu

u

v

v

uu

v

v

u

v

u

u

Av

Figure 2.9: Staggering of horizontal grid descriptors (lengths and areas). Thegrid lines indicate the tracer cell boundaries and are the reference grid for allpanels. a) The area of a tracer cell, Ac, is bordered by the lengths Δxg andΔyg. b) The area of a vorticity cell, Aζ , is bordered by the lengths Δxc andΔyc. c) The area of a u cell, Ac, is bordered by the lengths Δxv and Δyf . d)The area of a v cell, Ac, is bordered by the lengths Δxf and Δyu.


vorticity points. The quantities are staggered in space and the indexing is suchthat DXc(i,j) is positioned to the north of rAc(i,j) and DYc(i,j) positionedto the east.

Fig. 2.9c shows the “u” or western (w) cell. The length of the southernedge, Δxv, eastern edge, Δyf and surface area, Aw, presented in the verticalare stored in arrays DXv, DYf and rAw. The “v” suffix indicates that theAw: rAw

Δxv: DXvΔyf : DYf

length is measured between the v-points, the “f” suffix indicates that the lengthis measured between the (tracer) cell faces and the “w” suffix associates pointswith the u-points (w stands for west). The quantities are staggered in spaceand the indexing is such that DXv(i,j) is positioned to the south of rAw(i,j)and DYf(i,j) positioned to the east.

Fig. 2.9d shows the “v” or southern (s) cell. The length of the northernedge, Δxf , western edge, Δyu and surface area, As, presented in the verticalare stored in arrays DXf, DYu and rAs. The “u” suffix indicates that theAs: rAs

Δxf : DXfΔyu: DYu

length is measured between the u-points, the “f” suffix indicates that the lengthis measured between the (tracer) cell faces and the “s” suffix associates pointswith the v-points (s stands for south). The quantities are staggered in spaceand the indexing is such that DXf(i,j) is positioned to the north of rAs(i,j)and DYu(i,j) positioned to the west.

S/R INI CARTESIAN GRID (model/src/ini cartesian grid.F)S/R INI SPHERICAL POLAR GRID (model/src/ini spherical polar grid.F)S/R INI CURVILINEAR GRID (model/src/ini curvilinear grid.F)Ac, Aζ , Aw, As: rAc, rAz, rAw, rAs (GRID.h)Δxg , Δyg: DXg, DYg (GRID.h)Δxc, Δyc: DXc, DYc (GRID.h)Δxf , Δyf : DXf, DYf (GRID.h)Δxv, Δyu: DXv, DYu (GRID.h)

2.10.5.1 Reciprocals of horizontal grid descriptors

Lengths and areas appear in the denominator of expressions as much as in thenumerator. For efficiency and portability, we pre-calculate the reciprocal of thehorizontal grid quantities so that in-line divisions can be avoided.

For each grid descriptor (array) there is a reciprocal named using the prefixRECIP . This doubles the amount of storage in GRID.h but they are all only2-D descriptors.

S/R INI MASKS ETC (model/src/ini masks etc.F)A−1

c : RECIP Ac (GRID.h)A−1

ζ : RECIP Az (GRID.h)A−1

w : RECIP Aw (GRID.h)A−1

s : RECIP As (GRID.h)Δx−1

g , Δy−1g : RECIP DXg, RECIP DYg (GRID.h)

Δx−1c , Δy−1

c : RECIP DXc, RECIP DYc (GRID.h)Δx−1

f , Δy−1f : RECIP DXf, RECIP DYf (GRID.h)

Δx−1v , Δy−1

u : RECIP DXv, RECIP DYu (GRID.h)


2.10.5.2 Cartesian coordinates

Cartesian coordinates are selected when the logical flag usingCartesianGridin namelist PARM04 is set to true. The grid spacing can be set to uniformvia scalars dXspacing and dYspacing in namelist PARM04 or to variableresolution by the vectors DELX and DELY. Units are normally meters. Non-dimensional coordinates can be used by interpreting the gravitational constantas the Rayleigh number.

2.10.5.3 Spherical-polar coordinates

Spherical coordinates are selected when the logical flag usingSphericalPolarGridin namelist PARM04 is set to true. The grid spacing can be set to uniform viascalars dXspacing and dYspacing in namelist PARM04 or to variable res-olution by the vectors DELX and DELY. Units of these namelist variablesare alway degrees. The horizontal grid descriptors are calculated from thesenamelist variables have units of meters.

2.10.5.4 Curvilinear coordinates

Curvilinear coordinates are selected when the logical flag usingCurvilinear-Grid in namelist PARM04 is set to true. The grid spacing can not be set viathe namelist. Instead, the grid descriptors are read from data files, one foreach descriptor. As for other grids, the horizontal grid descriptors have units ofmeters.

2.10.6 Vertical grid

As for the horizontal grid, we use the suffixes “c” and “f” to indicates faces andcenters. Fig. 2.10a shows the default vertical grid used by the model. Δrf Δrf : DRf

Δrc: DRcis the difference in r (vertical coordinate) between the faces (i.e. Δrf ≡ −δkrwhere the minus sign appears due to the convention that the surface layer hasindex k = 1.).

The vertical grid is calculated in subroutine INI VERTICAL GRID andspecified via the vector DELR in namelist PARM04. The units of “r” areeither meters or Pascals depending on the isomorphism being used which inturn is dependent only on the choice of equation of state.

There are alternative namelist vectors DELZ and DELP which dictatewhether z- or p- coordinates are to be used but we intend to phase this out Caution!since they are redundant.

The reciprocals Δr−1f and Δr−1

c are pre-calculated (also in subroutine INI VERTICAL GRID).All vertical grid descriptors are stored in common blocks in GRID.h.

The above grid (Fig. 2.10a) is known as the cell centered approach becausethe tracer points are at cell centers; the cell centers are mid-way between thecell interfaces. This discretization is selected when the thickness of the levelsare provided (delR, parameter file data, namelist PARM04) An alternative, the


a)

Δrf

Δrc

w

w

w

w

b)

Δrc

Δrf

w

w

w

w

Figure 2.10: Two versions of the vertical grid. a) The cell centered approachwhere the interface depths are specified and the tracer points centered in be-tween the interfaces. b) The interface centered approach where tracer levels arespecified and the w-interfaces are centered in between.


hwhc

Δr f

Δr f

Δr f

x

r

Figure 2.11: A schematic of the x-r plane showing the location of the non-dimensional fractions hc and hw. The physical thickness of a tracer cell isgiven by hc(i, j, k)Δrf (k) and the physical thickness of the open side is givenby hw(i, j, k)Δrf (k).

vertex or interface centered approach, is shown in Fig. 2.10b. Here, the inte-rior interfaces are positioned mid-way between the tracer nodes (no longer cellcenters). This approach is formally more accurate for evaluation of hydrostaticpressure and vertical advection but historically the cell centered approach hasbeen used. An alternative form of subroutine INI VERTICAL GRID is usedto select the interface centered approach This form requires to specify Nr + 1vertical distances delRc (parameter file data, namelist PARM04, e.g. verifica-tion/ideal 2D oce/input/data) corresponding to surface to center, Nr−1 centerto center, and center to bottom distances.

S/R INI VERTICAL GRID (model/src/ini vertical grid.F)Δrf : DRf (GRID.h)Δrc: DRc (GRID.h)Δr−1

f : RECIP DRf (GRID.h)Δr−1

c : RECIP DRc (GRID.h)


2.10.7 Topography: partially filled cells

[5] presented two alternatives to the step-wise finite difference representation oftopography. The method is known to the engineering community as intersectingboundary method. It involves allowing the boundary to intersect a grid of cellsthereby modifying the shape of those cells intersected. We suggested allowingthe topography to take on a piece-wise linear representation (shaved cells) ora simpler piecewise constant representation (partial step). Both show dramaticimprovements in solution compared to the traditional full step representation,the piece-wise linear being the best. However, the storage requirements areexcessive so the simpler piece-wise constant or partial-step method is all that iscurrently supported.

Fig. 2.11 shows a schematic of the x-r plane indicating how the thickness of alevel is determined at tracer and u points. The physical thickness of a tracer cellhc: hFacC

hw: hFacWhs: hFacS

is given by hc(i, j, k)Δrf (k) and the physical thickness of the open side is givenby hw(i, j, k)Δrf (k). Three 3-D descriptors hc, hw and hs are used to describethe geometry: hFacC, hFacW and hFacS respectively. These are calculatedin subroutine INI MASKS ETC along with there reciprocals RECIP hFacC,RECIP hFacW and RECIP hFacS.

The non-dimensional fractions (or h-facs as we call them) are calculatedfrom the model depth array and then processed to avoid tiny volumes. Therule is that if a fraction is less than hFacMin then it is rounded to the nearerof 0 or hFacMin or if the physical thickness is less than hFacMinDr then itis similarly rounded. The larger of the two methods is used when there is aconflict. By setting hFacMinDr equal to or larger than the thinnest nominallayers, min (Δzf ), but setting hFacMin to some small fraction then the modelwill only lop thick layers but retain stability based on the thinnest unloppedthickness; min (Δzf ,hFacMinDr).

S/R INI MASKS ETC (model/src/ini masks etc.F)hc: hFacC (GRID.h)hw: hFacW (GRID.h)hs: hFacS (GRID.h)h−1

c : RECIP hFacC (GRID.h)h−1

w : RECIP hFacW (GRID.h)h−1

s : RECIP hFacS (GRID.h)

2.11. CONTINUITY AND HORIZONTAL PRESSURE GRADIENT TERMS79

2.11 Continuity and horizontal pressure gradi-

ent terms

The core algorithm is based on the “C grid” discretization of the continuityequation which can be summarized as:

∂tu+1

Δxcδi∂Φ∂r

∣∣∣∣s

η +εnh

ΔxcδiΦ′nh = Gu − 1

ΔxcδiΦ′h (2.89)

∂tv +1

Δycδj∂Φ∂r

∣∣∣∣s

η +εnh

ΔycδjΦ′nh = Gv − 1

ΔycδjΦ′h (2.90)

εnh

(∂tw +

1Δrc

δkΦ′nh

)= εnhGw + b

k − 1Δrc

δkΦ′h(2.91)

δiΔygΔrfhwu+ δjΔxgΔrfhsv + δkAcw = Acδk(P − E)r=0 (2.92)

where the continuity equation has been most naturally discretized by staggeringthe three components of velocity as shown in Fig. 2.8. The grid lengths Δxc andΔyc are the lengths between tracer points (cell centers). The grid lengths Δxg,Δyg are the grid lengths between cell corners. Δrf and Δrc are the distance(in units of r) between level interfaces (w-level) and level centers (tracer level).The surface area presented in the vertical is denoted Ac. The factors hw andhs are non-dimensional fractions (between 0 and 1) that represent the fractioncell depth that is “open” for fluid flow. hw: hFacW

hs: hFacSThe last equation, the discrete continuity equation, can be summed in thevertical to yield the free-surface equation:

Ac∂tη + δi∑

k

ΔygΔrfhwu+ δj∑

k

ΔxgΔrfhsv = Ac(P − E)r=0 (2.93)

The source term P−E on the rhs of continuity accounts for the local addition ofvolume due to excess precipitation and run-off over evaporation and only entersthe top-level of the ocean model.

2.12 Hydrostatic balance

The vertical momentum equation has the hydrostatic or quasi-hydrostatic bal-ance on the right hand side. This discretization guarantees that the conversionof potential to kinetic energy as derived from the buoyancy equation exactlymatches the form derived from the pressure gradient terms when forming thekinetic energy equation.

In the ocean, using z-coordinates, the hydrostatic balance terms are dis-cretized:

εnh∂tw + gρ′k

+1

ΔzδkΦ′h = . . . (2.94)

In the atmosphere, using p-coordinates, hydrostatic balance is discretized:

θ′k

+1

ΔΠδkΦ′h = 0 (2.95)


where ΔΠ is the difference in Exner function between the pressure points. Thenon-hydrostatic equations are not available in the atmosphere.

The difference in approach between ocean and atmosphere occurs because ofthe direct use of the ideal gas equation in forming the potential energy conversionterm αω. The form of these conversion terms is discussed at length in [13].

Because of the different representation of hydrostatic balance between oceanand atmosphere there is no elegant way to represent both systems using anarbitrary coordinate.

The integration for hydrostatic pressure is made in the positive r direction(increasing k-index). For the ocean, this is from the free-surface down and forthe atmosphere this is from the ground up.

The calculations are made in the subroutine CALC PHI HYD. Inside thisroutine, one of other of the atmospheric/oceanic form is selected based on thestring variable buoyancyRelation.

2.13 Flux-form momentum equations

The original finite volume model was based on the Eulerian flux form momentumequations. This is the default though the vector invariant form is optionallyavailable (and recommended in some cases).

The “G’s” (our colloquial name for all terms on rhs!) are broken into the var-ious advective, Coriolis, horizontal dissipation, vertical dissipation and metricforces:Gu: Gu

Gv: GvGw: Gw Gu = Gadv

u +Gcoru +Gh−diss

u +Gv−dissu +Gmetric

u +Gnh−metricu (2.96)

Gv = Gadvv +Gcor

v +Gh−dissv +Gv−diss

v +Gmetricv +Gnh−metric

v (2.97)Gw = Gadv

w +Gcorw +Gh−diss

w +Gv−dissw +Gmetric

w +Gnh−metricw (2.98)

In the hydrostatic limit, Gw = 0 and εnh = 0, reducing the vertical momentumto hydrostatic balance.

These terms are calculated in routines called from subroutine CALC MOM RHSa collected into the global arrays Gu, Gv, and Gw.

S/R CALC MOM RHS (pkg/mom fluxform/calc mom rhs.F)Gu: Gu (DYNVARS.h)Gv: Gv (DYNVARS.h)Gw: Gw (DYNVARS.h)

2.13.1 Advection of momentum

The advective operator is second order accurate in space:

AwΔrfhwGadvu = δiU

iui + δjV

iuj + δkW

iuk (2.99)

AsΔrfhsGadvv = δiU

jvi + δjV

jvj + δkW

jvk (2.100)

AcΔrcGadvw = δiU

kwi + δjV

kwj + δkW

kwk (2.101)

2.13. FLUX-FORM MOMENTUM EQUATIONS 81

and because of the flux form does not contribute to the global budget of linearmomentum. The quantities U , V and W are volume fluxes defined: U : uTrans

V : vTransW : rTransU = ΔygΔrfhwu (2.102)

V = ΔxgΔrfhsv (2.103)W = Acw (2.104)

The advection of momentum takes the same form as the advection of tracersbut by a translated advective flow. Consequently, the conservation of secondmoments, derived for tracers later, applies to u2 and v2 and w2 so that advectionof momentum correctly conserves kinetic energy.

S/R MOM U ADV UU (mom u adv uu.F)S/R MOM U ADV VU (mom u adv vu.F)S/R MOM U ADV WU (mom u adv wu.F)S/R MOM U ADV UV (mom u adv uv.F)S/R MOM U ADV VV (mom u adv vv.F)S/R MOM U ADV WV (mom u adv wv.F)uu, uv, vu, vv: aF (local to calc mom rhs.F)

2.13.2 Coriolis terms

The “pure C grid” Coriolis terms (i.e. in absence of C-D scheme) are discretized:

AwΔrfhwGCoru = fAcΔrfhcv

ji − εnhf ′AcΔrfhcw

ki

(2.105)

AsΔrfhsGCorv = −fAcΔrfhcu

ij

(2.106)

AcΔrcGCorw = εnhf ′AcΔrfhcu

ik

(2.107)

where the Coriolis parameters f and f ′ are defined:

f = 2Ω sinϕ (2.108)f ′ = 2Ω cosϕ (2.109)

where ϕ is geographic latitude when using spherical geometry, otherwise theβ-plane definition is used:

f = fo + βy (2.110)f ′ = 0 (2.111)

This discretization globally conserves kinetic energy. It should be noted thatdespite the use of this discretization in former publications, all calculations todate have used the following different discretization:

GCoru = fuv

ji − εnhf′uw

ik (2.112)GCor

v = −fvuij (2.113)

GCorw = εnhf

′wu

ik (2.114)


where the subscripts on f and f ′ indicate evaluation of the Coriolis parameters Need to change the defacode to match thisat the appropriate points in space. The above discretization does not conserve

anything, especially energy and for historical reasons is the default for the code.A flag controls this discretization: set run-time logical useEnergyConserv-ingCoriolis to true which otherwise defaults to false.

S/R MOM CDSCHEME (mom cdscheme.F)S/R MOM U CORIOLIS (mom u coriolis.F)S/R MOM V CORIOLIS (mom v coriolis.F)GCor

u , GCorv : cF (local to calc mom rhs.F)

2.13.3 Curvature metric terms

The most commonly used coordinate system on the sphere is the geographicsystem (λ, ϕ). The curvilinear nature of these coordinates on the sphere leadto some “metric” terms in the component momentum equations. Under thethin-atmosphere and hydrostatic approximations these terms are discretized:

AwΔrfhwGmetricu =

ui

atanϕAcΔrfhcv

j

i

(2.115)

AsΔrfhsGmetricv = −u

i

atanϕAcΔrfhcu

i

j

(2.116)

Gmetricw = 0 (2.117)

where a is the radius of the planet (sphericity is assumed) or the radial distanceof the particle (i.e. a function of height). It is easy to see that this discretizationsatisfies all the properties of the discrete Coriolis terms since the metric factorua tanϕ can be viewed as a modification of the vertical Coriolis parameter: f →f + u

a tanϕ.However, as for the Coriolis terms, a non-energy conserving form has exclu-

sively been used to date:

Gmetricu =

uvij

atanϕ (2.118)

Gmetricv =

uijuij

atanϕ (2.119)

where tanϕ is evaluated at the u and v points respectively.S/R MOM U METRIC SPHERE (mom u metric sphere.F)S/R MOM V METRIC SPHERE (mom v metric sphere.F)Gmetric

u , Gmetricv : mT (local to calc mom rhs.F)

2.13.4 Non-hydrostatic metric terms

For the non-hydrostatic equations, dropping the thin-atmosphere approximationre-introduces metric terms involving w and are required to conserve angular


momentum:

AwΔrfhwGmetricu = −u

iwk

aAcΔrfhc

i

(2.120)

AsΔrfhsGmetricv = −v

jwk

aAcΔrfhc

j

(2.121)

AcΔrcGmetricw =

ui2 + vj2

aAcΔrfhc

k

(2.122)

Because we are always consistent, even if consistently wrong, we have, in thepast, used a different discretization in the model which is:

Gmetricu = −u

awik (2.123)

Gmetricv = −v

awjk (2.124)

Gmetricw =

1a(uik2

+ vjk2) (2.125)

S/R MOM U METRIC NH (mom u metric nh.F)S/R MOM V METRIC NH (mom v metric nh.F)Gmetric

u , Gmetricv : mT (local to calc mom rhs.F)

2.13.5 Lateral dissipation

Historically, we have represented the SGS Reynolds stresses as simply downgradient momentum fluxes, ignoring constraints on the stress tensor such assymmetry.

AwΔrfhwGh−dissu = δiΔyfΔrfhcτ11 + δjΔxvΔrfhζτ12 (2.126)

AsΔrfhsGh−dissv = δiΔyuΔrfhζτ21 + δjΔxfΔrfhcτ22 (2.127)

Check signs of stress defini-tionsThe lateral viscous stresses are discretized:

τ11 = Ahc11Δ(ϕ)1

Δxfδiu−A4c11Δ2(ϕ)

1Δxf

δi∇2u (2.128)

τ12 = Ahc12Δ(ϕ)1

Δyuδju−A4c12Δ2(ϕ)

1Δyu

δj∇2u (2.129)

τ21 = Ahc21Δ(ϕ)1

Δxvδiv −A4c21Δ2(ϕ)

1Δxv

δi∇2v (2.130)

τ22 = Ahc22Δ(ϕ)1

Δyfδjv −A4c22Δ2(ϕ)

1Δyf

δj∇2v (2.131)

where the non-dimensional factors clmΔn(ϕ), {l,m, n} ∈ {1, 2} define the “co-sine” scaling with latitude which can be applied in various ad-hoc ways. For


instance, c11Δ = c21Δ = (cosϕ)3/2, c12Δ = c22Δ = 0 would represent thean-isotropic cosine scaling typically used on the “lat-lon” grid for Laplacianviscosity. Need to tidy up metho

controlling this in codeIt should be noted that despite the ad-hoc nature of the scaling, some scalingmust be done since on a lat-lon grid the converging meridians make it veryunlikely that a stable viscosity parameter exists across the entire model domain.

The Laplacian viscosity coefficient, Ah (viscAh), has units of m2s−1. Thebi-harmonic viscosity coefficient, A4 (viscA4), has units of m4s−1.

S/R MOM U XVISCFLUX (mom u xviscflux.F)S/R MOM U YVISCFLUX (mom u yviscflux.F)S/R MOM V XVISCFLUX (mom v xviscflux.F)S/R MOM V YVISCFLUX (mom v yviscflux.F)τ11, τ12, τ22, τ22: vF, v4F (local to calc mom rhs.F)

Two types of lateral boundary condition exist for the lateral viscous terms,no-slip and free-slip.

The free-slip condition is most convenient to code since it is equivalent tozero-stress on boundaries. Simple masking of the stress components sets themto zero. The fractional open stress is properly handled using the lopped cells.

The no-slip condition defines the normal gradient of a tangential flow suchthat the flow is zero on the boundary. Rather than modify the stresses by usingcomplicated functions of the masks and “ghost” points (see [3]) we add theboundary stresses as an additional source term in cells next to solid boundaries.This has the advantage of being able to cope with “thin walls” and also makesthe interior stress calculation (code) independent of the boundary conditions.The “body” force takes the form:

Gside−dragu =

4Δzf

(1 − hζ)Δxv

Δyu

j (Ahc12Δ(ϕ)u −A4c12Δ2(ϕ)∇2u

)(2.132)

Gside−dragv =

4Δzf

(1 − hζ)Δyu

Δxv

i (Ahc21Δ(ϕ)v −A4c21Δ2(ϕ)∇2v

)(2.133)

In fact, the above discretization is not quite complete because it assumesthat the bathymetry at velocity points is deeper than at neighboring vorticitypoints, e.g. 1 − hw < 1 − hζ

S/R MOM U SIDEDRAG (mom u sidedrag.F)S/R MOM V SIDEDRAG (mom v sidedrag.F)Gside−drag

u , Gside−dragv : vF (local to calc mom rhs.F)

2.13.6 Vertical dissipation

Vertical viscosity terms are discretized with only partial adherence to the vari-able grid lengths introduced by the finite volume formulation. This reduces theformal accuracy of these terms to just first order but only next to boundaries;


exactly where other terms appear such as linear and quadratic bottom drag.

Gv−dissu =

1Δrfhw

δkτ13 (2.134)

Gv−dissv =

1Δrfhs

δkτ23 (2.135)

Gv−dissw = εnh

1Δrfhd

δkτ33 (2.136)

represents the general discrete form of the vertical dissipation terms.In the interior the vertical stresses are discretized:

τ13 = Av1

Δrcδku (2.137)

τ23 = Av1

Δrcδkv (2.138)

τ33 = Av1

Δrfδkw (2.139)

It should be noted that in the non-hydrostatic form, the stress tensor is evenless consistent than for the hydrostatic (see [52]). It is well known how to dothis properly (see [24]) and is on the list of to-do’s.

S/R MOM U RVISCLFUX (mom u rviscflux.F)S/R MOM V RVISCLFUX (mom v rviscflux.F)τ13: urf (local to calc mom rhs.F)τ23: vrf (local to calc mom rhs.F)

As for the lateral viscous terms, the free-slip condition is equivalent to simplysetting the stress to zero on boundaries. The no-slip condition is implementedas an additional term acting on top of the interior and free-slip stresses. Bottomdrag represents additional friction, in addition to that imposed by the no-slipcondition at the bottom. The drag is cast as a stress expressed as a linear orquadratic function of the mean flow in the layer above the topography:

τbottom−drag13 =

(2Av

1Δrc

+ rb + Cd

√2KE

i

)u (2.140)

τbottom−drag23 =

(2Av

1Δrc

+ rb + Cd

√2KE

j

)v (2.141)

where these terms are only evaluated immediately above topography. rb (bot-tomDragLinear) has units of ms−1 and a typical value of the order 0.0002ms−1. Cd (bottomDragQuadratic) is dimensionless with typical values inthe range 0.001–0.003.

S/R MOM U BOTTOMDRAG (mom u bottomdrag.F)S/R MOM V BOTTOMDRAG (mom v bottomdrag.F)τbottom−drag13 , τbottom−drag

23 : vf (local to calc mom rhs.F)


2.13.7 Derivation of discrete energy conservation

These discrete equations conserve kinetic plus potential energy using the follow-ing definitions:

KE =12

(u2

i+ v2

j+ εnhw2

k)

(2.142)

2.14 Vector invariant momentum equations

The finite volume method lends itself to describing the continuity and tracerequations in curvilinear coordinate systems. However, in curvilinear coordi-nates many new metric terms appear in the momentum equations (written inLagrangian or flux-form) making generalization far from elegant. Fortunately,an alternative form of the equations, the vector invariant equations are exactlythat; invariant under coordinate transformations so that they can be applieduniformly in any orthogonal curvilinear coordinate system such as sphericalcoordinates, boundary following or the conformal spherical cube system.

The non-hydrostatic vector invariant equations read:

∂t�v + (2�Ω + �ζ) ∧ �v − br + �∇B = �∇ · �τ (2.143)

which describe motions in any orthogonal curvilinear coordinate system. Here,B is the Bernoulli function and �ζ = ∇ ∧ �v is the vorticity vector. We cantake advantage of the elegance of these equations when discretizing them anduse the discrete definitions of the grad, curl and divergence operators to satisfyconstraints. We can also consider the analogy to forming derived equations, suchas the vorticity equation, and examine how the discretization can be adjustedto give suitable vorticity advection among other things.

The underlying algorithm is the same as for the flux form equations. Allthat has changed is the contents of the “G’s”. For the time-being, only thehydrostatic terms have been coded but we will indicate the points where non-hydrostatic contributions will enter:

Gu = Gfvu +Gζ3v

u +Gζ2wu +G∂xB

u +G∂zτx

u +Gh−dissipu +Gv−dissip

u (2.144)Gv = Gfu

v +Gζ3uv +Gζ1w

v +G∂yBv +G∂zτy

v +Gh−dissipv +Gv−dissip

v (2.145)Gw = Gfu

w +Gζ1vw +Gζ2u

w +G∂zBw +Gh−dissip

w +Gv−dissipw (2.146)

S/R CALC MOM RHS (pkg/mom vecinv/calc mom rhs.F)Gu: Gu (DYNVARS.h)Gv: Gv (DYNVARS.h)Gw: Gw (DYNVARS.h)

2.14.1 Relative vorticity

The vertical component of relative vorticity is explicitly calculated and use in thediscretization. The particular form is crucial for numerical stability; alternativedefinitions break the conservation properties of the discrete equations.

2.14. VECTOR INVARIANT MOMENTUM EQUATIONS 87

Relative vorticity is defined:

ζ3 =ΓAζ

=1Aζ

(δiΔycv − δjΔxcu) (2.147)

where Aζ is the area of the vorticity cell presented in the vertical and Γ is thecirculation about that cell.

S/R MOM VI CALC RELVORT3 (mom vi calc relvort3.F)ζ3: vort3 (local to calc mom rhs.F)

2.14.2 Kinetic energy

The kinetic energy, denoted KE, is defined:

KE =12(u2

i+ v2

j+ εnhw2

k) (2.148)

S/R MOM VI CALC KE (mom vi calc ke.F)KE: KE (local to calc mom rhs.F)

2.14.3 Coriolis terms

The potential enstrophy conserving form of the linear Coriolis terms are written:

Gfvu =

1Δxc

f

hζ

j

Δxghsvji

(2.149)

Gfuv = − 1

Δyc

f

hζ

i

Δyghwuij

(2.150)

Here, the Coriolis parameter f is defined at vorticity (corner) points. f : fCoriGhζ : hFacZThe potential enstrophy conserving form of the non-linear Coriolis terms are

written:

Gζ3vu =

1Δxc

ζ3hζ

j

Δxghsvji

(2.151)

Gζ3uv = − 1

Δyc

ζ3hζ

i

Δyghwuij

(2.152)

ζ3: vort3The Coriolis terms can also be evaluated together and expressed in terms

of absolute vorticity f + ζ3. The potential enstrophy conserving form using theabsolute vorticity is written:

Gfvu +Gζ3v

u =1

Δxc

f + ζ3hζ

j

Δxghsvji

(2.153)

Gfuv +Gζ3u

v = − 1Δyc

f + ζ3hζ

i

Δyghwuij

(2.154)


Run-time control needsadded for these optionsThe distinction between using absolute vorticity or relative vorticity is useful

when constructing higher order advection schemes; monotone advection of rel-ative vorticity behaves differently to monotone advection of absolute vorticity.Currently the choice of relative/absolute vorticity, centered/upwind/high orderadvection is available only through commented subroutine calls.

S/R MOM VI CORIOLIS (mom vi coriolis.F)S/R MOM VI U CORIOLIS (mom vi u coriolis.F)S/R MOM VI V CORIOLIS (mom vi v coriolis.F)Gfv

u , Gζ3vu : uCf (local to calc mom rhs.F)

Gfuv , Gζ3u

v : vCf (local to calc mom rhs.F)

2.14.4 Shear terms

The shear terms (ζ2w and ζ1w) are are discretized to guarantee that no spuriousgeneration of kinetic energy is possible; the horizontal gradient of Bernoullifunction has to be consistent with the vertical advection of shear:N-H terms have not been

tried!Gζ2w

u =1

AwΔrfhwAcw

i(δku− εnhδjw)

k

(2.155)

Gζ1wv =

1AsΔrfhs

Acwi(δku− εnhδjw)

k

(2.156)

S/R MOM VI U VERTSHEAR (mom vi u vertshear.F)S/R MOM VI V VERTSHEAR (mom vi v vertshear.F)Gζ2w

u : uCf (local to calc mom rhs.F)Gζ1w

v : vCf (local to calc mom rhs.F)

2.14.5 Gradient of Bernoulli function

G∂xBu =

1Δxc

δi(φ′ +KE) (2.157)

G∂yBv =

1Δxy

δj(φ′ +KE) (2.158)

S/R MOM VI U GRAD KE (mom vi u grad ke.F)S/R MOM VI V GRAD KE (mom vi v grad ke.F)G∂xKE

u : uCf (local to calc mom rhs.F)G

∂yKEv : vCf (local to calc mom rhs.F)

2.14.6 Horizontal dissipation

The horizontal divergence, a complimentary quantity to relative vorticity, isused in parameterizing the Reynolds stresses and is discretized:

D =1

Achc(δiΔyghwu+ δjΔxghsv) (2.159)

2.14. VECTOR INVARIANT MOMENTUM EQUATIONS 89

S/R MOM VI CALC HDIV (mom vi calc hdiv.F)D: hDiv (local to calc mom rhs.F)

2.14.7 Horizontal dissipation

The following discretization of horizontal dissipation conserves potential vortic-ity (thickness weighted relative vorticity) and divergence and dissipates energy,enstrophy and divergence squared:

Gh−dissipu =

1Δxc

δi(ADD −AD4D∗) − 1

Δyuhwδjhζ(Aζζ −Aζ4ζ

∗)(2.160)

Gh−dissipv =

1Δxvhs

δihζ(Aζζ −Aζζ∗) +

1Δyc

δj(ADD −AD4D∗)(2.161)

where

D∗ =1

Achc(δiΔyghw∇2u+ δjΔxghs∇2v) (2.162)

ζ∗ =1Aζ

(δiΔyc∇2v − δjΔxc∇2u) (2.163)

S/R MOM VI HDISSIP (mom vi hdissip.F)Gh−dissip

u : uDiss (local to calc mom rhs.F)Gh−dissip

v : vDiss (local to calc mom rhs.F)

2.14.8 Vertical dissipation

Currently, this is exactly the same code as the flux form equations.

Gv−dissu =

1Δrfhw

δkτ13 (2.164)

Gv−dissv =

1Δrfhs

δkτ23 (2.165)

represents the general discrete form of the vertical dissipation terms.In the interior the vertical stresses are discretized:

τ13 = Av1

Δrcδku (2.166)

τ23 = Av1

Δrcδkv (2.167)

S/R MOM U RVISCLFUX (mom u rviscflux.F)S/R MOM V RVISCLFUX (mom v rviscflux.F)τ13: urf (local to calc mom rhs.F)τ23: vrf (local to calc mom rhs.F)


2.15 Tracer equations

The basic discretization used for the tracer equations is the second order piece-wise constant finite volume form of the forced advection-diffusion equations.There are many alternatives to second order method for advection and alterna-tive parameterizations for the sub-grid scale processes. The Gent-McWilliamseddy parameterization, KPP mixing scheme and PV flux parameterization areall dealt with in separate sections. The basic discretization of the advection-diffusion part of the tracer equations and the various advection schemes will bedescribed here.

2.15.1 Time-stepping of tracers: ABII

The default advection scheme is the centered second order method which re-quires a second order or quasi-second order time-stepping scheme to be sta-ble. Historically this has been the quasi-second order Adams-Bashforth method(ABII) and applied to all terms. For an arbitrary tracer, τ , the forced advection-diffusion equation reads:

∂tτ +Gτadv = Gτ

diff +Gτforc (2.168)

where Gτadv, G

τdiff and Gτ

forc are the tendencies due to advection, diffusion andforcing, respectively, namely:

Gτadv = ∂xuτ + ∂yvτ + ∂rwτ − τ∇ · v (2.169)

Gτdiff = ∇ · K∇τ (2.170)

and the forcing can be some arbitrary function of state, time and space.The term, τ∇ · v, is required to retain local conservation in conjunction

with the linear implicit free-surface. It only affects the surface layer since theflow is non-divergent everywhere else. This term is therefore referred to as thesurface correction term. Global conservation is not possible using the flux-form(as here) and a linearized free-surface ([24, 8]).

The continuity equation can be recovered by setting Gdiff = Gforc = 0 andτ = 1.

The driver routine that calls the routines to calculate tendencies are S/RCALC GT and S/R CALC GS for temperature and salt (moisture), respec-tively. These in turn call a generic advection diffusion routine S/R GAD CALC RHSthat is called with the flow field and relevant tracer as arguments and returns thecollective tendency due to advection and diffusion. Forcing is add subsequentlyin S/R CALC GT or S/R CALC GS to the same tendency array.

S/R GAD CALC RHS (pkg/generic advdiff/gad calc rhs.F)τ : tracer (argument)G(n): gTracer (argument)Fr: fVerT (argument)

2.16. LINEAR ADVECTION SCHEMES 91

The space and time discretization are treated separately (method of lines).Tendencies are calculated at time levels n and n−1 and extrapolated to n+1/2using the Adams-Bashforth method:B eps

G(n+1/2) = (32

+ ε)G(n) − (12

+ ε)G(n−1) (2.171)

where G(n) = Gτadv +Gτ

diff +Gτsrc at time step n. The tendency at n− 1 is not

re-calculated but rather the tendency at n is stored in a global array for laterre-use.

S/R ADAMS BASHFORTH2 (model/src/adams bashforth2.F)G(n+1/2): gTracer (argument on exit)G(n): gTracer (argument on entry)G(n−1): gTrNm1 (argument)ε: ABeps (PARAMS.h)

The tracers are stepped forward in time using the extrapolated tendency:

τ (n+1) = τ (n) + ΔtG(n+1/2) (2.172)

Δt: deltaTtracerS/R TIMESTEP TRACER (model/src/timestep tracer.F)τ (n+1): gTracer (argument on exit)τ (n): tracer (argument on entry)G(n+1/2): gTracer (argument)Δt: deltaTtracer (PARAMS.h)

Strictly speaking the ABII scheme should be applied only to the advectionterms. However, this scheme is only used in conjunction with the standardsecond, third and fourth order advection schemes. Selection of any other advec-tion scheme disables Adams-Bashforth for tracers so that explicit diffusion andforcing use the forward method.

2.16 Linear advection schemes

The advection schemes known as centered second order, centered fourth order,first order upwind and upwind biased third order are known as linear advectionschemes because the coefficient for interpolation of the advected tracer are linearand a function only of the flow, not the tracer field it self. We discuss these firstsince they are most commonly used in the field and most familiar.

2.16.1 Centered second order advection-diffusion

The basic discretization, centered second order, is the default. It is designedto be consistent with the continuity equation to facilitate conservation proper-ties analogous to the continuum. However, centered second order advection isnotoriously noisy and must be used in conjunction with some finite amount ofdiffusion to produce a sensible solution.


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

Θ

a) Analytic solutionupwind−1 DST−3 upwind−3 upwind−2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

Θ

b) Analytic solutionLax−Wendroff 4−DST centered−2 centered−4 4−FV

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

Θ

c) Analytic solutionminmod Superbee van Leer (MC) van Leer (alb)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

x

Θ

d) Analytic solution3−DST Sw μ=1 3−DST Sw μ(c) 4−DST Sw μ(c)

Figure 2.12: Comparison of 1-D advection schemes. Courant number is 0.05with 60 points and solutions are shown for T=1 (one complete period). a) Showsthe upwind biased schemes; first order upwind, DST3, third order upwind andsecond order upwind. b) Shows the centered schemes; Lax-Wendroff, DST4,centered second order, centered fourth order and finite volume fourth order. c)Shows the second order flux limiters: minmod, Superbee, MC limiter and thevan Leer limiter. d) Shows the DST3 method with flux limiters due to Swebywith μ = 1, μ = c/(1 − c) and a fourth order DST method with Sweby limiter,μ = c/(1 − c).


0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

Θ

a) Analytic solutionupwind−1 DST−3

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

Θ

b) Analytic solutionLax−Wendroff 4−DST

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

Θ

c) Analytic solutionminmod Superbee van Leer (MC) van Leer (alb)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.5

1

x

Θ

d) Analytic solution3−DST Sw μ=1 3−DST Sw μ(c) 4−DST Sw μ(c)

Figure 2.13: Comparison of 1-D advection schemes. Courant number is 0.89with 60 points and solutions are shown for T=1 (one complete period). a) Showsthe upwind biased schemes; first order upwind and DST3. Third order upwindand second order upwind are unstable at this Courant number. b) Shows thecentered schemes; Lax-Wendroff, DST4. Centered second order, centered fourthorder and finite volume fourth order and unstable at this Courant number. c)Shows the second order flux limiters: minmod, Superbee, MC limiter and thevan Leer limiter. d) Shows the DST3 method with flux limiters due to Swebywith μ = 1, μ = c/(1 − c) and a fourth order DST method with Sweby limiter,μ = c/(1 − c).


The advection operator is discretized:

AcΔrfhcGτadv = δiFx + δjFy + δkFr (2.173)

where the area integrated fluxes are given by:

Fx = Uτ i (2.174)Fy = V τ j (2.175)Fr = Wτk (2.176)

The quantities U , V and W are volume fluxes defined:U : uTransV : vTransW : rTrans U = ΔygΔrfhwu (2.177)

V = ΔxgΔrfhsv (2.178)W = Acw (2.179)

For non-divergent flow, this discretization can be shown to conserve thetracer both locally and globally and to globally conserve tracer variance, τ2.The proof is given in [1, 5].

S/R GAD C2 ADV X (gad c2 adv x.F)Fx: uT (argument)U : uTrans (argument)τ : tracer (argument)S/R GAD C2 ADV Y (gad c2 adv y.F)Fy : vT (argument)V : vTrans (argument)τ : tracer (argument)S/R GAD C2 ADV R (gad c2 adv r.F)Fr: wT (argument)W : rTrans (argument)τ : tracer (argument)

2.16.2 Third order upwind bias advection

Upwind biased third order advection offers a relatively good compromise be-tween accuracy and smoothness. It is not a “positive” scheme meaning falseextrema are permitted but the amplitude of such are significantly reduced overthe centered second order method.

The third order upwind fluxes are discretized:

Fx = Uτ − 16δiiτ

i

+12|U |δi 16δiiτ (2.180)

Fy = V τ − 16δiiτ

j

+12|V |δj 1

6δjjτ (2.181)

Fr = Wτ − 16δiiτ

k

+12|W |δk 1

6δkkτ (2.182)


At boundaries, δnτ is set to zero allowing δnn to be evaluated. We arecurrently examine the accuracy of this boundary condition and the effect on thesolution.

S/R GAD U3 ADV X (gad u3 adv x.F)Fx: uT (argument)U : uTrans (argument)τ : tracer (argument)S/R GAD U3 ADV Y (gad u3 adv y.F)Fy: vT (argument)V : vTrans (argument)τ : tracer (argument)S/R GAD U3 ADV R (gad u3 adv r.F)Fr: wT (argument)W : rTrans (argument)τ : tracer (argument)

2.16.3 Centered fourth order advection

Centered fourth order advection is formally the most accurate scheme we haveimplemented and can be used to great effect in high resolution simulation wheredynamical scales are well resolved. However, the scheme is noisy like the cen-tered second order method and so must be used with some finite amount ofdiffusion. Bi-harmonic is recommended since it is more scale selective and lesslikely to diffuse away the well resolved gradient the fourth order scheme workedso hard to create.

The centered fourth order fluxes are discretized:

Fx = Uτ − 16δiiτ

i

(2.183)

Fy = V τ − 16δiiτ

j

(2.184)

Fr = Wτ − 16δiiτ

k

(2.185)

As for the third order scheme, the best discretization near boundaries isunder investigation but currently δiτ = 0 on a boundary.


S/R GAD C4 ADV X (gad c4 adv x.F)Fx: uT (argument)U : uTrans (argument)τ : tracer (argument)S/R GAD C4 ADV Y (gad c4 adv y.F)Fy : vT (argument)V : vTrans (argument)τ : tracer (argument)S/R GAD C4 ADV R (gad c4 adv r.F)Fr: wT (argument)W : rTrans (argument)τ : tracer (argument)

2.16.4 First order upwind advection

Although the upwind scheme is the underlying scheme for the robust or non-linear methods given later, we haven’t actually supplied this method for generaluse. It would be very diffusive and it is unlikely that it could ever produce moreuseful results than the positive higher order schemes.

Upwind bias is introduced into many schemes using the abs function and isallows the first order upwind flux to be written:

Fx = Uτ i − 12|U |δiτ (2.186)

Fy = V τ j − 12|V |δjτ (2.187)

Fr = Wτk − 12|W |δkτ (2.188)

If for some reason, the above method is required, then the second order fluxlimiter scheme described later reduces to the above scheme if the limiter is setto zero.

2.17 Non-linear advection schemes

Non-linear advection schemes invoke non-linear interpolation and are widelyused in computational fluid dynamics (non-linear does not refer to the non-linearity of the advection operator). The flux limited advection schemes belongto the class of finite volume methods which neatly ties into the spatial discretiza-tion of the model.

When employing the flux limited schemes, first order upwind or direct-space-time method the time-stepping is switched to forward in time.

2.17.1 Second order flux limiters

The second order flux limiter method can be cast in several ways but is generallyexpressed in terms of other flux approximations. For example, in terms of a first

2.17. NON-LINEAR ADVECTION SCHEMES 97

order upwind flux and second order Lax-Wendroff flux, the limited flux is givenas:

F = F1 + ψ(r)FLW (2.189)

where ψ(r) is the limiter function,

F1 = uτ i − 12|u|δiτ (2.190)

is the upwind flux,

FLW = F1 +|u|2

(1 − c)δiτ (2.191)

is the Lax-Wendroff flux and c = uΔtΔx is the Courant (CFL) number.

The limiter function, ψ(r), takes the slope ratio

r =τi−1 − τi−2

τi − τi−1∀ u > 0 (2.192)

r =τi+1 − τiτi − τi−1

∀ u < 0 (2.193)

as it’s argument. There are many choices of limiter function but we only providethe Superbee limiter [45]:

ψ(r) = max[0,min[1, 2r],min[2, r]] (2.194)

S/R GAD FLUXLIMIT ADV X (gad fluxlimit adv x.F)Fx: uT (argument)U : uTrans (argument)τ : tracer (argument)S/R GAD FLUXLIMIT ADV Y (gad fluxlimit adv y.F)Fy: vT (argument)V : vTrans (argument)τ : tracer (argument)S/R GAD FLUXLIMIT ADV R (gad fluxlimit adv r.F)Fr: wT (argument)W : rTrans (argument)τ : tracer (argument)

2.17.2 Third order direct space time

The direct-space-time method deals with space and time discretization together(other methods that treat space and time separately are known collectively asthe “Method of Lines”). The Lax-Wendroff scheme falls into this category; itadds sufficient diffusion to a second order flux that the forward-in-time methodis stable. The upwind biased third order DST scheme is:

F = u (τi−1 + d0(τi − τi−1) + d1(τi−1 − τi−2)) ∀ u > 0 (2.195)F = u (τi − d0(τi − τi−1) − d1(τi+1 − τi)) ∀ u < 0 (2.196)


where

d1 =16(2 − |c|)(1 − |c|) (2.197)

d2 =16(1 − |c|)(1 + |c|) (2.198)

The coefficients d0 and d1 approach 1/3 and 1/6 respectively as the Courantnumber, c, vanishes. In this limit, the conventional third order upwind methodis recovered. For finite Courant number, the deviations from the linear methodare analogous to the diffusion added to centered second order advection in theLax-Wendroff scheme.

The DST3 method described above must be used in a forward-in-time man-ner and is stable for 0 ≤ |c| ≤ 1. Although the scheme appears to be forward-in-time, it is in fact third order in time and the accuracy increases with theCourant number! For low Courant number, DST3 produces very similar results(indistinguishable in Fig. 2.12) to the linear third order method but for largeCourant number, where the linear upwind third order method is unstable, thescheme is extremely accurate (Fig. 2.13) with only minor overshoots.

S/R GAD DST3 ADV X (gad dst3 adv x.F)Fx: uT (argument)U : uTrans (argument)τ : tracer (argument)S/R GAD DST3 ADV Y (gad dst3 adv y.F)Fy : vT (argument)V : vTrans (argument)τ : tracer (argument)S/R GAD DST3 ADV R (gad dst3 adv r.F)Fr: wT (argument)W : rTrans (argument)τ : tracer (argument)

2.17.3 Third order direct space time with flux limiting

The overshoots in the DST3 method can be controlled with a flux limiter. Thelimited flux is written:

F =12(u+ |u|) (τi−1 + ψ(r+)(τi − τi−1)

)+

12(u− |u|) (τi−1 + ψ(r−)(τi − τi−1)

)(2.199)

where

r+ =τi−1 − τi−2

τi − τi−1(2.200)

r− =τi+1 − τiτi − τi−1

(2.201)

and the limiter is the Sweby limiter:

ψ(r) = max[0,min[min(1, d0 + d1r],1 − c

cr]] (2.202)


S/R GAD DST3FL ADV X (gad dst3 adv x.F)Fx: uT (argument)U : uTrans (argument)τ : tracer (argument)S/R GAD DST3FL ADV Y (gad dst3 adv y.F)Fy: vT (argument)V : vTrans (argument)τ : tracer (argument)S/R GAD DST3FL ADV R (gad dst3 adv r.F)Fr: wT (argument)W : rTrans (argument)τ : tracer (argument)

2.17.4 Multi-dimensional advection

In many of the aforementioned advection schemes the behavior in multiple di-mensions is not necessarily as good as the one dimensional behavior. For in-stance, a shape preserving monotonic scheme in one dimension can have severeshape distortion in two dimensions if the two components of horizontal fluxesare treated independently. There is a large body of literature on the subjectdealing with this problem and among the fixes are operator and flux splittingmethods, corner flux methods and more. We have adopted a variant on thestandard splitting methods that allows the flux calculations to be implementedas if in one dimension:

τn+1/3 = τn − Δt(

1Δx

δiFx(τn) + τn 1

Δxδiu

)(2.203)

τn+2/3 = τn − Δt(

1Δy

δjFy(τn+1/3) + τn 1

Δyδiv

)(2.204)

τn+3/3 = τn − Δt(

1Δr

δkFx(τn+2/3) + τn 1

Δrδiw

)(2.205)

In order to incorporate this method into the general model algorithm, wecompute the effective tendency rather than update the tracer so that otherterms such as diffusion are using the n time-level and not the updated n+ 3/3quantities:

Gn+1/2adv =

1Δt

(τn+3/3 − τn) (2.206)

So that the over all time-stepping looks likes:

τn+1 = τn + Δt(G

n+1/2adv +Gdiff (τn) +Gn

forcing

)(2.207)


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

2nd order centered

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3rd order upwind

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

4th order centered

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Lax−Wendroff

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3−DST

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

4−DST

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Superbee flux limiter

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3−DST Sweby μ=μ(c)

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Superbee (no multi−dim)

Figure 2.14: Comparison of advection schemes in two dimensions; diagonaladvection of a resolved Gaussian feature. Courant number is 0.01 with 30×30points and solutions are shown for T=1/2. White lines indicate zero cross-ing (ie. the presence of false minima). The left column shows the secondorder schemes; top) centered second order with Adams-Bashforth, middle) Lax-Wendroff and bottom) Superbee flux limited. The middle column shows thethird order schemes; top) upwind biased third order with Adams-Bashforth,middle) third order direct space-time method and bottom) the same with fluxlimiting. The top right panel shows the centered fourth order scheme withAdams-Bashforth and right middle panel shows a fourth order variant on theDST method. Bottom right panel shows the Superbee flux limiter (second or-der) applied independently in each direction (method of lines).


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

2nd order centered

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3rd order upwind

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

4th order centered

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Lax−Wendroff

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3−DST

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

4−DST

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8


Figure 2.15: Comparison of advection schemes in two dimensions; diagonaladvection of a resolved Gaussian feature. Courant number is 0.27 with 30×30points and solutions are shown for T=1/2. White lines indicate zero cross-ing (ie. the presence of false minima). The left column shows the secondorder schemes; top) centered second order with Adams-Bashforth, middle) Lax-Wendroff and bottom) Superbee flux limited. The middle column shows thethird order schemes; top) upwind biased third order with Adams-Bashforth,middle) third order direct space-time method and bottom) the same with fluxlimiting. The top right panel shows the centered fourth order scheme withAdams-Bashforth and right middle panel shows a fourth order variant on theDST method. Bottom right panel shows the Superbee flux limiter (second or-der) applied independently in each direction (method of lines).


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

2nd order centered

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3rd order upwind

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

4th order centered

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Lax−Wendroff

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

3−DST

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

4−DST

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8


0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8


Figure 2.16: Comparison of advection schemes in two dimensions; diagonaladvection of a resolved Gaussian feature. Courant number is 0.47 with 30×30points and solutions are shown for T=1/2. White lines indicate zero crossingsand initial maximum values (ie. the presence of false extrema). The left col-umn shows the second order schemes; top) centered second order with Adams-Bashforth, middle) Lax-Wendroff and bottom) Superbee flux limited. The mid-dle column shows the third order schemes; top) upwind biased third order withAdams-Bashforth, middle) third order direct space-time method and bottom)the same with flux limiting. The top right panel shows the centered fourth or-der scheme with Adams-Bashforth and right middle panel shows a fourth ordervariant on the DST method. Bottom right panel shows the Superbee flux limiter(second order) applied independently in each direction (method of lines).

2.18. COMPARISON OF ADVECTION SCHEMES 103

S/R GAD ADVECTION (gad advection.F)τ : Tracer (argument)G

n+1/2adv : Gtracer (argument)

Fx, Fy, Fr: af (local)U : uTrans (local)V : vTrans (local)W : rTrans (local)

2.18 Comparison of advection schemes

Advection Scheme code use use Multi- Stencil commentsA.B. dimension (1 dim)

centered 2ndorder 2 Yes No 3 pts linear3rdorder upwind 3 Yes No 5 pts linear/τcentered 4thorder 4 Yes No 5 pts linear3rdorder DST 30 No Yes 5 pts linear/τ , non-linear/v2ndorder Flux Limiters 77 No Yes 5 pts non-linear3ndorder DST Flux limiter 33 No Yes 5 pts non-linear

Table 2.2: Summary of the different advection schemes available in MITgcm.“A.B.” stands for Adams-Bashforth and “DST” for direct space time. The codecorresponds to the number used to select the corresponding advection scheme inthe parameter file (e.g., tempAdvScheme=3 in file data selects the 3rd orderupwind advection scheme for temperature).

Figs. 2.14, 2.15 and 2.16 show solutions to a simple diagonal advection prob-lem using a selection of schemes for low, moderate and high Courant numbers,respectively. The top row shows the linear schemes, integrated with the Adams-Bashforth method. Theses schemes are clearly unstable for the high Courantnumber and weakly unstable for the moderate Courant number. The presence offalse extrema is very apparent for all Courant numbers. The middle row showssolutions obtained with the unlimited but multi-dimensional schemes. These so-lutions also exhibit false extrema though the pattern now shows symmetry dueto the multi-dimensional scheme. Also, the schemes are stable at high Courantnumber where the linear schemes weren’t. The bottom row (left and middle)shows the limited schemes and most obvious is the absence of false extrema. Theaccuracy and stability of the unlimited non-linear schemes is retained at highCourant number but at low Courant number the tendency is to loose amplitudein sharp peaks due to diffusion. The one dimensional tests shown in Figs. 2.12and 2.13 showed this phenomenon.

Finally, the bottom left and right panels use the same advection scheme butthe right does not use the multi-dimensional method. At low Courant numberthis appears to not matter but for moderate Courant number severe distortion ofthe feature is apparent. Moreover, the stability of the multi-dimensional scheme


is determined by the maximum Courant number applied of each dimension whilethe stability of the method of lines is determined by the sum. Hence, in thehigh Courant number plot, the scheme is unstable.

With many advection schemes implemented in the code two questions arise:“Which scheme is best?” and “Why don’t you just offer the best advectionscheme?”. Unfortunately, no one advection scheme is “the best” for all particu-lar applications and for new applications it is often a matter of trial to determinewhich is most suitable. Here are some guidelines but these are not the rule;

• If you have a coarsely resolved model, using a positive or upwind biasedscheme will introduce significant diffusion to the solution and using acentered higher order scheme will introduce more noise. In this case,simplest may be best.

• If you have a high resolution model, using a higher order scheme will givea more accurate solution but scale-selective diffusion might need to beemployed. The flux limited methods offer similar accuracy in this regime.

• If your solution has shocks or propagating fronts then a flux limited schemeis almost essential.

• If your time-step is limited by advection, the multi-dimensional non-linearschemes have the most stability (up to Courant number 1).

• If you need to know how much diffusion/dissipation has occurred you willhave a lot of trouble figuring it out with a non-linear method.

• The presence of false extrema is non-physical and this alone is the strongestargument for using a positive scheme.

2.19 Shapiro Filter

The Shapiro filter (Shapiro 1970, 1975) is a high order horizontal filter thatefficiently remove small scale grid noise without affecting the physical structuresof a field. It is applied at the end of the time step on both velocity and tracerfields.

Three different space operators are considered here (S1,S2 and S4). Theydiffers essentially by the sequence of derivative in both X and Y directions.Consequently they show different damping response function specially in thediagonal directions X+Y and X-Y.

Space derivatives can be computed in the real space, taken into account thegrid spacing. Alternatively, a pure computational filter can be defined, usingpure numerical differences and ignoring grid spacing. This later form is stablewhatever the grid is, and therefore specially useful for highly anisotropic gridsuch as spherical coordinate grid. A damping time-scale parameter τshap definesthe strength of the filter damping.

2.19. SHAPIRO FILTER 105

The 3 computational filter operators are :

S1c : [1 − 1/2Δtτshap

{(14δii)n + (

14δjj)n}]

S2c : [1 − Δtτshap

{18(δii + δjj)}n]

S4c : [1 − Δtτshap

(14δii)n][1 − Δt

τshap(14δjj)n]

In addition, the S2 operator can easily be extended to a physical space filter:

S2g : [1 − Δtτshap

{L2shap

8∇2}n]

with the Laplacian operator ∇2and a length scale parameter Lshap. The

stability of this S2g filter requires Lshap < Min(Global)(Δx,Δy). Add Response functions andfigures

Chapter 3

Getting started withMITgcm

In this section, we describe how to use the model. In the first section, we provideenough information to help you get started with the model. We believe the bestway to familiarize yourself with the model is to run the case study examplesprovided with the base version. Information on how to obtain, compile, andrun the code is found there as well as a brief description of the model structuredirectory and the case study examples. The latter and the code structure aredescribed more fully in chapters 2 and 4, respectively. Here, in this section, weprovide information on how to customize the code when you are ready to tryimplementing the configuration you have in mind.

3.1 Where to find information

A web site is maintained for release 2 (“Pelican”) of MITgcm:

http://mitgcm.org/pelican

Here you will find an on-line version of this document, a “browsable” copy ofthe code and a searchable database of the model and site, as well as links fordownloading the model and documentation, to data-sources, and other relatedsites.

There is also a web-archived support mailing list for the model that you canemail at [email protected] or browse at:

http://mitgcm.org/mailman/listinfo/mitgcm-support/http://mitgcm.org/pipermail/mitgcm-support/

Essentially all of the MITgcm web pages can be searched using a popular webcrawler such as Google or through our own search facility:

http://mitgcm.org/htdig/

107

108 CHAPTER 3. GETTING STARTED WITH MITGCM

3.2 Obtaining the code

MITgcm can be downloaded from our system by following the instructions be-low. As a courtesy we ask that you send e-mail to us at [email protected] enable us to keep track of who’s using the model and in what application.You can download the model two ways:

1. Using CVS software. CVS is a freely available source code managementtool. To use CVS you need to have the software installed. Many sys-tems come with CVS pre-installed, otherwise good places to look for thesoftware for a particular platform are cvshome.org and wincvs.org .

2. Using a tar file. This method is simple and does not require any spe-cial software. However, this method does not provide easy support formaintenance updates.

3.2.1 Method 1 - Checkout from CVS

If CVS is available on your system, we strongly encourage you to use it. CVSprovides an efficient and elegant way of organizing your code and keeping trackof your changes. If CVS is not available on your machine, you can also downloada tar file.

Before you can use CVS, the following environment variable(s) should be setwithin your shell. For a csh or tcsh shell, put the following

% setenv CVSROOT :pserver:[email protected]:/u/gcmpack

in your .cshrc or .tcshrc file. For bash or sh shells, put:

% export CVSROOT=’:pserver:[email protected]:/u/gcmpack’

in your .profile or .bashrc file.To get MITgcm through CVS, first register with the MITgcm CVS server

using command:

% cvs login ( CVS password: cvsanon )

You only need to do a “cvs login” once.To obtain the latest sources type:

% cvs co MITgcm

or to get a specific release type:

% cvs co -P -r checkpoint52i_post MITgcm

The MITgcm web site contains further directions concerning the source codeand CVS. It also contains a web interface to our CVS archive so that one mayeasily view the state of files, revisions, and other development milestones:

http://mitgcm.org/source_code.html

3.2. OBTAINING THE CODE 109

As a convenience, the MITgcm CVS server contains aliases which are namedsubsets of the codebase. These aliases can be especially helpful when used overslow internet connections or on machines with restricted storage space. Table3.1 contains a list of CVS aliases

Alias Name Information (directories) ContainedMITgcm code Only the source code – none of the verification exam-

ples.MITgcm verif basic Source code plus a small set of the verifica-

tion examples (global ocean.90x40x15, aim.5l cs,hs94.128x64x5, front relax, and plume on slope).

MITgcm verif atmos Source code plus all of the atmospheric examples.MITgcm verif ocean Source code plus all of the oceanic examples.MITgcm verif all Source code plus all of the verification examples.

Table 3.1: MITgcm CVS Modules

The checkout process creates a directory called MITgcm. If the directoryMITgcm exists this command updates your code based on the repository. Eachdirectory in the source tree contains a directory CVS. This information is re-quired by CVS to keep track of your file versions with respect to the repository.Don’t edit the files in CVS ! You can also use CVS to download code updates.More extensive information on using CVS for maintaining MITgcm code canbe found here . It is important to note that the CVS aliases in Table 3.1cannot be used in conjunction with the CVS -d DIRNAME option. However, theMITgcm directories they create can be changed to a different name following thecheck-out:

% cvs co MITgcm_verif_basic% mv MITgcm MITgcm_verif_basic

3.2.2 Method 2 - Tar file download

If you do not have CVS on your system, you can download the model as a tarfile from the web site at:

http://mitgcm.org/download/

The tar file still contains CVS information which we urge you not to delete; evenif you do not use CVS yourself the information can help us if you should needto send us your copy of the code. If a recent tar file does not exist, then pleasecontact the developers through the [email protected] mailing list.

3.2.2.1 Upgrading from an earlier version

If you already have an earlier version of the code you can “upgrade” your copyinstead of downloading the entire repository again. First, “cd” (change direc-tory) to the top of your working copy:


% cd MITgcm

and then issue the cvs update command such as:

% cvs -q update -r checkpoint52i_post -d -P

This will update the “tag” to “checkpoint52i post”, add any new directories (-d) and remove any empty directories (-P). The -q option means be quiet whichwill reduce the number of messages you’ll see in the terminal. If you havemodified the code prior to upgrading, CVS will try to merge your changes withthe upgrades. If there is a conflict between your modifications and the upgrade,it will report that file with a “C” in front, e.g.:

C model/src/ini_parms.F

If the list of conflicts scrolled off the screen, you can re-issue the cvs updatecommand and it will report the conflicts. Conflicts are indicated in the code bythe delimites “<<<<<<<”, “=======” and “>>>>>>>”. For example,

<<<<<<< ini_parms.F

& bottomDragLinear,myOwnBottomDragCoefficient,

=======

& bottomDragLinear,bottomDragQuadratic,

>>>>>>> 1.18

means that you added “myOwnBottomDragCoefficient” to a namelist atthe same time and place that we added “bottomDragQuadratic”. You need toresolve this conflict and in this case the line should be changed to:

& bottomDragLinear,bottomDragQuadratic,myOwnBottomDragCoefficient,

and the lines with the delimiters (<<<<<<,======,>>>>>>) be deleted.Unless you are making modifications which exactly parallel developments wemake, these types of conflicts should be rare.

Upgrading to the current pre-release version We don’t make a “release”for every little patch and bug fix in order to keep the frequency of upgrades to aminimum. However, if you have run into a problem for which “we have alreadyfixed in the latest code” and we haven’t made a “tag” or “release” since thatpatch then you’ll need to get the latest code:

% cvs -q update -A -d -P

Unlike, the “check-out” and “update” procedures above, there is no “tag” orrelease name. The -A tells CVS to upgrade to the very latest version. As a rule,we don’t recommend this since you might upgrade while we are in the processesof checking in the code so that you may only have part of a patch. Using thismethod of updating also means we can’t tell what version of the code you areworking with. So please be sure you understand what you’re doing.

3.3. MODEL AND DIRECTORY STRUCTURE 111

3.3 Model and directory structure

The “numerical” model is contained within a execution environment supportwrapper. This wrapper is designed to provide a general framework for grid-pointmodels. MITgcmUV is a specific numerical model that uses the framework. Un-der this structure the model is split into execution environment support codeand conventional numerical model code. The execution environment supportcode is held under the eesupp directory. The grid point model code is heldunder the model directory. Code execution actually starts in the eesupp rou-tines and not in the model routines. For this reason the top-level MAIN.F isin the eesupp/src directory. In general, end-users should not need to worryabout this level. The top-level routine for the numerical part of the code is inmodel/src/THE MODEL MAIN.F. Here is a brief description of the directorystructure of the model under the root tree (a detailed description is given insection 3: Code structure).

• bin: this directory is initially empty. It is the default directory in whichto compile the code.

• diags : contains the code relative to time-averaged diagnostics. It is sub-divided into two subdirectories inc and src that contain include files (*.hfiles) and Fortran subroutines (*.F files), respectively.

• doc: contains brief documentation notes.

• eesupp: contains the execution environment source code. Also subdividedinto two subdirectories inc and src.

• exe: this directory is initially empty. It is the default directory in whichto execute the code.

• model : this directory contains the main source code. Also subdivided intotwo subdirectories inc and src.

• pkg: contains the source code for the packages. Each package correspondsto a subdirectory. For example, gmredi contains the code related to theGent-McWilliams/Redi scheme, aim the code relative to the atmosphericintermediate physics. The packages are described in detail in section 3.

• tools : this directory contains various useful tools. For example, genmake2is a script written in csh (C-shell) that should be used to generate yourmakefile. The directory adjoint contains the makefile specific to the Tan-gent linear and Adjoint Compiler (TAMC) that generates the adjoint code.The latter is described in details in part V.

• utils : this directory contains various utilities. The subdirectory knud-sen2 contains code and a makefile that compute coefficients of the poly-nomial approximation to the knudsen formula for an ocean nonlinearequation of state. The matlab subdirectory contains matlab scripts for


reading model output directly into matlab. scripts contains C-shell post-processing scripts for joining processor-based and tiled-based model out-put.

• verification: this directory contains the model examples. See section 3.4.

3.4 Example experiments

The MITgcm distribution comes with more than a dozen pre-configured numer-ical experiments. Some of these example experiments are tests of individualparts of the model code, but many are fully fledged numerical simulations. Afew of the examples are used for tutorial documentation in sections 3.8 - 3.10.The other examples follow the same general structure as the tutorial examples.However, they only include brief instructions in a text file called README. Theexamples are located in subdirectories under the directory verification. Each ex-ample is briefly described below.

3.4.1 Full list of model examples

1. exp0 - single layer, ocean double gyre (barotropic with free-surface). Thisexperiment is described in detail in section 3.8.

2. exp1 - Four layer, ocean double gyre. This experiment is described indetail in section ??.

3. exp2 - 4x4 degree global ocean simulation with steady climatological forc-ing. This experiment is described in detail in section 3.10.

4. exp4 - Flow over a Gaussian bump in open-water or channel with openboundaries.

5. exp5 - Inhomogenously forced ocean convection in a doubly periodic box.

6. front relax - Relaxation of an ocean thermal front (test for Gent/McWilliamsscheme). 2D (Y-Z).

7. internal wave - Ocean internal wave forced by open boundary conditions.

8. natl box - Eastern subtropical North Atlantic with KPP scheme; 1 monthintegration

9. hs94.1x64x5 - Zonal averaged atmosphere using Held and Suarez ’94 forc-ing.

10. hs94.128x64x5 - 3D atmosphere dynamics using Held and Suarez ’94 forc-ing.

11. hs94.cs-32x32x5 - 3D atmosphere dynamics using Held and Suarez ’94forcing on the cubed sphere.

3.4. MITGCM EXAMPLE EXPERIMENTS 113

12. aim.5l zon-ave - Intermediate Atmospheric physics. Global Zonal Meanconfiguration, 1x64x5 resolution.

13. aim.5l XZ Equatorial Slice - Intermediate Atmospheric physics, equato-rial Slice configuration. 2D (X-Z).

14. aim.5l Equatorial Channel - Intermediate Atmospheric physics. 3D Equa-torial Channel configuration.

15. aim.5l LatLon - Intermediate Atmospheric physics. Global configuration,on latitude longitude grid with 128x64x5 grid points (2.8◦degree resolu-tion).

16. adjustment.128x64x1 Barotropic adjustment problem on latitude longi-tude grid with 128x64 grid points (2.8◦degree resolution).

17. adjustment.cs-32x32x1 Barotropic adjustment problem on cube spheregrid with 32x32 points per face ( roughly 2.8◦degree resolution).

18. advect cs Two-dimensional passive advection test on cube sphere grid.

19. advect xy Two-dimensional (horizontal plane) passive advection test onCartesian grid.

20. advect yz Two-dimensional (vertical plane) passive advection test on Carte-sian grid.

21. carbon Simple passive tracer experiment. Includes derivative calculation.Described in detail in section ??.

22. flt example Example of using float package.

23. global ocean.90x40x15 Global circulation with GM, flux boundary condi-tions and poles.

24. global ocean pressure Global circulation in pressure coordinate (non-Boussinesqocean model). Described in detail in section 3.11.

25. solid-body.cs-32x32x1 Solid body rotation test for cube sphere grid.

3.4.2 Directory structure of model examples

Each example directory has the following subdirectories:

• code: contains the code particular to the example. At a minimum, thisdirectory includes the following files:

– code/CPP EEOPTIONS.h: declares CPP keys relative to the “exe-cution environment” part of the code. The default version is locatedin eesupp/inc.


– code/CPP OPTIONS.h: declares CPP keys relative to the “numer-ical model” part of the code. The default version is located inmodel/inc.

– code/SIZE.h: declares size of underlying computational grid. Thedefault version is located in model/inc.

In addition, other include files and subroutines might be present in codedepending on the particular experiment. See Section 2 for more details.

• input : contains the input data files required to run the example. At aminimum, the input directory contains the following files:

– input/data: this file, written as a namelist, specifies the main param-eters for the experiment.

– input/data.pkg: contains parameters relative to the packages used inthe experiment.

– input/eedata: this file contains “execution environment” data. Atpresent, this consists of a specification of the number of threads touse in X and Y under multithreaded execution.

In addition, you will also find in this directory the forcing and topographyfiles as well as the files describing the initial state of the experiment. Thisvaries from experiment to experiment. See section 2 for more details.

• results : this directory contains the output file output.txt produced by thesimulation example. This file is useful for comparison with your ownoutput when you run the experiment.

Once you have chosen the example you want to run, you are ready to compilethe code.

3.5 Building the code

To compile the code, we use the make program. This uses a file (Makefile) thatallows us to pre-process source files, specify compiler and optimization optionsand also figures out any file dependencies. We supply a script (genmake2),described in section 3.16.2, that automatically creates the Makefile for you.You then need to build the dependencies and compile the code.

As an example, let’s assume that you want to build and run experimentverification/exp2. The are multiple ways and places to actually do this but herelet’s build the code in verification/exp2/input :

% cd verification/exp2/input

First, build the Makefile:

% ../../../tools/genmake2 -mods=../code

3.6. RUNNING MITGCM 115

The command line option tells genmake to override model source code with anyfiles in the directory ../code/.

On many systems, the genmake2 program will be able to automatically rec-ognize the hardware, find compilers and other tools within the user’s path (“echo$PATH”), and then choose an appropriate set of options from the files (“opt-files”) contained in the tools/build options directory. Under some circumstances,a user may have to create a new “optfile” in order to specify the exact combina-tion of compiler, compiler flags, libraries, and other options necessary to builda particular configuration of MITgcm. In such cases, it is generally helpful toread the existing “optfiles” and mimic their syntax.

Through the MITgcm-support list, the MITgcm developers are willing toprovide help writing or modifing “optfiles”. And we encourage users to post new“optfiles” (particularly ones for new machines or architectures) to the [email protected] list.

To specify an optfile to genmake2, the syntax is:

% ../../../tools/genmake2 -mods=../code -of /path/to/optfile

Once a Makefile has been generated, we create the dependencies:

% make depend

This modifies the Makefile by attaching a [long] list of files upon which otherfiles depend. The purpose of this is to reduce re-compilation if and when youstart to modify the code. The make depend command also creates links fromthe model source to this directory. It is important to note that the make dependstage will occasionally produce warnings or errors since the dependency parsingtool is unable to find all of the necessary header files (eg. netcdf.inc). In thesecircumstances, it is usually OK to ignore the warnings/errors and proceed tothe next step.

Next compile the code:

% make

The make command creates an executable called mitgcmuv. Additional make“targets” are defined within the makefile to aid in the production of adjoint andother versions of MITgcm.

Now you are ready to run the model. General instructions for doing so aregiven in section 3.6. Here, we can run the model with:

./mitgcmuv > output.txt

where we are re-directing the stream of text output to the file output.txt.

3.6 Running the model in prognostic mode

If compilation finished succesfuully (section 3.5) then an executable called mitgcmuvwill now exist in the local directory.

To run the model as a single process (ie. not in parallel) simply type:


% ./mitgcmuv

The “./” is a safe-guard to make sure you use the local executable in case youhave others that exist in your path (surely odd if you do!). The above commandwill spew out many lines of text output to your screen. This output containsdetails such as parameter values as well as diagnostics such as mean Kineticenergy, largest CFL number, etc. It is worth keeping this text output with thebinary output so we normally re-direct the stdout stream as follows:

% ./mitgcmuv > output.txt

In the event that the model encounters an error and stops, it is very helpful toinclude the last few line of this output.txt file along with the (stderr) errormessage within any bug reports.

For the example experiments in verification, an example of the output is keptin results/output.txt for comparison. You can compare your output.txt with thecorresponding one for that experiment to check that the set-up works.

3.6.1 Output files

The model produces various output files. Depending upon the I/O packageselected (either mdsio or mnc or both as determined by both the compile-timesettings and the run-time flags in data.pkg), the following output may appear.

3.6.1.1 MDSIO output files

The “traditional” output files are generated by the mdsio package. At a mini-mum, the instantaneous “state” of the model is written out, which is made ofthe following files:

• U.00000nIter - zonal component of velocity field (m/s and > 0 eastward).

• V.00000nIter - meridional component of velocity field (m/s and > 0 north-ward).

• W.00000nIter - vertical component of velocity field (ocean: m/s and >0 upward, atmosphere: Pa/s and > 0 towards increasing pressure i.e.downward).

• T.00000nIter - potential temperature (ocean: 0C, atmosphere: 0K).

• S.00000nIter - ocean: salinity (psu), atmosphere: water vapor (g/kg).

• Eta.00000nIter - ocean: surface elevation (m), atmosphere: surface pres-sure anomaly (Pa).

The chain 00000nIter consists of ten figures that specify the iteration numberat which the output is written out. For example, U.0000000300 is the zonalvelocity at iteration 300.

In addition, a “pickup” or “checkpoint” file called:

3.6. RUNNING MITGCM 117

• pickup.00000nIter

is written out. This file represents the state of the model in a condensedform and is used for restarting the integration. If the C-D scheme is used, thereis an additional “pickup” file:

• pickup cd.00000nIter

containing the D-grid velocity data and that has to be written out as wellin order to restart the integration. Rolling checkpoint files are the same as thepickup files but are named differently. Their name contain the chain ckptA orckptB instead of 00000nIter. They can be used to restart the model but areoverwritten every other time they are output to save disk space during longintegrations.

3.6.1.2 MNC output files

Unlike the mdsio output, the mnc–generated output is usually (though not neces-sarily) placed within a subdirectory with a name such as mnc test $DATE $SEQ.The files within this subdirectory are all in the “self-describing” netCDF formatand can thus be browsed and/or plotted using tools such as:

• At a minimum, the ncdump utility is typically included with every netCDFinstall:

http://www.unidata.ucar.edu/packages/netcdf/

• The ncview utility is a very convenient and quick way to plot netCDFdata and it runs on most OSes:

http://meteora.ucsd.edu/~pierce/ncview_home_page.html

• MatLAB(c) and other common post-processing environments provide var-ious netCDF interfaces including:

http://woodshole.er.usgs.gov/staffpages/cdenham/public_html/MexCDF/nc4ml5.html

3.6.2 Looking at the output

The “traditional” or mdsio model data are written according to a “meta/data”file format. Each variable is associated with two files with suffix names .dataand .meta. The .data file contains the data written in binary form (big endianby default). The .meta file is a “header” file that contains information aboutthe size and the structure of the .data file. This way of organizing the output isparticularly useful when running multi-processors calculations. The base versionof the model includes a few matlab utilities to read output files written in thisformat. The matlab scripts are located in the directory utils/matlab under theroot tree. The script rdmds.m reads the data. Look at the comments inside thescript to see how to use it.

Some examples of reading and visualizing some output in Matlab:


% matlab>> H=rdmds(’Depth’);>> contourf(H’);colorbar;>> title(’Depth of fluid as used by model’);

>> eta=rdmds(’Eta’,10);>> imagesc(eta’);axis ij;colorbar;>> title(’Surface height at iter=10’);

>> eta=rdmds(’Eta’,[0:10:100]);>> for n=1:11; imagesc(eta(:,:,n)’);axis ij;colorbar;pause(.5);end

Similar scripts for netCDF output (rdmnc.m) are available.

3.7. TUTORIALS 119

3.7 Tutorials

This is the first in a series of tutorials describing example MITgcm numericalexperiments. The example experiments include both straightforward examplesof idealized geophysical fluid simulations and more involved cases encompass-ing large scale modeling and automatic differentiation. Both hydrostatic andnon-hydrostatic experiments are presented, as well as experiments employingCartesian, spherical-polar and cube-sphere coordinate systems. These “casestudy” documents include information describing the experimental configura-tion and detailed information on how to configure the MITgcm code and inputfiles for each experiment.

3.8 Barotropic Ocean Gyre In Cartesian Coor-dinates

This example experiment demonstrates using the MITgcm to simulate a Barotropic,wind-forced, ocean gyre circulation. The experiment is a numerical rendition ofthe gyre circulation problem similar to the problems described analytically byStommel in 1966 [49] and numerically in Holland et. al [31].

In this experiment the model is configured to represent a rectangular enclosedbox of fluid, 1200 × 1200 km in lateral extent. The fluid is 5 km deep and isforced by a constant in time zonal wind stress, τx, that varies sinusoidally inthe “north-south” direction. Topologically the grid is Cartesian and the coriolisparameter f is defined according to a mid-latitude beta-plane equation

f(y) = f0 + βy (3.1)

where y is the distance along the “north-south” axis of the simulated domain.For this experiment f0 is set to 10−4s−1 in (3.1) and β = 10−11s−1m−1.

The sinusoidal wind-stress variations are defined according to

τx(y) = τ0 sin(πy

Ly) (3.2)

where Ly is the lateral domain extent (1200 km) and τ0 is set to 0.1Nm−2.

Figure 3.1 summarizes the configuration simulated.

3.8.1 Equations Solved

The model is configured in hydrostatic form. The implicit free surface form ofthe pressure equation described in Marshall et. al [39] is employed. A horizontalLaplacian operator ∇2

h provides viscous dissipation. The wind-stress momentuminput is added to the momentum equation for the “zonal flow”, u. Other termsin the model are explicitly switched off for this experiment configuration (see


5km

Ly=1200km

Lx=1200km

xτ

=0.1=0 xτxτ

1111

140

10

10−−−

−−

=

=

ms

sf

β

0ff =

yLff β+= 0

xy

z

Figure 3.1: Schematic of simulation domain and wind-stress forcing function forbarotropic gyre numerical experiment. The domain is enclosed bu solid walls atx = 0,1200km and at y = 0,1200km.

3.8. BAROTROPIC GYRE MITGCM EXAMPLE 121

section 3.15.4 ), yielding an active set of equations solved in this configurationas follows

Du

Dt− fv + g

∂η

∂x−Ah∇2

hu =τx

ρ0Δz(3.3)

Dv

Dt+ fu+ g

∂η

∂y−Ah∇2

hv = 0 (3.4)

∂η

∂t+ ∇h · �u = 0 (3.5)

where u and v and the x and y components of the flow vector �u.

3.8.2 Discrete Numerical Configuration

The domain is discretised with a uniform grid spacing in the horizontal set toΔx = Δy = 20 km, so that there are sixty grid cells in the x and y directions.Vertically the model is configured with a single layer with depth, Δz, of 5000 m.

3.8.2.1 Numerical Stability Criteria

The Laplacian dissipation coefficient, Ah, is set to 400ms−1. This value ischosen to yield a Munk layer width [1],

Mw = π(Ah

β)

13 (3.6)

of ≈ 100km. This is greater than the model resolution Δx, ensuring that thefrictional boundary layer is well resolved.

The model is stepped forward with a time step δt = 1200secs. With this timestep the stability parameter to the horizontal Laplacian friction [1]

Sl = 4Ahδt

Δx2 (3.7)

evaluates to 0.012, which is well below the 0.3 upper limit for stability.

The numerical stability for inertial oscillations [1]

Si = f2δt2 (3.8)



The advective CFL [1] for an extreme maximum horizontal flow speed of |�u| =2ms−1

Sa =|�u|δtΔx

(3.9)

evaluates to 0.12. This is approaching the stability limit of 0.5 and limits δt to1200s.

3.8.3 Code Configuration

The model configuration for this experiment resides under the directory verifi-cation/exp0/. The experiment files

• input/data

• input/data.pkg

• input/eedata,

• input/windx.sin y,

• input/topog.box,

• code/CPP EEOPTIONS.h

• code/CPP OPTIONS.h,

• code/SIZE.h.

contain the code customizations and parameter settings for this experiments.Below we describe the customizations to these files associated with this experi-ment.

3.8.3.1 File input/data

This file, reproduced completely below, specifies the main parameters for theexperiment. The parameters that are significant for this configuration are

• Line 7,

viscAh=4.E2,

this line sets the Laplacian friction coefficient to 400m2s−1

• Line 10,

beta=1.E-11,

this line sets β (the gradient of the coriolis parameter, f) to 10−11s−1m−1


• Lines 15 and 16

rigidLid=.FALSE.,implicitFreeSurface=.TRUE.,

these lines suppress the rigid lid formulation of the surface pressure in-verter and activate the implicit free surface form of the pressure inverter.

• Line 27,

startTime=0,

this line indicates that the experiment should start from t = 0 and implic-itly suppresses searching for checkpoint files associated with restarting annumerical integration from a previously saved state.

• Line 29,

endTime=12000,

this line indicates that the experiment should start finish at t = 12000s.A restart file will be written at this time that will enable the simulationto be continued from this point.

• Line 30,

deltaTmom=1200,

This line sets the momentum equation timestep to 1200s.

• Line 39,

usingCartesianGrid=.TRUE.,

This line requests that the simulation be performed in a Cartesian coor-dinate system.

• Line 41,

delX=60*20E3,

This line sets the horizontal grid spacing between each x-coordinate linein the discrete grid. The syntax indicates that the discrete grid should becomprise of 60 grid lines each separated by 20 × 103m (20 km).

• Line 42,

delY=60*20E3,


This line sets the horizontal grid spacing between each y-coordinate linein the discrete grid to 20 × 103m (20 km).

• Line 43,

delZ=5000,

This line sets the vertical grid spacing between each z-coordinate line inthe discrete grid to 5000m (5 km).

• Line 46,

bathyFile=’topog.box’

This line specifies the name of the file from which the domain bathymetryis read. This file is a two-dimensional (x, y) map of depths. This file isassumed to contain 64-bit binary numbers giving the depth of the modelat each grid cell, ordered with the x coordinate varying fastest. The pointsare ordered from low coordinate to high coordinate for both axes. Theunits and orientation of the depths in this file are the same as used inthe MITgcm code. In this experiment, a depth of 0m indicates a solidwall and a depth of −5000m indicates open ocean. The matlab programinput/gendata.m shows an example of how to generate a bathymetry file.

• Line 49,

zonalWindFile=’windx.sin_y’

This line specifies the name of the file from which the x-direction surfacewind stress is read. This file is also a two-dimensional (x, y) map and isenumerated and formatted in the same manner as the bathymetry file.The matlab program input/gendata.m includes example code to generatea valid zonalWindFile file.

other lines in the file input/data are standard values that are described in theMITgcm Getting Started and MITgcm Parameters notes.

1 # Model parameters

2 # Continuous equation parameters

3 &PARM01

4 tRef=20.,

5 sRef=10.,

6 viscAz=1.E-2,

7 viscAh=4.E2,

8 diffKhT=4.E2,

9 diffKzT=1.E-2,

10 beta=1.E-11,

11 tAlpha=2.E-4,


12 sBeta =0.,

13 gravity=9.81,

14 gBaro=9.81,

15 rigidLid=.FALSE.,

16 implicitFreeSurface=.TRUE.,

17 eosType=’LINEAR’,

18 readBinaryPrec=64,

19 &

20 # Elliptic solver parameters

21 &PARM02

22 cg2dMaxIters=1000,

23 cg2dTargetResidual=1.E-7,

24 &

25 # Time stepping parameters

26 &PARM03

27 startTime=0,

28 #endTime=311040000,

29 endTime=12000.0,

30 deltaTmom=1200.0,

31 deltaTtracer=1200.0,

32 abEps=0.1,

33 pChkptFreq=2592000.0,

34 chkptFreq=120000.0,

35 dumpFreq=2592000.0,

36 &

37 # Gridding parameters

38 &PARM04

39 usingCartesianGrid=.TRUE.,

40 usingSphericalPolarGrid=.FALSE.,

41 delX=60*20E3,

42 delY=60*20E3,

43 delZ=5000.,

44 &

45 &PARM05

46 bathyFile=’topog.box’,

47 hydrogThetaFile=,

48 hydrogSaltFile=,

49 zonalWindFile=’windx.sin_y’,

50 meridWindFile=,

51 &

3.8.3.2 File input/data.pkg

This file uses standard default values and does not contain customizations forthis experiment.


3.8.3.3 File input/eedata


3.8.3.4 File input/windx.sin y

The input/windx.sin y file specifies a two-dimensional (x, y) map of wind stress,τx, values. The units used are Nm−2. Although τx is only a function of ynin this experiment this file must still define a complete two-dimensional mapin order to be compatible with the standard code for loading forcing fields inMITgcm. The included matlab program input/gendata.m gives a complete codefor creating the input/windx.sin y file.

3.8.3.5 File input/topog.box

The input/topog.box file specifies a two-dimensional (x, y) map of depth values.For this experiment values are either 0m or -delZm, corresponding respectivelyto a wall or to deep ocean. The file contains a raw binary stream of data that isenumerated in the same way as standard MITgcm two-dimensional, horizontalarrays. The included matlab program input/gendata.m gives a complete codefor creating the input/topog.box file.

3.8.3.6 File code/SIZE.h

Two lines are customized in this file for the current experiment

• Line 39,

sNx=60,

this line sets the lateral domain extent in grid points for the axis alignedwith the x-coordinate.

• Line 40,

sNy=60,

this line sets the lateral domain extent in grid points for the axis alignedwith the y-coordinate.

1 C $Header: /u/gcmpack/manual/part3/case_studies/barotropic_gyre/code/SIZE.h.tex,v 1.1.1.1

2 C $Name: $

3 C

4 C /==========================================================\

5 C | SIZE.h Declare size of underlying computational grid. |

6 C |==========================================================|

7 C | The design here support a three-dimensional model grid |

8 C | with indices I,J and K. The three-dimensional domain |


9 C | is comprised of nPx*nSx blocks of size sNx along one axis|

10 C | nPy*nSy blocks of size sNy along another axis and one |

11 C | block of size Nz along the final axis. |

12 C | Blocks have overlap regions of size OLx and OLy along the|

13 C | dimensions that are subdivided. |

14 C \==========================================================/

15 C Voodoo numbers controlling data layout.

16 C sNx - No. X points in sub-grid.

17 C sNy - No. Y points in sub-grid.

18 C OLx - Overlap extent in X.

19 C OLy - Overlat extent in Y.

20 C nSx - No. sub-grids in X.

21 C nSy - No. sub-grids in Y.

22 C nPx - No. of processes to use in X.

23 C nPy - No. of processes to use in Y.

24 C Nx - No. points in X for the total domain.

25 C Ny - No. points in Y for the total domain.

26 C Nr - No. points in R for full process domain.

27 INTEGER sNx

28 INTEGER sNy

29 INTEGER OLx

30 INTEGER OLy

31 INTEGER nSx

32 INTEGER nSy

33 INTEGER nPx

34 INTEGER nPy

35 INTEGER Nx

36 INTEGER Ny

37 INTEGER Nr

38 PARAMETER (

39 & sNx = 60,

40 & sNy = 60,

41 & OLx = 3,

42 & OLy = 3,

43 & nSx = 1,

44 & nSy = 1,

45 & nPx = 1,

46 & nPy = 1,

47 & Nx = sNx*nSx*nPx,

48 & Ny = sNy*nSy*nPy,

49 & Nr = 1)

50 C MAX_OLX - Set to the maximum overlap region size of any array

51 C MAX_OLY that will be exchanged. Controls the sizing of exch

52 C routine buufers.

53 INTEGER MAX_OLX

54 INTEGER MAX_OLY

55 PARAMETER ( MAX_OLX = OLx,

56 & MAX_OLY = OLy )


3.8.3.7 File code/CPP OPTIONS.h


3.8.3.8 File code/CPP EEOPTIONS.h


3.9. BAROCLINIC GYRE MITGCM EXAMPLE 129

3.9 Four Layer Baroclinic Ocean Gyre In Spher-

ical Coordinates

This document describes an example experiment using MITgcm to simulate abaroclinic ocean gyre in spherical polar coordinates. The barotropic exampleexperiment in section 3.8 illustrated how to configure the code for a single layersimulation in a Cartesian grid. In this example a similar physical problem issimulated, but the code is now configured for four layers and in a spherical polarcoordinate system.

3.9.1 Overview

This example experiment demonstrates using the MITgcm to simulate a baro-clinic, wind-forced, ocean gyre circulation. The experiment is a numerical ren-dition of the gyre circulation problem similar to the problems described analyt-ically by Stommel in 1966 [49] and numerically in Holland et. al [31].

In this experiment the model is configured to represent a mid-latitude en-closed sector of fluid on a sphere, 60◦ × 60◦ in lateral extent. The fluid is 2 kmdeep and is forced by a constant in time zonal wind stress, τλ, that varies sinu-soidally in the north-south direction. Topologically the simulated domain is asector on a sphere and the coriolis parameter, f , is defined according to latitude,ϕ

f(ϕ) = 2Ω sin(ϕ) (3.10)

with the rotation rate, Ω set to 2π86400s .

The sinusoidal wind-stress variations are defined according to

τλ(ϕ) = τ0 sin(πϕ

Lϕ) (3.11)

where Lϕ is the lateral domain extent (60◦) and τ0 is set to 0.1Nm−2.

Figure 3.2 summarizes the configuration simulated. In contrast to the exam-ple in section 3.8, the current experiment simulates a spherical polar domain.As indicated by the axes in the lower left of the figure the model code worksinternally in a locally orthogonal coordinate (x, y, z). For this experiment de-scription the local orthogonal model coordinate (x, y, z) is synonymous with thecoordinates (λ, ϕ, r) shown in figure 1.16

The experiment has four levels in the vertical, each of equal thickness, Δz =500 m. Initially the fluid is stratified with a reference potential temperatureprofile, θ250 = 20◦ C, θ750 = 10◦ C, θ1250 = 8◦ C, θ1750 = 6◦ C. The equation of


2km

60o

60o

xτ

=0.1=0 xτxτ

0=f

)60sin(2Ω=f

xy

z

C°= 6θ

°= 0φ

°= 60φ N

C°= 8θC°=10θ

C°= 20θ

s86400

π=Ω

Figure 3.2: Schematic of simulation domain and wind-stress forcing functionfor the four-layer gyre numerical experiment. The domain is enclosed by solidwalls at 0◦ E, 60◦ E, 0◦ N and 60◦ N. An initial stratification is imposed bysetting the potential temperature, θ, in each layer. The vertical spacing, Δz, isconstant and equal to 500m.

state used in this experiment is linear

ρ = ρ0(1 − αθθ′) (3.12)

which is implemented in the model as a density anomaly equation

ρ′= −ρ0αθθ

′(3.13)

with ρ0 = 999.8 kgm−3 and αθ = 2×10−4 degrees−1. Integrated forward in thisconfiguration the model state variable theta is equivalent to either in-situ tem-perature, T , or potential temperature, θ. For consistency with later examples,in which the equation of state is non-linear, we use θ to represent temperaturehere. This is the quantity that is carried in the model core equations.

3.9.2 Equations solved

For this problem the implicit free surface, HPE (see section ??) form of theequations described in Marshall et. al [39] are employed. The flow is three-dimensional with just temperature, θ, as an active tracer. The equation of


state is linear. A horizontal Laplacian operator ∇2h provides viscous dissipation

and provides a diffusive sub-grid scale closure for the temperature equation. Awind-stress momentum forcing is added to the momentum equation for the zonalflow, u. Other terms in the model are explicitly switched off for this experimentconfiguration (see section 3.9.4 ). This yields an active set of equations solvedin this configuration, written in spherical polar coordinates as follows

Du

Dt− fv +

1ρ

∂p′

∂λ−Ah∇2

hu−Az∂2u

∂z2= Fλ (3.14)

Dv

Dt+ fu+

1ρ

∂p′

∂ϕ−Ah∇2

hv −Az∂2v

∂z2= 0 (3.15)

∂η

∂t+∂Hu

∂λ+∂Hv

∂ϕ= 0 (3.16)

Dθ

Dt−Kh∇2

hθ −Kz∂2θ

∂z2= 0 (3.17)

p′ = gρ0η +∫ 0

−z

ρ′dz (3.18)

ρ′ = −αθρ0θ′ (3.19)

Fλ|s =τλ

ρ0Δzs(3.20)

Fλ|i = 0 (3.21)

where u and v are the components of the horizontal flow vector �u on the sphere(u = λ, v = ϕ). The terms Hu and Hv are the components of the verticalintegral term given in equation 1.35 and explained in more detail in section2.3. However, for the problem presented here, the continuity relation (equation3.16) differs from the general form given in section 2.3, equation ??, becausethe source terms P − E + R are all 0.

The pressure field, p′, is separated into a barotropic part due to variationsin sea-surface height, η, and a hydrostatic part due to variations in density, ρ′,integrated through the water column.

The suffices s, i indicate surface layer and the interior of the domain. Thewindstress forcing, Fλ, is applied in the surface layer by a source term in thezonal momentum equation. In the ocean interior this term is zero.

In the momentum equations lateral and vertical boundary conditions forthe ∇2

h and ∂2

∂z2 operators are specified when the numerical simulation is run- see section 3.9.4. For temperature the boundary condition is “zero-flux” e.g.∂θ∂ϕ = ∂θ

∂λ = ∂θ∂z = 0.


The domain is discretised with a uniform grid spacing in latitude and longitudeΔλ = Δϕ = 1◦, so that there are sixty grid cells in the zonal and meridionaldirections. Vertically the model is configured with four layers with constant


depth, Δz, of 500 m. The internal, locally orthogonal, model coordinate vari-ables x and y are initialized from the values of λ, ϕ, Δλ and Δϕ in radiansaccording to

x = r cos(ϕ)λ, Δx = r cos(ϕ)Δλ (3.22)y = rϕ, Δy = rΔϕ (3.23)

The procedure for generating a set of internal grid variables from a sphericalpolar grid specification is discussed in section ??.S/R INI SPHERICAL POLAR GRID (model/src/ini spherical polar grid.F)Ac, Aζ , Aw, As: rAc, rAz, rAw, rAs (GRID.h)Δxg, Δyg: DXg, DYg (GRID.h)Δxc, Δyc: DXc, DYc (GRID.h)Δxf , Δyf : DXf, DYf (GRID.h)Δxv, Δyu: DXv, DYu (GRID.h)

As described in 2.15, the time evolution of potential temperature, θ, (equa-tion 3.17) is evaluated prognostically. The centered second-order scheme withAdams-Bashforth time stepping described in section 2.15.1 is used to step for-ward the temperature equation. Prognostic terms in the momentum equationsare solved using flux form as described in section ??. The pressure forces thatdrive the fluid motions, ( ∂p

′

∂λ and ∂p′

∂ϕ ), are found by summing pressure due tosurface elevation η and the hydrostatic pressure. The hydrostatic part of thepressure is diagnosed explicitly by integrating density. The sea-surface height,η, is diagnosed using an implicit scheme. The pressure field solution method isdescribed in sections 2.3 and ??.


The Laplacian viscosity coefficient, Ah, is set to 400ms−1. This value is chosento yield a Munk layer width,

Mw = π(Ah

β)

13 (3.24)

of ≈ 100km. This is greater than the model resolution in mid-latitudes Δx =r cos(ϕ)Δλ ≈ 80 km at ϕ = 45◦, ensuring that the frictional boundary layer iswell resolved.

The model is stepped forward with a time step δt = 1200secs. With this timestep the stability parameter to the horizontal Laplacian friction

Sl = 4Ahδt

Δx2 (3.25)


evaluates to 0.012, which is well below the 0.3 upper limit for stability for thisterm under ABII time-stepping.

The vertical dissipation coefficient, Az, is set to 1× 10−2m2s−1. The associatedstability limit

Sl = 4Azδt

Δz2 (3.26)

evaluates to 4.8 × 10−5 which is again well below the upper limit. The valuesof Ah and Az are also used for the horizontal (Kh) and vertical (Kz) diffusioncoefficients for temperature respectively.

The numerical stability for inertial oscillations

Si = f2δt2 (3.27)


The advective CFL for a extreme maximum horizontal flow speed of |�u| = 2ms−1

Ca =|�u|δtΔx

(3.28)

evaluates to 5 × 10−2. This is well below the stability limit of 0.5.

The stability parameter for internal gravity waves propagating at 2 m s−1

Sc =cgδt

Δx(3.29)

evaluates to ≈ 5 × 10−2. This is well below the linear stability limit of 0.25.



• input/data

• input/data.pkg

• input/eedata,






• code/SIZE.h.

contain the code customisations and parameter settings for this experiment.Below we describe the customisations to these files associated with this experi-ment.



• Line 4,

tRef=20.,10.,8.,6.,

this line sets the initial and reference values of potential temperature ateach model level in units of ◦C. The entries are ordered from surfaceto depth. For each depth level the initial and reference profiles will beuniform in x and y. The values specified here are read into the variabletRef in the model code, by procedure INI PARMS

S/R INI THETA(ini theta.F)ini theta.F

• Line 6,

viscAz=1.E-2,

this line sets the vertical Laplacian dissipation coefficient to 1×10−2m2s−1.Boundary conditions for this operator are specified later. The variableviscAz is read in the routine ini parms.F and is copied into model gen-eral vertical coordinate variable viscAr At each time step, the viscousterm contribution to the momentum equations is calculated in routineCALC DIFFUSIVITY

S/R CALC DIFFUSIVITY(calc diffusivity.F)

• Line 7,

viscAh=4.E2,

this line sets the horizontal laplacian frictional dissipation coefficient to 1×10−2m2s−1. Boundary conditions for this operator are specified later. Thevariable viscAh is read in the routine INI PARMS and applied in routinesCALC MOM RHS and CALC GW.

S/R CALC MOM RHS(calc mom rhs.F)

S/R CALC GW(calc gw.F)


• Line 8,

no_slip_sides=.FALSE.

this line selects a free-slip lateral boundary condition for the horizontallaplacian friction operator e.g. ∂u

∂y =0 along boundaries in y and ∂v∂x=0

along boundaries in x. The variable no slip sides is read in the routineINI PARMS and the boundary condition is evaluated in routine

S/R CALC MOM RHS(calc mom rhs.F)calc mom rhs.F

• Lines 9,

no_slip_bottom=.TRUE.

this line selects a no-slip boundary condition for bottom boundary condi-tion in the vertical laplacian friction operator e.g. u = v = 0 at z = −H ,where H is the local depth of the domain. The variable no slip bottom isread in the routine INI PARMS and is applied in the routine CALC MOM RHS.

S/R CALC MOM RHS(calc mom rhs.F)calc mom rhs.F

• Line 10,

diffKhT=4.E2,

this line sets the horizontal diffusion coefficient for temperature to 400 m2s−1.The boundary condition on this operator is ∂

∂x = ∂∂y = 0 at all bound-

aries. The variable diffKhT is read in the routine INI PARMS and used inroutine CALC GT.

S/R CALC GT(calc gt.F)calc gt.F

• Line 11,

diffKzT=1.E-2,

this line sets the vertical diffusion coefficient for temperature to 10−2 m2s−1.The boundary condition on this operator is ∂

∂z = 0 on all boundaries.The variable diffKzT is read in the routine INI PARMS. It is copied intomodel general vertical coordinate variable diffKrT which is used in rou-tine CALC DIFFUSIVITY.

S/R CALC DIFFUSIVITY(calc diffusivity.F)calc diffusivity.F

• Line 13,


tAlpha=2.E-4,

This line sets the thermal expansion coefficient for the fluid to 2×10−4 degrees−1

The variable tAlpha is read in the routine INI PARMS. The routine FIND RHOmakes use of tAlpha.

S/R FIND RHO(find rho.F)find rho.F

• Line 18,

eosType=’LINEAR’

This line selects the linear form of the equation of state. The variableeosType is read in the routine INI PARMS. The values of eosType setswhich formula in routine FIND RHO is used to calculate density.

S/R FIND RHO(find rho.F)find rho.F

• Line 40,

usingSphericalPolarGrid=.TRUE.,

This line requests that the simulation be performed in a spherical polarcoordinate system. It affects the interpretation of grid input parameters,for example delX and delY and causes the grid generation routines toinitialize an internal grid based on spherical polar geometry. The variableusingSphericalPolarGrid is read in the routine INI PARMS. When setto .TRUE. the settings of delX and delY are taken to be in degrees.These values are used in the routine

S/R INI SPEHRICAL POLAR GRID(ini spherical polar grid.F)ini spherical polar grid.F

• Line 41,

phiMin=0.,

This line sets the southern boundary of the modeled domain to 0◦ latitude.This value affects both the generation of the locally orthogonal grid thatthe model uses internally and affects the initialization of the coriolis force.Note - it is not required to set a longitude boundary, since the absolutelongitude does not alter the kernel equation discretisation. The variablephiMin is read in the routine INI PARMS and is used in routine


• Line 42,


delX=60*1.,

This line sets the horizontal grid spacing between each y-coordinate linein the discrete grid to 1◦ in longitude. The variable delX is read in theroutine INI PARMS and is used in routine


• Line 43,

delY=60*1.,

This line sets the horizontal grid spacing between each y-coordinate linein the discrete grid to 1◦ in latitude. The variable delY is read in theroutine INI PARMS and is used in routine


• Line 44,

delZ=500.,500.,500.,500.,

This line sets the vertical grid spacing between each z-coordinate line in thediscrete grid to 500 m, so that the total model depth is 2 km. The variabledelZ is read in the routine INI PARMS. It is copied into the internal modelcoordinate variable delR which is used in routine

S/R INI VERTICAL GRID(ini vertical grid.F)ini vertical grid.F

• Line 47,


This line specifies the name of the file from which the domain bathymetryis read. This file is a two-dimensional (x, y) map of depths. This file isassumed to contain 64-bit binary numbers giving the depth of the modelat each grid cell, ordered with the x coordinate varying fastest. The pointsare ordered from low coordinate to high coordinate for both axes. Theunits and orientation of the depths in this file are the same as used inthe MITgcm code. In this experiment, a depth of 0m indicates a solidwall and a depth of −2000m indicates open ocean. The matlab programinput/gendata.m shows an example of how to generate a bathymetry file.The variable bathyFile is read in the routine INI PARMS. The bathymetryfile is read in the routine

S/R INI DEPTHS(ini depths.F)ini depths.F


• Line 50,


This line specifies the name of the file from which the x-direction (zonal)surface wind stress is read. This file is also a two-dimensional (x, y) mapand is enumerated and formatted in the same manner as the bathymetryfile. The matlab program input/gendata.m includes example code to gen-erate a valid zonalWindFile file. The variable zonalWindFile is read inthe routine INI PARMS. The wind-stress file is read in the routine

S/R EXTERNAL FIELDS LOAD(external fields load.F)external fields load.F

other lines in the file input/data are standard values.



3 &PARM01

4 tRef=20.,10.,8.,6.,

5 sRef=10.,10.,10.,10.,

6 viscAz=1.E-2,

7 viscAh=4.E2,

8 no_slip_sides=.FALSE.,

9 no_slip_bottom=.TRUE.,

10 diffKhT=4.E2,

11 diffKzT=1.E-2,

12 beta=1.E-11,

13 tAlpha=2.E-4,

14 sBeta =0.,

15 gravity=9.81,





20 &


22 &PARM02



25 &


27 &PARM03

28 startTime=0.,

29 endTime=12000.,



32 abEps=0.1,

33 pChkptFreq=17000.0,

34 chkptFreq=0.0,


35 dumpFreq=2592000.0,

36 &


38 &PARM04

39 usingCartesianGrid=.FALSE.,

40 usingSphericalPolarGrid=.TRUE.,

41 phiMin=0.,

42 delX=60*1.,

43 delY=60*1.,

44 delZ=500.,500.,500.,500.,

45 &

46 &PARM05

47 bathyFile=’topog.box’,

48 hydrogThetaFile=,

49 hydrogSaltFile=,

50 zonalWindFile=’windx.sin_y’,

51 meridWindFile=,

52 &


This file uses standard default values and does not contain customisations forthis experiment.




The input/windx.sin y file specifies a two-dimensional (x, y) map of wind stress,τx, values. The units used are Nm−2 (the default for MITgcm). Although τxis only a function of latitude, y, in this experiment this file must still define acomplete two-dimensional map in order to be compatible with the standard codefor loading forcing fields in MITgcm (routine EXTERNAL FIELDS LOAD. Theincluded matlab program input/gendata.m gives a complete code for creatingthe input/windx.sin y file.


The input/topog.box file specifies a two-dimensional (x, y) map of depth values.For this experiment values are either 0 m or −2000 m, corresponding respectivelyto a wall or to deep ocean. The file contains a raw binary stream of data that isenumerated in the same way as standard MITgcm two-dimensional, horizontalarrays. The included matlab program input/gendata.m gives a complete codefor creating the input/topog.box file.




• Line 39,

sNx=60,


• Line 40,

sNy=60,


• Line 49,

Nr=4,

this line sets the vertical domain extent in grid points.






3.9.4.9 Other Files

Other files relevant to this experiment are

• model/src/ini cori.F. This file initializes the model coriolis variables fCorUand fCorV.

• model/src/ini spherical polar grid.F This file initializes the model grid dis-cretisation variables dxF, dyF, dxG, dyG, dxC, dyC.

• model/src/ini parms.F.

3.9.5 Running The Example

3.9.5.1 Code Download

In order to run the examples you must first download the code distribution.Instructions for downloading the code can be found in section 3.2.

3.9.5.2 Experiment Location

This example experiments is located under the release sub-directory

verification/exp2/

3.9.5.3 Running the Experiment

To run the experiment

1. Set the current directory to input/

% cd input

2. Verify that current directory is now correct

% pwd

You should see a response on the screen ending in

verification/exp2/input


3. Run the genmake script to create the experiment Makefile

% ../../../tools/genmake -mods=../code

4. Create a list of header file dependencies in Makefile

% make depend

5. Build the executable file.

% make

6. Run the mitgcmuv executable

% ./mitgcmuv

3.10. GLOBAL OCEAN MITGCM EXMAPLE 143

3.10 Global Ocean Simulation at 4◦ Resolution

This example experiment demonstrates using the MITgcm to simulate the plan-etary ocean circulation. The simulation is configured with realistic geographyand bathymetry on a 4◦ × 4◦ spherical polar grid. Twenty levels are used inthe vertical, ranging in thickness from 50 m at the surface to 815 m at depth,giving a maximum model depth of 6 km. At this resolution, the configurationcan be integrated forward for thousands of years on a single processor desktopcomputer.

3.10.1 Overview

The model is forced with climatological wind stress data and surface flux datafrom DaSilva [10]. Climatological data from Levitus [36] is used to initialize themodel hydrography. Levitus seasonal climatology data is also used throughoutthe calculation to provide additional air-sea fluxes. These fluxes are combinedwith the DaSilva climatological estimates of surface heat flux and fresh water,resulting in a mixed boundary condition of the style described in Haney [25].Altogether, this yields the following forcing applied in the model surface layer.

Fu =τx

ρ0Δzs(3.30)

Fv =τy

ρ0Δzs(3.31)

Fθ = −λθ(θ − θ∗) − 1Cpρ0Δzs

Q (3.32)

Fs = −λs(S − S∗) +S0

Δzs(E − P −R) (3.33)

where Fu, Fv, Fθ, Fs are the forcing terms in the zonal and meridional mo-mentum and in the potential temperature and salinity equations respectively.The term Δzs represents the top ocean layer thickness in meters. It is used inconjunction with a reference density, ρ0 (here set to 999.8 kgm−3), a referencesalinity, S0 (here set to 35 ppt), and a specific heat capacity, Cp (here set to4000 J ◦C−1 kg−1), to convert input dataset values into time tendencies of po-tential temperature (with units of ◦C s−1), salinity (with units ppt s−1) andvelocity (with units m s−2). The externally supplied forcing fields used in thisexperiment are τx, τy, θ∗, S∗, Q and E −P −R. The wind stress fields (τx, τy)have units of N m−2. The temperature forcing fields (θ∗ and Q) have units of◦C and W m−2 respectively. The salinity forcing fields (S∗ and E −P−R) haveunits of ppt and m s−1 respectively. The source files and procedures for ingest-ing this data into the simulation are described in the experiment configurationdiscussion in section 3.10.3.



The model is configured in hydrostatic form. The domain is discretised with auniform grid spacing in latitude and longitude on the sphere Δφ = Δλ = 4◦, sothat there are ninety grid cells in the zonal and forty in the meridional direction.The internal model coordinate variables x and y are initialized according to

x = r cos(φ), Δx = r cos(Δφ) (3.34)y = rλ, Δy = rΔλ (3.35)

Arctic polar regions are not included in this experiment. Meridionally themodel extends from 80◦S to 80◦N. Vertically the model is configured withtwenty layers with the following thicknesses Δz1 = 50 m, Δz2 = 50 m, Δz3 =55 m, Δz4 = 60 m, Δz5 = 65 m, Δz6 = 70 m, Δz7 = 80 m, Δz8 = 95 m, Δz9 =120 m, Δz10 = 155 m, Δz11 = 200 m, Δz12 = 260 m, Δz13 = 320 m, Δz14 =400 m, Δz15 = 480 m, Δz16 = 570 m, Δz17 = 655 m, Δz18 = 725 m, Δz19 =775 m, Δz20 = 815 m (here the numeric subscript indicates the model level in-dex number, k) to give a total depth, H , of −5450m. The implicit free surfaceform of the pressure equation described in Marshall et. al [39] is employed.A Laplacian operator, ∇2, provides viscous dissipation. Thermal and halinediffusion is also represented by a Laplacian operator.

Wind-stress forcing is added to the momentum equations in (3.36) for boththe zonal flow, u and the meridional flow v, according to equations (3.30) and(3.31). Thermodynamic forcing inputs are added to the equations in (3.36)for potential temperature, θ, and salinity, S, according to equations (3.32) and(3.33). This produces a set of equations solved in this configuration as follows:

Du

Dt− fv +

1ρ

∂p′

∂x−∇h ·Ah∇hu− ∂

∂zAz

∂u

∂z=

{Fu (surface)0 (interior)

(3.36)

Dv

Dt+ fu+

1ρ

∂p′

∂y−∇h · Ah∇hv − ∂

∂zAz

∂v

∂z=

{Fv (surface)0 (interior)

(3.37)

∂η

∂t+ ∇h · �u = 0 (3.38)

Dθ

Dt−∇h ·Kh∇hθ − ∂

∂zΓ(Kz)

∂θ

∂z=

{Fθ (surface)0 (interior)

(3.39)

Ds

Dt−∇h ·Kh∇hs− ∂

∂zΓ(Kz)

∂s

∂z=

{Fs (surface)0 (interior)

(3.40)

gρ0η +∫ 0

−z

ρ′dz = p

′(3.41)

where u = DxDt = r cos(φ)Dλ

Dt and v = DyDt = rDφ

Dt are the zonal and meridionalcomponents of the flow vector, �u, on the sphere. As described in MITgcm Nu-merical Solution Procedure 2, the time evolution of potential temperature, θ,


equation is solved prognostically. The total pressure, p, is diagnosed by sum-ming pressure due to surface elevation η and the hydrostatic pressure.


The Laplacian dissipation coefficient, Ah, is set to 5 × 105ms−1. This value ischosen to yield a Munk layer width [1],

Mw = π(Ah

β)

13 (3.42)

of ≈ 600km. This is greater than the model resolution in low-latitudes, Δx ≈400km, ensuring that the frictional boundary layer is adequately resolved.

The model is stepped forward with a time step δtθ = 30 hours for thermody-namic variables and δtv = 40 minutes for momentum terms. With this timestep, the stability parameter to the horizontal Laplacian friction [1]

Sl = 4Ahδtv

Δx2 (3.43)

evaluates to 0.16 at a latitude of φ = 80◦, which is below the 0.3 upperlimit for stability. The zonal grid spacing Δx is smallest at φ = 80◦ whereΔx = r cos(φ)Δφ ≈ 77km.


Sl = 4Azδtv

Δz2 (3.44)

evaluates to 0.015 for the smallest model level spacing (Δz1 = 50m) which isagain well below the upper stability limit.

The values of the horizontal (Kh) and vertical (Kz) diffusion coefficients forboth temperature and salinity are set to 1 × 103 m2s−1 and 3 × 10−5 m2s−1

respectively. The stability limit related to Kh will be at φ = 80◦ where Δx ≈77km. Here the stability parameter

Sl =4Khδtθ

Δx2 (3.45)

evaluates to 0.07, well below the stability limit of Sl ≈ 0.5. The stabilityparameter related to Kz

Sl =4Kzδtθ

Δz2 (3.46)

evaluates to 0.005 for min(Δz) = 50m, well below the stability limit of Sl ≈ 0.5.



Si = f2δtv2 (3.47)

evaluates to 0.24 for f = 2ω sin(80◦) = 1.43 × 10−4 s−1, which is close to theSi < 1 upper limit for stability.

The advective CFL [1] for a extreme maximum horizontal flow speed of |�u| =2ms−1

Sa =|�u|δtvΔx

(3.48)


The stability parameter for internal gravity waves propagating with a maximumspeed of cg = 10 ms−1 [1]

Sc =cgδtvΔx

(3.49)

evaluates to 3 × 10−1. This is close to the linear stability limit of 0.5.

3.10.3 Experiment Configuration

The model configuration for this experiment resides under the directory tuto-rial examples/global ocean circulation/. The experiment files

• input/data

• input/data.pkg

• input/eedata,

• input/windx.bin,

• input/windy.bin,

• input/salt.bin,

• input/theta.bin,

• input/SSS.bin,

• input/SST.bin,

• input/topog.bin,




• code/SIZE.h.

contain the code customizations and parameter settings for these experiments.Below we describe the customizations to these files associated with this experi-ment.

3.10.3.1 Driving Datasets

Figures (??-??) show the relaxation temperature (θ∗) and salinity (S∗) fields,the wind stress components (τx and τy), the heat flux (Q) and the net freshwater flux (E − P −R) used in equations ??-??. The figures also indicate thelateral extent and coastline used in the experiment. Figure (??) shows the depthcontours of the model domain.



• Lines 7-10 and 11-14

tRef= 16.0 , 15.2 , 14.5 , 13.9 , 13.3 ,

· · ·set reference values for potential temperature and salinity at each modellevel in units of ◦C and ppt. The entries are ordered from surface to depth.Density is calculated from anomalies at each level evaluated with respectto the reference values set here.S/R INI THETA(ini theta.F)

• Line 15,

viscAz=1.E-3,

this line sets the vertical Laplacian dissipation coefficient to 1×10−3m2s−1.Boundary conditions for this operator are specified later. This variable iscopied into model general vertical coordinate variable viscAr.


• Line 16,

viscAh=5.E5,

this line sets the horizontal Laplacian frictional dissipation coefficient to5 × 105m2s−1. Boundary conditions for this operator are specified later.


• Lines 17,


this line selects a free-slip lateral boundary condition for the horizontalLaplacian friction operator e.g. ∂u


along boundaries in x.

• Lines 9,


this line selects a no-slip boundary condition for bottom boundary condi-tion in the vertical Laplacian friction operator e.g. u = v = 0 at z = −H ,where H is the local depth of the domain.

• Line 19,

diffKhT=1.E3,


∂x = ∂∂y = 0 on all boundaries.

• Line 20,

diffKzT=3.E-5,

this line sets the vertical diffusion coefficient for temperature to 3×10−5 m2s−1.The boundary condition on this operator is ∂

∂z = 0 at both the upper andlower boundaries.

• Line 21,

diffKhS=1.E3,

this line sets the horizontal diffusion coefficient for salinity to 1000 m2s−1.The boundary condition on this operator is ∂


• Line 22,

diffKzS=3.E-5,

this line sets the vertical diffusion coefficient for salinity to 3×10−5 m2s−1.The boundary condition on this operator is ∂


• Lines 23-26

beta=1.E-11,


· · ·These settings do not apply for this experiment.

• Line 27,

gravity=9.81,

Sets the gravitational acceleration coefficient to 9.81ms−1.S/R CALC PHI HYD (calc phi hyd.F)S/R INI CG2D (ini cg2d.F)S/R INI CG3D (ini cg3d.F)S/R INI PARMS (ini parms.F)S/R SOLVE FOR PRESSURE (solve for pressure.F)

• Line 28-29,


Selects the barotropic pressure equation to be the implicit free surfaceformulation.

• Line 30,

eosType=’POLY3’,

Selects the third order polynomial form of the equation of state.S/R FIND RHO (find rho.F)S/R FIND ALPHA (find alpha.F)

• Line 31,

readBinaryPrec=32,

Sets format for reading binary input datasets holding model fields to use32-bit representation for floating-point numbers.S/R READ WRITE FLD (read write fld.F)S/R READ WRITE REC (read write rec.F)

• Line 36,

cg2dMaxIters=1000,

Sets maximum number of iterations the two-dimensional, conjugate gra-dient solver will use, irrespective of convergence criteria being met.S/R CG2D (cg2d.F)


• Line 37,

cg2dTargetResidual=1.E-13,

Sets the tolerance which the two-dimensional, conjugate gradient solverwill use to test for convergence in equation ?? to 1 × 10−13. Solver williterate until tolerance falls below this value or until the maximum numberof solver iterations is reached.S/R CG2D (cg2d.F)

• Line 42,

startTime=0,

Sets the starting time for the model internal time counter. When set tonon-zero this option implicitly requests a checkpoint file be read for initialstate. By default the checkpoint file is named according to the integernumber of time steps in the startTime value. The internal time counterworks in seconds.

• Line 43,

endTime=2808000.,

Sets the time (in seconds) at which this simulation will terminate. At theend of a simulation a checkpoint file is automatically written so that anumerical experiment can consist of multiple stages.

• Line 44,

#endTime=62208000000,

A commented out setting for endTime for a 2000 year simulation.

• Line 45,

deltaTmom=2400.0,

Sets the timestep δtv used in the momentum equations to 20 mins. Seesection ??.

S/R TIMESTEP(timestep.F)

• Line 46,

tauCD=321428.,


Sets the D-grid to C-grid coupling time scale τCD used in the momentumequations. See section ??.

S/R INI PARMS(ini parms.F)S/R CALC MOM RHS(calc mom rhs.F)

• Line 47,

deltaTtracer=108000.,

Sets the default timestep, δtθ, for tracer equations to 30 hours. See section??.

S/R TIMESTEP TRACER(timestep tracer.F)

• Line 47,



• Line 50,




1 # ====================

2 # | Model parameters |

3 # ====================

4 #


6 &PARM01

7 tRef= 16.0 , 15.2 , 14.5 , 13.9 , 13.3 ,


8 12.4 , 11.3 , 9.9 , 8.4 , 6.7 ,

9 5.2 , 3.8 , 2.9 , 2.3 , 1.8 ,

10 1.5 , 1.1 , 0.8 , 0.66, 0.63,

11 sRef= 34.65, 34.75, 34.82, 34.87, 34.90,

12 34.90, 34.86, 34.78, 34.69, 34.60,

13 34.58, 34.62, 34.68, 34.72, 34.73,

14 34.74, 34.73, 34.73, 34.72, 34.72,

15 viscAz=1.E-3,

16 viscAh=5.E5,



19 diffKhT=1.E3,

20 diffKzT=3.E-5,

21 diffKhS=1.E3,

22 diffKzS=3.E-5,

23 beta=1.E-11,

24 f0=1.e-4,

25 tAlpha=2.E-4,

26 sBeta =7.4E-4,

27 gravity=9.81,





32 &

33


35 &PARM02



38 &

39


41 &PARM03

42 startTime=0,

43 endTime=622080000,

44 #endTime=2808000.,


46 tauCD=321428.,


48 deltaTClock =108000.0,

49 cAdjFreq=-1.,

50 abEps=0.1,

51 pChkptFreq=0.0,

52 chkptFreq=0.0,

53 dumpFreq=2592000.0,

54 dumpFreq=31104000.,

55 tauThetaClimRelax=2592000.0,

56 tauSaltClimRelax=2592000.0,

57 &


58


60 &PARM04



63 delZ= 5.000000e+01, 5.000000e+01, 5.500000e+01, 6.000000e+01,

64 6.500000e+01, 7.000000e+01, 8.000000e+01, 9.500000e+01,

65 1.200000e+02, 1.550000e+02, 2.000000e+02, 2.600000e+02,

66 3.200000e+02, 4.000000e+02, 4.800000e+02, 5.700000e+02,

67 6.550000e+02, 7.250000e+02, 7.750000e+02, 8.150000e+02,

68 phiMin=-80.,

69 delY=40*4,

70 delX=90*4,

71 &

72

73 # Input datasets

74 &PARM05

75 hydrogThetaFile=’theta.bin’,

76 hydrogSaltFile=’salt.bin’,

77 bathyFile=’topog.bin’,

78 zonalWindFile=’windx.bin’,

79 meridWindFile=’windy.bin’,

80 thetaClimFile=’SST.bin’

81 saltClimFile=’SSS.bin’

82 &








The input/topog.box file specifies a two-dimensional (x, y) map of depth values.For this experiment values are either 0m or −2000 m, corresponding respectively


to a wall or to deep ocean. The file contains a raw binary stream of data that isenumerated in the same way as standard MITgcm two-dimensional, horizontalarrays. The included matlab program input/gendata.m gives a complete codefor creating the input/topog.box file.



• Line 39,

sNx=60,


• Line 40,

sNy=60,


• Line 49,

Nr=4,


1 C $Header: /u/gcmpack/manual/part3/case_studies/climatalogical_ogcm/code/SIZE.h.tex,v 1.1

2 C $Name: $

3 C

4 C /==========================================================\


6 C |==========================================================|








14 C \==========================================================/













26 C Nr - No. points in Z for full process domain.

27 INTEGER sNx

28 INTEGER sNy

29 INTEGER OLx

30 INTEGER OLy

31 INTEGER nSx

32 INTEGER nSy

33 INTEGER nPx

34 INTEGER nPy

35 INTEGER Nx

36 INTEGER Ny

37 INTEGER Nr

38 PARAMETER (

39 & sNx = 90,

40 & sNy = 40,

41 & OLx = 3,

42 & OLy = 3,

43 & nSx = 1,

44 & nSy = 1,

45 & nPx = 1,

46 & nPy = 1,



49 & Nr = 20)




53 INTEGER MAX_OLX

54 INTEGER MAX_OLY








3.10.3.10 Other Files


• model/src/ini cori.F. This file initializes the model coriolis variables fCorU.

• model/src/ini spherical polar grid.F

• model/src/ini parms.F,


contain the code customisations and parameter settings for this experiments.Below we describe the customisations to these files associated with this experi-ment.

3.11. P COORDINATE GLOBAL OCEAN MITGCM EXAMPLE 157

3.11 Global Ocean Simulation at 4◦ Resolution

in Pressure Coordinates

This example experiment demonstrates using the MITgcm to simulate the plan-etary ocean circulation in pressure coordinates, that is, without making theBoussinesq approximations. The simulation is configured with realistic geog-raphy and bathymetry on a 4◦ × 4◦ spherical polar grid. Fifteen levels areused in the vertical, ranging in thickness from 50.4089dbar ≈ 50m at thesurface to 710.33dbar ≈ 690m at depth, giving a maximum model depth of5302.3122dbar ≈ 5200km. At this resolution, the configuration can be inte-grated forward for thousands of years on a single processor desktop computer.

3.11.1 Overview

The model is forced with climatological wind stress data from Trenberth [50]and surface flux data from Jiang et al. [34]. Climatological data from Levitus[36] is used to initialize the model hydrography. Levitus seasonal climatologydata is also used throughout the calculation to provide additional air-sea fluxes.These fluxes are combined with the Jiang climatological estimates of surfaceheat flux, resulting in a mixed boundary condition of the style described inHaney [25]. Altogether, this yields the following forcing applied in the modelsurface layer.

Fu = gτx

Δps(3.50)

Fv = gτy

Δps(3.51)

Fθ = −gλθ(θ − θ∗) − 1CpΔps

Q (3.52)

Fs = +gρFWS

ρΔps(E − P −R) (3.53)

where Fu, Fv, Fθ, Fs are the forcing terms in the zonal and meridional mo-mentum and in the potential temperature and salinity equations respectively.The term Δps represents the top ocean layer thickness in Pa. It is used in con-junction with a reference density, ρFW (here set to 999.8 kgm−3), the surfacesalinity, S, and a specific heat capacity, Cp (here set to 4000 J ◦C−1 kg−1),to convert input dataset values into time tendencies of potential temperature(with units of ◦C s−1), salinity (with units ppt s−1) and velocity (with unitsm s−2). The externally supplied forcing fields used in this experiment are τx,τy, θ∗, Q and E − P −R. The wind stress fields (τx, τy) have units of N m−2.The temperature forcing fields (θ∗ and Q) have units of ◦C and W m−2 respec-tively. The salinity forcing fields (E − P − R) has units of m s−1 respectively.The source files and procedures for ingesting these data into the simulation aredescribed in the experiment configuration discussion in section 3.10.3.



Due to the pressure coordinate, the model can only be hydrostatic [12]. Thedomain is discretized with a uniform grid spacing in latitude and longitude onthe sphere Δφ = Δλ = 4◦, so that there are ninety grid cells in the zonal andforty in the meridional direction. The internal model coordinate variables x andy are initialized according to

x = r cos(φ), Δx = r cos(Δφ) (3.54)y = rλ, Δy = rΔλ (3.55)

Arctic polar regions are not included in this experiment. Meridionally themodel extends from 80◦S to 80◦N. Vertically the model is configured with fifteenlayers with the following thicknesses

Δp1 = 7103300.720021Pa,Δp2 = 6570548.440790Pa,Δp3 = 6041670.010249Pa,Δp4 = 5516436.666057Pa,Δp5 = 4994602.034410Pa,Δp6 = 4475903.435290Pa,Δp7 = 3960063.245801Pa,Δp8 = 3446790.312651Pa,Δp9 = 2935781.405664Pa,

Δp10 = 2426722.705046Pa,Δp11 = 1919291.315988Pa,Δp12 = 1413156.804970Pa,Δp13 = 1008846.750166Pa,Δp14 = 705919.025481Pa,Δp15 = 504089.693499Pa,

(here the numeric subscript indicates the model level index number, k; note,that the surface layer has the highest index number 15) to give a total depth,H , of −5200m. In pressure, this is p0

b = 53023122.566084Pa. The implicit freesurface form of the pressure equation described in Marshall et al. [39] with thenonlinear extension by Campin et al. [8] is employed. A Laplacian operator, ∇2,provides viscous dissipation. Thermal and haline diffusion is also representedby a Laplacian operator.

Wind-stress forcing is added to the momentum equations in (3.56) for boththe zonal flow, u and the meridional flow v, according to equations (3.50) and(3.51). Thermodynamic forcing inputs are added to the equations in (3.56)for potential temperature, θ, and salinity, S, according to equations (3.52) and(3.53). This produces a set of equations solved in this configuration as follows:


Du

Dt− fv +

1ρ

∂Φ′

∂x−∇h · Ah∇hu− (gρ0)2

∂

∂pAr

∂u

∂p=


(3.56)

Dv

Dt+ fu+

1ρ

∂Φ′

∂y−∇h ·Ah∇hv − (gρ0)2

∂

∂pAr

∂v

∂p=


(3.57)

∂pb

∂t+ ∇h · �u = 0 (3.58)

Dθ

Dt−∇h ·Kh∇hθ − (gρ0)2

∂

∂pΓ(Kr)

∂θ

∂p=


(3.59)

Ds

Dt−∇h ·Kh∇hs− (gρ0)2

∂

∂pΓ(Kr)

∂S

∂p=


(3.60)

Φ′(0)−H + α0pb +∫ p

0

α′dp = Φ′ (3.61)



Dt are the zonal and meridionalcomponents of the flow vector, �u, on the sphere. As described in MITgcmNumerical Solution Procedure 2, the time evolution of potential temperature,θ, equation is solved prognostically. The full geopotential height, Φ, is diagnosedby summing the geopotential height anomalies Φ′ due to bottom pressure pb anddensity variations. The integration of the hydrostatic equation is started at thebottom of the domain. The condition of p = 0 at the sea surface requires atime-independent integration constant for the height anomaly due to densityvariations Φ′(0)−H , which is provided as an input field.


The model configuration for this experiment resides under the directory tuto-rial examples/global ocean circulation/. The experiment files

• input/data

• input/data.pkg

• input/eedata,


• input/deltageopotjmd95.bin,

• input/lev s.bin,

• input/lev t.bin,

• input/trenberth taux.bin,


• input/trenberth tauy.bin,

• input/lev sst.bin,

• input/shi qnet.bin,

• input/shi empmr.bin,



• code/SIZE.h.


3.11.3.1 Driving Datasets

Figures (3.3-3.8) show the relaxation temperature (θ∗) and salinity (S∗) fields,the wind stress components (τx and τy), the heat flux (Q) and the net freshwater flux (E − P − R) used in equations 3.50-3.53. The figures also indicatethe lateral extent and coastline used in the experiment. Figure (3.9) shows thedepth contours of the model domain.



• Line 15,

viscAr=1.721611620915750E+05,

this line sets the vertical Laplacian dissipation coefficient to 1.72161162091575×105Pa2s−1. Note that, the factor (gρ)2 needs to be included in this line.Boundary conditions for this operator are specified later. This variable iscopied into model general vertical coordinate variable viscAr.


• Line 9–10,

viscAh=3.E5,no_slip_sides=.TRUE.

these lines set the horizontal Laplacian frictional dissipation coefficient to3 × 105m2s−1 and specify a no-slip boundary condition for this operator,that is, u = 0 along boundaries in y and v = 0 along boundaries in x.


0 5 10 15 20 25

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.3: Annual mean of relaxation temperature [◦C]

• Lines 11-13,

viscAr =1.721611620915750e5,#viscAz =1.67E-3,no_slip_bottom=.FALSE.,

These lines set the vertical Laplacian frictional dissipation coefficient to1.721611620915750× 105Pa2s−1, which corresponds to 1.67× 10−3 m2s−1

in the commented line, and specify a free slip boundary condition for thisoperator, that is, ∂u

∂p = ∂v∂p = 0 at p = p0

b , where p0b is the local bottom

pressure of the domain at rest. Note that, the factor (gρ)2 needs to beincluded in this line.

• Line 14,

diffKhT=1.E3,



• Line 15–16,


30 31 32 33 34 35 36 37

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.4: Annual mean of relaxation salinity [PSU]

diffKrT=5.154525811125000e3,#diffKzT=0.5E-4,

this line sets the vertical diffusion coefficient for temperature to 5.154525811125×103 Pa2s−1, which corresponds to 5 × 10−4 m2s−1 in the commented line.Note that, the factor (gρ)2 needs to be included in this line. The bound-ary condition on this operator is ∂

∂p = 0 at both the upper and lowerboundaries.

• Line 17–19,

diffKhS=1.E3,diffKrS=5.154525811125000e3,

#diffKzS=0.5E-4,

These lines set the same values for the diffusion coefficients for salinity asfor temperature.

• Line 20–22,

implicitDiffusion=.TRUE.,ivdc_kappa=1.030905162225000E9,

#ivdc_kappa=10.0,


−0.3 −0.2 −0.1 0 0.1 0.2 0.3

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.5: Annual mean of zonal wind stress component [Nm m−2]

Select implicit diffusion as a convection scheme and set coefficient for im-plicit vertical diffusion to 1.030905162225×109Pa2s−1, which correspondsto 10m2 s−1.

• Line 23-24,

gravity=9.81,gravitySign=-1.D0,

These lines set the gravitational acceleration coefficient to 9.81ms−1 anddefine the upward direction relative to the direction of increasing verticalcoordinate (in pressure coordinates, up is in the direction of decreasingpressure)

• Line 25,

rhoNil=1035.,

sets the reference density of sea water to 1035kgm−3.


−0.3 −0.2 −0.1 0 0.1 0.2 0.3

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.6: Annual mean of meridional wind stress component [Nm m−2]

S/R CALC PHI HYD (calc phi hyd.F)S/R INI CG2D (ini cg2d.F)S/R INI CG3D (ini cg3d.F)S/R INI PARMS (ini parms.F)S/R SOLVE FOR PRESSURE (solve for pressure.F)

• Line 28

eosType=’JMD95P’,

Selects the full equation of state according to Jackett and McDougall [33].The only other sensible choice is the equation of state by McDougall etal. [41], ’MDJFW’. All other equations of state do not make sense in thisconfiguration.S/R FIND RHO (find rho.F)S/R FIND ALPHA (find alpha.F)

• Line 28-29,



−200 −150 −100 −50 0 50 100 150 200

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.7: Annual mean heat flux [Wm−2]


• Line 30

exactConserv=.TRUE.,

Select a more accurate conservation of properties at the surface layer byincluding the horizontal velocity divergence to update the free surface.

• Line 31–33

nonlinFreeSurf=3,hFacInf=0.2,hFacSup=2.0,

Select the nonlinear free surface formulation and set lower and upper limitsfor the free surface excursions.

• Line 34

useRealFreshWaterFlux=.FALSE.,


−1 −0.5 0 0.5 1

x 10−7

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.8: Annual mean fresh water flux (Evaporation-Precipitation) [m s−1]

Select virtual salt flux boundary condition for salinity. The freshwaterflux at the surface only affect the surface salinity, but has no mass fluxassociated with it

• Line 35–36,

readBinaryPrec=64,writeBinaryPrec=64,

Sets format for reading binary input datasets and writing binary outputdatasets holding model fields to use 64-bit representation for floating-pointnumbers.S/R READ WRITE FLD (read write fld.F)S/R READ WRITE REC (read write rec.F)

• Line 42,

cg2dMaxIters=200,



0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 107

longitude [°E]

latit

ude

[°N

]

0 50 100 150 200 250 300 350−80

−60

−40

−20

0

20

40

60

80

Figure 3.9: Model bathymetry in pressure units [Pa]

• Line 43,



• Line 48,

startTime=0,


• Line 49–50,


endTime=8640000.,#endTime=62208000000,

Sets the time (in seconds) at which this simulation will terminate. At theend of a simulation a checkpoint file is automatically written so that anumerical experiment can consist of multiple stages. The commented outsetting for endTime is for a 2000 year simulation.

• Line 51–53,

deltaTmom = 1200.0,deltaTtracer = 172800.0,deltaTfreesurf = 172800.0,

Sets the timestep δtv used in the momentum equations to 20 mins and thetimesteps δtθ in the tracer equations and δtη in the implicit free surfaceequation to 48 hours. See section ??.S/R TIMESTEP(timestep.F)S/R INI PARMS(ini parms.F)S/R CALC MOM RHS(calc mom rhs.F)S/R TIMESTEP TRACER(timestep tracer.F)

• Line 55,

pChkptFreq =3110400000.,

write a pick-up file every 100 years of integration.

• Line 56–58

dumpFreq = 3110400000.,taveFreq = 3110400000.,monitorFreq = 31104000.,

write model output and time-averaged model output every 100 years, andmonitor statisitics every year.

• Line 59–61

periodicExternalForcing=.TRUE.,externForcingPeriod=2592000.,externForcingCycle=31104000.,

Allow periodic external forcing, set forcing period, during which one setof data is valid, to 1 month and the repeat cycle to 1 year.S/R EXTERNAL FORCING SURF(external forcing surf.F)

• Line 62


tauThetaClimRelax=5184000.0,

Set the restoring timescale to 2 months.S/R EXTERNAL FORCING SURF(external forcing surf.F)

• Line 63

abEps=0.1,

Adams-Bashford factor (see section 2.4)

• Line 68–69

usingCartesianGrid=.FALSE.,usingSphericalPolarGrid=.TRUE.,

Select spherical grid.

• Line 70–71

dXspacing=4.,dYspacing=4.,

Set the horizontal grid spacing in degrees spherical distance.

• Line 72

Ro_SeaLevel=53023122.566084,

specifies the total height (in r-units, i.e., pressure units) of the sea surfaceat rest. This is a reference value.

• Line 73

groundAtK1=.TRUE.,

specifies the reversal of the vertical indexing. The vertical index is 1 atthe bottom of the doman and maximal (i.e., 15) at the surface.

• Line 74–78

delR=7103300.720021, \ldots

set the layer thickness in pressure units, starting with the bottom layer.

• Line 84–93,


bathyFile=’topog.box’ploadFile=’deltageopotjmd95.bin’hydrogThetaFile=’lev_t.bin’,hydrogSaltFile =’lev_s.bin’,zonalWindFile =’trenberth_taux.bin’,meridWindFile =’trenberth_tauy.bin’,thetaClimFile =’lev_sst.bin’,surfQFile =’shi_qnet.bin’,EmPmRFile =’shi_empmr.bin’,

This line specifies the names of the files holding the bathymetry data set,the time-independent geopotential height anomaly at the bottom, initialconditions of temperature and salinity, wind stress forcing fields, sea sur-face temperature climatology, heat flux, and fresh water flux (evaporationminus precipitation minus run-off) at the surface. See file descriptions insection 3.11.3.


1 # ====================


3 # ====================

4 #


6 &PARM01

7 tRef=15*20.,

8 sRef=15*35.,

9 viscAh =3.E5,

10 no_slip_sides=.TRUE.,

11 viscAr =1.721611620915750e5,

12 #viscAz =1.67E-3,

13 no_slip_bottom=.FALSE.,

14 diffKhT=1.E3,

15 diffKrT=5.154525811125000e3,

16 #diffKzT=0.5E-4,

17 diffKhS=1.E3,

18 diffKrS=5.154525811125000e3,

19 #diffKzS=0.5E-4,

20 implicitDiffusion=.TRUE.,

21 ivdc_kappa=1.030905162225000e9,

22 #ivdc_kappa=10.0,

23 gravity=9.81,

24 gravitySign=-1.D0,

25 rhonil=1035.,

26 buoyancyRelation=’OCEANICP’,

27 eosType=’JMD95P’,




30 exactConserv=.TRUE.,

31 nonlinFreeSurf=3,

32 hFacInf=0.2,

33 hFacSup=2.0,

34 useRealFreshWaterFlux=.FALSE.,


36 writeBinaryPrec=64,

37 cosPower=0.5,

38 &

39


41 &PARM02



44 &

45


47 &PARM03

48 startTime = 0.,

49 endTime = 8640000.,

50 #endTime = 62208000000.,

51 deltaTmom = 1200.0,

52 deltaTtracer = 172800.0,

53 deltaTfreesurf = 172800.0,

54 deltaTClock = 172800.0,

55 pChkptFreq = 3110400000.,

56 dumpFreq = 3110400000.,

57 taveFreq = 3110400000.,

58 monitorFreq = 31104000.,

59 periodicExternalForcing=.TRUE.,

60 externForcingPeriod=2592000.,

61 externForcingCycle=31104000.,


63 abEps=0.1,

64 &

65


67 &PARM04



70 dXspacing=4.,

71 dYspacing=4.,

72 Ro_SeaLevel=53023122.566084,

73 groundAtK1=.TRUE.,

74 delR=7103300.720021, 6570548.440790, 6041670.010249,

75 5516436.666057, 4994602.034410, 4475903.435290,

76 3960063.245801, 3446790.312651, 2935781.405664,

77 2426722.705046, 1919291.315988, 1413156.804970,

78 1008846.750166, 705919.025481, 504089.693499,

79 phiMin=-80.,


80 &

81

82 # Input datasets

83 &PARM05

84 topoFile =’topog.bin’,

85 pLoadFile =’deltageopotjmd95.bin’,

86 hydrogThetaFile=’lev_t.bin’,

87 hydrogSaltFile =’lev_s.bin’,

88 zonalWindFile =’trenberth_taux.bin’,

89 meridWindFile =’trenberth_tauy.bin’,

90 thetaClimFile =’lev_sst.bin’,

91 #saltClimFile =’lev_sss.bin’,

92 surfQFile =’shi_qnet.bin’,

93 EmPmRFile =’shi_empmr.bin’,

94 &





3.11.3.5 File input/topog.bin

This file is a two-dimensional (x, y) map of depths. This file is assumed tocontain 64-bit binary numbers giving the depth of the model at each grid cell,ordered with the x coordinate varying fastest. The points are ordered fromlow coordinate to high coordinate for both axes. The units and orientation ofthe depths in this file are the same as used in the MITgcm code (Pa for thisexperiment). In this experiment, a depth of 0 Pa indicates a land point wall anda depth of > 0 Pa indicates open ocean.

3.11.3.6 File input/deltageopotjmd95.box

The file contains 12 identical two dimensional maps (x, y) of geopotential heightanomaly at the bottom at rest. The values have been obtained by verticallyintegrating the hydrostatic equation with the initial density field (from in-put/lev t/s.bin). This file has to be consitent with the temperature and salinityfield at rest and the choice of equation of state!

3.11.3.7 File input/lev t/s.bin

The files input/lev t/s.bin specify the initial conditions for temperature andsalinity for every grid point in a three dimensional array (x, y, z). The data are


obtain by interpolating Levitus [36] monthly mean values for January onto themodel grid. Keep in mind, that the first index corresponds to the bottom layerand highest index to the surface layer.

3.11.3.8 File input/trenberth taux/y.bin

Each of the input/trenberth taux/y.bin files specifies 12 two-dimensional (x, y, t)maps of zonal and meridional wind stress values, τx and τy , that is monthlymean values from Trenberth [50]. The units used are Nm−2.

3.11.3.9 File input/lev sst.bin

The file input/lev sst.bin contains 12 monthly surface temperature climatologiesfrom Levitus [36] in a three dimensional array (x, y, t).

3.11.3.10 File input/shi qnet/empmr.bin

The files input/shi qnet/empmr.bin contain 12 monthly surface fluxes of heat(qnet) and freshwater (empmr) by Jiang et al. [34] in three dimensional arrays(x, y, t). Both fluxes are normalized so that of one year there is no net flux intothe ocean. The freshwater flux is actually constant in time.


Three lines are customized in this file for the current experiment

• Line 39,

sNx=90,


• Line 40,

sNy=40,


• Line 49,

Nr=15,



2 C $Name: $

3 C

4 C /==========================================================\


6 C |==========================================================|








14 C \==========================================================/













27 INTEGER sNx

28 INTEGER sNy

29 INTEGER OLx

30 INTEGER OLy

31 INTEGER nSx

32 INTEGER nSy

33 INTEGER nPx

34 INTEGER nPy

35 INTEGER Nx

36 INTEGER Ny

37 INTEGER Nr

38 PARAMETER (

39 & sNx = 90,

40 & sNy = 40,

41 & OLx = 3,

42 & OLy = 3,

43 & nSx = 1,

44 & nSy = 1,

45 & nPx = 1,

46 & nPy = 1,



49 & Nr = 15)





53 INTEGER MAX_OLX

54 INTEGER MAX_OLY




This file uses mostly standard default values except for:

• #define ATMOSPHERIC_LOADINGenable pressure loading which is abused to include the initial geopotentialheight anomaly

• #define EXACT_CONSERVenable more accurate conservation properties to include the horizontalmass divergence in the free surface

• #define NONLIN_FRSURFenable the nonlinear free surface




3.12 Held-Suarez forcing atmospheric simulation

on a latitude-longitude grid 2.8◦ resolutionand on a cube-sphere grid with 32 square

cube faces.

This example illustrates the use of the MITgcm for large scale atmosphericcirculation simulation. Two simulations are described

• global atmospheric circulation on a latitude-longitude grid and

• global atmospheric circulation on a cube-sphere grid

The examples show how to use the isomorphic ’p-coordinate’ scheme in MITgcmto enable atmospheric simulation.

3.12.1 Overview

This example demonstrates using the MITgcm to simulate the planetary atmo-spheric circulation in two ways. In both cases the simulation is configured withflat orography. In the first case shown a 2.8◦ × 2.8◦ spherical polar horizontalgrid is employed. In the second case a cube-sphere horizontal grid is used thatprojects a cube with face size of 32× 32 onto a sphere. Five pressure corrdinatelevels are used in the vertical, ranging in thickness from 100 mb at the bottomof the atmosphere to 300 mb in the middle atmosphere. The total depth of theatmosphere is 1000mb. At this resolution, the configuration can be integratedforward for many years on a single processor desktop computer.

The model is forced by relaxation to a radiative equilibrium profile fromHeld and Suarez [26]. Initial conditions are a statically stable thermal gradientand no motion. The atmosphere in these experiments is dry and the only active“physics” are the terms in the Held and Suarez [26] formula. The MITgcm inter-mediate atmospheric physics package (see 6.10) and MITgcm high-end physicspackage ( see ??) are turned off. Altogether, this yields the following forcing(from Held and Suarez [26]) that is applied to the fluid:

�Fu = −kv(p)�u (3.62)Fθ = −kT (φ, p)[θ − θeq(φ, p)] (3.63)

(3.64)

where �Fu, Fθ, are the forcing terms in the zonal and meridional momentum andin the potential temperature equations respectively. The term kv in equation(3.62) applies a linear frictional drag (Rayleigh damping) that is active withinthe planetary boundary layer. It is defined so as to decay with height according

3.12. HELD-SUAREZ ATMOSPHERE MITGCM EXAMPLE 177

to

kv = kfmax(0, (pk/p0s − σb)/(1 − σb)) (3.65)

σb = 0.7 (3.66)kf = 1day−1 (3.67)

where pk is the pressure level of the cell center for level k and p0s is the

pressure at the base of the atmospheric column.


The model is configured in hydrostatic form. The domain is discretised with auniform grid spacing in latitude and longitude on the sphere Δφ = Δλ = 4◦, sothat there are ninety grid cells in the zonal and forty in the meridional direction.The internal model coordinate variables x and y are initialized according to

x = r cos(φ), Δx = r cos(Δφ) (3.68)y = rλ, Δx = rΔλ (3.69)

Arctic polar regions are not included in this experiment. Meridionally themodel extends from 80◦S to 80◦N. Vertically the model is configured withtwenty layers with the following thicknesses Δz1 = 50 m, Δz2 = 50 m, Δz3 =55 m, Δz4 = 60 m, Δz5 = 65 m, Δz6 = 70 m, Δz7 = 80 m, Δz8 = 95 m, Δz9 =120 m, Δz10 = 155 m, Δz11 = 200 m, Δz12 = 260 m, Δz13 = 320 m, Δz14 =400 m, Δz15 = 480 m, Δz16 = 570 m, Δz17 = 655 m, Δz18 = 725 m, Δz19 =775 m, Δz20 = 815 m (here the numeric subscript indicates the model level indexnumber, k). The implicit free surface form of the pressure equation describedin Marshall et. al [39] is employed. A Laplacian operator, ∇2, provides viscousdissipation. Thermal and haline diffusion is also represented by a Laplacianoperator.

Wind-stress forcing is added to the momentum equations for both the zonalflow, u and the meridional flow v, according to equations (3.62) and (??). Ther-modynamic forcing inputs are added to the equations for potential temperature,θ, and salinity, S, according to equations (3.63) and (??). This produces a setof equations solved in this configuration as follows:


Du

Dt− fv +

1ρ

∂p′


∂zAz

∂u

∂z=


(3.70)

Dv

Dt+ fu+

1ρ

∂p′

∂y−∇h · Ah∇hv − ∂

∂zAz

∂v

∂z=


(3.71)

∂η

∂t+ ∇h · �u = 0 (3.72)

Dθ


∂zΓ(Kz)

∂θ

∂z=


(3.73)

Ds

Dt−∇h ·Kh∇hs− ∂

∂zΓ(Kz)

∂s

∂z=


(3.74)

gρ0η +∫ 0

−z

ρ′dz = p

′(3.75)



Dt are the zonal and meridionalcomponents of the flow vector, �u, on the sphere. As described in MITgcm Nu-merical Solution Procedure 2, the time evolution of potential temperature, θ,equation is solved prognostically. The total pressure, p, is diagnosed by sum-ming pressure due to surface elevation η and the hydrostatic pressure.


The Laplacian dissipation coefficient, Ah, is set to 5 × 105ms−1. This value ischosen to yield a Munk layer width [1],

Mw = π(Ah

β)

13 (3.76)

of ≈ 600km. This is greater than the model resolution in low-latitudes, Δx ≈400km, ensuring that the frictional boundary layer is adequately resolved.

The model is stepped forward with a time step δtθ = 30 hours for thermody-namic variables and δtv = 40 minutes for momentum terms. With this timestep, the stability parameter to the horizontal Laplacian friction [1]

Sl = 4Ahδtv

Δx2 (3.77)

evaluates to 0.16 at a latitude of φ = 80◦, which is below the 0.3 upperlimit for stability. The zonal grid spacing Δx is smallest at φ = 80◦ whereΔx = r cos(φ)Δφ ≈ 77km.



Sl = 4Azδtv

Δz2 (3.78)

evaluates to 0.015 for the smallest model level spacing (Δz1 = 50m) which isagain well below the upper stability limit.

The values of the horizontal (Kh) and vertical (Kz) diffusion coefficients forboth temperature and salinity are set to 1 × 103 m2s−1 and 3 × 10−5 m2s−1

respectively. The stability limit related to Kh will be at φ = 80◦ where Δx ≈77km. Here the stability parameter

Sl =4Khδtθ

Δx2 (3.79)

evaluates to 0.07, well below the stability limit of Sl ≈ 0.5. The stabilityparameter related to Kz

Sl =4Kzδtθ

Δz2 (3.80)

evaluates to 0.005 for min(Δz) = 50m, well below the stability limit of Sl ≈ 0.5.


Si = f2δtv2 (3.81)

evaluates to 0.24 for f = 2ω sin(80◦) = 1.43 × 10−4 s−1, which is close to theSi < 1 upper limit for stability.


Sa =|�u|δtvΔx

(3.82)


The stability parameter for internal gravity waves propagating with a maximumspeed of cg = 10 ms−1 [1]

Sc =cgδtvΔx

(3.83)

evaluates to 3 × 10−1. This is close to the linear stability limit of 0.5.



The model configuration for this experiment resides under the directory verifi-cation/hs94.128x64x5. The experiment files

• input/data

• input/data.pkg

• input/eedata,

• input/windx.bin,

• input/windy.bin,

• input/salt.bin,

• input/theta.bin,

• input/SSS.bin,

• input/SST.bin,




• code/SIZE.h.




• Lines 7-10 and 11-14

tRef= 16.0 , 15.2 , 14.5 , 13.9 , 13.3 ,

· · ·set reference values for potential temperature and salinity at each modellevel in units of ◦C and ppt. The entries are ordered from surface to depth.Density is calculated from anomalies at each level evaluated with respectto the reference values set here.S/R INI THETA(ini theta.F)

• Line 15,


viscAz=1.E-3,



• Line 16,

viscAh=5.E5,

this line sets the horizontal Laplacian frictional dissipation coefficient to5 × 105m2s−1. Boundary conditions for this operator are specified later.

• Lines 17,





• Lines 9,



• Line 19,

diffKhT=1.E3,



• Line 20,

diffKzT=3.E-5,

this line sets the vertical diffusion coefficient for temperature to 3×10−5 m2s−1.The boundary condition on this operator is ∂


• Line 21,

diffKhS=1.E3,


this line sets the horizontal diffusion coefficient for salinity to 1000 m2s−1.The boundary condition on this operator is ∂


• Line 22,

diffKzS=3.E-5,

this line sets the vertical diffusion coefficient for salinity to 3×10−5 m2s−1.The boundary condition on this operator is ∂


• Lines 23-26

beta=1.E-11,· · ·These settings do not apply for this experiment.

• Line 27,

gravity=9.81,

Sets the gravitational acceleration coefficient to 9.81ms−1.S/R CALC PHI HYD (calc phi hyd.F)S/R INI CG2D (ini cg2d.F)S/R INI CG3D (ini cg3d.F)S/R INI PARMS (ini parms.F)S/R SOLVE FOR PRESSURE (solve for pressure.F)

• Line 28-29,



• Line 30,

eosType=’POLY3’,

Selects the third order polynomial form of the equation of state.S/R FIND RHO (find rho.F)S/R FIND ALPHA (find alpha.F)

• Line 31,

readBinaryPrec=32,


Sets format for reading binary input datasets holding model fields to use32-bit representation for floating-point numbers.S/R READ WRITE FLD (read write fld.F)S/R READ WRITE REC (read write rec.F)

• Line 36,

cg2dMaxIters=1000,


• Line 37,



• Line 42,

startTime=0,


• Line 43,

endTime=2808000.,

Sets the time (in seconds) at which this simulation will terminate. At theend of a simulation a checkpoint file is automatically written so that anumerical experiment can consist of multiple stages.

• Line 44,

#endTime=62208000000,

A commented out setting for endTime for a 2000 year simulation.

• Line 45,


deltaTmom=2400.0,

Sets the timestep δtv used in the momentum equations to 20 mins. Seesection ??.

S/R TIMESTEP(timestep.F)

• Line 46,

tauCD=321428.,

Sets the D-grid to C-grid coupling time scale τCD used in the momentumequations. See section ??.

S/R INI PARMS(ini parms.F)S/R CALC MOM RHS(calc mom rhs.F)

• Line 47,

deltaTtracer=108000.,

Sets the default timestep, δtθ, for tracer equations to 30 hours. See section??.

S/R TIMESTEP TRACER(timestep tracer.F)

• Line 47,



• Line 50,





1 # ====================


3 # ====================

4 #


6 &PARM01

7 tRef= 16.0 , 15.2 , 14.5 , 13.9 , 13.3 ,

8 12.4 , 11.3 , 9.9 , 8.4 , 6.7 ,

9 5.2 , 3.8 , 2.9 , 2.3 , 1.8 ,

10 1.5 , 1.1 , 0.8 , 0.66, 0.63,

11 sRef= 34.65, 34.75, 34.82, 34.87, 34.90,

12 34.90, 34.86, 34.78, 34.69, 34.60,

13 34.58, 34.62, 34.68, 34.72, 34.73,

14 34.74, 34.73, 34.73, 34.72, 34.72,

15 viscAz=1.E-3,

16 viscAh=5.E5,



19 diffKhT=1.E3,

20 diffKzT=3.E-5,

21 diffKhS=1.E3,

22 diffKzS=3.E-5,

23 beta=1.E-11,

24 f0=1.e-4,

25 tAlpha=2.E-4,

26 sBeta =7.4E-4,

27 gravity=9.81,





32 &

33


35 &PARM02



38 &

39


41 &PARM03

42 startTime=0,

43 endTime=622080000,

44 #endTime=2808000.,


46 tauCD=321428.,



48 deltaTClock =108000.0,

49 cAdjFreq=-1.,

50 abEps=0.1,

51 pChkptFreq=0.0,

52 chkptFreq=0.0,

53 dumpFreq=2592000.0,

54 dumpFreq=31104000.,


56 tauSaltClimRelax=2592000.0,

57 &

58


60 &PARM04



63 delZ= 5.000000e+01, 5.000000e+01, 5.500000e+01, 6.000000e+01,

64 6.500000e+01, 7.000000e+01, 8.000000e+01, 9.500000e+01,

65 1.200000e+02, 1.550000e+02, 2.000000e+02, 2.600000e+02,

66 3.200000e+02, 4.000000e+02, 4.800000e+02, 5.700000e+02,

67 6.550000e+02, 7.250000e+02, 7.750000e+02, 8.150000e+02,

68 phiMin=-80.,

69 delY=40*4,

70 delX=90*4,

71 &

72

73 # Input datasets

74 &PARM05

75 hydrogThetaFile=’theta.bin’,

76 hydrogSaltFile=’salt.bin’,

77 bathyFile=’topog.bin’,

78 zonalWindFile=’windx.bin’,

79 meridWindFile=’windy.bin’,

80 thetaClimFile=’SST.bin’

81 saltClimFile=’SSS.bin’

82 &









The input/topog.box file specifies a two-dimensional (x, y) map of depth values.For this experiment values are either 0m or −2000 m, corresponding respectivelyto a wall or to deep ocean. The file contains a raw binary stream of data that isenumerated in the same way as standard MITgcm two-dimensional, horizontalarrays. The included matlab program input/gendata.m gives a complete codefor creating the input/topog.box file.



• Line 39,

sNx=60,


• Line 40,

sNy=60,


• Line 49,

Nr=4,


1 C $Header: /u/gcmpack/manual/part3/case_studies/climatalogical_ogcm/code/SIZE.h.tex,v 1.1.1.1 2001/08

2 C $Name: $

3 C

4 C /==========================================================\


6 C |==========================================================|









14 C \==========================================================/













27 INTEGER sNx

28 INTEGER sNy

29 INTEGER OLx

30 INTEGER OLy

31 INTEGER nSx

32 INTEGER nSy

33 INTEGER nPx

34 INTEGER nPy

35 INTEGER Nx

36 INTEGER Ny

37 INTEGER Nr

38 PARAMETER (

39 & sNx = 90,

40 & sNy = 40,

41 & OLx = 3,

42 & OLy = 3,

43 & nSx = 1,

44 & nSy = 1,

45 & nPx = 1,

46 & nPy = 1,



49 & Nr = 20)




53 INTEGER MAX_OLX

54 INTEGER MAX_OLY














contain the code customisations and parameter settings for this experiments.Below we describe the customisations to these files associated with this experi-ment.


��

��

3.2 km

1000 m

z

x

y

Figure 3.10: Schematic of simulation domain for the surface driven convectionexperiment. The domain is doubly periodic with an initially uniform tempera-ture of 20 oC.

3.13 Surface Driven Convection

This experiment, figure 3.10, showcasing MITgcm’s non-hydrostatic capability,was designed to explore the temporal and spatial characteristics of convectionplumes as they might exist during a period of oceanic deep convection. It is

• non-hydrostatic

• doubly-periodic with cubic geometry

• has 50 m resolution

• Cartesian

• is on an f -plane

• with a linear equation of state

3.13.1 Overview

The model domain consists of an approximately 3 km square by 1 km deep boxof initially unstratified, resting fluid. The domain is doubly periodic.

The experiment has 20 levels in the vertical, each of equal thickness Δz =50 m (the horizontal resolution is also 50 m). The fluid is initially unstratifiedwith a uniform reference potential temperature θ = 20 oC. The equation of stateused in this experiment is linear

ρ = ρ0(1 − αθθ′) (3.84)

3.13. SURFACE DRIVEN CONVECTION 191

19.97

19.975

19.98

19.985

19.99

19.995

20

0.5 1 1.5 2 2.5 3

100

200

300

400

500

600

700

800

900

x km

dept

h km

Figure 3.11:

which is implemented in the model as a density anomaly equation

ρ′= −ρ0αθθ

′(3.85)

with ρ0 = 1000 kgm−3 and αθ = 2× 10−4 degrees−1. Integrated forward in thisconfiguration the model state variable theta is equivalent to either in-situ tem-perature, T , or potential temperature, θ. For consistency with other examples,in which the equation of state is non-linear, we use θ to represent temperaturehere. This is the quantity that is carried in the model core equations.

As the fluid in the surface layer is cooled (at a mean rate of 800 Wm2), itbecomes convectively unstable and overturns, at first close to the grid-scale, but,as the flow matures, on larger scales (figures 3.11 and 3.12), under the influenceof rotation (fo = 10−4 s−1) .

Model parameters are specified in file input/data. The grid dimensions areprescribed in code/SIZE.h. The forcing (file input/Qsurf.bin) is specified in abinary data file generated using the Matlab script input/gendata.m.


19.96

19.965

19.97

19.975

19.98

19.985

19.99

19.995

20

20.005

0.5 1 1.5 2 2.5 3

0.5

1

1.5

2

2.5

3

x km

y km

Figure 3.12:


3.13.2 Equations solved

The model is configured in nonhydrostatic form, that is, all terms in the NavierStokes equations are retained and the pressure field is found, subject to appro-priate bounday condintions, through inversion of a three-dimensional ellipticequation.

The implicit free surface form of the pressure equation described in Marshallet. al [39] is employed. A horizontal Laplacian operator ∇2

h provides viscousdissipation. The thermodynamic forcing appears as a sink in the potentialtemperature, θ, equation (??). This produces a set of equations solved in thisconfiguration as follows:

Du

Dt− fv +

1ρ

∂p′


∂zAz

∂u

∂z=

{0 (surface)0 (interior)

(3.86)

Dv

Dt+ fu+

1ρ

∂p′

∂y−∇h ·Ah∇hv − ∂

∂zAz

∂v

∂z=


(3.87)

Dw

Dt+ g

ρ′

ρ+

1ρ

∂p′

∂z−∇h ·Ah∇hw − ∂

∂zAz

∂w

∂z=


(3.88)

∂u

∂x+∂v

∂y+∂w

∂z+ = 0 (3.89)

Dθ


∂zKz

∂θ

∂z=


(3.90)

where u = DxDt , v = Dy

Dt and w = DzDt are the components of the flow vector in

directions x, y and z. The pressure is diagnosed through inversion (subject toappropriate boundary conditions) of a 3-D elliptic equation derived from thedivergence of the momentum equations and continuity (see section 1.3.6).

3.13.3 Discrete numerical configuration

The domain is discretised with a uniform grid spacing in each direction. Thereare 64 grid cells in directions x and y and 20 vertical levels thus the domaincomprises a total of just over 80 000 gridpoints.

3.13.4 Numerical stability criteria and other considera-tions

For a heat flux of 800 Wm2 and a rotation rate of 10−4 s−1 the plume-scale canbe expected to be a few hundred meters guiding our choice of grid resolution.This in turn restricts the timestep we can take. It is also desirable to minimisethe level of diffusion and viscosity we apply.


For this class of problem it is generally the advective time-scale which re-stricts the timestep.

For an extreme maximum flow speed of |�u| = 1ms−1, at a resolution of 50m, the implied maximum timestep for stability, δtu is

(3.91)

The choice of δt = 10 s is a safe 20 percent of this maximum.Interpreted in terms of a mixing-length hypothesis, a magnitude of Laplacian

diffusion coefficient κh(= κv) = 0.1 m2s−1 is consistent with an eddy velocityof 2 mm s−1 correlated over 50 m.

3.13.5 Experiment configuration

The model configuration for this experiment resides under the directory verifi-cation/convection/. The experiment files



• code/SIZE.h.

• input/data

• input/data.pkg

• input/eedata,

• input/Qsurf.bin,

contain the code customisations and parameter settings for this experiment.Below we describe these experiment-specific customisations.







Three lines are customized in this file. These prescribe the domain grid dimen-sions.

• Line 36,

sNx=64,


• Line 37,

sNy=64,


• Line 46,

Nr=20,


1 C /==========================================================\


3 C |==========================================================|








11 C \==========================================================/













24 INTEGER sNx

25 INTEGER sNy


26 INTEGER OLx

27 INTEGER OLy

28 INTEGER nSx

29 INTEGER nSy

30 INTEGER nPx

31 INTEGER nPy

32 INTEGER Nx

33 INTEGER Ny

34 INTEGER Nr

35 PARAMETER (

36 & sNx = 64,

37 & sNy = 64,

38 & OLx = 3,

39 & OLy = 3,

40 & nSx = 1,

41 & nSy = 1,

42 & nPx = 1,

43 & nPy = 1,



46 & Nr = 20)




50 INTEGER MAX_OLX

51 INTEGER MAX_OLY





• Line 4,

4 tRef=20*20.0,

this line sets the initial and reference values of potential temperature ateach model level in units of ◦C. Here the value is arbitrary since, in thiscase, the flow evolves independently of the absolute magnitude of the ref-erence temperature. For each depth level the initial and reference profileswill be uniform in x and y. The values specified are read into the variabletRef in the model code, by procedure S/R INI PARMS (ini parms.F). The temperature field is initialised, by procedure S/R INI THETA(ini theta.F) .

• Line 5,


5 sRef=20*35.0,

this line sets the initial and reference values of salinity at each modellevel in units of ppt. In this case salinity is set to an (arbitrary) uniformvalue of 35.0 ppt. However since, in this example, density is independentof salinity, an appropriatly defined initial salinity could provide a usefulpassive tracer. For each depth level the initial and reference profiles willbe uniform in x and y. The values specified are read into the variablesRef in the model code, by procedure S/R INI PARMS (ini parms.F). The salinity field is initialised, by procedure S/R INI SALT (ini salt.F).

• Line 6,

6 viscAh=0.1,

this line sets the horizontal laplacian dissipation coefficient to 0.1 m2s−1.Boundary conditions for this operator are specified later. The variableviscAh is read in the routine S/R INI PARMS (ini params.F) andapplied in routines S/R CALC MOM RHS (calc mom rhs.F) and S/RCALC GW (calc gw.F) .

• Line 7,

7 viscAz=0.1,

this line sets the vertical laplacian frictional dissipation coefficient to 0.1m2s−1. Boundary conditions for this operator are specified later. Thevariable viscAz is read in the routine S/R INI PARMS (ini parms.F)and is copied into model general vertical coordinate variable viscAr . Ateach time step, the viscous term contribution to the momentum equationsis calculated in routine S/R CALC DIFFUSIVITY (calc diffusivity.F) .

• Line 8,


this line selects a free-slip lateral boundary condition for the horizontallaplacian friction operator e.g. ∂u


along boundaries in x. The variable no slip sides is read in theroutine S/R INI PARMS (ini parms.F) and the boundary condition isevaluated in routine S/R CALC MOM RHS (calc mom rhs.F) .

• Lines 9,



this line selects a no-slip boundary condition for the bottom boundary con-dition in the vertical laplacian friction operator e.g. u = v = 0 at z = −H ,where H is the local depth of the domain. The variable no slip bottomis read in the routine S/R INI PARMS (ini parms.F) and is applied inthe routine S/R CALC MOM RHS (calc mom rhs.F) .

• Line 11,

diffKhT=0.1,

this line sets the horizontal diffusion coefficient for temperature to 0.1m2s−1. The boundary condition on this operator is ∂

∂x = ∂∂y = 0 at

all boundaries. The variable diffKhT is read in the routine S/RINI PARMS (ini parms.F) and used in routine S/R CALC GT (calc gt.F).

• Line 12,

diffKzT=0.1,

this line sets the vertical diffusion coefficient for temperature to 0.1 m2s−1.The boundary condition on this operator is ∂

∂z = 0 on all boundaries. Thevariable diffKzT is read in the routine S/R INI PARMS (ini parms.F). It is copied into model general vertical coordinate variable diffKrTwhich is used in routine S/R CALC DIFFUSIVITY (calc diffusivity.F).

• Line 13,

diffKhS=0.1,

this line sets the horizontal diffusion coefficient for salinity to 0.1 m2s−1.The boundary condition on this operator is ∂

∂x = ∂∂y = 0 on all bound-

aries. The variable diffKsT is read in the routine S/R INI PARMS(ini parms.F) and used in routine S/R CALC GS (calc gs.F) .

• Line 14,

diffKzS=0.1,

this line sets the vertical diffusion coefficient for temperature to 0.1 m2s−1.The boundary condition on this operator is ∂

∂z = 0 on all boundaries. Thevariable diffKzS is read in the routine S/R INI PARMS (ini parms.F). It is copied into model general vertical coordinate variable diffKrSwhich is used in routine S/R CALC DIFFUSIVITY (calc diffusivity.F).

• Line 15,


f0=1E-4,

this line sets the Coriolis parameter to 1 × 10−4 s−1. Since β = 0.0 thisvalue is then adopted throughout the domain.

• Line 16,

beta=0.E-11,

this line sets the the variation of Coriolis parameter with latitude to 0.

• Line 17,

tAlpha=2.E-4,

This line sets the thermal expansion coefficient for the fluid to 2 × 10−4

o C−1. The variable tAlpha is read in the routine S/R INI PARMS(ini parms.F) . The routine S/R FIND RHO (find rho.F) makes useof tAlpha.

• Line 18,

sBeta=0,

This line sets the saline expansion coefficient for the fluid to 0 consistentwith salt’s passive role in this example.

• Line 23-24,



• Line 25,

eosType=’LINEAR’,

Selects the linear form of the equation of state.

• Line 26,

nonHydrostatic=.TRUE.,

Selects for non-hydrostatic code.

• Line 27,


readBinaryPrec=64,

Sets format for reading binary input datasets holding model fields to use64-bit representation for floating-point numbers.

• Line 31,

cg2dMaxIters=1000,

Inactive - the pressure field in a non-hydrostatic simulation is invertedthrough a 3D elliptic equation.

• Line 32,


Inactive - the pressure field in a non-hydrostatic simulation is invertedthrough a 3D elliptic equation.

• Line 33,

cg3dMaxIters=40,

This line sets the maximum number of iterations the three-dimensional,conjugate gradient solver will use to 40, irrespective of the conver-gence criteria being met. Used in routine S/R CG3D (cg3d.F) .

• Line 34,


Sets the tolerance which the three-dimensional, conjugate gradient solverwill use to test for convergence in equation ?? to 1 × 10−9. The solverwill iterate until the tolerance falls below this value or until the maxi-mum number of solver iterations is reached. Used in routine S/R CG3D(cg3d.F) .

• Line 38,

startTime=0,


• Line 39,


nTimeSteps=8640.,

Sets the number of timesteps at which this simulation will terminate (inthis case 8640 timesteps or 1 day or δt = 10 s). At the end of a simulationa checkpoint file is automatically written so that a numerical experimentcan consist of multiple stages.

• Line 40,

deltaT=10,

Sets the timestep δt to 10 s.

• Line 51,

dXspacing=50.0,

Sets horizontal (x-direction) grid interval to 50 m.

• Line 52,

dYspacing=50.0,

Sets horizontal (y-direction) grid interval to 50 m.

• Line 53,

delZ=20*50.0,

Sets vertical grid spacing to 50 m. Must be consistent with code/SIZE.h.Here, 20 corresponds to the number of vertical levels.

• Line 57,

surfQfile=’Qsurf.bin’

This line specifies the name of the file from which the surface heat fluxis read. This file is a two-dimensional (x, y) map. It is assumed to con-tain 64-bit binary numbers giving the value of Q (W m2) to be appliedin each surface grid cell, ordered with the x coordinate varying fastest.The points are ordered from low coordinate to high coordinate for bothaxes. The matlab program input/gendata.m shows how to generate thesurface heat flux file used in the example. The variable Qsurf is readin the routine S/R INI PARMS (ini parms.F) and applied in S/R EX-TERNAL FORCING SURF (external forcing surf.F) where the flux isconverted to a temperature tendency.




3 &PARM01

4 tRef=20*20.0,

5 sRef=20*35.0,

6 viscAh=0.1,

7 viscAz=0.1,


9 no_slip_bottom=.FALSE.,

10 viscA4=0.E12,

11 diffKhT=0.1,

12 diffKzT=0.1,

13 diffKhS=0.1,

14 diffKzS=0.1,

15 f0=1E-4,

16 beta=0.E-11,

17 tAlpha=2.0E-4,

18 sBeta =0.,

19 gravity=9.81,

20 rhoConst=1000.0,

21 rhoNil=1000.0,

22 heatCapacity_Cp=3900.0,




26 nonHydrostatic=.TRUE.,


28 &


30 &PARM02



33 cg3dMaxIters=40,


35 &


37 &PARM03

38 nIter0=0,

39 nTimeSteps=1440,

40 deltaT=60,

41 abEps=0.1,

42 pChkptFreq=0.0,

43 chkptFreq=0.0,

44 dumpFreq=600,

45 monitorFreq=1.E-5,

46 &


48 &PARM04

49 usingCartesianGrid=.TRUE.,

50 usingSphericalPolarGrid=.FALSE.,


Figure 3.13:

51 dXspacing=50.0,

52 dYspacing=50.0,

53 delZ=20*50.0,

54 &

55 # Input datasets

56 &PARM05

57 surfQfile=’Qsurf.bin’,

58 &





3.13.5.7 File input/Qsurf.bin

The file input/Qsurf.bin specifies a two-dimensional (x, y) map of heat flux val-ues where Q = Qo × (0.5 + random number between 0 and 1).

In the example Qo = 800 W m−2 so that values of Q lie in the range 400 to1200 W m−2 with a mean of Qo. Although the flux models a loss, because it isdirected upwards, according to the model’s sign convention, Q is positive.

3.13.6 Running the example

3.13.6.1 Code download

In order to run the examples you must first download the code distribution.Instructions for downloading the code can be found in 3.2.

3.13.6.2 Experiment Location

This example experiments is located under the release sub-directory

verification/convection/

3.13.6.3 Running the Experiment

To run the experiment


1. Set the current directory to input/

% cd input

2. Verify that current directory is now correct

% pwd

You should see a response on the screen ending in

verification/convection/input

3. Run the genmake script to create the experiment Makefile

% ../../../tools/genmake -mods=../code

4. Create a list of header file dependencies in Makefile

% make depend

5. Build the executable file.

% make

6. Run the mitgcmuv executable

% ./mitgcmuv

3.14. GRAVITY PLUME ON A CONTINENTAL SLOPE 205

Figure 3.14: Temperature after 23 hours of cooling. The cold dense water ismixed with ambient water as it accelerates down the slope and hence is warmedthan the unmixed plume.

3.14 Gravity Plume On a Continental Slope

An important test of any ocean model is the ability to represent the flow ofdense fluid down a slope. One example of such a flow is a non-rotating gravityplume on a continental slope, forced by a limited area of surface cooling above acontinental shelf. Because the flow is non-rotating, a two dimensional model canbe used in the across slope direction. The experiment is non-hydrostatic anduses open-boundaries to radiate transients at the deep water end. (Dense flowdown a slope can also be forced by a dense inflow prescribed on the continentalshelf; this configuration is being implemented by the DOME (Dynamics of Over-flow Mixing and Entrainment) collaboration to compare solutions in differentmodels).

The fluid is initially unstratified. The surface buoyancy loss B0 (dimensionsof L2T−3) over a cross-shelf distance R causes vertical convective mixing andmodifies the density of the fluid by an amount

Δρ =B0ρ0t

gH(3.92)

where H is the depth of the shelf, g is the acceleration due to gravity, t is timesince onset of cooling and ρ0 is the reference density. Dense fluid slumps undergravity, with a flow speed close to the gravity wave speed:

U ∼√g′H ∼

√gΔρHρ0

∼√B0t (3.93)

A steady state is rapidly established in which the buoyancy flux out of the


cooling region is balanced by the surface buoyancy loss. Then

U ∼ (B0R)1/3 ; Δρ ∼ ρ0

gH(B0R)2/3 (3.94)

The Froude number of the flow on the shelf is close to unity (but in practiceslightly less than unity, giving subcritical flow). When the flow reaches theslope, it accelerates, so that it may become supercritical (provided the slopeangle α is steep enough). In this case, a hydraulic control is established at theshelf break. On the slope, where the Froude number is greater than one, andgradient Richardson number (defined as Ri ∼ g′h∗/U2 where h∗ is the thicknessof the interface between dense and ambient fluid) is reduced below 1/4, Kelvin-Helmholtz instability is possible, and leads to entrainment of ambient fluid intothe plume, modifying the density, and hence the acceleration down the slope.Kelvin-Helmholtz instability is suppressed at low Reynolds and Peclet numbersgiven by

Re ∼ Uh

ν∼ (B0R)1/3h

ν; Pe = RePr (3.95)

where h is the depth of the dense fluid on the slope. Hence this experiment iscarried out in the high Re, Pe regime. A further constraint is that the convectiveheat flux must be much greater than the diffusive heat flux (Nusselt number>> 1). Then

Nu =Uh∗

κ>> 1 (3.96)

Finally, since we have assumed that the convective mixing on the shelf occurs ina much shorter time than the horizontal equilibration, this implies H/R << 1.

Hence to summarize the important nondimensional parameters, and the lim-its we are considering:

H

R<< 1 ; Re >> 1 ; Pe >> 1 ; Nu >> 1 ; ; Ri < 1/4 (3.97)

In addition we are assuming that the slope is steep enough to provide suffi-cient acceleration to the gravity plume, but nonetheless much less that 1 : 1,since many Kelvin-Helmholtz billows appear on the slope, implying horizontallengthscale of the slope >> the depth of the dense fluid.

3.14.1 Configuration

The topography, spatial grid, forcing and initial conditions are all specified in bi-nary data files generated using a Matlab script called gendata.m and detailed insection 3.14.2. Other model parameters are specified in file data and data.obcsand detailed in section 3.14.4.

3.14.2 Binary input data

The domain is 200 m deep and 6.4 km across. Uniform resolution of 60×31/3 mis used in the vertical and variable resolution of the form shown in Fig. 3.15 with


0 1 2 3 4 5 610

15

20

25

30

35

40

X (km)

Δ x (m

)

Figure 3.15: Horizontal grid spacing, Δx, in the across-slope direction for thegravity plume experiment.

0 1 2 3 4 5 6

−200

−180

−160

−140

−120

−100

−80

−60

−40

−20

0

X (km)

Z (m)

Figure 3.16: Topography, h(x), used for the gravity plume experiment.


0 1 2 3 4 5 60

20

40

60

80

100

120

140

160

180

200

X (km)

Q (W

/m2 )

Figure 3.17: Upward surface heat flux, Q(x), used as forcing in the gravityplume experiment.

320 points is usedin the horizontal. The formula for Δx is:

Δx(i) = Δx1 + (Δx2 − Δx1)(1 + tanh(i− isw

))/2

where

Nx = 320Lx = 6400 (m)

Δx1 =23Lx

Nx(m)

Δx2 =Lx/2

Nx− Lx/2Δx1(m)

is = Lx/(2Δx1)w = 40

Here, Δx1 is the resolution on the shelf, Δx2 is the resolution in deep water andNx is the number of points in the horizontal.

The topography, shown in Fig. 3.16, is given by:

H(x) = −Ho + (Ho − hs)(1 + tanh(x− xs

Ls

))/2


where

Ho = 200 (m)hs = 40 (m)xs = 1500 + Lx/2 (m)

Ls =(Ho − hs)

2s(m)

s = 0.15

Here, s is the maximum slope, Ho is the maximum depth, hs is the shelf depth,xs is the lateral position of the shelf-break and Ls is the length-scale of theslope.

The forcing is through heat loss over the shelf, shown in Fig. 3.17 and takesthe form of a fixed flux with profile:

Q(x) = Qo(1 + tanh(x− xq

Lq

))/2

where

Qo = 200 (W m−2)xq = 2500 + Lx/2 (m)Lq = 100 (m)

Here, Qo, is the maximum heat flux, xq is the position of the cut-off and Lq isthe width of the cut-off.

The initial tempeture field is unstratified but with random perturbations,to induce convection early on in the run. The random perturbation are calcu-lated in computational space and because of the variable resolution introducesome spatial correlations but this does not matter for this experiment. Theperturbations have range 0 − 0.01 ◦K.

3.14.3 Code configuration

The computational domain (number of points) is specified in code/SIZE.h andis configured as a single tile of dimensions 320×1×60. There are no experimentspecific source files.

Optional code required to for this experiment are the non-hydrostatic algo-rithm and open-boundaries:

• Non-hydrostatic terms and algorithm are enabled with #define AL-LOW NONHYDROSTATIC in code/CPP OPTIONS.h and activatedwith nonHydrostatic=.TRUE., in namelist PARM01 of input/data.

• Open boundaries are enabled with #define ALLOW OBCS in code/CPP OPTIONS.hand activated with use OBCS=.TRUE, in namelist PACKAGES ofinput/data.pkg.


g 9.81 m s−2 acceleration due to gravityρo 999.8 kg m−3 reference densityα 2 × 10−4 K−1 expansion coefficientAh 1 × 10−2 m2s−1 horizontal viscosityAv 1 × 10−3 m2s−1 vertical viscosityκh 0 m2s−1 (explicit) horizontal diffusionκv 0 m2s−1 (explicit) vertical diffusion

Δt 20 s time stepΔz 3.33 m vertical grid spacingΔx 13.3 − 39.5 m horizontal grid spacing

Table 3.2: Model parameters used in the gravity plume experiment.

3.14.4 Model parameters

The model parameters (Table 3.2) are specified in input/data and if not as-sume the default values defined in model/src/set defaults.F. A linear equa-tion of state is used, eosType=’LINEAR’, but only temperature is active,sBeta=0.E-4. For the given heat flux, Qo, the buoyancy forcing is Bo =gαQρocp

∼ 10−7 m2s−3. Using R = 103 m, the shelf width, then this gives a veloc-ity scale of U ∼ 5 × 10−2 m s−1 for the initial front but will accelerate by anorder of magnitude over the slope. The temperature anomaly will be of orderΔθ ∼ 3 × 10−2 K. The viscosity is constant and gives a Reynolds number of100, using h = 20 m for the initial front and will be an order magnitude biggerover the slope. There is no explicit diffusion but a non-linear advection schemeis used for temperature which adds enough diffusion so as to keep the modelstable. The time-step is set to 20 s and gives Courant number order one whenthe flow reaches the bottom of the slope.

3.14.5 Build and run the model

Build the model per usual. For example:

% cd verification/plume_on_slope% mkdir build% cd build% ../../../tools/genmake -mods=../code -disable=gmredi,kpp,zonal_filt,shap_filt

% make depend% make

When compilation is complete, run the model as usual, for example:

% cd ../% mkdir run


% cp input/* build/mitgcmuv run/% cd run% ./mitgcmuv > output.txt


3.15 Centennial Time Scale Tracer Injection

3.15.1 Introduction

This document describes the fourth example MITgcm experiment. This exampleillustrates the use of the MITgcm to perform sensitivity analysis in a large scaleocean circulation simulation.

3.15.2 Overview

This example experiment demonstrates using the MITgcm to simulate the plan-etary ocean circulation. The simulation is configured with realistic geographyand bathymetry on a 4◦ × 4◦ spherical polar grid. Twenty vertical layers areused in the vertical, ranging in thickness from 50 m at the surface to 815 m atdepth, giving a maximum model depth of 6 km. At this resolution, the config-uration can be integrated forward for thousands of years on a single processordesktop computer.

The model is forced with climatological wind stress data and surface flux datafrom Da Silva [10]. Climatological data from Levitus [36] is used to initializethe model hydrography. Levitus data is also used throughout the calculationto derive air-sea fluxes of heat at the ocean surface. These fluxes are combinedwith climatological estimates of surface heat flux and fresh water, resulting ina mixed boundary condition of the style described in Haney [25]. Altogether,this yields the following forcing applied in the model surface layer.

Fu =τx

ρ0Δzs(3.98)

Fv =τy

ρ0Δzs(3.99)

Fθ = −λθ(θ − θ∗) − 1Cpρ0Δzs

Q (3.100)

Fs = −λs(S − S∗) +S0

Δzs(E − P −R) (3.101)

where Fu, Fv, Fθ, Fs are the forcing terms in the zonal and meridional momen-tum and in the potential temperature and salinity equations respectively. Theterm Δzs represents the top ocean layer thickness. It is used in conjunctionwith the reference density, ρ0 (here set to 999.8 kgm−3), the reference salinity,S0 (here set to 35ppt), and a specific heat capacity Cp to convert wind-stressfluxes given in Nm−2,

The configuration is illustrated in figure ??.

3.15. CENTENNIAL TIME SCALE TRACER INJECTION 213


The model is configured in hydrostatic form. The domain is discretised with auniform grid spacing in latitude and longitude of Δx = Δy = 4◦, so that thereare ninety grid cells in the x and forty in the y direction (Arctic polar regionsare not included in this experiment). Vertically the model is configured withtwenty layers with the following thicknesses Δz1 = 50 m, Δz2 = 50 m, Δz3 =55 m, Δz4 = 60 m, Δz5 = 65 m, Δz6 = 70 m, Δz7 = 80 m, Δz8 = 95 m, Δz9 =120 m, Δz10 = 155 m, Δz11 = 200 m, Δz12 = 260 m, Δz13 = 320 m, Δz14 =400 m, Δz15 = 480 m, Δz16 = 570 m, Δz17 = 655 m, Δz18 = 725 m, Δz19 =775 m, Δz20 = 815 m (here the numeric subscript indicates the model level indexnumber, k). The implicit free surface form of the pressure equation describedin Marshall et. al [39] is employed. A Laplacian operator, ∇2, provides viscousdissipation. Thermal and haline diffusion is also represented by a Laplacianoperator.

Wind-stress momentum inputs are added to the momentum equations forboth the zonal flow, u and the meridional flow v, according to equations (3.98)and (3.99). Thermodynamic forcing inputs are added to the equations for poten-tial temperature, θ, and salinity, S, according to equations (3.100) and (3.101).This produces a set of equations solved in this configuration as follows:

Du

Dt− fv +

1ρ

∂p′

∂x−Ah∇2

hu−Az∂2u

∂z2= Fu (3.102)

Dv

Dt+ fu+

1ρ

∂p′

∂y−Ah∇2

hv −Az∂2v

∂z2= Fv (3.103)

∂η

∂t+ ∇h · �u = 0 (3.104)

Dθ

Dt−Kh∇2

hθ − Γ(Kz)∂2θ

∂z2= Fθ (3.105)

Ds

Dt−Kh∇2

hs− Γ(Kz)∂2s

∂z2= Fs (3.106)

gρ0η +∫ 0

−z

ρ′dz = p

′(3.107)

(3.108)

where u and v are the x and y components of the flow vector �u. The suffices s, iindicate surface and interior model levels respectively. As described in MITgcmNumerical Solution Procedure 2, the time evolution of potential temperature,θ, equation is solved prognostically. The total pressure, p, is diagnosed by sum-ming pressure due to surface elevation η and the hydrostatic pressure.



The Laplacian dissipation coefficient, Ah, is set to 400ms−1. This value ischosen to yield a Munk layer width [1],

Mw = π(Ah

β)

13 (3.109)

of ≈ 100km. This is greater than the model resolution in mid-latitudes Δx,ensuring that the frictional boundary layer is well resolved.

The model is stepped forward with a time step δt = 1200secs. With this timestep the stability parameter to the horizontal Laplacian friction [1]

Sl = 4Ahδt

Δx2 (3.110)



Sl = 4Azδt

Δz2(3.111)

evaluates to 4.8 × 10−5 which is again well below the upper limit. The valuesof Ah and Az are also used for the horizontal (Kh) and vertical (Kz) diffusioncoefficients for temperature respectively.


Si = f2δt2 (3.112)



Sa =|�u|δtΔx

(3.113)


The stability parameter for internal gravity waves [1]

Sc =cgδt

Δx(3.114)

evaluates to 5 × 10−2. This is well below the linear stability limit of 0.25.




• input/data

• input/data.pkg

• input/eedata,





• code/SIZE.h.




• Line 4,

tRef=20.,10.,8.,6.,

this line sets the initial and reference values of potential temperature ateach model level in units of ◦C. The entries are ordered from surfaceto depth. For each depth level the initial and reference profiles will beuniform in x and y.

S/R INI THETA(ini theta.F)

• Line 6,

viscAz=1.E-2,



• Line 7,


viscAh=4.E2,

this line sets the horizontal Laplacian frictional dissipation coefficient to1× 10−2m2s−1. Boundary conditions for this operator are specified later.

• Lines 8,





• Lines 9,



• Line 10,

diffKhT=4.E2,


∂x = ∂∂y = 0 at all boundaries.

• Line 11,

diffKzT=1.E-2,

this line sets the vertical diffusion coefficient for temperature to 10−2 m2s−1.The boundary condition on this operator is ∂

∂z = 0 on all boundaries.

• Line 13,

tAlpha=2.E-4,

This line sets the thermal expansion coefficient for the fluid to 2×10−4 degrees−1

S/R FIND RHO(find rho.F)

• Line 18,

eosType=’LINEAR’

This line selects the linear form of the equation of state.

S/R FIND RHO(find rho.F)


• Line 40,

usingSphericalPolarGrid=.TRUE.,

This line requests that the simulation be performed in a spherical polarcoordinate system. It affects the interpretation of grid input parameters,for example delX and delY and causes the grid generation routines toinitialize an internal grid based on spherical polar geometry.

S/R INI SPEHRICAL POLAR GRID(ini spherical polar grid.F)

• Line 41,

phiMin=0.,

This line sets the southern boundary of the modeled domain to 0◦ latitude.This value affects both the generation of the locally orthogonal grid thatthe model uses internally and affects the initialization of the coriolis force.Note - it is not required to set a longitude boundary, since the absolutelongitude does not alter the kernel equation discretisation.

• Line 42,

delX=60*1.,

This line sets the horizontal grid spacing between each y-coordinate linein the discrete grid to 1◦ in longitude.

• Line 43,

delY=60*1.,

This line sets the horizontal grid spacing between each y-coordinate linein the discrete grid to 1◦ in latitude.

• Line 44,

delZ=500.,500.,500.,500.,

This line sets the vertical grid spacing between each z-coordinate line inthe discrete grid to 500 m, so that the total model depth is 2 km. Thevariable delZ is copied into the internal model coordinate variable delR

S/R INI VERTICAL GRID(ini vertical grid.F)

• Line 47,




• Line 50,











The input/topog.box file specifies a two-dimensional (x, y) map of depth values.For this experiment values are either 0m or −2000 m, corresponding respectivelyto a wall or to deep ocean. The file contains a raw binary stream of data that isenumerated in the same way as standard MITgcm two-dimensional, horizontal


arrays. The included matlab program input/gendata.m gives a complete codefor creating the input/topog.box file.



• Line 39,

sNx=60,


• Line 40,

sNy=60,


• Line 49,

Nr=4,














3.16 Doing it yourself: customizing the code

When you are ready to run the model in the configuration you want, the easiestthing is to use and adapt the setup of the case studies experiment (describedpreviously) that is the closest to your configuration. Then, the amount of setupwill be minimized. In this section, we focus on the setup relative to the “numer-ical model” part of the code (the setup relative to the “execution environment”part is covered in the parallel implementation section) and on the variables andparameters that you are likely to change.

3.16.1 Building/compiling the code elsewhere

In the example above (section 3.5) we built the executable in the input directoryof the experiment for convenience. You can also configure and compile the codein other locations, for example on a scratch disk with out having to copy theentire source tree. The only requirement to do so is you have genmake2 in yourpath or you know the absolute path to genmake2.

The following sections outline some possible methods of organizing yoursource and data.

3.16.1.1 Building from the ../code directory

This is just as simple as building in the input/ directory:

% cd verification/exp2/code% ../../../tools/genmake2% make depend% make

However, to run the model the executable (mitgcmuv) and input files must bein the same place. If you only have one calculation to make:

% cd ../input% cp ../code/mitgcmuv ./% ./mitgcmuv > output.txt

or if you will be making multiple runs with the same executable:

% cd ../% cp -r input run1% cp code/mitgcmuv run1% cd run1% ./mitgcmuv > output.txt

3.16.1.2 Building from a new directory

Since the input directory contains input files it is often more useful to keep inputpristine and build in a new directory within verification/exp2/:

3.16. CUSTOMIZING MITGCM 221

% cd verification/exp2% mkdir build% cd build% ../../../tools/genmake2 -mods=../code% make depend% make

This builds the code exactly as before but this time you need to copy either theexecutable or the input files or both in order to run the model. For example,

% cp ../input/* ./% ./mitgcmuv > output.txt

or if you tend to make multiple runs with the same executable then running ina new directory each time might be more appropriate:

% cd ../% mkdir run1% cp build/mitgcmuv run1/% cp input/* run1/% cd run1% ./mitgcmuv > output.txt

3.16.1.3 Building on a scratch disk

Model object files and output data can use up large amounts of disk space so itis often the case that you will be operating on a large scratch disk. Assumingthe model source is in /MITgcm then the following commands will build themodel in /scratch/exp2-run1:

% cd /scratch/exp2-run1% ~/MITgcm/tools/genmake2 -rootdir=~/MITgcm \-mods=~/MITgcm/verification/exp2/code

% make depend% make

To run the model here, you’ll need the input files:

% cp ~/MITgcm/verification/exp2/input/* ./% ./mitgcmuv > output.txt

As before, you could build in one directory and make multiple runs of theone experiment:

% cd /scratch/exp2% mkdir build% cd build% ~/MITgcm/tools/genmake2 -rootdir=~/MITgcm \-mods=~/MITgcm/verification/exp2/code


% make depend% make% cd ../% cp -r ~/MITgcm/verification/exp2/input run2% cd run2% ./mitgcmuv > output.txt

3.16.2 Using genmake2

To compile the code, first use the program genmake2 (located in the tools di-rectory) to generate a Makefile. genmake2 is a shell script written to work withall “sh”–compatible shells including bash v1, bash v2, and Bourne. Internally,genmake2 determines the locations of needed files, the compiler, compiler op-tions, libraries, and Unix tools. It relies upon a number of “optfiles” located inthe tools/build options directory.

The purpose of the optfiles is to provide all the compilation options forparticular “platforms” (where “platform” roughly means the combination of thehardware and the compiler) and code configurations. Given the combinationsof possible compilers and library dependencies (eg. MPI and NetCDF) theremay be numerous optfiles available for a single machine. The naming schemefor the majority of the optfiles shipped with the code is

OS HARDWARE COMPILER

where

OS is the name of the operating system (generally the lower-case output of the’uname’ command)

HARDWARE is a string that describes the CPU type and corresponds tooutput from the ’uname -m’ command:

ia32 is for “x86” machines such as i386, i486, i586, i686, and athlon

ia64 is for Intel IA64 systems (eg. Itanium, Itanium2)

amd64 is AMD x86 64 systems

ppc is for Mac PowerPC systems

COMPILER is the compiler name (generally, the name of the FORTRANexecutable)

In many cases, the default optfiles are sufficient and will result in usableMakefiles. However, for some machines or code configurations, new “optfiles”must be written. To create a new optfile, it is generally best to start with one ofthe defaults and modify it to suit your needs. Like genmake2, the optfiles are allwritten using a simple “sh”–compatible syntax. While nearly all variables usedwithin genmake2 may be specified in the optfiles, the critical ones that shouldbe defined are:


FC the FORTRAN compiler (executable) to use

DEFINES the command-line DEFINE options passed to the compiler

CPP the C pre-processor to use

NOOPTFLAGS options flags for special files that should not be optimized

For example, the optfile for a typical Red Hat Linux machine (“ia32” archi-tecture) using the GCC (g77) compiler is

FC=g77DEFINES=’-D_BYTESWAPIO -DWORDLENGTH=4’CPP=’cpp -traditional -P’NOOPTFLAGS=’-O0’# For IEEE, use the "-ffloat-store" optionif test "x$IEEE" = x ; then

FFLAGS=’-Wimplicit -Wunused -Wuninitialized’FOPTIM=’-O3 -malign-double -funroll-loops’

elseFFLAGS=’-Wimplicit -Wunused -ffloat-store’FOPTIM=’-O0 -malign-double’

fi

If you write an optfile for an unrepresented machine or compiler, you arestrongly encouraged to submit the optfile to the MITgcm project for inclusion.Please send the file to the

[email protected]

mailing list.In addition to the optfiles, genmake2 supports a number of helpful command-

line options. A complete list of these options can be obtained from:

% genmake2 -h

The most important command-line options are:

--optfile=/PATH/FILENAME specifies the optfile that should be used for a par-ticular build.

If no ”optfile” is specified (either through the command line or the MIT-GCM OPTFILE environment variable), genmake2 will try to make a rea-sonable guess from the list provided in tools/build options. The methodused for making this guess is to first determine the combination of op-erating system and hardware (eg. ”linux ia32”) and then find a workingFORTRAN compiler within the user’s path. When these three items havebeen identified, genmake2 will try to find an optfile that has a matchingname.


--pdefault=’PKG1 PKG2 PKG3 ...’ specifies the default set of packages to beused. The normal order of precedence for packages is as follows:

1. If available, the command line (--pdefault) settings over-rule anyothers.

2. Next, genmake2 will look for a file named “packages.conf” in thelocal directory or in any of the directories specified with the --modsoption.

3. Finally, if neither of the above are available, genmake2 will use the/pkg/pkg default file.

--pdepend=/PATH/FILENAME specifies the dependency file used for packages.

If not specified, the default dependency file pkg/pkg depend is used. Thesyntax for this file is parsed on a line-by-line basis where each line containeseither a comment (”#”) or a simple ”PKGNAME1 (+—-)PKGNAME2”pairwise rule where the ”+” or ”-” symbol specifies a ”must be used with”or a ”must not be used with” relationship, respectively. If no rule isspecified, then it is assumed that the two packages are compatible andwill function either with or without each other.

--adof=/path/to/file specifies the ”adjoint” or automatic differentiation op-tions file to be used. The file is analogous to the “optfile” defined abovebut it specifies information for the AD build process.

The default file is located in tools/adjoint options/adjoint default and itdefines the ”TAF” and ”TAMC” compilers. An alternate version is alsoavailable at tools/adjoint options/adjoint staf that selects the newer ”STAF”compiler. As with any compilers, it is helpful to have their directorieslisted in your $PATH environment variable.

--mods=’DIR1 DIR2 DIR3 ...’ specifies a list of directories containing “mod-ifications”. These directories contain files with names that may (or maynot) exist in the main MITgcm source tree but will be overridden by anyidentically-named sources within the “MODS” directories.

The order of precedence for this ”name-hiding” is as follows:

• “MODS” directories (in the order given)

• Packages either explicitly specified or provided by default (in theorder given)

• Packages included due to package dependencies (in the order thatthat package dependencies are parsed)

• The ”standard dirs” (which may have been specified by the “-standarddirs”option)

--mpi This option enables certain MPI features (using CPP #defines) withinthe code and is necessary for MPI builds (see Section 3.16.3).


--make=/path/to/gmake Due to the poor handling of soft-links and other bugscommon with the make versions provided by commercial Unix vendors,GNU make (sometimes called gmake) should be preferred. This optionprovides a means for specifying the make executable to be used.

--bash=/path/to/sh On some (usually older UNIX) machines, the “bash”shell is unavailable. To run on these systems, genmake2 can be invokedusing an “sh” (that is, a Bourne, POSIX, or compatible) shell. The syntaxin these circumstances is:

% /bin/sh genmake2 -bash=/bin/sh [...options...]

where /bin/sh can be replaced with the full path and name of the desiredshell.

3.16.3 Building with MPI

Building MITgcm to use MPI libraries can be complicated due to the varietyof different MPI implementations available, their dependencies or interactionswith different compilers, and their often ad-hoc locations within file systems.For these reasons, its generally a good idea to start by finding and reading thedocumentation for your machine(s) and, if necessary, seeking help from yourlocal systems administrator.

The steps for building MITgcm with MPI support are:

1. Determine the locations of your MPI-enabled compiler and/or MPI li-braries and put them into an options file as described in Section 3.16.2.One can start with one of the examples in:

MITgcm/tools/build options/

such as linux ia32 g77+mpi cg01 or linux ia64 efc+mpi and then editit to suit the machine at hand. You may need help from your user guideor local systems administrator to determine the exact location of the MPIlibraries. If libraries are not installed, MPI implementations and relatedtools are available including:

• MPICH

• LAM/MPI

• MPIexec

2. Build the code with the genmake2 -mpi option (see Section 3.16.2) usingcommands such as:

% ../../../tools/genmake2 -mods=../code -mpi -of=YOUR_OPTFILE

% make depend

% make


3. Run the code with the appropriate MPI “run” or “exec” program providedwith your particular implementation of MPI. Typical MPI packages suchas MPICH will use something like:

% mpirun -np 4 -machinefile mf ./mitgcmuv

Sightly more complicated scripts may be needed for many machines sinceexecution of the code may be controlled by both the MPI library and a jobscheduling and queueing system such as PBS, LoadLeveller, Condor, orany of a number of similar tools. A few example scripts (those used for ourregular verification runs) are available at: http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm contrib/test

An example of the above process on the MITgcm cluster (“cg01”) using theGNU g77 compiler and the mpich MPI library is:

% cd MITgcm/verification/exp5

% mkdir build

% cd build

% ../../../tools/genmake2 -mpi -mods=../code \

-of=../../../tools/build_options/linux_ia32_g77+mpi_cg01

% make depend

% make

% cd ../input

% /usr/local/pkg/mpi/mpi-1.2.4..8a-gm-1.5/g77/bin/mpirun.ch_gm \

-machinefile mf --gm-kill 5 -v -np 2 ../build/mitgcmuv

3.16.4 Configuration and setup

The CPP keys relative to the “numerical model” part of the code are all definedand set in the file CPP OPTIONS.h in the directory model/inc or in one of thecode directories of the case study experiments under verification. The modelparameters are defined and declared in the file model/inc/PARAMS.h and theirdefault values are set in the routine model/src/set defaults.F. The default valuescan be modified in the namelist file data which needs to be located in thedirectory where you will run the model. The parameters are initialized in theroutine model/src/ini parms.F. Look at this routine to see in what part of thenamelist the parameters are located.

In what follows the parameters are grouped into categories related to thecomputational domain, the equations solved in the model, and the simulationcontrols.

3.16.5 Computational domain, geometry and time-discretization

dimensions

The number of points in the x, y, and r directions are represented by thevariables sNx, sNy and Nr respectively which are declared and set inthe file model/inc/SIZE.h. (Again, this assumes a mono-processor calcu-lation. For multiprocessor calculations see the section on parallel imple-mentation.)


grid

Three different grids are available: cartesian, spherical polar, and curvi-linear (which includes the cubed sphere). The grid is set through thelogical variables usingCartesianGrid, usingSphericalPolarGrid, andusingCurvilinearGrid. In the case of spherical and curvilinear grids, thesouthern boundary is defined through the variable phiMin which corre-sponds to the latitude of the southern most cell face (in degrees). Theresolution along the x and y directions is controlled by the 1D arrays delxand dely (in meters in the case of a cartesian grid, in degrees otherwise).The vertical grid spacing is set through the 1D array delz for the ocean (inmeters) or delp for the atmosphere (in Pa). The variable Ro SeaLevelrepresents the standard position of Sea-Level in “R” coordinate. This istypically set to 0m for the ocean (default value) and 105Pa for the atmo-sphere. For the atmosphere, also set the logical variable groundAtK1 to’.TRUE.’ which puts the first level (k=1) at the lower boundary (ground).

For the cartesian grid case, the Coriolis parameter f is set through thevariables f0 and beta which correspond to the reference Coriolis parame-ter (in s−1) and ∂f

∂y (in m−1s−1) respectively. If beta is set to a nonzerovalue, f0 is the value of f at the southern edge of the domain.

topography - full and partial cells

The domain bathymetry is read from a file that contains a 2D (x,y) map ofdepths (in m) for the ocean or pressures (in Pa) for the atmosphere. Thefile name is represented by the variable bathyFile. The file is assumedto contain binary numbers giving the depth (pressure) of the model ateach grid cell, ordered with the x coordinate varying fastest. The pointsare ordered from low coordinate to high coordinate for both axes. Themodel code applies without modification to enclosed, periodic, and doubleperiodic domains. Periodicity is assumed by default and is suppressed bysetting the depths to 0m for the cells at the limits of the computationaldomain (note: not sure this is the case for the atmosphere). The precisionwith which to read the binary data is controlled by the integer variablereadBinaryPrec which can take the value 32 (single precision) or 64(double precision). See the matlab program gendata.m in the input direc-tories under verification to see how the bathymetry files are generated forthe case study experiments.

To use the partial cell capability, the variable hFacMin needs to be setto a value between 0 and 1 (it is set to 1 by default) corresponding tothe minimum fractional size of the cell. For example if the bottom cellis 500m thick and hFacMin is set to 0.1, the actual thickness of the cell(i.e. used in the code) can cover a range of discrete values 50m apart from50m to 500m depending on the value of the bottom depth (in bathyFile)at this point.

Note that the bottom depths (or pressures) need not coincide with themodels levels as deduced from delz or delp. The model will interpolate


the numbers in bathyFile so that they match the levels obtained fromdelz or delp and hFacMin.

(Note: the atmospheric case is a bit more complicated than what is writtenhere I think. To come soon...)

time-discretization

The time steps are set through the real variables deltaTMom and deltaT-tracer (in s) which represent the time step for the momentum and tracerequations, respectively. For synchronous integrations, simply set the twovariables to the same value (or you can prescribe one time step onlythrough the variable deltaT). The Adams-Bashforth stabilizing parame-ter is set through the variable abEps (dimensionless). The stagger baro-clinic time stepping can be activated by setting the logical variable stag-gerTimeStep to ’.TRUE.’.

3.16.6 Equation of state

First, because the model equations are written in terms of perturbations, areference thermodynamic state needs to be specified. This is done through the1D arrays tRef and sRef. tRef specifies the reference potential temperatureprofile (in oC for the ocean and oK for the atmosphere) starting from the levelk=1. Similarly, sRef specifies the reference salinity profile (in ppt) for the oceanor the reference specific humidity profile (in g/kg) for the atmosphere.

The form of the equation of state is controlled by the character variablesbuoyancyRelation and eosType. buoyancyRelation is set to ’OCEANIC’by default and needs to be set to ’ATMOSPHERIC’ for atmosphere simulations.In this case, eosType must be set to ’IDEALGAS’. For the ocean, two formsof the equation of state are available: linear (set eosType to ’LINEAR’) anda polynomial approximation to the full nonlinear equation ( set eosType to’POLYNOMIAL’). In the linear case, you need to specify the thermal and ha-line expansion coefficients represented by the variables tAlpha (in K−1) andsBeta (in ppt−1). For the nonlinear case, you need to generate a file ofpolynomial coefficients called POLY3.COEFFS. To do this, use the programutils/knudsen2/knudsen2.f under the model tree (a Makefile is available in thesame directory and you will need to edit the number and the values of thevertical levels in knudsen2.f so that they match those of your configuration).

There there are also higher polynomials for the equation of state:

’UNESCO’: The UNESCO equation of state formula of Fofonoff and Millard[15]. This equation of state assumes in-situ temperature, which is not amodel variable; its use is therefore discouraged, and it is only listed forcompleteness.

’JMD95Z’: A modified UNESCO formula by Jackett and McDougall [33], whichuses the model variable potential temperature as input. The ’Z’ indicatesthat this equation of state uses a horizontally and temporally constantpressure p0 = −gρ0z.


’JMD95P’: A modified UNESCO formula by Jackett and McDougall [33], whichuses the model variable potential temperature as input. The ’P’ indicatesthat this equation of state uses the actual hydrostatic pressure of the lasttime step. Lagging the pressure in this way requires an additional pickupfile for restarts.

’MDJWF’: The new, more accurate and less expensive equation of state by Mc-Dougall et al. [41]. It also requires lagging the pressure and therefore anadditional pickup file for restarts.

For none of these options an reference profile of temperature or salinity is re-quired.

3.16.7 Momentum equations

In this section, we only focus for now on the parameters that you are likelyto change, i.e. the ones relative to forcing and dissipation for example. Thedetails relevant to the vector-invariant form of the equations and the variousadvection schemes are not covered for the moment. We assume that you usethe standard form of the momentum equations (i.e. the flux-form) with the de-fault advection scheme. Also, there are a few logical variables that allow you toturn on/off various terms in the momentum equation. These variables are calledmomViscosity, momAdvection, momForcing, useCoriolis, momPres-sureForcing, momStepping and metricTerms and are assumed to be set to’.TRUE.’ here. Look at the file model/inc/PARAMS.h for a precise definitionof these variables.

initialization

The velocity components are initialized to 0 unless the simulation is start-ing from a pickup file (see section on simulation control parameters).

forcing

This section only applies to the ocean. You need to generate wind-stressdata into two files zonalWindFile and meridWindFile correspondingto the zonal and meridional components of the wind stress, respectively(if you want the stress to be along the direction of only one of the modelhorizontal axes, you only need to generate one file). The format of thefiles is similar to the bathymetry file. The zonal (meridional) stress dataare assumed to be in Pa and located at U-points (V-points). As for thebathymetry, the precision with which to read the binary data is controlledby the variable readBinaryPrec. See the matlab program gendata.m inthe input directories under verification to see how simple analytical windforcing data are generated for the case study experiments.

There is also the possibility of prescribing time-dependent periodic forc-ing. To do this, concatenate the successive time records into a single file


(for each stress component) ordered in a (x,y,t) fashion and set the fol-lowing variables: periodicExternalForcing to ’.TRUE.’, externForc-ingPeriod to the period (in s) of which the forcing varies (typically 1month), and externForcingCycle to the repeat time (in s) of the forc-ing (typically 1 year – note: externForcingCycle must be a multipleof externForcingPeriod). With these variables set up, the model willinterpolate the forcing linearly at each iteration.

dissipation

The lateral eddy viscosity coefficient is specified through the variableviscAh (in m2s−1). The vertical eddy viscosity coefficient is specifiedthrough the variable viscAz (in m2s−1) for the ocean and viscAp (inPa2s−1) for the atmosphere. The vertical diffusive fluxes can be computedimplicitly by setting the logical variable implicitViscosity to ’.TRUE.’.In addition, biharmonic mixing can be added as well through the variableviscA4 (in m4s−1). On a spherical polar grid, you might also need to setthe variable cosPower which is set to 0 by default and which representsthe power of cosine of latitude to multiply viscosity. Slip or no-slip con-ditions at lateral and bottom boundaries are specified through the logicalvariables no slip sides and no slip bottom. If set to ’.FALSE.’, free-slip boundary conditions are applied. If no-slip boundary conditions areapplied at the bottom, a bottom drag can be applied as well. Two formsare available: linear (set the variable bottomDragLinear in s−1) andquadratic (set the variable bottomDragQuadratic in m−1).The Fourier and Shapiro filters are described elsewhere.

C-D scheme

If you run at a sufficiently coarse resolution, you will need the C-D schemefor the computation of the Coriolis terms. The variable tauCD, whichrepresents the C-D scheme coupling timescale (in s) needs to be set.

calculation of pressure/geopotential

First, to run a non-hydrostatic ocean simulation, set the logical vari-able nonHydrostatic to ’.TRUE.’. The pressure field is then invertedthrough a 3D elliptic equation. (Note: this capability is not availablefor the atmosphere yet.) By default, a hydrostatic simulation is assumedand a 2D elliptic equation is used to invert the pressure field. The pa-rameters controlling the behaviour of the elliptic solvers are the vari-ables cg2dMaxIters and cg2dTargetResidual for the 2D case andcg3dMaxIters and cg3dTargetResidual for the 3D case. You proba-bly won’t need to alter the default values (are we sure of this?).For the calculation of the surface pressure (for the ocean) or surfacegeopotential (for the atmosphere) you need to set the logical variablesrigidLid and implicitFreeSurface (set one to ’.TRUE.’ and the otherto ’.FALSE.’ depending on how you want to deal with the ocean upperor atmosphere lower boundary).


3.16.8 Tracer equations

This section covers the tracer equations i.e. the potential temperature equationand the salinity (for the ocean) or specific humidity (for the atmosphere) equa-tion. As for the momentum equations, we only describe for now the parametersthat you are likely to change. The logical variables tempDiffusion tempAd-vection tempForcing, and tempStepping allow you to turn on/off termsin the temperature equation (same thing for salinity or specific humidity withvariables saltDiffusion, saltAdvection etc.). These variables are all assumedhere to be set to ’.TRUE.’. Look at file model/inc/PARAMS.h for a precisedefinition.

initialization

The initial tracer data can be contained in the binary files hydrogTh-etaFile and hydrogSaltFile. These files should contain 3D data orderedin an (x,y,r) fashion with k=1 as the first vertical level. If no file namesare provided, the tracers are then initialized with the values of tRef andsRef mentioned above (in the equation of state section). In this case, theinitial tracer data are uniform in x and y for each depth level.

forcing

This part is more relevant for the ocean, the procedure for the atmospherenot being completely stabilized at the moment.

A combination of fluxes data and relaxation terms can be used for driv-ing the tracer equations. For potential temperature, heat flux data (inW/m2) can be stored in the 2D binary file surfQfile. Alternatively orin addition, the forcing can be specified through a relaxation term. TheSST data to which the model surface temperatures are restored to aresupposed to be stored in the 2D binary file thetaClimFile. The corre-sponding relaxation time scale coefficient is set through the variable tau-ThetaClimRelax (in s). The same procedure applies for salinity withthe variable names EmPmRfile, saltClimFile, and tauSaltClimRe-lax for freshwater flux (in m/s) and surface salinity (in ppt) data filesand relaxation time scale coefficient (in s), respectively. Also for salinity,if the CPP key USE NATURAL BCS is turned on, natural boundaryconditions are applied i.e. when computing the surface salinity tendency,the freshwater flux is multiplied by the model surface salinity instead of aconstant salinity value.

As for the other input files, the precision with which to read the data iscontrolled by the variable readBinaryPrec. Time-dependent, periodicforcing can be applied as well following the same procedure used for thewind forcing data (see above).

dissipation

Lateral eddy diffusivities for temperature and salinity/specific humidityare specified through the variables diffKhT and diffKhS (in m2/s). Ver-


tical eddy diffusivities are specified through the variables diffKzT anddiffKzS (in m2/s) for the ocean and diffKpT and diffKpS (in Pa2/s)for the atmosphere. The vertical diffusive fluxes can be computed implic-itly by setting the logical variable implicitDiffusion to ’.TRUE.’. Inaddition, biharmonic diffusivities can be specified as well through the co-efficients diffK4T and diffK4S (in m4/s). Note that the cosine powerscaling (specified through cosPower—see the momentum equations sec-tion) is applied to the tracer diffusivities (Laplacian and biharmonic) aswell. The Gent and McWilliams parameterization for oceanic tracers isdescribed in the package section. Finally, note that tracers can be alsosubject to Fourier and Shapiro filtering (see the corresponding section onthese filters).

ocean convection

Two options are available to parameterize ocean convection: one is touse the convective adjustment scheme. In this case, you need to set thevariable cadjFreq, which represents the frequency (in s) with which theadjustment algorithm is called, to a non-zero value (if set to a negativevalue by the user, the model will set it to the tracer time step). Theother option is to parameterize convection with implicit vertical diffusion.To do this, set the logical variable implicitDiffusion to ’.TRUE.’ andthe real variable ivdc kappa to a value (in m2/s) you wish the tracervertical diffusivities to have when mixing tracers vertically due to staticinstabilities. Note that cadjFreq and ivdc kappacan not both have non-zero value.

3.16.9 Simulation controls

The model ”clock” is defined by the variable deltaTClock (in s) which de-termines the IO frequencies and is used in tagging output. Typically, you willset it to the tracer time step for accelerated runs (otherwise it is simply set tothe default time step deltaT). Frequency of checkpointing and dumping of themodel state are referenced to this clock (see below).

run duration

The beginning of a simulation is set by specifying a start time (in s)through the real variable startTime or by specifying an initial iterationnumber through the integer variable nIter0. If these variables are set tononzero values, the model will look for a ”pickup” file pickup.0000nIter0to restart the integration. The end of a simulation is set through the realvariable endTime (in s). Alternatively, you can specify instead the num-ber of time steps to execute through the integer variable nTimeSteps.

frequency of output

Real variables defining frequencies (in s) with which output files are writ-ten on disk need to be set up. dumpFreq controls the frequency with


which the instantaneous state of the model is saved. chkPtFreq andpchkPtFreq control the output frequency of rolling and permanent check-point files, respectively. See section 1.5.1 Output files for the definitionof model state and checkpoint files. In addition, time-averaged fields canbe written out by setting the variable taveFreq (in s). The precisionwith which to write the binary data is controlled by the integer variablewriteBinaryPrec (set it to 32 or 64).


3.17 Testing

A script (testreport) for automated testing is included in the model withinthe verification directory. While intended mostly for advanced users, the scriptcan be helpful for beginners.

3.17.1 Using testreport

On many systems, the program can be run with the command:

% cd verification% ./testreport -ieee

which will do the following:

1. Locate all “valid” test directories. Here, valid tests are defined to be thosedirectories within the current directory (which is generally verification)that contain a subdirectory and file with the names results/output.txt.

2. Then within each valid test:

(a) run genmake2 to produce a Makefile

(b) build an executable

(c) run the executable

(d) compare and the output of the executable with the contents of certainvariables within TESTNAME/results/output.txt

(e) print and, if requested (with the -addr=EMAIL ADDRESS option), senda MIME-encoded email with the testing results

For further details, please see the MITgcm Developers’ HOWTO at:

http://mitgcm.org/docs.html

3.17.2 Automated testing

Automated testing results are produced on a regular basis and they can beviewed at:

http://mitgcm.org/testing.html

which also includes links to various scripts for batch job submission on a varietyof different machines.

Chapter 4

Software Architecture

This chapter focuses on describing the WRAPPER environment within whichboth the core numerics and the pluggable packages operate. The descriptionpresented here is intended to be a detailed exposition and contains significantbackground material, as well as advanced details on working with the WRAP-PER. The tutorial sections of this manual (see sections 3.7 and 5.3) contain moresuccinct, step-by-step instructions on running basic numerical experiments, ofvarous types, both sequentially and in parallel. For many projects simply start-ing from an example code and adapting it to suit a particular situation willbe all that is required. The first part of this chapter discusses the MITgcmarchitecture at an abstract level. In the second part of the chapter we describedpractical details of the MITgcm implementation and of current tools and oper-ating system features that are employed.

4.1 Overall architectural goals

Broadly, the goals of the software architecture employed in MITgcm are three-fold

• We wish to be able to study a very broad range of interesting and chal-lenging rotating fluids problems.

• We wish the model code to be readily targeted to a wide range of platforms

• On any given platform we would like to be able to achieve performancecomparable to an implementation developed and specialized specificallyfor that platform.

These points are summarized in figure 4.1 which conveys the goals of theMITgcm design. The goals lead to a software architecture which at the high-level can be viewed as consisting of

1. A core set of numerical and support code. This is discussed in detail insection ??.

235

236 CHAPTER 4. SOFTWARE ARCHITECTURE

Figure 4.1: The MITgcm architecture is designed to allow simulation of a widerange of physical problems on a wide range of hardware. The computationalresource requirements of the applications targeted range from around 107 bytes( ≈ 10 megabytes ) of memory to 1011 bytes ( ≈ 100 gigabytes). Arithmeticoperation counts for the applications of interest range from 109 floating pointoperations to more than 1017 floating point operations.

2. A scheme for supporting optional ”pluggable” packages (containing forexample mixed-layer schemes, biogeochemical schemes, atmospheric physics).These packages are used both to overlay alternate dynamics and to in-troduce specialized physical content onto the core numerical code. Anoverview of the package scheme is given at the start of part ??.

3. A support framework called WRAPPER (Wrappable Application Paral-lel Programming Environment Resource), within which the core numericsand pluggable packages operate.

This chapter focuses on describing the WRAPPER environment underwhich both the core numerics and the pluggable packages function. The de-scription presented here is intended to be a detailed exposition and containssignificant background material, as well as advanced details on working withthe WRAPPER. The examples section of this manual (part ??) contains moresuccinct, step-by-step instructions on running basic numerical experiments bothsequentially and in parallel. For many projects simply starting from an examplecode and adapting it to suit a particular situation will be all that is required.

4.2. WRAPPER 237

4.2 WRAPPER

A significant element of the software architecture utilized in MITgcm is a soft-ware superstructure and substructure collectively called the WRAPPER (Wrap-pable Application Parallel Programming Environment Resource). All numericaland support code in MITgcm is written to “fit” within the WRAPPER infras-tructure. Writing code to “fit” within the WRAPPER means that coding hasto follow certain, relatively straightforward, rules and conventions ( these arediscussed further in section 4.3.1 ).

The approach taken by the WRAPPER is illustrated in figure 4.2 whichshows how the WRAPPER serves to insulate code that fits within it from ar-chitectural differences between hardware platforms and operating systems. Thisallows numerical code to be easily retargetted.

4.2.1 Target hardware

The WRAPPER is designed to target as broad as possible a range of com-puter systems. The original development of the WRAPPER took place on amulti-processor, CRAY Y-MP system. On that system, numerical code per-formance and scaling under the WRAPPER was in excess of that of an im-plementation that was tightly bound to the CRAY systems proprietary multi-tasking and micro-tasking approach. Later developments have been carried outon uniprocessor and multi-processor Sun systems with both uniform memoryaccess (UMA) and non-uniform memory access (NUMA) designs. Significantwork has also been undertaken on x86 cluster systems, Alpha processor basedclustered SMP systems, and on cache-coherent NUMA (CC-NUMA) systemsfrom Silicon Graphics. The MITgcm code, operating within the WRAPPER, isalso routinely used on large scale MPP systems (for example T3E systems andIBM SP systems). In all cases numerical code, operating within the WRAP-PER, performs and scales very competitively with equivalent numerical codethat has been modified to contain native optimizations for a particular system??.

4.2.2 Supporting hardware neutrality

The different systems listed in section 4.2.1 can be categorized in many differentways. For example, one common distinction is between shared-memory parallelsystems (SMP’s, PVP’s) and distributed memory parallel systems (for exam-ple x86 clusters and large MPP systems). This is one example of a differencebetween compute platforms that can impact an application. Another commondistinction is between vector processing systems with highly specialized CPU’sand memory subsystems and commodity microprocessor based systems. Thereare numerous other differences, especially in relation to how parallel executionis supported. To capture the essential differences between different platformsthe WRAPPER uses a machine model.


Figure 4.2: Numerical code is written to fit within a software support infras-tructure called WRAPPER. The WRAPPER is portable and can be specializedfor a wide range of specific target hardware and programming environments,without impacting numerical code that fits within the WRAPPER. Codes thatfit within the WRAPPER can generally be made to run as fast on a particularplatform as codes specially optimized for that platform.

4.2. WRAPPER 239

4.2.3 WRAPPER machine model

Applications using the WRAPPER are not written to target just one particularmachine (for example an IBM SP2) or just one particular family or class ofmachines (for example Parallel Vector Processor Systems). Instead the WRAP-PER provides applications with an abstract machine model. The machine modelis very general, however, it can easily be specialized to fit, in a computationallyefficient manner, any computer architecture currently available to the scientificcomputing community.

4.2.4 Machine model parallelism

Codes operating under the WRAPPER target an abstract machine that is as-sumed to consist of one or more logical processors that can compute concur-rently. Computational work is divided among the logical processors by allocat-ing “ownership” to each processor of a certain set (or sets) of calculations. Eachset of calculations owned by a particular processor is associated with a specificregion of the physical space that is being simulated, only one processor will beassociated with each such region (domain decomposition).

In a strict sense the logical processors over which work is divided do notneed to correspond to physical processors. It is perfectly possible to executea configuration decomposed for multiple logical processors on a single phys-ical processor. This helps ensure that numerical code that is written to fitwithin the WRAPPER will parallelize with no additional effort and is also use-ful when debugging codes. Generally, however, the computational domain willbe subdivided over multiple logical processors in order to then bind those logicalprocessors to physical processor resources that can compute in parallel.

4.2.4.1 Tiles

Computationally, associated with each region of physical space allocated to aparticular logical processor, there will be data structures (arrays, scalar vari-ables etc...) that hold the simulated state of that region. We refer to these datastructures as being owned by the processor to which their associated regionof physical space has been allocated. Individual regions that are allocated toprocessors are called tiles. A processor can own more than one tile. Figure4.3 shows a physical domain being mapped to a set of logical processors, witheach processors owning a single region of the domain (a single tile). Exceptfor periods of communication and coordination, each processor computes au-tonomously, working only with data from the tile (or tiles) that the processorowns. When multiple tiles are alloted to a single processor, each tile is computedon independently of the other tiles, in a sequential fashion.

4.2.4.2 Tile layout

Tiles consist of an interior region and an overlap region. The overlap region of atile corresponds to the interior region of an adjacent tile. In figure 4.4 each tile


Figure 4.3: The WRAPPER provides support for one and two dimensional de-compositions of grid-point domains. The figure shows a hypothetical domain oftotal size NxNyNz. This hypothetical domain is decomposed in two-dimensionsalong the Nx and Ny directions. The resulting tiles are owned by differentprocessors. The owning processors perform the arithmetic operations associ-ated with a tile. Although not illustrated here, a single processor can ownseveral tiles. Whenever a processor wishes to transfer data between tiles orcommunicate with other processors it calls a WRAPPER supplied function.

would own the region within the black square and hold duplicate informationfor overlap regions extending into the tiles to the north, south, east and west.During computational phases a processor will reference data in an overlap regionwhenever it requires values that outside the domain it owns. Periodically pro-cessors will make calls to WRAPPER functions to communicate data betweentiles, in order to keep the overlap regions up to date (see section 4.2.8). TheWRAPPER functions can use a variety of different mechanisms to communicatedata between tiles.

4.2.5 Communication mechanisms

Logical processors are assumed to be able to exchange information between tilesand between each other using at least one of two possible mechanisms.

• Shared memory communication. Under this mode of communicationdata transfers are assumed to be possible using direct addressing of regionsof memory. In this case a CPU is able to read (and write) directly toregions of memory ”owned” by another CPU using simple programminglanguage level assignment operations of the the sort shown in figure 4.5.In this way one CPU (CPU1 in the figure) can communicate informationto another CPU (CPU2 in the figure) by assigning a particular value to aparticular memory location.

4.2. WRAPPER 241

Figure 4.4: A global grid subdivided into tiles. Tiles contain a interior regionand an overlap region. Overlap regions are periodically updated from neighbor-ing tiles.

• Distributed memory communication. Under this mode of communi-cation there is no mechanism, at the application code level, for directlyaddressing regions of memory owned and visible to another CPU. Insteada communication library must be used as illustrated in figure 4.6. In thiscase CPU’s must call a function in the API of the communication libraryto communicate data from a tile that it owns to a tile that another CPUowns. By default the WRAPPER binds to the MPI communication library?? for this style of communication.

The WRAPPER assumes that communication will use one of these two stylesof communication. The underlying hardware and operating system support forthe style used is not specified and can vary from system to system.

4.2.6 Shared memory communication

Under shared communication independent CPU’s are operating on the exactsame global address space at the application level. This means that CPU 1 candirectly write into global data structures that CPU 2 “owns” using a simpleassignment at the application level. This is the model of memory access issupported at the basic system design level in “shared-memory” systems suchas PVP systems, SMP systems, and on distributed shared memory systems(the SGI Origin). On such systems the WRAPPER will generally use simple


CPU1 | CPU2==== | ====

|a(3) = 8 | WHILE ( a(3) .NE. 8 )

| WAIT| END WHILE|

Figure 4.5: In the WRAPPER shared memory communication model, simplewrites to an array can be made to be visible to other CPU’s at the applicationcode level. So that for example, if one CPU (CPU1 in the figure above) writesthe value 8 to element 3 of array a, then other CPU’s (for example CPU2 inthe figure above) will be able to see the value 8 when they read from a(3). Thisprovides a very low latency and high bandwidth communication mechanism.

CPU1 | CPU2==== | ====

|a(3) = 8 | WHILE ( a(3) .NE. 8 )CALL SEND( CPU2,a(3) ) | CALL RECV( CPU1, a(3) )

| END WHILE|

Figure 4.6: In the WRAPPER distributed memory communication model datacan not be made directly visible to other CPU’s. If one CPU writes the value 8to element 3 of array a, then at least one of CPU1 and/or CPU2 in the figureabove will need to call a bespoke communication library in order for the updatedvalue to be communicated between CPU’s.

4.2. WRAPPER 243

read and write statements to access directly application data structures whencommunicating between CPU’s.

In a system where assignments statements, like the one in figure 4.5 mapdirectly to hardware instructions that transport data between CPU and memorybanks, this can be a very efficient mechanism for communication. In this casetwo CPU’s, CPU1 and CPU2, can communicate simply be reading and writingto an agreed location and following a few basic rules. The latency of this sortof communication is generally not that much higher than the hardware latencyof other memory accesses on the system. The bandwidth available betweenCPU’s communicating in this way can be close to the bandwidth of the systemsmain-memory interconnect. This can make this method of communication veryefficient provided it is used appropriately.

4.2.6.1 Memory consistency

When using shared memory communication between multiple processors theWRAPPER level shields user applications from certain counter-intuitive sys-tem behaviors. In particular, one issue the WRAPPER layer must deal with isa systems memory model. In general the order of reads and writes expressed bythe textual order of an application code may not be the ordering of instructionsexecuted by the processor performing the application. The processor perform-ing the application instructions will always operate so that, for the applicationinstructions the processor is executing, any reordering is not apparent. How-ever, in general machines are often designed so that reordering of instructionsis not hidden from other second processors. This means that, in general, evenon a shared memory system two processors can observe inconsistent memoryvalues.

The issue of memory consistency between multiple processors is discussedat length in many computer science papers, however, from a practical point ofview, in order to deal with this issue, shared memory machines all provide somemechanism to enforce memory consistency when it is needed. The exact mech-anism employed will vary between systems. For communication using sharedmemory, the WRAPPER provides a place to invoke the appropriate mechanismto ensure memory consistency for a particular platform.

4.2.6.2 Cache effects and false sharing

Shared-memory machines often have local to processor memory caches whichcontain mirrored copies of main memory. Automatic cache-coherence proto-cols are used to maintain consistency between caches on different processors.These cache-coherence protocols typically enforce consistency between regionsof memory with large granularity (typically 128 or 256 byte chunks). The co-herency protocols employed can be expensive relative to other memory accessesand so care is taken in the WRAPPER (by padding synchronization structuresappropriately) to avoid unnecessary coherence traffic.


4.2.6.3 Operating system support for shared memory.

Applications running under multiple threads within a single process can useshared memory communication. In this case all the memory locations in an ap-plication are potentially visible to all the compute threads. Multiple threads op-erating within a single process is the standard mechanism for supporting sharedmemory that the WRAPPER utilizes. Configuring and launching code to run inmulti-threaded mode on specific platforms is discussed in section ??. However,on many systems, potentially very efficient mechanisms for using shared mem-ory communication between multiple processes (in contrast to multiple threadswithin a single process) also exist. In most cases this works by making a lim-ited region of memory shared between processes. The MMAP ?? and IPC ??facilities in UNIX systems provide this capability as do vendor specific tools likeLAPI ?? and IMC ??. Extensions exist for the WRAPPER that allow thesemechanisms to be used for shared memory communication. However, thesemechanisms are not distributed with the default WRAPPER sources, becauseof their proprietary nature.

4.2.7 Distributed memory communication

Many parallel systems are not constructed in a way where it is possible or prac-tical for an application to use shared memory for communication. For examplecluster systems consist of individual computers connected by a fast network. Onsuch systems their is no notion of shared memory at the system level. For thissort of system the WRAPPER provides support for communication based ona bespoke communication library (see figure 4.6). The default communicationlibrary used is MPI ??. However, it is relatively straightforward to implementbindings to optimized platform specific communication libraries. For examplethe work described in ?? substituted standard MPI communication for a highlyoptimized library.

4.2.8 Communication primitives

Optimized communication support is assumed to be possibly available for asmall number of communication operations. It is assumed that communicationperformance optimizations can be achieved by optimizing a small number ofcommunication primitives. Three optimizable primitives are provided by theWRAPPER

• EXCHANGE This operation is used to transfer data between interiorand overlap regions of neighboring tiles. A number of different forms ofthis operation are supported. These different forms handle

– Data type differences. Sixty-four bit and thirty-two bit fields may behandled separately.

4.2. WRAPPER 245

Figure 4.7: Three performance critical parallel primitives are provided by theWRAPPER. These primitives are always used to communicate data betweentiles. The figure shows four tiles. The curved arrows indicate exchange primi-tives which transfer data between the overlap regions at tile edges and interiorregions for nearest-neighbor tiles. The straight arrows symbolize global sumoperations which connect all tiles. The global sum operation provides both akey arithmetic primitive and can serve as a synchronization primitive. A thirdbarrier primitive is also provided, it behaves much like the global sum primitive.


– Bindings to different communication methods. Exchange primitivesselect between using shared memory or distributed memory commu-nication.

– Transformation operations required when transporting data betweendifferent grid regions. Transferring data between faces of a cube-sphere grid, for example, involves a rotation of vector components.

– Forward and reverse mode computations. Derivative calculations re-quire tangent linear and adjoint forms of the exchange primitives.

• GLOBAL SUM The global sum operation is a central arithmetic op-eration for the pressure inversion phase of the MITgcm algorithm. Forcertain configurations scaling can be highly sensitive to the performanceof the global sum primitive. This operation is a collective operation in-volving all tiles of the simulated domain. Different forms of the globalsum primitive exist for handling

– Data type differences. Sixty-four bit and thirty-two bit fields may behandled separately.

– Bindings to different communication methods. Exchange primitivesselect between using shared memory or distributed memory commu-nication.

– Forward and reverse mode computations. Derivative calculations re-quire tangent linear and adjoint forms of the exchange primitives.

• BARRIER The WRAPPER provides a global synchronization functioncalled barrier. This is used to synchronize computations over all tiles. TheBARRIER and GLOBAL SUM primitives have much in common andin some cases use the same underlying code.

4.2.9 Memory architecture

The WRAPPER machine model is aimed to target efficiently systems withhighly pipelined memory architectures and systems with deep memory hierar-chies that favor memory reuse. This is achieved by supporting a flexible tilingstrategy as shown in figure 4.8. Within a CPU computations are carried outsequentially on each tile in turn. By reshaping tiles according to the targetplatform it is possible to automatically tune code to improve memory perfor-mance. On a vector machine a given domain might be sub-divided into a fewlong, thin regions. On a commodity microprocessor based system, however, thesame region could be simulated use many more smaller sub-domains.

4.2. WRAPPER 247

Figure 4.8: The tiling strategy that the WRAPPER supports allows tiles to beshaped to suit the underlying system memory architecture. Compact tiles thatlead to greater memory reuse can be used on cache based systems (upper halfof figure) with deep memory hierarchies, long tiles with large inner loops can beused to exploit vector systems having highly pipelined memory systems.


4.2.10 Summary

Following the discussion above, the machine model that the WRAPPER presentsto an application has the following characteristics

• The machine consists of one or more logical processors.• Each processor operates on tiles that it owns.• A processor may own more than one tile.• Processors may compute concurrently.• Exchange of information between tiles is handled by the machine (WRAP-

PER) not by the application.

Behind the scenes this allows the WRAPPER to adapt the machine modelfunctions to exploit hardware on which

• Processors may be able to communicate very efficiently with each otherusing shared memory.

• An alternative communication mechanism based on a relatively simpleinter-process communication API may be required.

• Shared memory may not necessarily obey sequential consistency, howeversome mechanism will exist for enforcing memory consistency.

• Memory consistency that is enforced at the hardware level may be expen-sive. Unnecessary triggering of consistency protocols should be avoided.

• Memory access patterns may need to either repetitive or highly pipelinedfor optimum hardware performance.

This generic model captures the essential hardware ingredients of almost allsuccessful scientific computer systems designed in the last 50 years.

4.3 Using the WRAPPER

In order to support maximum portability the WRAPPER is implemented pri-marily in sequential Fortran 77. At a practical level the key steps provided bythe WRAPPER are

1. specifying how a domain will be decomposed

2. starting a code in either sequential or parallel modes of operations

3. controlling communication between tiles and between concurrently com-puting CPU’s.

This section describes the details of each of these operations. Section 4.3.1 ex-plains how the way in which a domain is decomposed (or composed) is expressed.Section ?? describes practical details of running codes in various different paral-lel modes on contemporary computer systems. Section ?? explains the internalinformation that the WRAPPER uses to control how information is communi-cated between tiles.

4.3. USING THE WRAPPER 249

4.3.1 Specifying a domain decomposition

At its heart much of the WRAPPER works only in terms of a collection oftiles which are interconnected to each other. This is also true of applicationcode operating within the WRAPPER. Application code is written as a seriesof compute operations, each of which operates on a single tile. If applicationcode needs to perform operations involving data associated with another tile,it uses a WRAPPER function to obtain that data. The specification of how aglobal domain is constructed from tiles or alternatively how a global domain isdecomposed into tiles is made in the file SIZE.h. This file defines the followingparameters

Parameters: sNx, sNy, OLx, OLy, nSx, nSy, nPx, nPyFile: model/inc/SIZE.h

Together these parameters define a tiling decomposition of the style shownin figure 4.9. The parameters sNx and sNy define the size of an individual tile.The parameters OLx and OLy define the maximum size of the overlap extent.This must be set to the maximum width of the computation stencil that the nu-merical code finite-difference operations require between overlap region updates.The maximum overlap required by any of the operations in the MITgcm codedistributed with this release is three grid points. This is set by the requirementsof the ∇4 dissipation and diffusion operator. Code modifications and enhance-ments that involve adding wide finite-difference stencils may require increasingOLx and OLy. Setting OLx and OLy to a too large value will decrease codeperformance (because redundant computations will be performed), however itwill not cause any other problems.

The parameters nSx and nSy specify the number of tiles that will be createdwithin a single process. Each of these tiles will have internal dimensions ofsNx and sNy. If, when the code is executed, these tiles are allocated to differentthreads of a process that are then bound to different physical processors ( see themulti-threaded execution discussion in section 4.3.2 ) then computation will beperformed concurrently on each tile. However, it is also possible to run the samedecomposition within a process running a single thread on a single processor.In this case the tiles will be computed over sequentially. If the decomposition isrun in a single process running multiple threads but attached to a single physicalprocessor, then, in general, the computation for different tiles will be interleavedby system level software. This too is a valid mode of operation.

The parameters sNx, sNy, OLx, OLy, nSx andnSy are used extensively bynumerical code. The settings of sNx, sNy, OLx and OLy are used to formthe loop ranges for many numerical calculations and to provide dimensions forarrays holding numerical state. The nSx andnSy are used in conjunction withthe thread number parameter myThid. Much of the numerical code operatingwithin the WRAPPER takes the form

DO bj=myByLo(myThid),myByHi(myThid)DO bi=myBxLo(myThid),myBxHi(myThid)


One tile

nSx × nSytiles per process

nPx × nPyprocesses

nPy=2

nPx=2

nSy=1

nSx=3

sNx

sNy

OLxOLy

SIZE.h

Figure 4.9: The three level domain decomposition hierarchy employed by theWRAPPER. A domain is composed of tiles. Multiple tiles can be allocated toa single process. Multiple processes can exist, each with multiple tiles. Tileswithin a process can be spread over multiple compute threads.


:a block of computations rangingover 1,sNx +/- OLx and 1,sNy +/- OLy grid points:

ENDDOENDDO

communication code to sum a number or maybe updatetile overlap regions

DO bj=myByLo(myThid),myByHi(myThid)DO bi=myBxLo(myThid),myBxHi(myThid)

:another block of computations rangingover 1,sNx +/- OLx and 1,sNy +/- OLy grid points:

ENDDOENDDO

The variables myBxLo(myThid), myBxHi(myThid), myByLo(myThid) and my-ByHi(myThid) set the bounds of the loops in bi and bj in this schematic. Thesevariables specify the subset of the tiles in the range 1,nSx and 1,nSy that thelogical processor bound to thread number myThid owns. The thread numbervariable myThid ranges from 1 to the total number of threads requested atexecution time. For each value of myThid the loop scheme above will step se-quentially through the tiles owned by that thread. However, different threadswill have different ranges of tiles assigned to them, so that separate threads cancompute iterations of the bi, bj loop concurrently. Within a bi, bj loop compu-tation is performed concurrently over as many processes and threads as thereare physical processors available to compute.

An exception to the the use of bi and bj in loops arises in the exchangeroutines used when the exch2 package is used with the cubed sphere. In thiscase bj is generally set to 1 and the loop runs from 1,bi. Within the loop biis used to retrieve the tile number, which is then used to reference exchangeparameters.

The amount of computation that can be embedded a single loop over bi andbj varies for different parts of the MITgcm algorithm. Figure 4.10 shows a codeextract from the two-dimensional implicit elliptic solver. This portion of thecode computes the l2Norm of a vector whose elements are held in the arraycg2d r writing the final result to scalar variable err. In this case, because thel2norm requires a global reduction, the bi,bj loop only contains one statement.This computation phase is then followed by a communication phase in whichall threads and processes must participate. However, in other areas of theMITgcm code entries subsections of code are within a single bi,bj loop. Forexample the evaluation of all the momentum equation prognostic terms ( seeS/R DYNAMICS()) is within a single bi,bj loop.


REAL*8 cg2d_r(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)REAL*8 err

::

other computations::

err = 0.DO bj=myByLo(myThid),myByHi(myThid)DO bi=myBxLo(myThid),myBxHi(myThid)DO J=1,sNyDO I=1,sNxerr = err +

& cg2d_r(I,J,bi,bj)*cg2d_r(I,J,bi,bj)ENDDOENDDO

ENDDOENDDO

CALL GLOBAL_SUM_R8( err , myThid )err = SQRT(err)

Figure 4.10: Example of numerical code for calculating the l2-norm of a vectorwithin the WRAPPER. Notice that under the WRAPPER arrays such as cg2d rhave two extra trailing dimensions. These right most indices are tile indexes.Different threads with a single process operate on different ranges of tile index,as controlled by the settings of myByLo, myByHi, myBxLo and myBxHi.


The final decomposition parameters are nPx and nPy. These parametersare used to indicate to the WRAPPER level how many processes (each withnSx×nSy tiles) will be used for this simulation. This information is neededduring initialization and during I/O phases. However, unlike the variables sNx,sNy, OLx, OLy, nSx and nSy the values of nPx and nPy are absent from thecore numerical and support code.

4.3.1.1 Examples of SIZE.h specifications

The following different SIZE.h parameter setting illustrate how to interpret thevalues of sNx, sNy, OLx, OLy, nSx, nSy, nPx and nPy.

1. PARAMETER (& sNx = 90,& sNy = 40,& OLx = 3,& OLy = 3,& nSx = 1,& nSy = 1,& nPx = 1,& nPy = 1)

This sets up a single tile with x-dimension of ninety grid points, y-dimensionof forty grid points, and x and y overlaps of three grid points each.


This sets up tiles with x-dimension of forty-five grid points, y-dimensionof twenty grid points, and x and y overlaps of three grid points each.There are four tiles allocated to four separate processes (nPx=2,nPy=2)and arranged so that the global domain size is again ninety grid pointsin x and forty grid points in y. In general the formula for global grid size(held in model variables Nx and Ny) is

Nx = sNx*nSx*nPxNy = sNy*nSy*nPy

3. PARAMETER (& sNx = 90,& sNy = 10,


& OLx = 3,& OLy = 3,& nSx = 1,& nSy = 2,& nPx = 1,& nPy = 2)

This sets up tiles with x-dimension of ninety grid points, y-dimension often grid points, and x and y overlaps of three grid points each. There arefour tiles allocated to two separate processes (nPy=2) each of which hastwo separate sub-domains nSy=2, The global domain size is again ninetygrid points in x and forty grid points in y. The two sub-domains in eachprocess will be computed sequentially if they are given to a single threadwithin a single process. Alternatively if the code is invoked with multiplethreads per process the two domains in y may be computed concurrently.


This sets up tiles with x-dimension of thirty-two grid points, y-dimensionof thirty-two grid points, and x and y overlaps of three grid points each.There are six tiles allocated to six separate logical processors (nSx=6).This set of values can be used for a cube sphere calculation. Each tileof size 32 × 32 represents a face of the cube. Initializing the tile connec-tivity correctly ( see section 4.3.3.3. allows the rotations associated withmoving between the six cube faces to be embedded within the tile-tilecommunication code.

4.3.2 Starting the code

When code is started under the WRAPPER, execution begins in a main routineeesupp/src/main.F that is owned by the WRAPPER. Control is transferredto the application through a routine called THE MODEL MAIN() once theWRAPPER has initialized correctly and has created the necessary variables tosupport subsequent calls to communication routines by the application code.The startup calling sequence followed by the WRAPPER is shown in figure4.11.


MAIN

|

|--EEBOOT :: WRAPPER initialization

| |

| |-- EEBOOT_MINMAL :: Minimal startup. Just enough to

| | allow basic I/O.

| |-- EEINTRO_MSG :: Write startup greeting.

| |

| |-- EESET_PARMS :: Set WRAPPER parameters

| |

| |-- EEWRITE_EEENV :: Print WRAPPER parameter settings

| |

| |-- INI_PROCS :: Associate processes with grid regions.

| |

| |-- INI_THREADING_ENVIRONMENT :: Associate threads with grid regions.

| |

| |--INI_COMMUNICATION_PATTERNS :: Initialize between tile

| :: communication data structures

|

|

|--CHECK_THREADS :: Validate multiple thread start up.

|

|--THE_MODEL_MAIN :: Numerical code top-level driver routine

Figure 4.11: Main stages of the WRAPPER startup procedure. This processproceeds transfer of control to application code, which occurs through the pro-cedure THE MODEL MAIN().


4.3.2.1 Multi-threaded execution

Prior to transferring control to the procedure THE MODEL MAIN() the WRAP-PER may cause several coarse grain threads to be initialized. The routineTHE MODEL MAIN() is called once for each thread and is passed a singlestack argument which is the thread number, stored in the variable myThid. Inaddition to specifying a decomposition with multiple tiles per process ( see sec-tion 4.3.1) configuring and starting a code to run using multiple threads requiresthe following steps.

Compilation First the code must be compiled with appropriate multi-threadingdirectives active in the file main.F and with appropriate compiler flags to re-quest multi-threading support. The header files MAIN PDIRECTIVES1.h andMAIN PDIRECTIVES2.h contain directives compatible with compilers for Sun,Compaq, SGI, Hewlett-Packard SMP systems and CRAY PVP systems. Thesedirectives can be activated by using compile time directives -DTARGET SUN, -DTARGET DEC, -DTARGET SGI, -DTARGET HP or -DTARGET CRAY VECTORrespectively. Compiler options for invoking multi-threaded compilation varyfrom system to system and from compiler to compiler. The options will bedescribed in the individual compiler documentation. For the Fortran compilerfrom Sun the following options are needed to correctly compile multi-threadedcode

-stackvar -explicitpar -vpara -noautopar

These options are specific to the Sun compiler. Other compilers will use differentsyntax that will be described in their documentation. The effect of these optionsis as follows

1. -stackvar Causes all local variables to be allocated in stack storage. Thisis necessary for local variables to ensure that they are private to theirthread. Note, when using this option it may be necessary to override thedefault limit on stack-size that the operating system assigns to a process.This can normally be done by changing the settings of the command shellsstack-size limit variable. However, on some systems changing this limitwill require privileged administrator access to modify system parameters.

2. -explicitpar Requests that multiple threads be spawned in response toexplicit directives in the application code. These directives are insertedwith syntax appropriate to the particular target platform when, for ex-ample, the -DTARGET SUN flag is selected.

3. -vpara This causes the compiler to describe the multi-threaded configu-ration it is creating. This is not required but it can be useful when troubleshooting.

4. -noautopar This inhibits any automatic multi-threaded parallelizationthe compiler may otherwise generate.


An example of valid settings for the eedata file for a domain with two sub-domains in y and running with two threads is shown below

nTx=1,nTy=2

This set of values will cause computations to stay within a single thread whenmoving across the nSx sub-domains. In the y-direction, however, sub-domainswill be split equally between two threads.

Multi-threading files and parameters The following files and variablesare used in setting up multi-threaded execution.

File: eesupp/inc/MAIN PDIRECTIVES1.hFile: eesupp/inc/MAIN PDIRECTIVES2.hFile: model/src/THE MODEL MAIN.FFile: eesupp/src/MAIN.FFile: tools/genmake2File: eedataCPP: TARGET SUNCPP: TARGET DECCPP: TARGET HPCPP: TARGET SGICPP: TARGET CRAY VECTORParameter: nTxParameter: nTy

4.3.2.2 Multi-process execution

Despite its appealing programming model, multi-threaded execution remainsless common then multi-process execution. One major reason for this is thatmany system libraries are still not “thread-safe”. This means that for exampleon some systems it is not safe to call system routines to do I/O when runningin multi-threaded mode, except for in a limited set of circumstances. Anotherreason is that support for multi-threaded programming models varies betweensystems.

Multi-process execution is more ubiquitous. In order to run code in a multi-process configuration a decomposition specification ( see section 4.3.1) is given( in which the at least one of the parameters nPx or nPy will be greater thanone) and then, as for multi-threaded operation, appropriate compile time andrun time steps must be taken.

Compilation Multi-process execution under the WRAPPER assumes thatthe portable, MPI libraries are available for controlling the start-up of multipleprocesses. The MPI libraries are not required, although they are usually used,for performance critical communication. However, in order to simplify the taskof controlling and coordinating the start up of a large number (hundreds and


possibly even thousands) of copies of the same program, MPI is used. The callsto the MPI multi-process startup routines must be activated at compile time.Currently MPI libraries are invoked by specifying the appropriate options filewith the -of flag when running the genmake2 script, which generates the Make-file for compiling and linking MITgcm. (Previously this was done by setting theALLOW USE MPI and ALWAYS USE MPI flags in the CPP EEOPTIONS.hfile.) More detailed information about the use of genmake2 for specifying localcompiler flags is located in section 3.16.2.

Directory: tools/build optionsFile: tools/genmake2

Execution The mechanics of starting a program in multi-process mode underMPI is not standardized. Documentation associated with the distribution ofMPI installed on a system will describe how to start a program using thatdistribution. For the free, open-source MPICH system the MITgcm program isstarted using a command such as

mpirun -np 64 -machinefile mf ./mitgcmuv

In this example the text -np 64 specifies the number of processes that will becreated. The numeric value 64 must be equal to the product of the processorgrid settings of nPx and nPy in the file SIZE.h. The parameter mf specifies thata text file called “mf” will be read to get a list of processor names on which thesixty-four processes will execute. The syntax of this file is specified by the MPIdistribution.

File: SIZE.hParameter: nPxParameter: nPy

Environment variables On most systems multi-threaded execution also re-quires the setting of a special environment variable. On many machines thisvariable is called PARALLEL and its values should be set to the number ofparallel threads required. Generally the help pages associated with the multi-threaded compiler on a machine will explain how to set the required environmentvariables for that machines.

Runtime input parameters Finally the file eedata needs to be configuredto indicate the number of threads to be used in the x and y directions. Thevariables nTx and nTy in this file are used to specify the information required.The product of nTx and nTy must be equal to the number of threads spawnedi.e. the setting of the environment variable PARALLEL. The value of nTx mustsubdivide the number of sub-domains in x (nSx) exactly. The value of nTy


must subdivide the number of sub-domains in y (nSy) exactly. The multipro-cess startup of the MITgcm executable mitgcmuv is controlled by the routinesEEBOOT MINIMAL() and INI PROCS(). The first routine performs basicsteps required to make sure each process is started and has a textual outputstream associated with it. By default two output files are opened for each pro-cess with names STDOUT.NNNN and STDERR.NNNN. The NNNNNpart of the name is filled in with the process number so that process number 0will create output files STDOUT.0000 and STDERR.0000, process number1 will create output files STDOUT.0001 and STDERR.0001 etc... Thesefiles are used for reporting status and configuration information and for report-ing error conditions on a process by process basis. The EEBOOT MINIMAL()procedure also sets the variables myProcId and MPI COMM MODEL. Thesevariables are related to processor identification are are used later in the routineINI PROCS() to allocate tiles to processes.

Allocation of processes to tiles in controlled by the routine INI PROCS().For each process this routine sets the variables myXGlobalLo and myYGlobalLo.These variables specify in index space the coordinates of the southernmost andwesternmost corner of the southernmost and westernmost tile owned by thisprocess. The variables pidW, pidE, pidS and pidN are also set in this routine.These are used to identify processes holding tiles to the west, east, south andnorth of this process. These values are stored in global storage in the header fileEESUPPORT.h for use by communication routines. The above does not holdwhen the exch2 package is used – exch2 sets its own parameters to specify theglobal indices of tiles and their relationships to each other. See the documenta-tion on the exch2 package (6.18) for details.

File: eesupp/src/eeboot minimal.FFile: eesupp/src/ini procs.FFile: eesupp/inc/EESUPPORT.hParameter: myProcIdParameter: MPI COMM MODELParameter: myXGlobalLoParameter: myYGlobalLoParameter: pidWParameter: pidEParameter: pidSParameter: pidN

4.3.3 Controlling communication

The WRAPPER maintains internal information that is used for communicationoperations and that can be customized for different platforms. This sectiondescribes the information that is held and used.

1. Tile-tile connectivity information For each tile the WRAPPER sets aflag that sets the tile number to the north, south, east and west of that tile.


This number is unique over all tiles in a configuration. Except when usingthe cubed sphere and the exch2 package, the number is held in the vari-ables tileNo ( this holds the tiles own number), tileNoN, tileNoS, tileNoEand tileNoW. A parameter is also stored with each tile that specifies thetype of communication that is used between tiles. This information is heldin the variables tileCommModeN, tileCommModeS, tileCommModeE andtileCommModeW. This latter set of variables can take one of the followingvalues COMM NONE, COMM MSG, COMM PUT and COMM GET. Avalue of COMM NONE is used to indicate that a tile has no neighborto communicate with on a particular face. A value of COMM MSG isused to indicated that some form of distributed memory communicationis required to communicate between these tile faces ( see section 4.2.7). Avalue of COMM PUT or COMM GET is used to indicate forms of sharedmemory communication ( see section 4.2.6). The COMM PUT value indi-cates that a CPU should communicate by writing to data structures ownedby another CPU. A COMM GET value indicates that a CPU should com-municate by reading from data structures owned by another CPU. Theseflags affect the behavior of the WRAPPER exchange primitive (see figure4.7). The routine ini communication patterns() is responsible for settingthe communication mode values for each tile.

When using the cubed sphere configuration with the exch2 package, therelationships between tiles and their communication methods are set bythe package in other variables. See the exch2 package documentation (6.18for details.

File: eesupp/src/ini communication patterns.FFile: eesupp/inc/EESUPPORT.hParameter: tileNoParameter: tileNoEParameter: tileNoWParameter: tileNoNParameter: tileNoSParameter: tileCommModeEParameter: tileCommModeWParameter: tileCommModeNParameter: tileCommModeS

2. MP directives The WRAPPER transfers control to numerical applica-tion code through the routine THE MODEL MAIN. This routine is calledin a way that allows for it to be invoked by several threads. Support forthis is based on using multi-processing (MP) compiler directives. Mostcommercially available Fortran compilers support the generation of codeto spawn multiple threads through some form of compiler directives. Asthis is generally much more convenient than writing code to interface tooperating system libraries to explicitly spawn threads, and on some sys-


tems this may be the only method available the WRAPPER is distributedwith template MP directives for a number of systems.

These directives are inserted into the code just before and after the transferof control to numerical algorithm code through the routine THE MODEL MAIN.Figure 4.12 shows an example of the code that performs this process for aSilicon Graphics system. This code is extracted from the files main.F andMAIN PDIRECTIVES1.h. The variable nThreads specifies how many in-stances of the routine THE MODEL MAIN will be created. The value ofnThreads is set in the routine INI THREADING ENVIRONMENT. Thevalue is set equal to the the product of the parameters nTx and nTy thatare read from the file eedata. If the value of nThreads is inconsistent withthe number of threads requested from the operating system (for exampleby using an environment variable as described in section ??) then usuallyan error will be reported by the routine CHECK THREADS.

File: eesupp/src/ini threading environment.FFile: eesupp/src/check threads.FFile: eesupp/src/main.FFile: eesupp/inc/MAIN PDIRECTIVES1.hFile: eedataParameter: nThreadsParameter: nTxParameter: nTy

3. memsync flags As discussed in section 4.2.6.1, when using shared mem-ory, a low-level system function may be need to force memory consis-tency. The routine MEMSYNC() is used for this purpose. This routineshould not need modifying and the information below is only providedfor completeness. A logical parameter exchNeedsMemSync set in the rou-tine INI COMMUNICATION PATTERNS() controls whether the MEM-SYNC() primitive is called. In general this routine is only used for multi-threaded execution. The code that goes into the MEMSYNC() routineis specific to the compiler and processor being used for multi-threadedexecution and in general must be written using a short code snippet ofassembly language. For an Ultra Sparc system the following code snippetis used

asm("membar #LoadStore|#StoreStore");

for an Alpha based system the equivalent code reads

asm("mb");

while on an x86 system the following code is required


asm("lock; addl $0,0(%%esp)": : :"memory")

4. Cache line size As discussed in section 4.2.6.2, milti-threaded codesexplicitly avoid penalties associated with excessive coherence traffic onan SMP system. To do this the shared memory data structures used bythe GLOBAL SUM, GLOBAL MAX and BARRIER routines are padded.The variables that control the padding are set in the header file EEPARAMS.h.These variables are called cacheLineSize, lShare1, lShare4 and lShare8.The default values should not normally need changing.

5. BARRIER This is a CPP macro that is expanded to a call to a routinewhich synchronizes all the logical processors running under the WRAP-PER. Using a macro here preserves flexibility to insert a specialized callin-line into application code. By default this resolves to calling the proce-dure BARRIER(). The default setting for the BARRIER macro is givenin the file CPP EEMACROS.h.

6. GSUM This is a CPP macro that is expanded to a call to a routinewhich sums up a floating point number over all the logical processorsrunning under the WRAPPER. Using a macro here provides extra flexi-bility to insert a specialized call in-line into application code. By defaultthis resolves to calling the procedure GLOBAL SUM R8() ( for 64-bitfloating point operands) or GLOBAL SUM R4() (for 32-bit floating pointoperands). The default setting for the GSUM macro is given in the fileCPP EEMACROS.h. The GSUM macro is a performance critical oper-ation, especially for large processor count, small tile size configurations.The custom communication example discussed in section 4.3.3.2 showshow the macro is used to invoke a custom global sum routine for a specificset of hardware.

7. EXCH The EXCH CPP macro is used to update tile overlap regions.It is qualified by a suffix indicating whether overlap updates are for two-dimensional ( EXCH XY ) or three dimensional ( EXCH XYZ ) phys-ical fields and whether fields are 32-bit floating point ( EXCH XY R4,EXCH XYZ R4 ) or 64-bit floating point ( EXCH XY R8, EXCH XYZ R8). The macro mappings are defined in the header file CPP EEMACROS.h.As with GSUM, the EXCH operation plays a crucial role in scaling tosmall tile, large logical and physical processor count configurations. Theexample in section 4.3.3.2 discusses defining an optimized and specializedform on the EXCH operation.

The EXCH operation is also central to supporting grids such as the cube-sphere grid. In this class of grid a rotation may be required between tiles.Aligning the coordinate requiring rotation with the tile decomposition,allows the coordinate transformation to be embedded within a customform of the EXCH primitive. In these cases EXCH is mapped to exch2routines, as detailed in the exch2 package documentation 6.18.


8. Reverse Mode The communication primitives EXCH and GSUM bothemploy hand-written adjoint forms (or reverse mode) forms. These reversemode forms can be found in the source code directory pkg/autodiff. For theglobal sum primitive the reverse mode form calls are to GLOBAL ADSUM R4and GLOBAL ADSUM R8. The reverse mode form of the exchange prim-itives are found in routines prefixed ADEXCH. The exchange routinesmake calls to the same low-level communication primitives as the forwardmode operations. However, the routine argument simulationMode is setto the value REVERSE SIMULATION. This signifies ti the low-level rou-tines that the adjoint forms of the appropriate communication operationshould be performed.

9. MAX NO THREADS The variable MAX NO THREADS is used toindicate the maximum number of OS threads that a code will use. Thisvalue defaults to thirty-two and is set in the file EEPARAMS.h. For singlethreaded execution it can be reduced to one if required. The value; islargely private to the WRAPPER and application code will nor normallyreference the value, except in the following scenario.For certain physical parametrization schemes it is necessary to have asubstantial number of work arrays. Where these arrays are allocated inheap storage ( for example COMMON blocks ) multi-threaded executionwill require multiple instances of the COMMON block data. This can beachieved using a Fortran 90 module construct, however, if this might beunavailable then the work arrays can be extended with dimensions use thetile dimensioning scheme of nSx and nSy ( as described in section 4.3.1).However, if the configuration being specified involves many more tilesthan OS threads then it can save memory resources to reduce the variableMAX NO THREADS to be equal to the actual number of threads thatwill be used and to declare the physical parameterization work arrayswith a single MAX NO THREADS extra dimension. An example of thisis given in the verification experiment aim.5l cs. Here the default settingof MAX NO THREADS is altered to

INTEGER MAX_NO_THREADSPARAMETER ( MAX_NO_THREADS = 6 )

and several work arrays for storing intermediate calculations are createdwith declarations of the form.

common /FORCIN/ sst1(ngp,MAX_NO_THREADS)

This declaration scheme is not used widely, because most global datais used for permanent not temporary storage of state information. Inthe case of permanent state information this approach cannot be usedbecause there has to be enough storage allocated for all tiles. However,the technique can sometimes be a useful scheme for reducing memoryrequirements in complex physical parameterizations.


C--C-- Parallel directives for MIPS Pro Fortran compilerC--C Parallel compiler directives for SGI with IRIXC$PAR PARALLEL DOC$PAR& CHUNK=1,MP_SCHEDTYPE=INTERLEAVE,C$PAR& SHARE(nThreads),LOCAL(myThid,I)C

DO I=1,nThreadsmyThid = I

C-- Invoke nThreads instances of the numerical modelCALL THE_MODEL_MAIN(myThid)

ENDDO

Figure 4.12: Prior to transferring control to the procedureTHE MODEL MAIN() the WRAPPER may use MP directives to spawnmultiple threads.

4.3.3.1 Specializing the Communication Code

The isolation of performance critical communication primitives and the sub-division of the simulation domain into tiles is a powerful tool. Here we showhow it can be used to improve application performance and how it can be usedto adapt to new griding approaches.

4.3.3.2 JAM example

On some platforms a big performance boost can be obtained by binding thecommunication routines EXCH and GSUM to specialized native libraries ) froexample the shmem library on CRAY T3E systems). The LETS MAKE JAMCPP flag is used as an illustration of a specialized communication configurationthat substitutes for standard, portable forms of EXCH and GSUM. It affectsthree source files eeboot.F, CPP EEMACROS.h and cg2d.F. When the flag isdefined is has the following effects.

• An extra phase is included at boot time to initialize the custom commu-nications library ( see ini jam.F).

• The GSUM and EXCH macro definitions are replaced with calls to cus-tom routines ( see gsum jam.F and exch jam.F)

• a highly specialized form of the exchange operator (optimized for overlapregions of width one) is substituted into the elliptic solver routine cg2d.F.

Developing specialized code for other libraries follows a similar pattern.

4.4. MITGCM EXECUTION UNDER WRAPPER 265

4.3.3.3 Cube sphere communication

Actual EXCH routine code is generated automatically from a series of templatefiles, for example exch rx.template. This is done to allow a large number of vari-ations on the exchange process to be maintained. One set of variations supportsthe cube sphere grid. Support for a cube sphere grid in MITgcm is based onhaving each face of the cube as a separate tile or tiles. The exchange routinesare then able to absorb much of the detailed rotation and reorientation requiredwhen moving around the cube grid. The set of EXCH routines that containthe word cube in their name perform these transformations. They are invokedwhen the run-time logical parameter useCubedSphereExchange is set true. Tofacilitate the transformations on a staggered C-grid, exchange operations aredefined separately for both vector and scalar quantities and for grid-centeredand for grid-face and corner quantities. Three sets of exchange routines aredefined. Routines with names of the form exch rx are used to exchange cellcentered scalar quantities. Routines with names of the form exch uv rx are usedto exchange vector quantities located at the C-grid velocity points. The vec-tor quantities exchanged by the exch uv rx routines can either be signed (forexample velocity components) or un-signed (for example grid-cell separations).Routines with names of the form exch z rx are used to exchange quantities atthe C-grid vorticity point locations.

4.4 MITgcm execution under WRAPPER

Fitting together the WRAPPER elements, package elements and MITgcm coreequation elements of the source code produces calling sequence shown in section4.4.1

4.4.1 Annotated call tree for MITgcm and WRAPPER

WRAPPER layer.

MAIN

|

|--EEBOOT :: WRAPPER initialization

| |

| |-- EEBOOT_MINMAL :: Minimal startup. Just enough to

| | allow basic I/O.

| |-- EEINTRO_MSG :: Write startup greeting.

| |

| |-- EESET_PARMS :: Set WRAPPER parameters

| |

| |-- EEWRITE_EEENV :: Print WRAPPER parameter settings

| |

| |-- INI_PROCS :: Associate processes with grid regions.

| |

| |-- INI_THREADING_ENVIRONMENT :: Associate threads with grid regions.

| |

| |--INI_COMMUNICATION_PATTERNS :: Initialize between tile


C/\ | | | |-TIMEAVE_CUMUATE :: Time averaging package ( see pkg/timeave ).

C/\ | | | |-DEBUG_STATS_RL :: Quick debug package ( see pkg/debug ).

C/\ | | |

C/\ | | |-CALC_GW :: vert. momentum tendency terms ( NH, QH only ).

C/\ | | |

C/\ | | |-UPDATE_SURF_DR :: Update the surface-level thickness fraction.

C/\ | | |

C/\ | | |-UPDATE_CG2D :: Update 2d conjugate grad. for Free-Surf.

C/\ | | |

C/\ | | |-SOLVE_FOR_PRESSURE :: Find surface pressure.

C/\ | | | |-CALC_DIV_GHAT :: Form the RHS of the surface pressure eqn.

C/\ | | | |-CG2D :: Two-dim pre-con. conjugate-gradient.

C/\ | | | |-CG3D :: Three-dim pre-con. conjugate-gradient solver.

C/\ | | |

C/\ | | |-THE_CORRECTION_STEP :: Step forward to next time step.

C/\ | | | |

C/\ | | | |-CALC_GRAD_PHI_SURF :: Return DDx and DDy of surface pressure

C/\ | | | |-CORRECTION_STEP :: Pressure correction to momentum

C/\ | | | |-CYCLE_TRACER :: Move tracers forward in time.

C/\ | | | |-OBCS_APPLY :: Open bndy package. see pkg/obcs

C/\ | | | |-SHAP_FILT_APPLY :: Shapiro filter package. see pkg/shap_filt

C/\ | | | |-ZONAL_FILT_APPLY :: FFT filter package. see pkg/zonal_filt

C/\ | | | |-CONVECTIVE_ADJUSTMENT :: Control static instability mixing.

C/\ | | | | |-FIND_RHO :: Find adjacent densities.

C/\ | | | | |-CONVECT :: Mix static instability.

C/\ | | | | |-TIMEAVE_CUMULATE :: Update convection statistics.

C/\ | | | |

C/\ | | | |-CALC_EXACT_ETA :: Change SSH to flow divergence.

C/\ | | |

C/\ | | |-DO_FIELDS_BLOCKING_EXCHANGES :: Sync up overlap regions.

C/\ | | | |-EXCH

C/\ | | |

C/\ | | |-FLT_MAIN :: Float package ( pkg/flt ).

C/\ | | |

C/\ | | |-MONITOR :: Monitor package ( pkg/monitor ).

C/\ | | |

C/\ | | |-DO_THE_MODEL_IO :: Standard diagnostic I/O.

C/\ | | | |-WRITE_STATE :: Core state I/O

C/\ | | | |-TIMEAVE_STATV_WRITE :: Time averages. see pkg/timeave

C/\ | | | |-AIM_WRITE_DIAGS :: Intermed. atmos diags. see pkg/aim

C/\ | | | |-GMREDI_DIAGS :: GM diags. see pkg/gmredi

C/\ | | | |-KPP_DO_DIAGS :: KPP diags. see pkg/kpp

C/\ | | |

C/\ | | |-WRITE_CHECKPOINT :: Do I/O for restart files.

C/\ | |

C/\ | |-COST_TILE :: Cost function package. ( see pkg/cost )

C<===|=|

C<===|=| **************************

C<===|=| END MAIN TIMESTEPPING LOOP

C<===|=| **************************

C<===|=|

C | |-COST_FINAL :: Cost function package. ( see pkg/cost )

C |

C |-WRITE_CHECKPOINT :: Final state storage, for restart.

C |

C |-TIMER_PRINTALL :: Computational timing summary

C |


C |-COMM_STATS :: Summarise inter-proc and inter-thread communication

C :: events.

C

4.4.2 Measuring and Characterizing Performance

TO BE DONE (CNH)

4.4.3 Estimating Resource Requirements

TO BE DONE (CNH)

4.4.3.1 Atlantic 1/6 degree example

4.4.3.2 Dry Run testing

4.4.3.3 Adjoint Resource Requirements

4.4.3.4 State Estimation Environment Resources

Chapter 5

Automatic Differentiation

Automatic differentiation (AD), also referred to as algorithmic (or, more loosely,computational) differentiation, involves automatically deriving code to calculatepartial derivatives from an existing fully non-linear prognostic code. (see [23]).A software tool is used that parses and transforms source files according toa set of linguistic and mathematical rules. AD tools are like source-to-sourcetranslators in that they parse a program code as input and produce a newprogram code as output. However, unlike a pure source-to-source translation,the output program represents a new algorithm, such as the evaluation of theJacobian, the Hessian, or higher derivative operators. In principle, a variety ofderived algorithms can be generated automatically in this way.

The MITGCM has been adapted for use with the Tangent linear and AdjointModel Compiler (TAMC) and its successor TAF (Transformation of Algorithmsin Fortran), developed by Ralf Giering ([20], [18, 19]). The first application ofthe adjoint of the MITGCM for sensitivity studies has been published by [37].[48, 47] use the MITGCM and its adjoint for ocean state estimation studies.In the following we shall refer to TAMC and TAF synonymously, except wereexplicitly stated otherwise.

TAMC exploits the chain rule for computing the first derivative of a functionwith respect to a set of input variables. Treating a given forward code as acomposition of operations – each line representing a compositional element, thechain rule is rigorously applied to the code, line by line. The resulting tangentlinear or adjoint code, then, may be thought of as the composition in forwardor reverse order, respectively, of the Jacobian matrices of the forward code’scompositional elements.

5.1 Some basic algebra

Let M be a general nonlinear, model, i.e. a mapping from the m-dimensionalspace U ⊂ IRm of input variables �u = (u1, . . . , um) (model parameters, initialconditions, boundary conditions such as forcing functions) to the n-dimensional

273

274 CHAPTER 5. AUTOMATIC DIFFERENTIATION

space V ⊂ IRn of model output variable �v = (v1, . . . , vn) (model state, modeldiagnostics, objective function, ...) under consideration,

M :U −→ V

�u �−→ �v = M(�u)(5.1)

The vectors �u ∈ U and v ∈ V may be represented w.r.t. some given basisvectors span(U) = {�ei}i=1,...,m and span(V ) = {�fj}j=1,...,n as

�u =m∑

i=1

ui �ei, �v =n∑

j=1

vj�fj

Two routes may be followed to determine the sensitivity of the output vari-able �v to its input �u.

5.1.1 Forward or direct sensitivity

Consider a perturbation to the input variables δ�u (typically a single componentδ�u = δui �ei). Their effect on the output may be obtained via the linear approx-imation of the model M in terms of its Jacobian matrix M , evaluated in thepoint u(0) according to

δ�v = M |�u(0) δ�u (5.2)

with resulting output perturbation δ�v. In components Mji = ∂Mj/∂ui, itreads

δvj =∑

i

∂Mj

∂ui

∣∣∣∣u(0)

δui (5.3)

Eq. (5.2) is the tangent linear model (TLM). In contrast to the full nonlinearmodel M, the operator M is just a matrix which can readily be used to findthe forward sensitivity of �v to perturbations in u, but if there are very manyinput variables (� O(106) for large-scale oceanographic application), it quicklybecomes prohibitive to proceed directly as in (5.2), if the impact of each com-ponent ei is to be assessed.

5.1.2 Reverse or adjoint sensitivity

Let us consider the special case of a scalar objective function J (�v) of the modeloutput (e.g. the total meridional heat transport, the total uptake of CO2 inthe Southern Ocean over a time interval, or a measure of some model-to-datamisfit)

J : U −→ V −→ IR�u �−→ �v = M(�u) �−→ J (�u) = J (M(�u)) (5.4)

The perturbation of J around a fixed point J0,

J = J0 + δJ

5.1. SOME BASIC ALGEBRA 275

can be expressed in both bases of �u and �v w.r.t. their corresponding innerproduct 〈 , 〉

J = J |�u(0) +⟨∇uJ T |�u(0) , δ�u

⟩+ O(δ�u2)

= J |�v(0) +⟨∇vJ T |�v(0) , δ�v

⟩+ O(δ�v2)

(5.5)

(note, that the gradient ∇f is a co-vector, therefore its transpose is required inthe above inner product). Then, using the representation of δJ =

⟨∇vJ T , δ�v⟩,

the definition of an adjoint operator A∗ of a given operator A,

〈A∗�x , �y 〉 = 〈 �x , A�y 〉which for finite-dimensional vector spaces is just the transpose of A,

A∗ = AT

and from eq. (5.2), (5.5), we note that (omitting |’s):δJ =

⟨∇vJ T , δ�v⟩

=⟨∇vJ T , M δ�u

⟩=⟨MT ∇vJ T , δ�u

⟩(5.6)

With the identity (5.5), we then find that the gradient ∇uJ can be readilyinferred by invoking the adjoint M∗ of the tangent linear model M

∇uJ T |�u = MT |�u · ∇vJ T |�v= MT |�u · δ�v∗= δ�u∗

(5.7)

Eq. (5.7) is the adjoint model (ADM), in which MT is the adjoint (here, thetranspose) of the tangent linear operator M , δ�v∗ the adjoint variable of themodel state �v, and δ�u∗ the adjoint variable of the control variable �u.

The reverse nature of the adjoint calculation can be readily seen as fol-lows. Consider a model integration which consists of Λ consecutive operationsMΛ(MΛ−1(......(Mλ(......(M1(M0(�u))))), where the M’s could be the elemen-tary steps, i.e. single lines in the code of the model, or successive time stepsof the model integration, starting at step 0 and moving up to step Λ, with in-termediate Mλ(�u) = �v(λ+1) and final MΛ(�u) = �v(Λ+1) = �v. Let J be a costfunction which explicitly depends on the final state �v only (this restriction is forclarity reasons only). J (u) may be decomposed according to:

J (M(�u)) = J (MΛ(MΛ−1(......(Mλ(......(M1(M0(�u)))))) (5.8)

Then, according to the chain rule, the forward calculation reads, in terms of theJacobi matrices (we’ve omitted the |’s which, nevertheless are important to theaspect of tangent linearity; note also that by definition 〈∇vJ T , δ�v 〉 = ∇vJ ·δ�v)

∇vJ (M(δ�u)) = ∇vJ ·MΛ · ...... ·Mλ · ...... ·M1 ·M0 · δ�u= ∇vJ · δ�v (5.9)


whereas in reverse mode we have

MT (∇vJ T ) = MT0 ·MT

1 · ...... ·MTλ · ...... ·MT

Λ · ∇vJ T

= MT0 ·MT

1 · ...... · ∇v(λ)J T

= ∇uJ T

(5.10)

clearly expressing the reverse nature of the calculation. Eq. (5.10) is at theheart of automatic adjoint compilers. If the intermediate steps λ in eqn. (5.8)– (5.10) represent the model state (forward or adjoint) at each intermediatetime step as noted above, then correspondingly, MT (δ�v(λ) ∗) = δ�v(λ−1) ∗ forthe adjoint variables. It thus becomes evident that the adjoint calculation alsoyields the adjoint of each model state component �v(λ) at each intermediate stepλ, namely

∇v(λ)J T |�v(λ) = MTλ |�v(λ) · ...... ·MT

Λ |�v(λ) · δ�v∗= δ�v(λ) ∗ (5.11)

in close analogy to eq. (5.7) We note in passing that that the δ�v(λ) ∗ are theLagrange multipliers of the model equations which determine �v(λ).

In components, eq. (5.7) reads as follows. Let

δ�u = (δu1, . . . , δum)T, δ�u∗ = ∇uJ T =

(∂J∂u1

, . . . , ∂J∂um

)Tδ�v = (δv1, . . . , δun)T

, δ�v∗ = ∇vJ T =(

∂J∂v1

, . . . , ∂J∂vn

)Tdenote the perturbations in �u and �v, respectively, and their adjoint variables;further

M =

⎛⎜⎝∂M1∂u1

. . . ∂M1∂um

......

∂Mn

∂u1. . . ∂Mn

∂um

⎞⎟⎠is the Jacobi matrix of M (an n×m matrix) such that δ�v = M · δ�u, or

δvj =m∑

i=1

Mji δui =m∑

i=1

∂Mj

∂uiδui

Then eq. (5.7) takes the form

δu∗i =n∑

j=1

Mji δv∗j =

n∑j=1

∂Mj

∂uiδv∗j

or ⎛⎜⎜⎜⎝∂

∂u1J∣∣∣�u(0)

...∂

∂umJ∣∣∣�u(0)

⎞⎟⎟⎟⎠ =

⎛⎜⎜⎜⎝∂M1∂u1

∣∣∣�u(0)

. . . ∂Mn

∂u1

∣∣∣�u(0)

......

∂M1∂um

∣∣∣�u(0)

. . . ∂Mn

∂um

∣∣∣�u(0)

⎞⎟⎟⎟⎠ ·

⎛⎜⎜⎜⎝∂

∂v1J∣∣∣�v

...∂

∂vnJ∣∣∣�v

⎞⎟⎟⎟⎠


Furthermore, the adjoint δv(λ) ∗ of any intermediate state v(λ) may be obtained,using the intermediate Jacobian (an nλ+1 × nλ matrix)

Mλ =

⎛⎜⎜⎜⎝∂(Mλ)1

∂v(λ)1

. . . ∂(Mλ)1

∂v(λ)nλ

......

∂(Mλ)nλ+1

∂v(λ)1

. . .∂(Mλ)nλ+1

∂v(λ)nλ

⎞⎟⎟⎟⎠and the shorthand notation for the adjoint variables δv(λ) ∗

j = ∂

∂v(λ)j

J T , j =

1, . . . , nλ, for intermediate components, yielding

0BB@

δv(λ) ∗1

...

δv(λ) ∗nλ

1CCA =

0BBBB@

∂(Mλ)1

∂v(λ)1

. . . . . .∂(Mλ)nλ+1

∂v(λ)1

......

∂(Mλ)1

∂v(λ)nλ

. . . . . .∂(Mλ)nλ+1

∂v(λ)nλ

1CCCCA

·

0BBBBBBBB@

∂(Mλ+1)1

∂v(λ+1)1

. . .∂(Mλ+1)nλ+2

∂v(λ+1)1

......

......

∂(Mλ+1)1

∂v(λ+1)nλ+1

. . .∂(Mλ+1)nλ+2

∂v(λ+1)nλ+1

1CCCCCCCCA

· . . . ·

0B@

δv∗1

...δv∗

n

1CA

(5.12)

Eq. (5.9) and (5.10) are perhaps clearest in showing the advantage of thereverse over the forward mode if the gradient ∇uJ , i.e. the sensitivity of thecost function J with respect to all input variables u (or the sensitivity of thecost function with respect to all intermediate states �v(λ)) are sought. In orderto be able to solve for each component of the gradient ∂J /∂ui in (5.9) a forwardcalculation has to be performed for each component separately, i.e. δ�u = δui�ei

for the i-th forward calculation. Then, (5.9) represents the projection of ∇uJonto the i-th component. The full gradient is retrieved from the m forwardcalculations. In contrast, eq. (5.10) yields the full gradient ∇uJ (and allintermediate gradients ∇v(λ)J ) within a single reverse calculation.

Note, that if J is a vector-valued function of dimension l > 1, eq. (5.10) hasto be modified according to

MT(∇vJ T

(δ �J))

= ∇uJ T · δ �J

where now δ �J ∈ IRl is a vector of dimension l. In this case l reverse simulationshave to be performed for each δJk, k = 1, . . . , l. Then, the reverse mode is moreefficient as long as l < n, otherwise the forward mode is preferable. Strictly, thereverse mode is called adjoint mode only for l = 1.

A detailed analysis of the underlying numerical operations shows that thecomputation of ∇uJ in this way requires about 2 to 5 times the computation


of the cost function. Alternatively, the gradient vector could be approximatedby finite differences, requiring m computations of the perturbed cost function.

To conclude we give two examples of commonly used types of cost functions:

Example 1: J = vj(T )The cost function consists of the j-th component of the model state �v at timeT . Then ∇vJ T = �fj is just the j-th unit vector. The ∇uJ T is the projectionof the adjoint operator onto the j-th component fj,

∇uJ T = MT · ∇vJ T =∑

i

MTji �ei

Example 2: J = 〈H(�v) − �d , H(�v) − �d 〉The cost function represents the quadratic model vs. data misfit. Here, �d is thedata vector and H represents the operator which maps the model state spaceonto the data space. Then, ∇vJ takes the form

∇vJ T = 2 H ·(H(�v) − �d

)= 2∑

j

{∑k

∂Hk

∂vj(Hk(�v) − dk)

}�fj

where Hkj = ∂Hk/∂vj is the Jacobi matrix of the data projection operator.Thus, the gradient ∇uJ is given by the adjoint operator, driven by the modelvs. data misfit:

∇uJ T = 2MT ·H ·(H(�v) − �d

)5.1.3 Storing vs. recomputation in reverse mode

We note an important aspect of the forward vs. reverse mode calculation. Be-cause of the local character of the derivative (a derivative is defined w.r.t. apoint along the trajectory), the intermediate results of the model trajectory�v(λ+1) = Mλ(v(λ)) may be required to evaluate the intermediate JacobianMλ|�v(λ) δ�v(λ). This is the case e.g. for nonlinear expressions (momentum ad-vection, nonlinear equation of state), state-dependent conditional statements(parameterization schemes). In the forward mode, the intermediate results arerequired in the same order as computed by the full forward model M, but inthe reverse mode they are required in the reverse order. Thus, in the reversemode the trajectory of the forward model integration M has to be stored to beavailable in the reverse calculation. Alternatively, the complete model state upto the point of evaluation has to be recomputed whenever its value is required.

A method to balance the amount of recomputations vs. storage require-ments is called checkpointing (e.g. [22], [43]). It is depicted in 5.1 for a 3-levelcheckpointing [as an example, we give explicit numbers for a 3-day integrationwith a 1-hourly timestep in square brackets].


lev3 In a first step, the model trajectory is subdivided into nlev3 subsections[nlev3=3 1-day intervals], with the label lev3 for this outermost loop. Themodel is then integrated along the full trajectory, and the model statestored to disk only at every klev3

i -th timestep [i.e. 3 times, at i = 0, 1, 2corresponding to klev3

i = 0, 24, 48]. In addition, the cost function is com-puted, if needed.

lev2 In a second step each subsection itself is divided into nlev2 subsections[nlev2=4 6-hour intervals per subsection]. The model picks up at the lastoutermost dumped state vklev3

nand is integrated forward in time along

the last subsection, with the label lev2 for this intermediate loop. Themodel state is now stored to disk at every klev2

i -th timestep [i.e. 4 times,at i = 0, 1, 2, 3 corresponding to klev2

i = 48, 54, 60, 66].

lev1 Finally, the model picks up at the last intermediate dump state vklev2n

andis integrated forward in time along the last subsection, with the label lev1for this intermediate loop. Within this sub-subsection only, parts of themodel state is stored to memory at every timestep [i.e. every hour i =0, ..., 5 corresponding to klev1

i = 66, 67, . . . , 71]. The final state vn = vklev1n

is reached and the model state of all preceding timesteps along the lastinnermost subsection are available, enabling integration backwards in timealong the last subsection. The adjoint can thus be computed along thislast subsection klev2

n .

This procedure is repeated consecutively for each previous subsection klev2n−1, . . . , k

lev21

carrying the adjoint computation to the initial time of the subsection klev3n .

Then, the procedure is repeated for the previous subsection klev3n−1 carrying the

adjoint computation to the initial time klev31 .

For the full model trajectory of nlev3 · nlev2 · nlev1 timesteps the requiredstoring of the model state was significantly reduced to nlev2 + nlev3 to diskand roughly nlev1 to memory [i.e. for the 3-day integration with a total oof72 timesteps the model state was stored 7 times to disk and roughly 6 timesto memory]. This saving in memory comes at a cost of a required 3 full for-ward integrations of the model (one for each checkpointing level). The optimalbalance of storage vs. recomputation certainly depends on the computing re-sources available and may be adjusted by adjusting the partitioning among thenlev3, nlev2, nlev1.


v_kn−1^lev2v_k1^lev2

v_kn^lev3v_kn−1^lev3v_k1^lev3

v_k1^lev1 v_kn^lev1

v_kn^lev2

Figure 5.1: Schematic view of intermediate dump and restart for 3-level check-pointing.

5.2 TLM and ADM generation in general

In this section we describe in a general fashion the parts of the code that arerelevant for automatic differentiation using the software tool TAF.

The basic flow is depicted in 5.2. If CPP option ALLOW AUTODIFF TAMCis defined, the driver routine the model main, instead of calling the main loop,invokes the adjoint of this routine, adthe main loop, which is the toplevel routinein terms of automatic differentiation. The routine adthe main loop has beengenerated by TAF. It contains both the forward integration of the full model,the cost function calculation, any additional storing that is required for efficientcheckpointing, and the reverse integration of the adjoint model.

[DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM]In Fig. 5.2 the structure of adthe main loop has been strongly simplified to

5.2. TLM AND ADM GENERATION IN GENERAL 281

focus on the essentials; in particular, no checkpointing procedures are shownhere. Prior to the call of adthe main loop, the routine ctrl unpack is invokedto unpack the control vector or initialise the control variables. Following thecall of adthe main loop, the routine ctrl pack is invoked to pack the controlvector (cf. Section 5.2.5). If gradient checks are to be performed, the optionALLOW GRADIENT CHECK is defined. In this case the driver routine grdchk mainis called after the gradient has been computed via the adjoint (cf. Section ??).

5.2.1 General setup

In order to configure AD-related setups the following packages need to be en-abled: The packages are enabled by adding them to your experiment-specificconfiguration file packages.conf (see Section ???).

The following AD-specific CPP option files need to be customized:

the_model_main||--- initialise_fixed||--- #ifdef ALLOW_ADJOINT_RUN| || |--- ctrl_unpack| || |--- adthe_main_loop| | || | |--- initialise_varia| | |--- ctrl_map_forcing| | |--- do iloop = 1, nTimeSteps| | | |--- forward_step| | | |--- cost_tile| | | end do| | |--- cost_final| | || | |--- adcost_final| | |--- do iloop = nTimeSteps, 1, -1| | | |--- adcost_tile| | | |--- adforward_step| | | end do| | |--- adctrl_map_forcing| | |--- adinitialise_varia| | o| || |--- ctrl_pack| ||--- #else| || |--- the_main_loop| || #endif||--- #ifdef ALLOW_GRADIENT_CHECK| || |--- grdchk_main| o| #endifo

Figure 5.2:


autodiffctrlcostgrdchk

• ECCO CPPOPTIONS.hThis header file collects CPP options for the packages autodiff, cost, ctrlas well as AD-unrelated options for the external forcing package exf. 1

• tamc.hThis header configures the splitting of the time stepping loop w.r.t. the3-level checkpointing (see section ???).

5.2.2 Building the AD code

The build process of an AD code is very similar to building the forward model.However, depending on which AD code one wishes to generate, and on whichAD tool is available (TAF or TAMC), the following make targets are available:

AD-target output description

(1) <MODE><TOOL>only <MODE> <TOOL> output.f generates code for <MODE> using <TOOL>no make dependencies on .F .h

useful for compiling on remote platforms(2) <MODE><TOOL> <MODE> <TOOL> output.f generates code for <MODE> using <TOOL>

includes make dependencies on .F .h

i.e. input for <TOOL> may be re-generated(3) <MODE>all mitgcmuv <MODE> generates code for <MODE> using <TOOL>

and compiles all code(use of TAF is set as default)

Here, the following placeholders are used

<TOOL> – TAF

– TAMC

<MODE> – ad generates the adjoint model (ADM)

– ftl generates the tangent linear model (TLM)

– svd generates both ADM and TLM forsingular value decomposition (SVD) type calculations

For example, to generate the adjoint model using TAF after routines (.F) orheaders (.h) have been modified, but without compilation, type make adtaf;or, to generate the tangent linear model using TAMC without re-generating theinput code, type make ftltamconly.

1NOTE: These options are not set in their package-specific headers suchas COST CPPOPTIONS.h, but are instead collected in the single header fileECCO CPPOPTIONS.h. The package-specific header files serve as simple placehold-ers at this point.


A typical full build process to generate the ADM via TAF would look likefollows:

% mkdir build% cd build% ../../../tools/genmake2 -mods=../code_ad% make depend% make adall

5.2.3 The AD build process in detail

The make <MODE>all target consists of the following procedures:

1. A header file AD CONFIG.h is generated which contains a CPP option onwhich code ought to be generated. Depending on the make target, thecontents is

• #define ALLOW ADJOINT RUN

• #define ALLOW TANGENTLINEAR RUN

• #define ALLOW ECCO OPTIMIZATION

2. A single file <MODE> input code.f is concatenated consisting of all .f filesthat are part of the list AD FILES and all .flow files that are part ofthe list AD FLOW FILES.

3. The AD tool is invoked with the ¡MODE¿ ¡TOOL¿ FLAGS. The de-fault AD tool flags in genmake2 can be overrwritten by an adjoint optionsfile (similar to the platform-specific build options, see Section ???. TheAD tool writes the resulting AD code into the file <MODE> input code ad.f

4. A short sed script adjoint sed is applied to <MODE> input code ad.f toreinstate myThid into the CALL argument list of active file I/O. Theresult is written to file <MODE> <TOOL> output.f.

5. All routines are compiled and an executable is generated (see Table ???).

5.2.3.1 The list AD FILES and .list files

Not all routines are presented to the AD tool. Routines typically hidden arediagnostics routines which do not influence the cost function, but may createartificial flow dependencies such as I/O of active variables.

genmake2 generates a list (or variable) AD FILES which contains all rou-tines that are shown to the AD tool. This list is put together from all files withsuffix .list that genmake2 finds in its search directories. The list file for thecore MITgcm routines is in model/src/ is called model ad diff.list. Notethat no wrapper routine is shown to TAF. These are either not visible at all tothe AD code, or hand-written AD code is available (see next section).


Each package directory contains its package-specific list file <PKG> ad diff.list.For example, pkg/ptracers/ contains the file ptracers ad diff.list. Thus,enabling a package will automatically extend the AD FILES list of genmake2to incorporate the package-specific routines. Note that you will need to regener-ate the Makefile if you enable a package (e.g. by adding it to packages.conf)and a Makefile already exists.

5.2.3.2 The list AD FLOW FILES and .flow files

TAMC and TAF can evaluate user-specified directives that start with a specificsyntax (CADJ, C$TAF, !$TAF). The main categories of directives are STOREdirectives and FLOW directives. Here, we are concerned with flow directives,store directives are treated elsewhere.

Flow directives enable the AD tool to evaluate how it should treat routinesthat are ’hidden’ by the user, i.e. routines which are not contained in theAD FILES list (see previous section), but which are called in part of the codethat the AD tool does see. The flow directive tell the AD tool

• which subroutine arguments are input/output

• which subroutine arguments are active

• which subroutine arguments are required to compute the cost

• which subroutine arguments are dependent

The syntax for the flow directives can be found in the AD tool manuals.genmake2 generates a list (or variable) AD FLOW FILES which contains

all files with suffix.flow that it finds in its search directories. The flow direc-tives for the core MITgcm routines of eesupp/src/ and model/src/ reside inpkg/autodiff/. This directory also contains hand-written adjoint code for theMITgcm WRAPPER (see Section ???).

Flow directives for package-specific routines are contained in the correspond-ing package directories in the file <PKG> ad.flow, e.g. ptracers-specific direc-tives are in ptracers ad.flow.

5.2.3.3 Store directives for 3-level checkpointing

The storing that is required at each period of the 3-level checkpointing is con-troled by three top-level headers.

do ilev_3 = 1, nchklev_3# include ‘‘checkpoint_lev3.h’’




...

end doend do

end do

All files checkpoint lev?.h are contained in directory pkg/autodiff/.

5.2.3.4 Changing the default AD tool flags: ad options files

5.2.3.5 Hand-written adjoint code

5.2.4 The cost function (dependent variable)

The cost function J is referred to as the dependent variable. It is a functionof the input variables �u via the composition J (�u) = J (M(�u)). The input arereferred to as the independent variables or control variables. All aspects relevantto the treatment of the cost function J (parameter setting, initialization, accu-mulation, final evaluation), are controlled by the package pkg/cost. The aspectsrelevant to the treatment of the independent variables are controlled by thepackage pkg/ctrl and will be treated in the next section.

the_model_main||-- initialise_fixed| || |-- packages_readparms| || |-- cost_readparms| o||-- the_main_loop

... ||-- initialise_varia| || |-- packages_init_variables| || |-- cost_init| o||-- do iloop = 1,nTimeSteps| |-- forward_step| |-- cost_tile| | || | |-- cost_tracer| end do||-- cost_finalo

Figure 5.3:

5.2.4.1 Enabling the package

packages.conf, ECCO CPPOPTIONS.h


• The package is enabled by adding cost to your file packages.conf (see Sec-tion ???)

•N.B.: In general the following packages ought to be enabled simultane-

ously: autodiff, cost, ctrl. The basic CPP option to enable the cost functionis ALLOW COST. Each specific cost function contribution has its own op-tion. For the present example the option is ALLOW COST TRACER. Allcost-specific options are set in ECCO CPPOPTIONS.h Since the cost function isusually used in conjunction with automatic differentiation, the CPP option AL-LOW ADJOINT RUN (file CPP OPTIONS.h) and ALLOW AUTODIFF TAMC(file ECCO CPPOPTIONS.h) should be defined.

5.2.4.2 Initialization

The initialization of the cost package is readily enabled as soon as the CPPoption ALLOW COST is defined.

• Parameters: cost readparmsThis S/R reads runtime flags and parameters from file data.cost. For thepresent example the only relevant parameter read is mult tracer. Thismultiplier enables different cost function contributions to be switched on( = 1.) or off ( = 0.) at runtime. For more complex cost functions whichinvolve model vs. data misfits, the corresponding data filenames and dataspecifications (start date and time, period, ...) are read in this S/R.

• Variables: cost initThis S/R initializes the different cost function contributions. The contri-bution for the present example is objf tracer which is defined on eachtile (bi,bj).

5.2.4.3 Accumulation

• cost tile, cost tracer

The ’driver’ routine cost tile is called at the end of each time step. Within this’driver’ routine, S/R are called for each of the chosen cost function contribu-tions. In the present example (ALLOW COST TRACER), S/R cost traceris called. It accumulates objf tracer according to eqn. (??).

5.2.4.4 Finalize all contributions

• cost final

At the end of the forward integration S/R cost final is called. It accumulatesthe total cost function fc from each contribution and sums over all tiles:

J = fc = mult tracer∑

global sum

nSx, nSy∑bi, bj

objf tracer(bi, bj) + ... (5.13)


The total cost function fc will be the ’dependent’ variable in the argument listfor TAMC, i.e.

tamc -output ’fc’ ...

5.2.5 The control variables (independent variables)

The control variables are a subset of the model input (initial conditions, bound-ary conditions, model parameters). Here we identify them with the variable�u. All intermediate variables whose derivative w.r.t. control variables do notvanish are called active variables. All subroutines whose derivative w.r.t. thecontrol variables don’t vanish are called active routines. Read and write opera-tions from and to file can be viewed as variable assignments. Therefore, files towhich active variables are written and from which active variables are read arecalled active files. All aspects relevant to the treatment of the control variables(parameter setting, initialization, perturbation) are controlled by the packagepkg/ctrl.

5.2.5.1 genmake and CPP options

• genmake, CPP OPTIONS.h, ECCO CPPOPTIONS.h

To enable the directory to be included to the compile list, ctrl has to be addedto the enable list in .genmakerc or in genmake itself (analogous to cost package,cf. previous section). Each control variable is enabled via its own CPP optionin ECCO CPPOPTIONS.h.

5.2.5.2 Initialization

• Parameters: ctrl readparmsThis S/R reads runtime flags and parameters from file data.ctrl. For thepresent example the file contains the file names of each control variablethat is used. In addition, the number of wet points for each control vari-able and the net dimension of the space of control variables (counting wetpoints only) nvarlength is determined. Masks for wet points for each tile(bi, bj) and vertical layer k are generated for the three relevant categorieson the C-grid: nWetCtile for tracer fields, nWetWtile for zonal velocityfields, nWetStile for meridional velocity fields.

• Control variables, control vector, and their gradients: ctrl unpackTwo important issues related to the handling of the control variables inthe MITGCM need to be addressed. First, in order to save memory, thecontrol variable arrays are not kept in memory, but rather read from fileand added to the initial fields during the model initialization phase. Sim-ilarly, the corresponding adjoint fields which represent the gradient of thecost function w.r.t. the control variables are written to file at the end of


*************the_main_loop*************||--- initialise_varia| || ...| |--- packages_init_varia| | || | ...| | |--- #ifdef ALLOW_ADJOINT_RUN| | | call ctrl_map_ini| | | call cost_ini| | | #endif| | ...| | o| ...| o

...|--- #ifdef ALLOW_ADJOINT_RUN| call ctrl_map_forcing| #endif

...|--- #ifdef ALLOW_TAMC_CHECKPOINTING

do ilev_3 = 1,nchklev_3| do ilev_2 = 1,nchklev_2| do ilev_1 = 1,nchklev_1| iloop = (ilev_3-1)*nchklev_2*nchklev_1 +| (ilev_2-1)*nchklev_1 + ilev_1| #else| do iloop = 1, nTimeSteps| #endif| || |--- call forward_step| || |--- #ifdef ALLOW_COST| | call cost_tile| | #endif| || | enddo| o||--- #ifdef ALLOW_COST| call cost_final| #endifo

Figure 5.4:


the_model_main||-- initialise_fixed| || |-- packages_readparms| || |-- ctrl_init - initialise control| o package||-- ctrl_unpack - unpack control vector||-- adthe_main_loop - forward/adjoint run| || |-- initialise_variables| | || | |-- packages_init_variables| | || | |-- ctrl_map_ini - link init. state and| | o parameters to control| | variables| |-- ctrl_map_forcing - link forcing fields to| ... control variables||-- ctrl_pack - pack control vector

Figure 5.5:

the adjoint integration. Second, in addition to the files holding the 2-dim.and 3-dim. control variables and the corresponding cost gradients, a 1-dim. control vector and gradient vector are written to file. They containonly the wet points of the control variables and the corresponding gradi-ent. This leads to a significant data compression. Furthermore, an optionis available (ALLOW NONDIMENSIONAL CONTROL IO) to non-dimensionalisethe control and gradient vector, which otherwise would contain differentpieces of different magnitudes and units. Finally, the control and gradientvector can be passed to a minimization routine if an update of the controlvariables is sought as part of a minimization exercise.

The files holding fields and vectors of the control variables and gradientare generated and initialised in S/R ctrl unpack.

5.2.5.3 Perturbation of the independent variables

The dependency flow for differentiation w.r.t. the controls starts with adding aperturbation onto the input variable, thus defining the independent or controlvariables for TAMC. Three types of controls may be considered:

• ctrl map ini (initial value sensitivity):Consider as an example the initial tracer distribution tr1 as control vari-able. After tr1 has been initialised in ini tr1 (dynamical variables suchas temperature and salinity are initialised in ini fields), a perturbation


anomaly is added to the field in S/R ctrl map ini

u = u[0] + Δutr1(...) = tr1ini(...) + xx tr1(...)

(5.14)

xx tr1 is a 3-dim. global array holding the perturbation. In the caseof a simple sensitivity study this array is identical to zero. However,it’s specification is essential in the context of automatic differentiationsince TAMC treats the corresponding line in the code symbolically whendetermining the differentiation chain and its origin. Thus, the variablenames are part of the argument list when calling TAMC:

tamc -input ’xx_tr1 ...’ ...

Now, as mentioned above, the MITGCM avoids maintaining an array foreach control variable by reading the perturbation to a temporary arrayfrom file. To ensure the symbolic link to be recognized by TAMC, ascalar dummy variable xx tr1 dummy is introduced and an ’active read’routine of the adjoint support package pkg/autodiff is invoked. The read-procedure is tagged with the variable xx tr1 dummy enabling TAMCto recognize the initialization of the perturbation. The modified call ofTAMC thus reads

tamc -input ’xx_tr1_dummy ...’ ...

and the modified operation to (5.14) in the code takes on the form

call active_read_xyz(& ..., tmpfld3d, ..., xx_tr1_dummy, ... )

tr1(...) = tr1(...) + tmpfld3d(...)

Note, that reading an active variable corresponds to a variable assignment.Its derivative corresponds to a write statement of the adjoint variable,followed by a reset. The ’active file’ routines have been designed to supportactive read and corresponding adjoint active write operations (and viceversa).

• ctrl map forcing (boundary value sensitivity):The handling of boundary values as control variables proceeds exactlyanalogous to the initial values with the symbolic perturbation taking placein S/R ctrl map forcing. Note however an important difference: Since theboundary values are time dependent with a new forcing field applied ateach time steps, the general problem may be thought of as a new con-trol variable at each time step (or, if the perturbation is averaged over acertain period, at each N timesteps), i.e.

uforcing = { uforcing(tn) }n =1,...,nTimeSteps


In the current example an equilibrium state is considered, and only aninitial perturbation to surface forcing is applied with respect to the equi-librium state. A time dependent treatment of the surface forcing is im-plemented in the ECCO environment, involving the calendar (cal ) andexternal forcing (exf ) packages.

• ctrl map params (parameter sensitivity):This routine is not yet implemented, but would proceed proceed along thesame lines as the initial value sensitivity. The mixing parameters diffkrand kapgm are currently added as controls in ctrl map ini.F.

5.2.5.4 Output of adjoint variables and gradient

Several ways exist to generate output of adjoint fields.

• ctrl map ini, ctrl map forcing:

– xx ...: the control variable fieldsBefore the forward integration, the control variables are read fromfile xx ... and added to the model field.

– adxx ...: the adjoint variable fields, i.e. the gradient ∇uJ for eachcontrol variableAfter the adjoint integration the corresponding adjoint variables arewritten to adxx ....

• ctrl unpack, ctrl pack:

– vector ctrl: the control vectorAt the very beginning of the model initialization, the updated com-pressed control vector is read (or initialised) and distributed to 2-dim.and 3-dim. control variable fields.

– vector grad: the gradient vectorAt the very end of the adjoint integration, the 2-dim. and 3-dim.adjoint variables are read, compressed to a single vector and writtento file.

• addummy in stepping:In addition to writing the gradient at the end of the forward/adjoint in-tegration, many more adjoint variables of the model state at intermediatetimes can be written using S/R addummy in stepping. This routine is partof the adjoint support package pkg/autodiff (cf.f. below). The procedure isenabled using via the CPP-option ALLOW AUTODIFF MONITOR(file ECCO CPPOPTIONS.h). To be part of the adjoint code, the corre-sponding S/R dummy in stepping has to be called in the forward model


(S/R the main loop) at the appropriate place. The adjoint common blocksare extracted from the adjoint code via the header file adcommon.h.

dummy in stepping is essentially empty, the corresponding adjoint routineis hand-written rather than generated automatically. Appropriate flow di-rectives (dummy in stepping.flow) ensure that TAMC does not automati-cally generate addummy in stepping by trying to differentiate dummy in stepping,but instead refers to the hand-written routine.

dummy in stepping is called in the forward code at the beginning of eachtimestep, before the call to dynamics, thus ensuring that addummy in steppingis called at the end of each timestep in the adjoint calculation, after thecall to addynamics.

addummy in stepping includes the header files adcommon.h. This headerfile is also hand-written. It contains the common blocks /addynvars r/,/addynvars cd/, /addynvars diffkr/, /addynvars kapgm/, /adtr1 r/,/adffields/, which have been extracted from the adjoint code to enableaccess to the adjoint variables.

WARNING: If the structure of the common blocks /dynvars r/, /dyn-vars cd/, etc., changes similar changes will occur in the adjoint commonblocks. Therefore, consistency between the TAMC-generated commonblocks and those in adcommon.h have to be checked.

5.2.5.5 Control variable handling for optimization applications

In optimization mode the cost function J (u) is sought to be minimized with re-spect to a set of control variables δJ = 0, in an iterative manner. The gradient∇uJ |u[k] together with the value of the cost function itself J (u[k]) at iterationstep k serve as input to a minimization routine (e.g. quasi-Newton method,conjugate gradient, ... [21]) to compute an update in the control variable foriteration step k + 1

u[k+1] = u[0] + Δu[k+1] satisfying J (u[k+1]

)< J (u[k]

)u[k+1] then serves as input for a forward/adjoint run to determine J and ∇uJat iteration step k + 1. Tab. ?? sketches the flow between forward/adjointmodel and the minimization routine.


u[0] , Δu[k]??y

u[k] = u[0] + Δu[k]forward−→ v[k] = M

`u[k]

´ forward−→ J[k] = J `M

`u[k]

´´

??y

∇uJ[k](δJ ) = T∗ · ∇vJ |v[k](δJ )

adjoint←− ad v[k](δJ ) = ∇vJ |v[k](δJ )

adjoint←− adJ = δJ

????yJ[k], ∇uJ[k]

J[k] , ∇uJ[k] −→ minimisation −→ Δu[k+1]

??y

Δu[k+1]

The routines ctrl unpack and ctrl pack provide the link between the modeland the minimization routine. As described in Section ?? the unpack and packroutines read and write control and gradient vectors which are compressed tocontain only wet points, in addition to the full 2-dim. and 3-dim. fields. Thecorresponding I/O flow looks as follows:

vector ctrl <k>?y

ctrl unpack?y

xx theta0...<k>

xx salt0...<k>read−→ forward integration

.

.

.↓

adxx theta0...<k>

adjoint integrationwrite−→ adxx salt0...<k>

.

.

. ?y

ctrl pack?y

vector grad <k>

ctrl unpack reads the updated control vector vector ctrl <k>. It dis-tributes the different control variables to 2-dim. and 3-dim. files xx ...<k>. Atthe start of the forward integration the control variables are read from xx ...<k>and added to the field. Correspondingly, at the end of the adjoint integrationthe adjoint fields are written to adxx ...<k>, again via the active file routines.Finally, ctrl pack collects all adjoint files and writes them to the compressedvector file vector grad <k>.


5.3 Sensitivity of Air-Sea Exchange to Tracer

Injection Site

The MITGCM has been adapted to enable AD using TAMC or TAF. Thepresent description, therefore, is specific to the use of TAMC or TAF as ADtool. The following sections describe the steps which are necessary to generatea tangent linear or adjoint model of the MITGCM. We take as an example thesensitivity of carbon sequestration in the ocean. The AD-relevant hooks in thecode are sketched in 5.2, 5.4.

5.3.1 Overview of the experiment

We describe an adjoint sensitivity analysis of out-gassing from the ocean intothe atmosphere of a carbon-like tracer injected into the ocean interior (see [29]).

5.3.1.1 Passive tracer equation

For this work the MITGCM was augmented with a thermodynamically inac-tive tracer, C. Tracer residing in the ocean model surface layer is out-gassedaccording to a relaxation time scale, μ. Within the ocean interior, the tracer ispassively advected by the ocean model currents. The full equation for the timeevolution

∂C

∂t= −U · ∇C − μC + Γ(C) + S (5.15)

also includes a source term S. This term represents interior sources of C suchas would arise due to direct injection. The velocity term, U , is the sum ofthe model Eulerian circulation and an eddy-induced velocity, the latter param-eterized according to Gent/McWilliams ([16, 17]). The convection function, Γ,mixes C vertically wherever the fluid is locally statically unstable.

The out-gassing time scale, μ, in eqn. (5.15) is set so that 1/μ ∼ 1 year forthe surface ocean and μ = 0 elsewhere. With this value, eqn. (5.15) is valid asa prognostic equation for small perturbations in oceanic carbon concentrations.This configuration provides a powerful tool for examining the impact of large-scale ocean circulation on CO2 out-gassing due to interior injections. As sourcewe choose a constant in time injection of S = 1 mol/s.

5.3.1.2 Model configuration

The model configuration employed has a constant 4◦ × 4◦ resolution horizontalgrid and realistic geography and bathymetry. Twenty vertical layers are usedwith vertical spacing ranging from 50 m near the surface to 815 m at depth.Driven to steady-state by climatological wind-stress, heat and fresh-water forc-ing the model reproduces well known large-scale features of the ocean generalcirculation.

5.3. SENSITIVITY OF AIR-SEA EXCHANGE TO TRACER INJECTION SITE 295

5.3.1.3 Out-gassing cost function

To quantify and understand out-gassing due to injections of C in eqn. (5.15),we define a cost function J that measures the total amount of tracer out-gassedat each timestep:

J (t = T ) =∫ t=T

t=0

∫A

μC dAdt (5.16)

Equation(5.16) integrates the out-gassing term, μC, from (5.15) over the entireocean surface area, A, and accumulates it up to time T . Physically, J can bethought of as representing the amount of CO2 that our model predicts wouldbe out-gassed following an injection at rate S. The sensitivity of J to thespatial location of S, ∂J

∂S , can be used to identify regions from which circulationwould cause CO2 to rapidly out-gas following injection and regions in whichCO2 injections would remain effectively sequestered within the ocean.


The model configuration for this experiment resides under the directory verifica-tion/carbon/. The code customization routines are in verification/carbon/code/:

• .genmakerc

• COST CPPOPTIONS.h

• CPP EEOPTIONS.h

• CPP OPTIONS.h

• CTRL OPTIONS.h

• ECCO OPTIONS.h

• SIZE.h

• adcommon.h

• tamc.h

The runtime flag and parameters settings are contained in verification/carbon/input/,together with the forcing fields and and restart files:

• data

• data.cost

• data.ctrl

• data.gmredi

• data.grdchk

• data.optim


• data.pkg

• eedata

• topog.bin

• windx.bin, windy.bin

• salt.bin, theta.bin

• SSS.bin, SST.bin

• pickup*

Finally, the file to generate the adjoint code resides in adjoint/:

• makefile

Below we describe the customizations of this files which are specific to thisexperiment.

5.3.2.1 File .genmakerc

This file overwrites default settings of genmake. In the present example it is usedto switch on the following packages which are related to automatic differentiationand are disabled by default:

set ENABLE=( autodiff cost ctrl ecco gmredi grdchk kpp )Other packages which are not needed are switched off:

set DISABLE=( aim obcs zonal filt shap filt cal exf )

5.3.2.2 File COST CPPOPTIONS.h, CTRL OPTIONS.h

These files used to contain package-specific CPP-options (see Section ??). Fortechnical reasons those options have been grouped together in the file ECCO OPTIONS.h.To retain the modularity, the files have been kept and contain the standard in-clude of the CPP OPTIONS.h file.

5.3.2.3 File CPP EEOPTIONS.h

This file contains ’wrapper’-specific CPP options. It only needs to be changedif the code is to be run in a parallel environment (see Section ??).

5.3.2.4 File CPP OPTIONS.h

This file contains model-specific CPP options (see Section ??). Most optionsare related to the forward model setup. They are identical to the global steadycirculation setup of verification/global ocean.90x40x15/. The three options spe-cific to this experiment are

#define ALLOW PASSIVE TRACERThis flag enables the code to carry through the advection/diffusion of a passive


tracer along the model integration.#define ALLOW MIT ADJOINT RUN

This flag enables the inclusion of some AD-related fields concerning initializa-tion, link between control variables and forward model variables, and the call tothe top-level forward/adjoint subroutine adthe main loop instead of the main loop.

#define ALLOW GRADIENT CHECKThis flag enables the gradient check package. After computing the unperturbedcost function and its gradient, a series of computations are performed for which• an element of the control vector is perturbed• the cost function w.r.t. the perturbed element is computed• the difference between the perturbed and unperturbed cost function is com-puted to compute the finite difference gradient• the finite difference gradient is compared with the adjoint-generated gradient.The gradient check package is further described in Section ???.

5.3.2.5 File ECCO OPTIONS.h

The CPP options of several AD-related packages are grouped in this file:

• Overall ECCO-related execution modus:These determine whether a pure forward run, a sensitivity run or an iter-ation of optimization is performed. These options are not needed in thepresent context.

• Adjoint support package: pkg/autodiff/This package contains hand-written adjoint code such as active file han-dling, flow directives for files which must not be differentiated, and TAMC-specific header files.

#define ALLOW AUTODIFF TAMCdefines TAMC-related features in the code.

#define ALLOW TAMC CHECKPOINTINGenables the checkpointing feature of TAMC (see Section ??). In thepresent example a 3-level checkpointing is implemented. The code containsthe relevant store directives, common block and tape initializations, stor-ing key computation, and loop index handling. The checkpointing lengthat each level is defined in file tamc.h, cf. below. The out and intermediateloop directivs are contained in the files checkpoint lev3 directives.h, check-point lev2 directives.h (package pkg/autodiff).

#define ALLOW AUTODIFF MONITOORenables the monitoring of intermediate adjoint variables (see Section ??).

#define ALLOW DIVIDED ADJOINTenables adjoint dump and restart (see Section ??).

• Cost function package: pkg/cost/This package contains all relevant routines for initializing, accumulatingand finalizing the cost function (see Section ??).

#define ALLOW COST


enables all general aspects of the cost function handling, in particular thehooks in the forward code for initializing, accumulating and finalizing thecost function.

#define ALLOW COST TRACERincludes the call to the cost function for this particular experiment, eqn.(5.16).

• Control variable package: pkg/ctrl/This package contains all relevant routines for the handling of the controlvector. Each control variable can be enabled/disabled with its own flag:

#define ALLOW THETA0 CONTROL initial temperature#define ALLOW SALT0 CONTROL initial salinity#define ALLOW TR0 CONTROL initial passive tracer concentration#define ALLOW TAUU0 CONTROL zonal wind stress#define ALLOW TAUV0 CONTROL meridional wind stress#define ALLOW SFLUX0 CONTROL freshwater flux#define ALLOW HFLUX0 CONTROL heat flux#define ALLOW DIFFKR CONTROL diapycnal diffusivity#undef ALLOW KAPPAGM CONTROL isopycnal diffusivity

5.3.2.6 File SIZE.h

The file contains the grid point dimensions of the forward model. It is identicalto the verification/exp2/:

sNx = 90sNy = 40Nr = 20

It corresponds to a single-tile/single-processor setup: nSx = nSy = 1, nPx =nPy = 1, with standard overlap dimensioning OLx = OLy = 3.

5.3.2.7 File adcommon.h

This file contains common blocks of some adjoint variables that are generatedby TAMC. The common blocks are used by the adjoint support routine ad-dummy in stepping which needs to access those variables:

common /addynvars r/ is related to DYNVARS.hcommon /addynvars cd/ is related to DYNVARS.hcommon /addynvars diffkr/ is related to DYNVARS.hcommon /addynvars kapgm/ is related to DYNVARS.hcommon /adtr1 r/ is related to TR1.hcommon /adffields/ is related to FFIELDS.h

Note that if the structure of the common block changes in the above headerfiles of the forward code, the structure of the adjoint common blocks will changeaccordingly. Thus, it has to be made sure that the structure of the adjoint com-mon block in the hand-written file adcommon.h complies with the automaticallygenerated adjoint common blocks in adjoint model.F. The header file is enabledvia the CPP-option ALLOW AUTODIFF MONITOR.


5.3.2.8 File tamc.h

This routine contains the dimensions for TAMC checkpointing and some indicesrelevant for storing ky computations.

• #ifdef ALLOW TAMC CHECKPOINTING3-level checkpointing is enabled, i.e. the timestepping is divided intothree different levels (see Section ??). The model state of the outer-most (nchklev 3) and the intermediate (nchklev 2) timestepping loop arestored to file (handled in the main loop). The innermost loop (nchklev 1)avoids I/O by storing all required variables to common blocks. This storingmay also be necessary if no checkpointing is chosen (nonlinear functions,if-statements, iterative loops, ...). In the present example the dimensionsare chosen as follows:

nchklev 1 = 36nchklev 2 = 30nchklev 3 = 60

To guarantee that the checkpointing intervals span the entire integrationperiod the following relation must be satisfied:

nchklev 1*nchklev 2*nchklev 3 ≥ nTimeStepswhere nTimeSteps is either specified in data or computed via

nTimeSteps = (endTime-startTime)/deltaTClock .

• #undef ALLOW TAMC CHECKPOINTINGNo checkpointing is enabled. In this case the relevant counter is nchklev 0.Similar to above, the following relation has to be satisfied

nchklev 0 ≥ nTimeSteps.

The following parameters may be worth describing:isbytemaxpass

5.3.2.9 File makefile

This file contains all relevant parameter flags and lists to run TAMC or TAF.It is assumed that TAMC is available to you, either locally, being installed onyour network, or remotely through the ’TAMC Utility’. TAMC is called withthe command tamc followed by a number of options. They are described indetail in the TAMC manual [18]. Here we briefly discuss the main flags used inthe makefile. The standard output for TAF is written to file taf.log.

tamc -input <variable names> -output <variable name> -i4 -r4 ...-toplevel <S/R name> -reverse <file names>

taf -input <variable names> -output <variable name> -i4 -r4 ...-toplevel <S/R name> -reverse <file names>-flow taf flow.log -nonew arg


• -toplevel <S/R name>Name of the toplevel routine, with respect to which the control flow anal-ysis is performed.

• -input <variable names>List of independent variables u with respect to which the dependent vari-able J is differentiated.

• -output <variable name>Dependent variable J which is to be differentiated.

• -reverse <file names>Adjoint code is generated to compute the sensitivity of an independentvariable w.r.t. many dependent variables. In the discussion of Section??? the generated adjoint top-level routine computes the product of thetransposed Jacobian matrix MT times the gradient vector ∇vJ .<file names> refers to the list of files .f which are to be analyzed byTAMC. This list is generally smaller than the full list of code to be com-piled. The files not contained are either above the top-level routine (someinitializations), or are deliberately hidden from TAMC, either becausehand-written adjoint routines exist, or the routines must not (or don’thave to) be differentiated. For each routine which is part of the flow treeof the top-level routine, but deliberately hidden from TAMC (or for eachpackage which contains such routines), a corresponding file .flow existscontaining flow directives for TAMC.

• -i4 -r4

• -flow taf flow.logWill cause TAF to produce a flow listing file named taf flow.log in whichthe set of active and passive variables are identified for each subroutine.

• -nonew argThe default in the order of the parameter list of adjoint routines haschanged. Before TAF 1.3 the default was compatible with the TAMC-generated list. As of TAF 1.3 the order of adjoint routine parameter listsis no longer copatible with TAMC. To restore compatibility when usingTAF 1.3 and higher, this argument is needed. It is currently crucial to usesince all hand-written adjoint routines refer to the TAMC default.

5.3.2.10 The input parameter files

File data

File data.cost

File data.ctrl


File data.gmredi

File data.grdchk

File data.optim

File data.pkg

File eedata

File topog.binContains two-dimendional bathymetry information

File windx.bin, windy.bin, salt.bin, theta.bin, SSS.bin, SST.binThese contain the initial values (salinity, temperature, salt.bin, theta.bin), sur-face boundary values (surface wind stresses, (windx.bin, windy.bin), and surfacerestoring fields (SSS.bin, SST.bin).

File pickup*Contains model state after model spinup.

5.3.3 Compiling the model and its adjoint

The built process of the adjoint model is slightly more complex than that ofcompiling the forward code. The main reason is that the adjoint code generationrequires a specific list of routines that are to be differentiated (as opposed tothe automatic generation of a list of files to be compiled by genmake). Thislist excludes routines that don’t have to be or must not be differentiated. Forsome of the latter routines flow directives may be necessary, a list of which hasto be given as well. For this reason, a separate makefile is currently maintainedin the directory adjoint/. This makefile is responsible for the adjoint codegeneration.

In the following we describe the build process step by step, assuming youare in the directory bin/. A summary of steps to follow is given at the end.

Adjoint code generation and compilation – step by step

1. ln -s ../verification/???/code/.genmakerc .ln -s ../verification/???/code/*.[Fh] .Link your customized genmake options, header files, and modified code tothe compile directory.

2. ../tools/genmake -makefileGenerate your Makefile (cf. Section ???).


3. make dependDependency analysis for the CPP pre-compiler (cf. Section ???).

4. cd ../adjointmake adtaf or make adtamcDepending on whether you have TAF or TAMC at your disposal, you’llchoose adtaf or adtamc as your make target for the makefile in the direc-tory adjoint/. Several things happen at this stage.

(a) make adrestore, make ftlrestoreThe initial template files adjoint model.F and tangentlinear model.Fin pkg/autodiff which are part of the compiling list created by gen-make are restored.

(b) make depend, make small fThe bin/ directory is brought up to date, i.e. for recent changes inheader or source code .[Fh], corresponding .f routines are generatedor re-generated. Note that here, only CPP precompiling is performed;no object code .o is generated as yet. Precompiling is necessary forTAMC to see the full code.

(c) make allcodeAll Fortran routines *.f in bin/ are concatenated into a single filecalled tamc code.f.

(d) make admodeltaf/admodeltamcAdjoint code is generated by TAMC or TAF. The adjoint code iswritten to the file tamc code ad.f. It contains all adjoint routinesof the forward routines concatenated in tamc code.f. For a givenforward routines subroutine routinename the adjoint routine isnamed adsubroutine routinename by default (that default can bechanged via the flag -admark <markname>). Furthermore, it maycontain modified code which incorporates the translation of adjointstore directives into specific Fortran code. For a given forward rou-tines subroutine routinename the modified routine is named mdsubroutineroutinename. TAMC or TAF info is written to file tamc code.protor taf.log, respectively.

(e) make adchangeThe multi-threading capability of the MITGCM requires a slightchange in the parameter list of some routines that are related to toactive file handling. This post-processing invokes the sed script ad-joint ecco sed.com to insert the threading counter myThId into theparameter list of those subroutines. The resulting code is writtento file tamc code sed ad.f and appended to the file adjoint model.F.This concludes the adjoint code generation.

5. cd ../binmake


The file adjoint model.F now contains the full adjoint code. All routinesare now compiled.

N.B.: The targets make adtaf/adtamc now comprise a series of targets thatin previous versions had to be invoked separarely. This was probably preferableat a more experimental stage, but has now been dropped in favour of a morestraightforward build process.

Adjoint code generation and compilation – summary

cd bin

ln -s ../verification/my experiment/code/.genmakerc .

ln -s ../verification/my experiment/code/*.[Fh] .

../tools/genmake -makefile

make depend

cd ../adjoint

make adtaf <OR: make adtamc>

contains the targets:

adrestore small f allcode admodeltaf/admodeltamc adchange

cd ../bin

make


5.4 The gradient check package

An indispensable test to validate the gradient computed via the adjoint isa comparison against finite difference gradients. The gradient check packagepkg/grdchk enables such tests in a straightforward and easy manner. The driverroutine grdchk main is called from the model main after the gradient has beencomputed via the adjoint model (cf. flow chart ???).

The gradient check proceeds as follows: The i−th component of the gradient(∇uJ T )i is compared with the following finite-difference gradient:(∇uJ T

)i

vs.∂J∂ui

=J (ui + ε) − J (ui)

ε

A gradient check at point ui may generally considered to be successful if thedeviation of the ratio between the adjoint and the finite difference gradient fromunity is less than 1 percent,

1 − (gradJ )i(adjoint)(gradJ )i(finite difference)

< 1%

5.4.1 Code description


The relevant CPP precompile options are set in the following files:

• .genmakercoption grdchk is added to the enable list (alternatively, genmake maybe invoked with the option -enable=grdchk).

• CPP OPTIONS.hTogether with the flag ALLOW ADJOINT RUN, define the flag AL-LOW GRADIENT CHECK.

The relevant runtime flags are set in the files

• data.pkgSet useGrdchk = .TRUE.

• data.grdchk

– grdchk eps

– nbeg

– nstep

– nend

– grdchkvarindex

5.5. ADJOINT DUMP & RESTART – DIVIDED ADJOINT (DIVA) 305

the_model_main||-- ctrl_unpack|-- adthe_main_loop - unperturbed cost function and|-- ctrl_pack adjoint gradient are computed here||-- grdchk_main

||-- grdchk_init|-- do icomp=... - loop over control vector elements

||-- grdchk_loc - determine location of icomp on grid||-- grdchk_getxx - get control vector component from file| perturb it and write back to file|-- grdchk_getadxx - get gradient component calculated| via adjoint|-- the_main_loop - forward run and cost evaluation| with perturbed control vector element|-- calculate ratio of adj. vs. finite difference gradient||-- grdchk_setxx - Reset control vector element||-- grdchk_print - print results

Figure 5.6:

5.5 Adjoint dump & restart – divided adjoint(DIVA)

Patrick Heimbach & Geoffrey Gebbie, MIT/EAPS, 07-Mar-2003NOTE:

THIS SECTION IS SUBJECT TO CHANGE. IT REFERS TO TAF-1.4.26.

Previous TAF versions are incomplete and have problems withboth TAF options ’-pure’ and ’-mpi’.

The code which is tuned to the DIVA implementation of this TAFversion is checkpoint50 (MITgcm) and ecco c50 e28 (ECCO).

5.5.1 Introduction

Most high performance computing (HPC) centres require the use of batch jobsfor code execution. Limits in maximum available CPU time and memory mayprevent the adjoint code execution from fitting into any of the available queues.This presents a serious limit for large scale / long time adjoint ocean and climatemodel integrations. The MITgcm itself enables the split of the total modelintegration into sub-intervals through standard dump/restart of/from the fullmodel state. For a similar procedure to run in reverse mode, the adjoint modelrequires, in addition to the model state, the adjoint model state, i.e. all variableswith derivative information which are needed in an adjoint restart. This adjointdump & restart is also termed ’divided adjoint (DIVA).


For this to work in conjunction with automatic differentiation, an AD toolneeds to perform the following tasks:

1. identify an adjoint state, i.e. those sensitivities whose accumulation isinterrupted by a dump/restart and which influence the outcome of thegradient. Ideally, this state consists of

• the adjoint of the model state,

• the adjoint of other intermediate results (such as control variables,cost function contributions, etc.)

• bookkeeping indices (such as loop indices, etc.)

2. generate code for storing and reading adjoint state variables

3. generate code for bookkeeping , i.e. maintaining a file with index infor-mation

4. generate a suitable adjoint loop to propagate adjoint values for dump/restartwith a minimum overhad of adjoint intermediate values.

TAF (but not TAMC!) generates adjoint code which performs the abovespecified tasks. It is closely tied to the adjoint multi-level checkpointing. Theadjoint state is dumped (and restarted) at each step of the outermost checkpoint-ing level and adjoint intergration is performed over one outermost checkpointinginterval. Prior to the adjoint computations, a full foward sweep is performed togenerate the outermost (forward state) tapes and to calculate the cost function.In the current implementation, the forward sweep is immediately followed bythe first adjoint leg. Thus, in theory, the following steps are performed (auto-matically)

• 1st model call:This is the case if file costfinal does not exist. S/R mdthe main loop iscalled.

1. calculate forward trajectory and dump model state after each outer-most checkpointing interval to files tapelev3

2. calculate cost function fc and write it to file costfinal

• 2nd and all remaining model call:This is the case if file costfinal does exist. S/R adthe main loop iscalled.

1. (forward run and cost function call is avoided since all values areknown)

– if 1st adjoint leg:create index file divided.ctrl which contains info on currentcheckpointing index ilev3

5.5. ADJOINT DUMP & RESTART – DIVIDED ADJOINT (DIVA) 307

– if not i-th adjoint leg:adjoint picks up at ilev3 = nlev3 − i+ 1 and runs to nlev3 − i

2. perform adjoint leg from nlev3 − i+ 1 to nlev3 − i

3. dump adjoint state to file snapshot

4. dump index file divided.ctrl for next adjoint leg

5. in the last step the gradient is written.

A few modififications were performed in the forward code, obvious ones suchas adding the corresponding TAF-directive at the appropriate place, and lessobvious ones (avoid some re-initializations, when in an intermediate adjointintegration interval).

[For TAF-1.4.20 a number of hand-modifications were necessary to compen-sate for TAF bugs. Since we refer to TAF-1.4.26 onwards, these modificationsare not documented here].

5.5.2 Recipe 1: single processor

1. In ECCO CPPOPTIONS.h set:

#define ALLOW_DIVIDED_ADJOINT

#undef ALLOW_DIVIDED_ADJOINT_MPI

2. Generate adjoint code. Using the TAF option ’-pure’, two codes aregenerated:

• mdthe main loop:Is responsible for the forward trajectory, storing of outermost check-point levels to file, computation of cost function, and storing of costfunction to file (1st step).

• adthe main loop:Is responsible for computing one adjoint leg, dump adjoint state tofile and write index info to file (2nd and consecutive steps).for adjoint code generation, e.g. add ’-pure’ to TAF option list

make adtaf

• One modification needs to be made to adjoint codes in S/R adecco the main loop:There’s a remaining issue with the ’-pure’ option. The ’call ad...’between ’call ad...’ and the read of the snapshot file should becalled only in the firt adjoint leg between nlev3 and nlev3 − 1. Inthe ecco-branch, the following lines should be bracketed by an if(idivbeg .GE. nchklev 3) then, thus:

...

xx_psbar_mean_dummy = onetape_xx_psbar_mean_dummy_3h(1)

xx_tbar_mean_dummy = onetape_xx_tbar_mean_dummy_4h(1)

xx_sbar_mean_dummy = onetape_xx_sbar_mean_dummy_5h(1)


call barrier( mythid )

cAdd(

if (idivbeg .GE. nchklev_3) then

cAdd)

call adcost_final( mythid )


call adcost_sst( mythid )

call adcost_ssh( mythid )

call adcost_hyd( mythid )

call adcost_averagesfields( mytime,myiter,mythid )


cAdd(

endif

cAdd)

C----------------------------------------------

C read snapshot

C----------------------------------------------

if (idivbeg .lt. nchklev_3) then

open(unit=77,file=’snapshot’,status=’old’,form=’unformatted’,

$iostat=iers)

...

For the main code, in all likelihood the block which needs to bebracketed consists of adcost final only.

• Now the code can be copied as usual to adjoint model.F and thenbe compiled:

make adchange

then compile

5.5.3 Recipe 2: multi processor (MPI)

1. On the machine where you execute the code (most likely not the ma-chine where you run TAF) find the includes directory for MPI containingmpif.h. Either copy mpif.h to the machine where you generate the .ffiles before TAF-ing, or add the path to the includes directory to yougenmake platform setup, TAF needs some MPI parameter settings (es-sentially mpi comm world and mpi integer) to incorporate those in theadjoint code.

2. In ECCO CPPOPTIONS.h set

#define ALLOW_DIVIDED_ADJOINT

#define ALLOW_DIVIDED_ADJOINT_MPI

This will include the header file mpif.h into the top level routine for TAF.

3. Add the TAF option ’-mpi’ to the TAF argument list in the makefile.

4. Follow the same steps as in Recipe 1 (previous section).

That’s it. Good luck & have fun.

Chapter 6

Physical Parameterizationand Packages

Within this chapter, the MITgcm “packages” are described. Initially, “pack-ages” were conceived to group source code files together based upon their func-tionality. Each package was assigned a separate subdirectory (within pkg) and,usually, contained source code for implimenting different physical parametiza-tions. This was a convenient method for both segregating and rapidly includingor excluding parameterizations during the software build process.

Over time, package use has increased. The number of packages has grownand they have evolved to contain much of the model functionality includingmomentum schemes, I/O utilities, diagnostics, “exchange” algorithms for do-main decomposition, and numerous physical parameterizations. The followingsections describe how to use the existing packages and how to modify them andcreate new ones.

309

310 CHAPTER 6. PHYSICAL PARAMETERIZATION AND PACKAGES

6.1 Using MITgcm Packages

The set of packages that will be used within a partiucular model can be con-figured using a combination of both “compile–time” and “run–time” options.Compile–time options are those used to select which packages will be “compiledin” or implemented within the program. Packages excluded at compile timeare completely absent from the executable program(s) and thus cannot be lateractivated by any set of subsequent run–time options.

6.1.1 Package Inclusion/Exclusion

There are numerous ways that one can specify compile–time package inclusionor exclusion and they are all implemented by the genmake2 program which waspreviously described in Section 3.5. The options are as follows:

1. Setting the genamake2 options --enable PKG and/or --disable PKG spec-ifies inclusion or exclusion. This method is intended as a convenient wayto perform a single (perhaps for a quick test) compilation.

2. By creating a text file with the name packages.conf in either the lo-cal build directory or the -mods=DIR directory, one can specify a list ofpackages (one package per line, with ’#’ as the comment character) tobe included. Since the packages.conf file can be saved, this is the pre-ferred method for setting and recording (for future reference) the packageconfiguration.

3. For convenience, a list of “standard” package groups is contained in thepkg/pkg groups file. By selecting one of the package group names in thepackages.conf file, one automatically obtains all packages in that group.

4. By default (that is, if a packages.conf file is not found), the genmake2program will use the contents of the pkg/pkg default file to obtain a listof packages.

5. To help prevent users from creating unusable package groups, the genmake2program will parse the contents of the pkg/pkg depend file to determine:

• whether any two requested packages cannot be simultaneously in-cluded (eg. seaice and thsice are mutually exclusive),

• whether additional packages must be included in order to satisfypackage dependencies (eg. rw depends upon functionality within themdsio package), and

• whether the set of all requested packages is compatible with the de-pendencies (and producing an error if they aren’t).

Thus, as a result of the dependencies, additional packages may be addedto those originally requested.

6.2. PACKAGE CODING STANDARDS 311

6.1.2 Package Activation

For run–time package control, MITgcm uses flags set through a data.pkg file.While some packages (eg. debug, mnc, exch2) may have their own usage con-ventions, most follow a simple flag naming convention of the form:

usePackageName=.TRUE.

where the usePackageName variable can activate or disable the package at run-time. As mentioned previously, packages must be included in order to be acti-vated. Generally, such mistakes will be detected and reported as errors by thecode. However, users should still be aware of the dependency.

6.2 Package Coding Standards

The following sections describe how to modify and/or create new MITgcm pack-ages.

6.2.1 Packages are Not Libraries

To a beginner, the MITgcm packages may resemble libraries as used in myriadsoftware projects. While future versions are likely to implement packages aslibraries (perhaps using FORTRAN90/95 syntax) the current packages (FOR-TRAN77) are not based upon any concept of libraries.

6.2.1.1 File Inclusion Rules

Instead, packages should be viewed only as directories containing “sets of sourcefiles” that are built using some simple mechanisms provided by genmake2. Con-ceptually, the build process adds files as they are found and proceeds accordingto the following rules:

1. genmake2 locates a “core” or main set of source files (the -standarddirsoption sets these locations and the default value contains the directorieseesupp and model).

2. genmake2 then finds additional source files by inspecting the contents ofeach of the package directories:

(a) As the new files are found, they are added to a list of source files.

(b) If there is a file name “collision” (that is, if one of the files in a packagehas the same name as one of the files previously encountered) then thefile within the newer (more recently visited) package will superseed(or “hide”) any previous file(s) with the same name.

(c) Packages are visited (and thus files discovered) in the order that thepackages are enabled within genmake2. Thus, the files in PackB maysuperseed the files in PackA if PackA is enabled before PackB. Thus,


package ordering can be significant! For this reason, genmake2 honorsthe order in which packages are specified.

These rules were adopted since they provide a relatively simple means forrapidly including (or “hiding”) existing files with modified versions.

6.2.1.2 Conditional Compilation and PACKAGES CONFIG.h

Given that packages are simply groups of files that may be added or removedto form a whole, one may wonder how linking (that is, FORTRAN symbolresolution) is handled. This is the second way that genmake2 supports theconcept of packages. Basically, genmake2 creates a Makefile that, in turn, isable to create a file called PACKAGES CONFIG.h that contains a set of C pre-processor (or “CPP”) directives such as:

#undef ALLOW_KPP#undef ALLOW_LAND...#define ALLOW_GENERIC_ADVDIFF#define ALLOW_MDSIO...

These CPP symbols are then used throughout the code to conditionally isolatevariable definitions, function calls, or any other code that depends upon thepresence or absence of any particular package.

An example illustrating the use of these defines is:

#ifdef ALLOW_GMREDIIF (useGMRedi) CALL GMREDI_CALC_DIFF(I bi,bj,iMin,iMax,jMin,jMax,K,I maskUp,O KappaRT,KappaRS,I myThid)

#endif

which is included from the file calc diffusivity.F and shows how both thecompile–time ALLOW GMREDI flag and the run–time useGMRedi are nested.

There are some benefits to using the technique described here. The first isthat code snippets or subroutines associated with packages can be placed orcalled from almost anywhere else within the code. The second benefit is relatedto memory footprint and performance. Since unused code can be removed,there is no performance penalty due to unnecessary memory allocation, unusedfunction calls, or extra run-time IF (...) conditions. The major problemswith this approach are the potentially difficult-to-read and difficult-to-debugcode caused by an overuse of CPP statements. So while it can be done, devel-opers should exerecise some discipline and avoid unnecesarily “smearing” theirpackage implementation details across numerous files.

6.2. PACKAGE CODING STANDARDS 313

6.2.1.3 Package Startup or Boot Sequence

Calls to package routines within the core code timestepping loop can vary. How-ever, all packages should follow a required ”boot” sequence outlined here:

1. S/R PACKAGES_BOOT()

:

CALL OPEN_COPY_DATA_FILE( ’data.pkg’, ’PACKAGES_BOOT’, ... )

2. S/R PACKAGES_READPARMS()

:

#ifdef ALLOW_${PKG}

if ( use${Pkg} )

& CALL ${PKG}_READPARMS( retCode )

#endif

3. S/R PACKAGES_INIT_FIXED()

:

#ifdef ALLOW_${PKG}

if ( use${Pkg} )

& CALL ${PKG}_INIT_FIXED( retCode )

#endif

4. S/R PACKAGES_CHECK()

:

#ifdef ALLOW_${PKG}

if ( use${Pkg} )

& CALL ${PKG}_CHECK( retCode )

#else

if ( use${Pkg} )

& CALL PACKAGES_CHECK_ERROR(’${PKG}’)

#endif

5. S/R PACKAGES_INIT_VARIABLES()

:

#ifdef ALLOW_${PKG}

if ( use${Pkg} )

& CALL ${PKG}_INIT_VARIA( )

#endif

6. S/R DO_THE_MODEL_IO

#ifdef ALLOW_${PKG}

if ( use${Pkg} )

& CALL ${PKG}_DIAGS( )

#endif

7. S/R PACKAGES_WRITE_PICKUP()

#ifdef ALLOW_${PKG}

if ( use${Pkg} )

& CALL ${PKG}_WRITE_PICKUP( )

#endif


6.2.1.4 Adding a package to PARAMS.h and packages boot()

An MITgcm package directory contains all the code needed for that packageapart from one variable for each package. This variable is the use${Pkg} flag.This flag, which is of type logical, must be declared in the shared header filePARAMS.h in the PARM PACKAGES block. This convention is used to sup-port a single runtime control file data.pkg which is read by the startup routinepackages boot() and that sets a flag controlling the runtime use of a package.This routine needs to be able to read the flags for packages that were not builtat compile time. Therefore when adding a new package, in addition to creatingthe per-package directory in the pkg/ subdirectory a developer should add ause${Pkg} flag to PARAMS.h and a use${Pkg} entry to the packages boot()PACKAGES namelist. The only other package specific code that should appearoutside the individual package directory are calls to the specific package API.

6.3. GENT/MCWILIAMS/REDI SGS EDDY PARAMETERIZATION 315

6.3 Gent/McWiliams/Redi SGS Eddy parame-terization

There are two parts to the Redi/GM parameterization of geostrophic eddies.The first aims to mix tracer properties along isentropes (neutral surfaces) bymeans of a diffusion operator oriented along the local isentropic surface (Redi).The second part, adiabatically re-arranges tracers through an advective fluxwhere the advecting flow is a function of slope of the isentropic surfaces (GM).

The first GCM implementation of the Redi scheme was by Cox 1987 in theGFDL ocean circulation model. The original approach failed to distinguishbetween isopycnals and surfaces of locally referenced potential density (nowcalled neutral surfaces) which are proper isentropes for the ocean. As will bediscussed later, it also appears that the Cox implementation is susceptible to acomputational mode. Due to this mode, the Cox scheme requires a backgroundlateral diffusion to be present to conserve the integrity of the model fields.

The GM parameterization was then added to the GFDL code in the formof a non-divergent bolus velocity. The method defines two stream-functions ex-pressed in terms of the isoneutral slopes subject to the boundary condition ofzero value on upper and lower boundaries. The horizontal bolus velocities arethen the vertical derivative of these functions. Here in lies a problem highlightedby Griffies et al., 1997: the bolus velocities involve multiple derivatives on thepotential density field, which can consequently give rise to noise. Griffies et al.point out that the GM bolus fluxes can be identically written as a skew fluxwhich involves fewer differential operators. Further, combining the skew fluxformulation and Redi scheme, substantial cancellations take place to the pointthat the horizontal fluxes are unmodified from the lateral diffusion parameteri-zation.

6.3.1 Redi scheme: Isopycnal diffusion

The Redi scheme diffuses tracers along isopycnals and introduces a term in thetendency (rhs) of such a tracer (here τ) of the form:

∇ · κρKRedi∇τ (6.1)

where κρ is the along isopycnal diffusivity and KRedi is a rank 2 tensor thatprojects the gradient of τ onto the isopycnal surface. The unapproximatedprojection tensor is:

KRedi =

⎛⎝ 1 + Sx SxSy Sx

SxSy 1 + Sy Sy

Sx Sy |S|2

⎞⎠ (6.2)

Here, Sx = −∂xσ/∂zσ and Sy = −∂yσ/∂zσ are the components of the isoneutralslope.

The first point to note is that a typical slope in the ocean interior is small,say of the order 10−4. A maximum slope might be of order 10−2 and only


exceeds such in unstratified regions where the slope is ill defined. It is thereforejustifiable, and customary, to make the small slope approximation, |S| << 1.The Redi projection tensor then becomes:

KRedi =

⎛⎝ 1 0 Sx

0 1 Sy

Sx Sy |S|2

⎞⎠ (6.3)

6.3.2 GM parameterization

The GM parameterization aims to parameterise the “advective” or “transport”effect of geostrophic eddies by means of a “bolus” velocity, u∗. The divergenceof this advective flux is added to the tracer tendency equation (on the rhs):

−∇ · τu∗ (6.4)

The bolus velocity is defined as:

u∗ = −∂zFx (6.5)v∗ = −∂zFy (6.6)w∗ = ∂xFx + ∂yFy (6.7)

where Fx and Fy are stream-functions with boundary conditions Fx = Fy = 0on upper and lower boundaries. The virtue of casting the bolus velocity interms of these stream-functions is that they are automatically non-divergent(∂xu

∗+ ∂yv∗ = −∂xzFx − ∂yzFy = −∂zw

∗). Fx and Fy are specified in terms ofthe isoneutral slopes Sx and Sy:

Fx = κGMSx (6.8)Fy = κGMSy (6.9)

This is the form of the GM parameterization as applied by Donabasaglu, 1997,in MOM versions 1 and 2.

6.3.3 Griffies Skew Flux

Griffies notes that the discretisation of bolus velocities involves multiple layersof differencing and interpolation that potentially lead to noisy fields and com-putational modes. He pointed out that the bolus flux can be re-written in termsof a non-divergent flux and a skew-flux:

u∗τ =

⎛⎝ −∂z(κGMSx)τ−∂z(κGMSy)τ

(∂xκGMSx + ∂yκGMSy)τ

⎞⎠=

⎛⎝ −∂z(κGMSxτ)−∂z(κGMSyτ)

∂x(κGMSxτ) + ∂y(κGMSy)τ)

⎞⎠+

⎛⎝ κGMSx∂zτκGMSy∂zτ

−κGMSx∂xτ − κGMSy)∂yτ

⎞⎠


The first vector is non-divergent and thus has no effect on the tracer field andcan be dropped. The remaining flux can be written:

u∗τ = −κGMKGM∇τ (6.10)

where

KGM =

⎛⎝ 0 0 −Sx

0 0 −Sy

Sx Sy 0

⎞⎠ (6.11)

is an anti-symmetric tensor.This formulation of the GM parameterization involves fewer derivatives than

the original and also involves only terms that already appear in the Redi mixingscheme. Indeed, a somewhat fortunate cancellation becomes apparent when weuse the GM parameterization in conjunction with the Redi isoneutral mixingscheme:

κρKRedi∇τ − u∗τ = (κρKRedi + κGMKGM)∇τ (6.12)

In the instance that κGM = κρ then

κρKRedi + κGMKGM = κρ

⎛⎝ 1 0 00 1 0

2Sx 2Sy |S|2

⎞⎠ (6.13)

which differs from the variable Laplacian diffusion tensor by only two non-zeroelements in the z-row.

S/R GMREDI CALC TENSOR (pkg/gmredi/gmredi calc tensor.F)σx: SlopeX (argument on entry)σy: SlopeY (argument on entry)σz : SlopeY (argument)Sx: SlopeX (argument on exit)Sy: SlopeY (argument on exit)

6.3.4 Variable κGM

Visbeck et al., 1996, suggest making the eddy coefficient, κGM , a function of theEady growth rate, |f |/√Ri. The formula involves a non-dimensional constant,α, and a length-scale L:

κGM = αL2 |f |√Ri

z

where the Eady growth rate has been depth averaged (indicated by the over-line). A local Richardson number is defined Ri = N2/(∂u/∂z)2 which, whencombined with thermal wind gives:

1Ri

=(∂u

∂z )2

N2=

( gfρo

|∇σ|)2N2

=M4

|f |2N2


10−4

10−3

10−2

10−1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Slope |S|

Tape

r fn f(

S)

DM95 {1+tanh[ (Sc−|S|)/S

d ] }/2

GKW91 (Smax

/|S|)2

Figure 6.1: Taper functions used in GKW99 and DM95.

whereM2 is defined M2 = gρo|∇σ|. Substituting into the formula for κGM gives:

κGM = αL2M2

N

z

= αL2M2

N2N

z

= αL2|S|Nz

6.3.5 Tapering and stability

Experience with the GFDL model showed that the GM scheme has to bematched to the convective parameterization. This was originally expressed inconnection with the introduction of the KPP boundary layer scheme (Large etal., 97) but in fact, as subsequent experience with the MIT model has found, isnecessary for any convective parameterization.

S/R GMREDI SLOPE LIMIT (pkg/gmredi/gmredi slope limit.F)σx, sx: SlopeX (argument)σy , sy: SlopeY (argument)σz : dSigmadRReal (argument)z∗σ: dRdSigmaLtd (argument)

6.3.5.1 Slope clipping

Deep convection sites and the mixed layer are indicated by homogenized, un-stable or nearly unstable stratification. The slopes in such regions can be eitherinfinite, very large with a sign reversal or simply very large. From a numerical


−0.02 −0.015 −0.01 −0.005 0 0.005 0.01 0.015 0.02−0.01

−0.008

−0.006

−0.004

−0.002

0

0.002

0.004

0.006

0.008

0.01

Slope S

Effe

ctive

slo

pe S

* = f(

S) S

DM95 {1+tanh[ (Sc−|S|)/S

d ] }/2 S

GKW91 (Smax

/|S|)2 S

Cox Slope Clipping

Figure 6.2: Effective slope as a function of “true” slope using Cox slope clipping,GKW91 limiting and DM95 limiting.

point of view, large slopes lead to large variations in the tensor elements (imply-ing large bolus flow) and can be numerically unstable. This was first recognizedby Cox, 1987, who implemented “slope clipping” in the isopycnal mixing tensor.Here, the slope magnitude is simply restricted by an upper limit:

|∇σ| =√σ2

x + σ2y (6.14)

Slim = − |∇σ|Smax

where Smax is a parameter (6.15)

σ�z = min(σz , Slim) (6.16)

[sx, sy] = − [σx, σy]σ�

z

(6.17)

Notice that this algorithm assumes stable stratification through the “min” func-tion. In the case where the fluid is well stratified (σz < Slim) then the slopesevaluate to:

[sx, sy] = − [σx, σy]σz

(6.18)

while in the limited regions (σz > Slim) the slopes become:

[sx, sy] =[σx, σy]

|∇σ|/Smax(6.19)


so that the slope magnitude is limited√s2x + s2y = Smax.

The slope clipping scheme is activated in the model by setting GM tap-er scheme = ’clipping’ in data.gmredi.

Even using slope clipping, it is normally the case that the vertical diffusionterm (with coefficient κρK33 = κρS

2max) is large and must be time-stepped us-

ing an implicit procedure (see section on discretisation and code later). Fig.6.3 shows the mixed layer depth resulting from a) using the GM scheme withclipping and b) no GM scheme (horizontal diffusion). The classic result of dra-matically reduced mixed layers is evident. Indeed, the deep convection sites tojust one or two points each and are much shallower than we might prefer. This,it turns out, is due to the over zealous re-stratification due to the bolus trans-port parameterization. Limiting the slopes also breaks the adiabatic nature ofthe GM/Redi parameterization, re-introducing diabatic fluxes in regions wherethe limiting is in effect.

6.3.5.2 Tapering: Gerdes, Koberle and Willebrand, Clim. Dyn.1991

The tapering scheme used in Gerdes et al., 1999, ([42]) addressed two issueswith the clipping method: the introduction of large vertical fluxes in additionto convective adjustment fluxes is avoided by tapering the GM/Redi slopes backto zero in low-stratification regions; the adjustment of slopes is replaced by atapering of the entire GM/Redi tensor. This means the direction of fluxes isunaffected as the amplitude is scaled.

The scheme inserts a tapering function, f1(S), in front of the GM/Reditensor:

f1(S) = min

[1,(Smax

|S|)2]

(6.20)

where Smax is the maximum slope you want allowed. Where the slopes, |S| <Smax then f1(S) = 1 and the tensor is un-tapered but where |S| ≥ Smax

then f1(S) scales down the tensor so that the effective vertical diffusivity termκf1(S)|S|2 = κS2

max.The GKW tapering scheme is activated in the model by setting GM tap-

er scheme = ’gkw91’ in data.gmredi.

6.3.6 Tapering: Danabasoglu and McWilliams, J. Clim.1995

The tapering scheme used by Danabasoglu and McWilliams, 1995, [11], followeda similar procedure but used a different tapering function, f1(S):

f1(S) =12

(1 + tanh

[Sc − |S|Sd

])(6.21)

where Sc = 0.004 is a cut-off slope and Sd = 0.001 is a scale over which theslopes are smoothly tapered. Functionally, the operates in the same way as the


Figure missing.

Figure 6.3: Mixed layer depth using GM parameterization with a) Cox slopeclipping and for comparison b) using horizontal constant diffusion.

GKW91 scheme but has a substantially lower cut-off, turning off the GM/RediSGS parameterization for weaker slopes.

The DM tapering scheme is activated in the model by setting GM tap-er scheme = ’dm95’ in data.gmredi.

6.3.7 Tapering: Large, Danabasoglu and Doney, JPO 1997

The tapering used in Large et al., 1997, [35], is based on the DM95 taperingscheme, but also tapers the scheme with an additional function of height, f2(z),so that the GM/Redi SGS fluxes are reduced near the surface:

f2(S) =12

(1 + sin(π

z

D− π/2)

)(6.22)

where D = Lρ|S| is a depth-scale and Lρ = c/f with c = 2 m s−1. This taperingwith height was introduced to fix some spurious interaction with the mixed-layerKPP parameterization.

The LDD tapering scheme is activated in the model by setting GM tap-er scheme = ’ldd97’ in data.gmredi.

6.3.8 Package Reference


6.4 DIC Package

6.4.1 Introduction

This is one of the biogeochemical packages handled from the pkg gchem. Themain purpose of this package is to consider the cycling of carbon in the ocean.It also looks at the cycling of phosphorous and oxygen. There are five tracersDIC, ALK, PO4, DOP and O2. The air-sea exchange of CO2 and O2 arehandled as in the OCMIP experiments (reference). The export of biologicalmatter is computed as a function of available light and PO4. This export isremineralized at depth according to a Martin curve (again, this is the same asin the OCMIP experiments). There is also a representation of the carbonateflux handled as in the OCMIP experiments. The air-sea exchange on CO2 isaffected by temperature, salinity and the pH of the surface waters. The pH isdetermined following the method of Follows et al.

6.4.2 Key subroutines and parameters

INITIALIZATIONDIC ABIOTIC.h contains the common block for the parameters and fieldsneeded to calculate the air-sea flux of CO2 and O2. The fixed parameters areset in dic abiotic param which is called from gchem init fixed.F. The parametersneeded for the biotic part of the calculations are initialized in dic biotic paramand are stored in DIC BIOTIC.h. The first guess of pH is calculated in dic surfforcing init.F.

LOADING FIELDSThe air-sea exchange of CO2 and O2 need wind, atmospheric pressure (althoughthe current version has this hardwired to 1), and sea-ice coverage. The calcula-tion of pH needs silica fields. These fields are read in in dic fields load.F. Thesefields are initialized to zero in dic ini forcing.F. The fields for interpolating arein common block in DIC LOAD.h.

FORCINGThe tracers are advected-diffused in ptracers integrate.F. The updated trac-ers are passed to dic biotic forcing.F where the effects of the air-sea exchangeand biological activity and remineralization are calculated and the tracers areupdated for a second time. Below we discuss the subroutines called fromdic biotic forcing.F.

Air-sea exchange of CO2 is calculated in dic surfforcing. Air-Sea Exchange ofCO2 depends on T,S and pH. The determination of pH is done in carbon chem.F.There are three subroutines in this file: carbon coeffs which determines thecoefficients for the carbon chemistry equations; calc pco2 which calculates thepH using a Newton-Raphson method; and calc pco2 approx which uses the muchmore efficient method of Follows et al. The latter is hard-wired into this package,the former is kept here for completeness.

6.4. DIC PACKAGE 323

Biological productivity is determined following McKinely et al. and is calcu-lated in bio export.F The light in each latitude band is calculate in insol.F. Theformation of hard tissue (carbonate) is linked to the biological productivity andhas an effect on the alkalinity - the flux of carbonate is calculated in car flux.F.The flux of phosphate to depth where it instantly remineralized is calculated inphos flux.F.

Alkalinity tendency comes from changes to the salinity from addition/subtractionof freshwater in the surface. This is handled in alk surfforcing.F.

Oxygen air-sea exchange is calculated in o2 surfforcing.F.

DIAGNOSTICSAverages of air-sea exchanges, biological productivity, carbonate activity andpH are calculated. These are initialized to zero in dic biotic init and are storedin common block in DIC BIOTIC.h.

6.4.3 Do’s and Don’ts

This package must be run with both ptracers and gchem enabled. It is set up for5 tracers, but there is the provision of a 6th tracer (iron) that is not discussedhere.

6.4.4 Reference Material


6.5 Ocean vertical mixing – the nonlocal K-profile

parameterization scheme KPP

6.6. THERMODYNAMIC SEA ICE PACKAGE: “THSICE” 325

6.6 Thermodynamic Sea Ice Package: “thsice”

Important note: This document has been written by Stephanie Dutkiewiczand describes an earlier implementation of the sea-ice package. This needs tobe updated to reflect the recent changes (JMC).This thermodynamic ice model is based on the 3-layer model by Winton (2000).and the energy-conserving LANL CICE model (Bitz and Lipscomb, 1999). Themodel considers two equally thick ice layers; the upper layer has a variable spe-cific heat resulting from brine pockets, the lower layer has a fixed heat capacity.A zero heat capacity snow layer lies above the ice. Heat fluxes at the top andbottom surfaces are used to calculate the change in ice and snow layer thickness.Grid cells of the ocean model are either fully covered in ice or are open water.There is a provision to parametrize ice fraction (and leads) in this package.Modifications are discussed in small font following the subroutine descriptions.

The ice model is called from thermodynamics.F, subroutine ice forcing.F iscalled in place of external forcing surf.F.

subroutine ICE FORCINGIn ice forcing.F, we calculate the freezing potential of the ocean model surfacelayer of water:

frzmlt = (Tf − SST )cswρswΔz

Δtwhere csw is seawater heat capacity, ρsw is the seawater density, Δz is the oceanmodel upper layer thickness and Δt is the model (tracer) timestep. The freezingtemperature, Tf = μS is a function of the salinity.

1) Provided there is no ice present and frzmlt is less than 0, the surfacetendencies of wind, heat and freshwater are calculated as usual (ie. as in exter-nal forcing surf.F).

2) If there is ice present in the grid cell we call the main ice model routineice therm.F (see below). Output from this routine gives net heat and freshwaterflux affecting the top of the ocean.

Subroutine ice forcing.F uses these values to find the sea surface tendenciesin grid cells. When there is ice present, the surface stress tendencies are set tozero; the ice model is purely thermodynamic and the effect of ice motion on thesea-surface is not examined.

Relaxation of surface T and S is only allowed equatorward of relaxlat (seeDATA.ICE below), and no relaxation is allowed under the ice at any latitude.(Note that there is provision for allowing grid cells to have both open water and seaice; if compact is between 0

and 1)

subroutine ICE FREEZE


This routine is called from thermodynamics.F after the new temperaturecalculation, calc gt.F, but before calc gs.F. In ice freeze.F, any ocean upperlayer grid cell with no ice cover, but with temperature below freezing, Tf = μShas ice initialized. We calculate frzmlt from all the grid cells in the watercolumn that have a temperature less than freezing. In this routine, any waterbelow the surface that is below freezing is set to Tf . A call to ice start.F ismade if frzmlt > 0, and salinity tendancy is updated for brine release.(There is a provision for fractional ice: In the case where the grid cell has less ice coverage than icemaskmax we

allow ice start.F to be called).

subroutine ICE STARTThe energy available from freezing the sea surface is brought into this routineas esurp. The enthalpy of the 2 layers of any new ice is calculated as:

q1 = −ci ∗ Tf + Li

q2 = −cfTmlt + ci(Tmlt − Tf) + Li(1 − Tmlt

Tf

(6.23)

where cf is specific heat of liquid fresh water, ci is the specific heat of fresh ice,Li is latent heat of freezing, ρi is density of ice and Tmlt is melting temperatureof ice with salinity of 1. The height of a new layer of ice is

hinew =esurpΔtqi0av

where qi0av = − ρi

2 (q1 + q2).The surface skin temperature Ts and ice temperatures T1, T2 and the sea

surface temperature are set at Tf .( There is provision for fractional ice: new ice is formed over open water; the first freezing in the cell must havea height of himin0; this determines the ice fraction compact. If there is already ice in the grid cell, the new icemust have the same height and the new ice fraction is

if = (1 − if )hinew

hi

where if is ice fraction from previous timestep and hi is current ice height. Snow is redistributed over the new ice

fraction. The ice fraction is not allowed to become larger than iceMaskmax and if the ice height is above hihig

then freezing energy comes from the full grid cell, ice growth does not occur under orginal ice due to freezing water.

subroutine ICE THERMThe main subroutine of this package is ice therm.F where the ice temperaturesare calculated and the changes in ice and snow thicknesses are determined.Output provides the net heat and fresh water fluxes that force the top layer ofthe ocean model.

If the current ice height is less than himin then the ice layer is set to zeroand the ocean model upper layer temperature is allowed to drop lower than


its freezing temperature; and atmospheric fluxes are allowed to effect the gridcell. If the ice height is greater than himin we proceed with the ice modelcalculation.

We follow the procedure of Winton (1999) – see equations 3 to 21 – tocalculate the surface and internal ice temperatures. The surface temperature isfound from the balance of the flux at the surface Fs, the shortwave heat fluxabsorbed by the ice, fswint, and the upward conduction of heat through thesnow and/or ice, Fu. We linearize Fs about the surface temperature, Ts, at theprevious timestep (whereˆindicates the value at the previous timestep):

Fs(Ts) = Fs(Ts) +∂Fs(Ts)∂Ts

(Ts − Ts)

where,

Fs = Fsensible + Flatent + F downlongwave + Fup

longwave + (1 − α)Fshortwave

anddFs

dT=dFsensible

dT+dFlatent

dT+dFup

longwave

dT.

Fs and dFs

dT are currently calculated from the BULKF package described sep-arately, but could also be provided by an atmospheric model. The surfacealbedo is calculated from the ice height and/or surface temperature (see below,srf albedo.F) and the shortwave flux absorbed in the ice is

fswint = (1 − eκihi)(1 − α)Fshortwave

where κi is bulk extinction coefficient.The conductive flux to the surface is

Fu = K1/2(T1 − Ts)

where K1/2 is the effective conductive coupling of the snow-ice layer betweenthe surface and the mid-point of the upper layer of ice K1/2 = 4KiKs

Kshi+4Kihs. Ki

and Ks are constant thermal conductivities of seaice and snow.From the above equations we can develop a system of equations to find the

skin surface temperature, Ts and the two ice layer temperatures (see Winton,1999, for details). We solve these equations iteratively until the change in Ts issmall. When the surface temperature is greater then the melting temperatureof the surface, the temperatures are recalculated setting Ts to 0. The enthalpyof the ice layers are calculated in order to keep track of the energy in the icemodel. Enthalpy is defined, here, as the energy required to melt a unit mass ofseaice with temperature T . For the upper layer (1) with brine pockets and thelower fresh layer (2):

q1 = −cfTf + ci(Tf − T ) + Li(1 − Tf

T)

q2 = −ciT + Li


where cf is specific heat of liquid fresh water, ci is the specific heat of fresh ice,and Li is latent heat of melting fresh ice.

From the new ice temperatures, we can calculate the energy flux at thesurface available for melting (if Ts=0) and the energy at the ocean-ice interfacefor either melting or freezing.

Etop = (Fs −K1/2(Ts − T1))Δt

Ebot = (4Ki(T2 − Tf )

hi− Fb)Δt

where Fb is the heat flux at the ice bottom due to the sea surface temperaturevariations from freezing. If Tsst is above freezing, Fb = cswρswγ(Tsst − Tf)u∗,γ is the heat transfer coefficient and u∗ = QQ is frictional velocity between iceand water. If Tsst is below freezing, Fb = (Tf −Tsst)cfρfΔz/Δt and set Tsst toTf . We also include the energy from lower layers that drop below freezing, andset those layers to Tf .

If Etop > 0 we melt snow from the surface, if all the snow is melted andthere is energy left, we melt the ice. If the ice is all gone and there is still energyleft, we apply the left over energy to heating the ocean model upper layer (SeeWinton, 1999, equations 27-29). Similarly if Ebot > 0 we melt ice from thebottom. If all the ice is melted, the snow is melted (with energy from the oceanmodel upper layer if necessary). If Ebot < 0 we grow ice at the bottom

Δhi =−Ebot

(qbotρi)

where qbot = −ciTf + Li is the enthalpy of the new ice, The enthalpy of thesecond ice layer, q2 needs to be modified:

q2 =hi/2q2 + Δhiqbot

hi/2 + Δhi

If there is a ice layer and the overlying air temperature is below 0oC thenany precipitation, P joins the snow layer:

Δhs = −P ρf

ρsΔt,

ρf and ρs are the fresh water and snow densities. Any evaporation, similarly,removes snow or ice from the surface. We also calculate the snow age here, incase it is needed for the surface albedo calculation (see srf albedo.F below).

For practical reasons we limit the ice growth to hilim and snow is limitedto hslim. We converts any ice and/or snow above these limits back to water,maintaining the salt balance. Note however, that heat is not conserved in thisconversion; sea surface temperatures below the ice are not recalculated.

If the snow/ice interface is below the waterline, snow is converted to ice(see Winton, 1999, equations 35 and 36). The subroutine new layers winton.F,


described below, repartitions the ice into equal thickness layers while conservingenergy.

The subroutine ice therm.F now calculates the heat and fresh water fluxesaffecting the ocean model surface layer. The heat flux:

qnet = fswocn− Fb − esurpΔt

is composed of the shortwave flux that has passed through the ice layer andis absorbed by the water, fswocn= QQ, the ocean flux to the ice Fb, andthe surplus energy left over from the melting, esurp. The fresh water flux isdetermined from the amount of fresh water and salt in the ice/snow systembefore and after the timestep.(There is a provision for fractional ice: If ice height is above hihig then all energy from freezing at sea surface

is used only in the open water aparts of the cell (ie. Fb will only have the conduction term). The melt energy

is partitioned by frac energy between melting ice height and ice extent. However, once ice height drops below

himon0 then all energy melts ice extent.

subroutine SFC ALBEDOThe routine ice therm.F calls this routine to determine the surface albedo. Thereare two calculations provided here:1) from LANL CICE model

α = fsαs + (1 − fs)(αimin + (αimax − αimin)(1 − e−hi/hα))

where fs is 1 if there is snow, 0 if not; the snow albedo, αs has two valuesdepending on whether Ts < 0 or not; αimin and αimax are ice albedos for thinmelting ice, and thick bare ice respectively, and hα is a scale height.2) From GISS model (Hansen et al 1983)

α = αie−hs/ha + αs(1 − e−hs/ha)

where αi is a constant albedo for bare ice, ha is a scale height and αs is avariable snow albedo.

αs = α1 + α2e−λaas

where α1 is a constant, α2 depends on Ts, as is the snow age, and λa is a scalefrequency. The snow age is calculated in ice therm.F and is given in equation41 in Hansen et al (1983).

subroutine NEW LAYERS WINTONThe subroutine new layers winton.F repartitions the ice into equal thicknesslayers while conserving energy. We pass to this subroutine, the ice layer en-thalpies after melting/growth and the new height of the ice layers. The endinglayer height should be half the sum of the new ice heights from ice therm.F. The


enthalpies of the ice layers are adjusted accordingly to maintain total energy inthe ice model. If layer 2 height is greater than layer 1 height then layer 2 givesice to layer 1 and:

q1 = f1q1 + (1 − f1)q2

where f1 is the fraction of the new to old upper layer heights. T1 will thereforealso have changed. Similarly for when ice layer height 2 is less than layer 1height, except here we need to to be careful that the new T2 does not fall belowthe melting temperature.

Initializing subroutinesice init.F: Set ice variables to zero, or reads in pickup information from pickup.ic(which was written out in checkpoint.F)ice readparms.F: Reads data.ice

Diagnostic subroutinesice ave.F: Keeps track of means of the ice variablesice diags.F: Finds averages and writes out diagnostics

Common BlocksICE.h: Ice Varibles, also relaxlat and startIceModelICE DIAGS.h: matrices for diagnostics: averages of fields from ice diags.FBULKF ICE CONSTANTS.h (in BULKF package): all the parameters needby the ice model

Input file DATA.ICEHere we need to set StartIceModel: which is 1 if the model starts from no ice;and 0 if there is a pickup file with the ice matrices (pickup.ic) which is read inice init.F and written out in checkpoint.F. The parameter relaxlat defines thelatitude poleward of which there is no relaxing of surface T or S to observations.This avoids the relaxation forcing the ice model at these high latitudes.(Note: hicemin is set to 0 here. If the provision for allowing grid cells to have both open water and seaice is ever

implemented, this would be greater than 0)

Important Notes1) heat fluxes have different signs in the ocean and ice models.2) StartIceModel must be changed in data.ice: 1 (if starting from no ice), 0(if using pickup.ic file).


ReferencesBitz, C.M. and W.H. Lipscombe, 1999: An Energy-Conserving ThermodynamicModel of Sea Ice. Journal of Geophysical Research, 104, 15,669 – 15,677.

Hansen, J., G. Russell, D. Rind, P. Stone, A. Lacis, S. Lebedeff, R. Ruedy andL.Travis, 1983: Efficient Three-Dimensional Global Models for Climate Studies:Models I and II. Monthly Weather Review, 111, 609 – 662.

Hunke, E.C and W.H. Lipscomb, circa 2001: CICE: the Los Alamos Sea IceModel Documentation and Software User’s Manual. LACC-98-16v.2.(note: this documentation is no longer available as CICE has progressed to avery different version 3)

Winton, M, 2000: A reformulated Three-layer Sea Ice Model. Journal of Atmo-spheric and Ocean Technology, 17, 525 – 531.


6.7 Sea Ice Package: “seaice”

Package “seaice” provides a dynamic and thermodynamic interactive sea-icemodel. Sea-ice model thermodynamics are based on Hibler [28], that is, a 2-category model that simulates ice thickness and concentration. Snow is simu-lated as per Zhang et al. [54]. Although recent years have seen an increased useof multi-category thickness distribution sea-ice models for climate studies, theHibler 2-category ice model is still the most widely used model and has resultedin realistic simulation of sea-ice variability on regional and global scales. Be-ing less complicated, compared to multi-category models, the 2-category modelpermits easier application of adjoint model optimization methods.

Note, however, that the Hibler 2-category model and its variants use a so-called zero-layer thermodynamic model to estimate ice growth and decay. Thezero-layer thermodynamic model assumes that ice does not store heat and, there-fore, tends to exaggerate the seasonal variability in ice thickness. This exag-geration can be significantly reduced by using Semtner’s [46] three-layer ther-modynamic model that permits heat storage in ice. Recently, the three-layerthermodynamic model has been reformulated by Winton [53]. The reformula-tion improves model physics by representing the brine content of the upper icewith a variable heat capacity. It also improves model numerics and consumesless computer time and memory. The Winton sea-ice thermodynamics have beenported to the MIT GCM; they currently reside under pkg/thsice. At presentpkg/thsice is not fully compatible with pkg/seaice and with pkg/exf. But theeventual objective is to have fully compatible and interchangeable thermody-namic packages for sea-ice, so that it becomes possible to use Hibler dynamicswith Winton thermodyanmics.

The ice dynamics models that are most widely used for large-scale climatestudies are the viscous-plastic (VP) model [27], the cavitating fluid (CF) model[14], and the elastic-viscous-plastic (EVP) model [32]. Compared to the VPmodel, the CF model does not allow ice shear in calculating ice motion, stress,and deformation. EVP models approximate VP by adding an elastic term tothe equations for easier adaptation to parallel computers. Because of its higheraccuracy in plastic solution and relatively simpler formulation, compared to theEVP model, we decided to use the VP model as the dynamic component ofour ice model. To do this we extended the alternating-direction-implicit (ADI)method of Zhang and Rothrock [55] for use in a parallel configuration.

The sea ice model requires the following input fields: 10-m winds, 2-m airtemperature and specific humidity, downward longwave and shortwave radia-tions, precipitation, evaporation, and river and glacier runoff. The sea ice modelalso requires surface temperature from the ocean model and third level horizon-tal velocity which is used as a proxy for surface geostrophic velocity. Outputfields are surface wind stress, evaporation minus precipitation minus runoff, netsurface heat flux, and net shortwave flux. The sea-ice model is global: in ice-freeregions bulk formulae are used to estimate oceanic forcing from the atmosphericfields.

6.8. BULK FORMULA PACKAGE 333

6.8 Bulk Formula Package

author: Stephanie Dutkiewicz

Instead of forcing the model with heat and fresh water flux data, this packagecalculates these fluxes using the changing sea surface temperature. We needto read in some atmospheric data: air temperature, air humidity, downshortwave radiation, down longwave radiation, precipitation, windspeed. The current setup also reads in wind stress, but this can be changedso that the stresses are calculated from the wind speed.

The current setup requires that there is the thermodynamic-seaice package(pkg/thsice, also refered below as seaice) is also used. It would be useful thoughto have it also setup to run with some very simple parametrization of the seaice.

The heat and fresh water fluxes are calculated in bulkf forcing.F called fromforward step.F. These fluxes are used over open water, fluxes over seaice arerecalculated in the sea-ice package. Before the call to bulkf forcing.F we callbulkf fields load.F to find the current atmospheric conditions. The only otherchanges to the model code come from the initializing and writing diagnostics ofthese fluxes.

subroutine BULKF FIELDS LOADHere we find the atmospheric data needed for the bulk formula calculations.These are read in at periodic intervals and values are interpolated to the currenttime. The data file names come from data.blk. The values that can be readin are: air temperature, air humidity, precipitation, down solar radiation, downlong wave radiation, zonal and meridional wind speeds, total wind speed, netheat flux, net freshwater forcing, cloud cover, snow fall, zonal and meridionalwind stresses, and SST and SSS used for relaxation terms. Not all these filesare necessary or used. For instance cloud cover and snow fall are not used in thecurrent bulk formula calculation. If total wind speed is not supplied, wind speedis calculate from the zonal and meridional components. If wind stresses are notread in, then the stresses are calculated from the wind speed. Net heat fluxand net freshwater can be read in and used over open ocean instead of the bulkformula calculations (but over seaice the bulkf formula is always used). This is”hardwired” into bulkf forcing and the ”ch” in the variable names suggests thatthis is ”cheating”. SST and SSS need to be read in if there is any relaxationused.

subroutine BULKF FORCINGIn bulkf forcing.F, we calculate heat and fresh water fluxes (and wind stress, if


necessary) for each grid cell. First we determine if the grid cell is open water orseaice and this information is carried by iceornot. There is a provision here fora different designation if there is snow cover (but currently this does not makeany difference). We then call bulkf formula lanl.F which provides values for: uplong wave radiation, latent and sensible heat fluxes, the derivative of these threewith respect to surface temperature, wind stress, evaporation. Net long waveradiation is calculated from the combination of the down long wave read in andthe up long wave calculated.

We then find the albedo of the surface - with a call to sfc albedo if there issea-ice (see the seaice package for information on the subroutine). If the gridcell is open ocean the albedo is set as 0.1. Note that this is a parameter thatcan be used to tune the results. The net short wave radiation is then the downshortwave radiation minus the amount reflected.

If the wind stress needed to be calculated in bulkf formula lanl.F, it wascalculated to grid cell center points, so in bulkf forcing.F we regrid to u and vpoints. We let the model know if it has read in stresses or calculated stressesby the switch readwindstress which is can be set in data.blk, and defaults to.TRUE..

We then calculate Qnet and EmPmR that will be used as the fluxes overthe open ocean. There is a provision for using runoff. If we are ”cheating” andusing observed fluxes over the open ocean, then there is a provision here to useread in Qnet and EmPmR.

The final call is to calculate averages of the terms found in this subroutine.

subroutine BULKF FORMULA LANLThis is the main program of the package where the heat fluxes and freshwaterfluxes over ice and open water are calculated. Note that this subroutine isalso called from the seaice package during the iterations to find the ice surfacetemperature.

Latent heat (L) used in this subroutine depends on the state of the sur-face: vaporization for open water, fusion and vaporization for ice surfaces. Airtemperature is converted from Celsius to Kelvin. If there is no wind speed(us) given, then the wind speed is calculated from the zonal and meridionalcomponents.

We calculate the virtual temperature:

To = Tair(1 + γqair)

where Tair is the air temperature at hT , qair is humidity at hq and γ is aconstant.

The saturated vapor pressure is calculate (QQ ref):

qsat =a

poe

L(b− cTsrf

)

where a, b, c are constants, Tsrf is surface temperature and po is the surfacepressure.


The two values crucial for the bulk formula calculations are the differencebetween air at sea surface and sea surface temperature:

ΔT = Tair − Tsrf + αhT

where α is adiabatic lapse rate and hT is the height where the air tempera-ture was taken; and the difference between the air humidity and the saturatedhumidity

Δq = qair − qsat.

We then calculate the turbulent exchange coefficients following Bryan et al(1996) and the numerical scheme of Hunke and Lipscombe (1998). We estimateinitial values for the exchange coefficients, cu, cT and cq as

κ

ln(zref/zrou)

where κ is the Von Karman constant, zref is a reference height and zrou is aroughness length scale which could be a function of type of surface, but is hereset as a constant. Turbulent scales are:

u∗ = cuus

T ∗ = cT ΔTq∗ = cqΔq

We find the ”integrated flux profile” for momentum and stability if there arestable QQ conditions (Υ > 0) :

ψm = ψs = −5Υ

and for unstable QQ conditions (Υ < 0):

ψm = 2ln(0.5(1 + χ)) + ln(0.5(1 + χ2)) − 2 tan−1 χ+ π/2ψs = 2ln(0.5(1 + χ2))

whereΥ =

κgzref

u∗2(T ∗

To+

q∗

1/γ + qa)

and χ = (1 − 16Υ)1/2.The coefficients are updated through 5 iterations as:

cu =cu

1 + cu(λ− ψm)/κ

cT =cT

1 + cT (λ− ψs)/κcq = c′T (6.24)

where λ = ln(hT/zref ).


We can then find the bulk formula heat fluxes:

Sensible heat flux:Qs = ρaircpairuscucT ΔT

Latent heat flux:Ql = ρairLuscucqΔq

Up long wave radiationQup

lw = εσT 4srf

where ε is emissivity (which can be different for open ocean, ice and snow), σis Stefan-Boltzman constant.

We calculate the derivatives of the three above functions with respect tosurface temperature

dQs

dT= ρaircpairuscucT

dQl

dT=

ρairL2uscucqc

T 2srf

dQup]lw

dT= 4εσt3srf

And total derivative dQo

dT = dQs

dT + dQl

dT + dQuplw

dT .If we do not read in the wind stress, it is calculated here.

Initializing subroutinesbulkf init.F: Set bulkf variables to zero.bulkf readparms.F: Reads data.blk

Diagnostic subroutinesbulkf ave.F: Keeps track of means of the bulkf variablesbulkf diags.F: Finds averages and writes out diagnostics

Common BlocksBULKF.h: BULKF Variables, data file names, and logicals readwindstressand readsurfaceBULKF DIAGS.h: matrices for diagnostics: averages of fields from bulkf diags.FBULKF ICE CONSTANTS.h: all the parameters need by the ice model and inthe bulkf formula calculations.

Input file DATA.ICE


We read in the file names of atmospheric data used in the bulk formula calcu-lations. Here we can also set the logicals: readwindstress if we read in thewind stress rather than calculate it from the wind speed; and readsurface toread in the surface temperature and salinity if these will be used as part of arelaxing term.

Important Notes1) heat fluxes have different signs in the ocean and ice models.2) StartIceModel must be changed in data.ice: 1 (if starting from no ice), 0(if using pickup.ic file).

References

Bryan F.O., B.G Kauffman, W.G. Large, P.R. Gent, 1996: The NCAR CSMflux coupler. Technical note TN-425+STR, NCAR.

Hunke, E.C and W.H. Lipscomb, circa 2001: CICE: the Los Alamos Sea IceModel Documentation and Software User’s Manual. LACC-98-16v.2.(note: this documentation is no longer available as CICE has progressed to avery different version 3)


6.9 Generic Advection/Diffusion

The generic advdiff package contains high-level subroutines to solve the advection-diffusion equation of any tracer, either active (potential temperature, salinityor water vapor) or passive (see pkg/ptracers). (see also sections 2.15 to 2.18).

6.9.1 Introduction

Package generic advdiff is a...

6.9.2 Key subroutines, parameters and files

The generic advdiff package has...

6.10. ATMOSPHERIC INTERMEDIATE PHYSICS: AIM 339

6.10 Atmospheric Intermediate Physics: AIM

Note: kpg/aim corresponds to an old version of the AIM physics (release.1).The folowing document below describes the more recent aim v23 package thatis based on the version v23 of the SPEEDY code ([51]).



6.11 Land package

This package provides a simple land model based on Rong Zhang [e-mail:[email protected]]2 layers model (see documentation below).

It is primarily implemented for AIM ( v23) atmospheric physics but couldbe adapted to work with a different atmospheric physics. Two subroutines(aim aim2land.F aim land2aim.F in pkg/aim v23) are used as interface withAIM physics.

Number of layers is a parameter (land nLev in LAND SIZE.h) and can bechanged.

Note on Land Modeldate: June 1999

author: Rong Zhang

This is a simple 2-layer land model. The top layer depth z1 = 0.1m, thesecond layer depth z2 = 4m.

Let Tg1, Tg2 be the temperature of each layer, W1,W2 be the soil moistureof each layer. The field capacity f1, f2 are the maximum water amount in eachlayer, so Wi is the ratio of available water to field capacity. fi = γzi, γ = 0.24is the field capapcity per meter soil, so f1 = 0.024m, f2 = 0.96m.

The land temperature is determined by total surface downward heat flux F,

z1C1dTg1

dt= F − λ

Tg1 − Tg2

(z1 + z2)/2(6.25)

z2C2dTg2

dt= λ

Tg1 − Tg2

(z1 + z2)/2(6.26)

here C1, C2 are the heat capacity of each layer , λ is the thermal conductivity,λ = 0.42Wm−1K−1.

C1 = CwW1γ + Cs (6.27)

C2 = CwW2γ + Cs (6.28)

Cw , Cs are the heat capacity of water and dry soil respectively. Cw = 4.2 ×106Jm−3K−1, Cs = 1.13 × 106Jm−3K−1.

The soil moisture is determined by precipitation P (m/s),surface evaporationE(m/s) and runoff R(m/s).

dW1

dt=P − E −R

f1+W2 −W1

τ(6.29)

6.11. LAND PACKAGE 341

τ = 2 days is the time constant for diffusion of moisture between layers.

dW2

dt=f1f2

W1 −W2

τ(6.30)

In the code, R = 0 gives better result, W1,W2 are set to be within [0, 1]. IfW1 is greater than 1, then let δW1 = W1 − 1,W1 = 1 and W2 = W2 + pδW1

f1f2

,i.e. the runoff of top layer is put into second layer. p = 0.5 is the fraction oftop layer runoff that is put into second layer.

The time step is 1 hour, it takes several years to reach equalibrium offline.

ReferencesHansen J. et al. Efficient three-dimensional global models for climate studies:

models I and II. Monthly Weather Review, vol.111, no.4, pp. 609-62, 1983


6.12 Coupling interface for Atmospheric Inter-

mediate code


6.13. COUPLER FOR MAPPING BETWEEN AIM AND OCEAN 343

6.13 Coupler for mapping between AIM and ocean



6.14 Toolkit for building couplers


6.15. NETCDF I/O INTEGRATION: MNC 345

6.15 NetCDF I/O Integration: MNC

The mnc package is a set of convenience routines written to expedite the pro-cess of creating, appending, and reading NetCDF files. NetCDF is an increas-ingly popular self-describing file format [44] intended primarily for scientific datasets. An extensive collection of NetCDF reference papers, user guides, software,FAQs, and other information can be obtained from UCAR’s web site at:

http://www.unidata.ucar.edu/packages/netcdf/

6.15.1 Using MNC

6.15.1.1 MNC Configuration

As with all MITgcm packages, MNC can be turned on or off at compile timeusing the packages.conf file or the genmake2 -enable=mnc or -disable=mncswitches.

While MNC is likely to work “as is”, there are a few compile–time constantsthat may need to be increased for simulations that employ large numbers oftiles within each process. Note that the important quantity is the maximumnumber of tiles per process. Since MPI configurations tend to distribute largenumbers of tiles over relatively large numbers of MPI processes, these constantswill rarely need to be increased.

If MNC runs out of space within its “lookup” tables during a simulation, thenit will provide an error message along with a recommendation of which param-eter to increase. The parameters are all located within pkg/mnc/mnc common.hand the ones that may need to be increased are:

Name Default Description

MNC MAX ID 1000 IDs for various low-level entitiesMNC MAX INFO 400 IDs (mostly for object sizes)MNC CW MAX I 150 IDs for the “wrapper” layer

In those rare cases where MNC “out-of-memory” error messages are encoun-tered, it is a good idea to increase the too-small parameter by a factor of 2–10in order to avoid wasting time on an iterative compile–test sequence.

6.15.1.2 MNC Inputs

For run-time configuration, most of the MNC–related model parameters arecontained within a Fortran namelist file called data.mnc. If this file does notexist, then the MNC package will interpret that as an indication that it is notto be used. If the data.mnc file does exist, then it may contain the followingparameters:


Name T Default Description

useMNC L .FALSE. overall MNC ON/OFF switchmnc echo gvtypes L .FALSE. echo pre-defined “types” (debugging)mnc use outdir L .FALSE. create a directory for outputmnc outdir str S ’mnc ’ output directory namemnc outdir date L .FALSE. embed date in the output dir namepickup write mnc L .FALSE. use MNC to write (create) pickup filespickup read mnc L .FALSE. use MNC to read pickup filesmnc use indir L .FALSE. use a directory (path) for inputmnc indir str S ’’ input directory (or path) namesnapshot mnc L .FALSE. write snapshot (instantaneous) w/MNCmonitor mnc L .FALSE. write monitor w/MNCtimeave mnc L .FALSE. write timeave w/MNCautodiff mnc L .FALSE. write autodiff w/MNC

Additional MNC–related parameters are contained within the main datanamelist file and in some of the namelist files for individual packages. Theseoptions are:

Name T Default Description

Main namelist file: “data”snapshot ioinc L .FALSE. write snapshot “inclusively”timeave ioinc L .FALSE. write timeave “inclusively”monitor ioinc L .FALSE. write monitor “inclusively”the run name C “name...” name is included in all MNC output

Diagnostics namelist file: “data.diagnostics”diag mnc L .FALSE. write diagnostics w/MNCdiag ioinc L .FALSE. write diagnostics “inclusively”

By default, turning on MNC for a particular output type will result in turn-ing off all the corresponding (usually, default) MDSIO or STDOUT outputmechanisms. In other words, output defaults to being an exclusive selection.To enable multiple kinds of simultaneous output, flags of the form NAME ioinchave been created where NAME corresponds to the various MNC output flags.When a NAME ioinc flag is set to .TRUE., then multiple simultaneous forms ofoutput are allowed for the NAME output mechanism. The intent of this design isthat typical users will only want one kind of output while people debugging thecode (particularly the I/O routines) may want simultaneous types of output.

This “inclusive” versus “exclusive” design is easily applied in cases wherethree or more kinds of output may be generated. Thus, it can be readily ex-tended to additional new output types (eg. HDF5).

Input types are always exclusive.

6.15.1.3 MNC Output

While NetCDF files are supposed to be “self-describing”, it is helpful to notethe following:

• The constraints placed upon the “unlimited” (or “record”) dimension in-herent with NetCDF v3.x make it very inefficient to put variables written


at potentially different intervals within the same file. For this reason,MNC output is split into a few file “base names” which try to reflect thenature of their content.

• All MNC output is currently done in a “tile-per-file” fashion since mostNetCDF v3.x implementions cannot write safely within MPI or multi-threaded environments. This tiling is done in a global fashion and the tilenumbers are appended to the base names described above. Some scripts to“assemble” output are available (MITgcm/utils/matlab). More generalmanipulations can be accomplished with the

NetCDF Operators (or ‘‘NCO’’) at http://nco.sourceforge.net

which is a very powerful and convenient set of tools for working with allNetCDF files.

• On many systems, NetCDF has practical file size limits on the order of2–4GB (the maximium memory addressable with 32bit pointers) due to alack of operating system, compiler, and/or library support. In cases wherethis limit is reached, it is generally a good idea to reduce write frequenciesor restart from pickups.

• MNC does not (yet) provide a mechanism for reading information from asingle “global” file as can be done with the MDSIO package. This is inprogress.

6.15.2 MNC Internals

The mnc package is a two-level convenience library (or “wrapper”) for most ofthe NetCDF Fortran API. Its purpose is to streamline the user interface toNetCDF by maintaining internal relations (look-up tables) keyed with strings(or names) and entities such as NetCDF files, variables, and attributes.

The two levels of the mnc package are:

Upper level

The upper level contains information about two kinds of associations:

grid type is lookup table indexed with a grid type name. Each gridtype name is associated with a number of dimensions, the dimensionsizes (one of which may be unlimited), and starting and ending in-dex arrays. The intent is to store all the necessary size and shapeinformation for the Fortran arrays containing MITgcm–style “tile”variables (that is, a central region surrounded by a variably-sized“halo” or exchange region as shown in Figures 4.7 and 4.8).

variable type is a lookup table indexed by a variable type name. Foreach name, the table contains a reference to a grid type for the vari-able and the names and values of various attributes.


Within the upper level, these associations are not permanently tied toany particular NetCDF file. This allows the information to be re-usedover multiple file reads and writes.

Lower level

In the lower (or internal) level, associations are stored for NetCDF filesand many of the entities that they contain including dimensions, variables,and global attributes. All associations are on a per-file basis. Thus, eachentity is tied to a unique NetCDF file and will be created or destroyedwhen files are, respectively, opened or closed.

6.15.2.1 MNC Grid–Types and Variable–Types

As a convenience for users, the MNC package includes numerous routines to aidin the writing of data to NetCDF format. Probably the biggest convenienceis the use of pre-defined “grid types” and “variable types”. These “types”are simply look-up tables that store dimensions, indicies, attributes, and otherinformation that can all be retrieved using a single character string.

The “grid types” are a way of mapping variables within MITgcm to NetCDFarrays. Within MITgcm, most spatial variables are defined using two– or three–dimensional arrays with “overlap” regions (see Figures 4.7, a possible verticalindex, and 4.8) and tile indicies such as the following “U” velocity:

_RL uVel (1-OLx:sNx+OLx,1-OLy:sNy+OLy,Nr,nSx,nSy)

as defined in model/inc/DYNVARS.hThe grid type is a character string that encodes the presence and types

associated with the four possible dimensions. The character string follows theformat

H0 H1 H2 V T

where the terms H0, H1, H2, V, T can be almost any combination of thefollowing:

Horizontal Vertical TimeH0: location H1: dimensions H2: halo V: location T: level

- xy Hn - -U x Hy i tV y c

CenCor

A example list of all pre-defined combinations is contained in the file

pkg/mnc/pre-defined grids.txt.

The variable type is an association between a variable type name and thefollowing items:


Item Purposegrid type defines the in-memory arrangementbi,bj dimensions tiling indices, if present

and is used by the mnc cw * [R|W] subroutines for reading and writing variables.

6.15.2.2 Using MNC: Examples

Writing variables to NetCDF files can be accomplished in as few as two functioncalls. The first function call defines a variable type, associates it with a name(character string), and provides additional information about the indicies forthe tile (bi,bj) dimensions. The second function call will write the data at, ifnecessary, the current time level within the model.

Examples of the initialization calls can be found in the file model/src/ini mnc io.Fwhere these function calls:

C Create MNC definitions for DYNVARS.h variables

CALL MNC_CW_ADD_VNAME(’iter’, ’-_-_--__-__t’, 0,0, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’iter’,1,

& ’long_name’,’iteration_count’, myThid)

CALL MNC_CW_ADD_VNAME(’model_time’, ’-_-_--__-__t’, 0,0, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’model_time’,1,

& ’long_name’,’Model Time’, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’model_time’,1,’units’,’s’, myThid)

CALL MNC_CW_ADD_VNAME(’U’, ’U_xy_Hn__C__t’, 4,5, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’U’,1,’units’,’m/s’, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’U’,1,

& ’coordinates’,’XU YU RC iter’, myThid)

CALL MNC_CW_ADD_VNAME(’T’, ’Cen_xy_Hn__C__t’, 4,5, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’T’,1,’units’,’degC’, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’T’,1,’long_name’,

& ’potential_temperature’, myThid)

CALL MNC_CW_ADD_VATTR_TEXT(’T’,1,

& ’coordinates’,’XC YC RC iter’, myThid)

initialize four VNAMEs and add one or more NetCDF attributes to each.The four variables defined above are subsequently written at specific time

steps within model/src/write state.F using the function calls:

C Write dynvars using the MNC package

CALL MNC_CW_SET_UDIM(’state’, -1, myThid)

CALL MNC_CW_I_W(’I’,’state’,0,0,’iter’, myIter, myThid)

CALL MNC_CW_SET_UDIM(’state’, 0, myThid)

CALL MNC_CW_RL_W(’D’,’state’,0,0,’model_time’,myTime, myThid)

CALL MNC_CW_RL_W(’D’,’state’,0,0,’U’, uVel, myThid)

CALL MNC_CW_RL_W(’D’,’state’,0,0,’T’, theta, myThid)

While it is easiest to write variables within typical 2D and 3D fields whereall data is known at a given time, it is also possible to write fields where onlya portion (eg. a “slab” or “slice”) is known at a given instant. An example isprovided within pkg/mom vecinv/mom vecinv.F where an offset vector is used:


IF (useMNC .AND. snapshot_mnc) THEN

CALL MNC_CW_RL_W_OFFSET(’D’,’mom_vi’,bi,bj, ’fV’, uCf,

& offsets, myThid)

CALL MNC_CW_RL_W_OFFSET(’D’,’mom_vi’,bi,bj, ’fU’, vCf,

& offsets, myThid)

ENDIF

to write a 3D field one depth slice at a time.Each element in the offset vector corresponds (in order) to the dimensions

of the “full” (or virtual) array and specifies which are known at the time of thecall. A zero within the offset array means that all values along that dimensionare available while a positive integer means that only values along that index ofthe dimension are available. In all cases, the matrix passed is assumed to start(that is, have an in-memory structure) coinciding with the start of the specifiedslice. Thus, using this offset array mechanism, a slice can be written along anysingle dimension or combinations of dimensions.

6.16. MDSIO 351

6.16 MDSIO

The mdsio package contains a group of Fortran routines intended as a generalinterface for reading and writing direct-access (“binary”) Fortran files. Themdsio routines are used by the rw package.


6.17 Simulation state monitoring toolkit


6.18. EXCH2: EXTENDED CUBED SPHERE TOPOLOGY 353

6.18 exch2: Extended Cubed Sphere Topology

6.18.1 Introduction

The exch2 package extends the original cubed sphere topology configuration toallow more flexible domain decomposition and parallelization. Cube faces (alsocalled subdomains) may be divided into any number of tiles that divide evenlyinto the grid point dimensions of the subdomain. Furthermore, the tiles canrun on separate processors individually or in groups, which provides for manualcompile-time load balancing across a relatively arbitrary number of processors.

The exchange parameters are declared in pkg/exch2/W2 EXCH2 TOPOLOGY.hand assigned in pkg/exch2/w2 e2setup.F. The validity of the cube topologydepends on the SIZE.h file as detailed below. The default files provided in therelease configure a cubed sphere topology of six tiles, one per subdomain, eachwith 32×32 grid points, with all tiles running on a single processor. Both files aregenerated by Matlab scripts in utils/exch2/matlab-topology-generator; seeSection 6.18.3 Generating Topology Files for exch2 for details on creating alter-nate topologies. Pregenerated examples of these files with alternate topologiesare provided under utils/exch2/code-mods along with the appropriate SIZE.hfile for single-processor execution.

6.18.2 Invoking exch2

To use exch2 with the cubed sphere, the following conditions must be met:

• The exch2 package is included when genmake2 is run. The easiest wayto do this is to add the line exch2 to the profile.conf file – see Section 3.5Building the code for general details.

• An example of W2 EXCH2 TOPOLOGY.h and w2 e2setup.F must reside ina directory containing files symbolically linked by the genmake2 script. Thesafest place to put these is the directory indicated in the -mods=DIR commandline modifier (typically ../code), or the build directory. The default versions ofthese files reside in pkg/exch2 and are linked automatically if no other versionsexist elsewhere in the build path, but they should be left untouched to avoidbreaking configurations other than the one you intend to modify.

• Files containing grid parameters, named tile00n.mitgridwhere n=(1:6)(one per subdomain), must be in the working directory when the MITgcm ex-ecutable is run. These files are provided in the example experiments for cubedsphere configurations with 32×32 cube sides – please contact MITgcm supportif you want to generate files for other configurations.

• As always when compiling MITgcm, the file SIZE.h must be placed wheregenmake2 will find it. In particular for exch2, the domain decomposition spec-


ified in SIZE.h must correspond with the particular configuration’s topologyspecified in W2 EXCH2 TOPOLOGY.h and w2 e2setup.F. Domain decompositionissues particular to exch2 are addressed in Section 6.18.3 Generating TopologyFiles for exch2 and 6.18.4 exch2, SIZE.h, and Multiprocessing ; a more generalbackground on the subject relevant to MITgcm is presented in Section 4.3.1Specifying a decomposition.

At the time of this writing the following examples use exch2 and may beused for guidance:

verification/adjust_nlfs.cs-32x32x1verification/adjustment.cs-32x32x1verification/aim.5l_csverification/global_ocean.cs32x15verification/hs94.cs-32x32x5

6.18.3 Generating Topology Files for exch2

Alternate cubed sphere topologies may be created using the Matlab scriptsin utils/exch2/matlab-topology-generator. Running the m-file driver.mfrom the Matlab prompt (there are no parameters to pass) generates exch2topology files W2 EXCH2 TOPOLOGY.h and w2 e2setup.F in the working direc-tory and displays a figure of the topology via Matlab – figures 6.6, 6.5, and6.4 are examples of the generated diagrams. The other m-files in the directoryare subroutines called from driver.m and should not be run “bare” except fordevelopment purposes.

The parameters that determine the dimensions and topology of the gener-ated configuration are nr, nb, ng, tnx and tny, and all are assigned early in thescript.

The first three determine the height and width of the subdomains and hencethe size of the overall domain. Each one determines the number of grid points,and therefore the resolution, along the subdomain sides in a “great circle”around each the three spatial axes of the cube. At the time of this writingMITgcm requires these three parameters to be equal, but they provide for fu-ture releases to accomodate different resolutions around the axes to allow sub-domains with differing resolutions.

The parameters tnx and tny determine the width and height of the tilesinto which the subdomains are decomposed, and must evenly divide the integerassigned to nr, nb and ng. The result is a rectangular tiling of the subdomain.Figure 6.4 shows one possible topology for a twenty-four-tile cube, and figure6.5 shows one for twelve tiles.


0 20 40 60 80 100 120 1400

10

20

30

40

50

60

70

80

90

100

f1

t1f5n

f6n

t2

f6n

f2w

t3f5n

f3w

t4

f3wf2

w

f2

t5f1e

f6e

t6

f6e

f4s

t7f1e

f3s

t8

f3sf4

s

f3

t9f1n

f2n

t10

f2n

f4w

t11f1n

f5w

t12

f5wf4

w

f4

t13f3e

f2e

t14

f2e

f6s

t15f3e

f5s

t16

f5sf6

s

f5

t17f3n

f4n

t18

f4n

f6w

t19f3n

f1w

t20

f1wf6

w

f6

t21f5e

f4e

t22

f4e

f2s

t23f5e

f1s

t24

f1sf2

s

Figure 6.4: Plot of a cubed sphere topology with a 32×192 domain divided intosix 32×32 subdomains, each of which is divided into four tiles of width tnx=16and height tny=16 for a total of twenty-four tiles. The colored borders of thesubdomains represent the parameters nr (red), nb (blue), and ng (green).


0 20 40 60 80 100 120 1400

10

20

30

40

50

60

70

80

90

100

f1t1f5n

f3w

f6n

t2

f3w

f6n

f2w f2t3f1

e

f3s

f6e

t4

f3s

f6e

f4s

f3t5f1n

f5w

f2n

t6

f5w

f2n

f4w f4t7f3

e

f5s

f2e

t8

f5s

f2e

f6s

f5t9f3n

f1w

f4n

t10

f1w

f4n

f6w f6t11f5

e

f1s

f4e

t12

f1s

f4e

f2s

Figure 6.5: Plot of a cubed sphere topology with a 32×192 domain divided intosix 32×32 subdomains of two tiles each (tnx=16, tny=32).

Tiles can be selected from the topology to be omitted from being allocatedmemory and processors. This tuning is useful in ocean modeling for omit-ting tiles that fall entirely on land. The tiles omitted are specified in the fileblanklist.txt by their tile number in the topology, separated by a newline.

6.18.4 exch2, SIZE.h, and Multiprocessing

Once the topology configuration files are created, the Fortran PARAMETERs inSIZE.h must be configured to match. Section 4.3.1 Specifying a decomposi-tion provides a general description of domain decomposition within MITgcmand its relation to SIZE.h. The current section specifies constraints that theexch2 package imposes and describes how to enable parallel execution with MPI.

As in the general case, the parameters sNx and sNy define the size of theindividual tiles, and so must be assigned the same respective values as tnx andtny in driver.m.

The halo width parameters OLx and OLy have no special bearing on exch2and may be assigned as in the general case. The same holds for Nr, the numberof vertical levels in the model.


0 20 40 60 80 100 120 1400

10

20

30

40

50

60

70

80

90

100

f1t1f5n

f3w

f6n

f2w f2t2f1

e

f3s

f6e

f4s

f3t3f1n

f5w

f2n

f4w f4t4f3

e

f5s

f2e

f6s

f5t5f3n

f1w

f4n

f6w f6t6f5

e

f1s

f4e

f2s

Figure 6.6: Plot of a cubed sphere topology with a 32×192 domain divided intosix 32×32 subdomains with one tile each (tnx=32, tny=32). This is the defaultconfiguration.


The parameters nSx, nSy, nPx, and nPy relate to the number of tiles andhow they are distributed on processors. When using exch2, the tiles are storedin the x dimension, and so nSy=1 in all cases. Since the tiles as configured byexch2 cannot be split up accross processors without regenerating the topology,nPy=1 as well.

The number of tiles MITgcm allocates and how they are distributed betweenprocessors depends on nPx and nSx. nSx is the number of tiles per processorand nPx is the number of processors. The total number of tiles in the topologyminus those listed in blanklist.txt must equal nSx*nPx. Note that in orderto obtain maximum usage from a given number of processors in some cases,this restriction might entail sharing a processor with a tile that would otherwisebe excluded because it is topographically outside of the domain and thereforein blanklist.txt. For example, suppose you have five processors and a do-main decomposition of thirty-six tiles that allows you to exclude seven tiles.To evenly distribute the remaining twenty-nine tiles among five processors, youwould have to run one “dummy” tile to make an even six tiles per processor.Such dummy tiles are not listed in blanklist.txt.

The following is an example of SIZE.h for the twelve-tile configuration il-lustrated in figure 6.5 running on one processor:

PARAMETER (& sNx = 16,& sNy = 32,& OLx = 2,& OLy = 2,& nSx = 12,& nSy = 1,& nPx = 1,& nPy = 1,& Nx = sNx*nSx*nPx,& Ny = sNy*nSy*nPy,& Nr = 5)

The following is an example for the twenty-four-tile topology in figure 6.4running on six processors:

PARAMETER (& sNx = 16,& sNy = 16,& OLx = 2,& OLy = 2,& nSx = 4,


& nSy = 1,& nPx = 6,& nPy = 1,& Nx = sNx*nSx*nPx,& Ny = sNy*nSy*nPy,& Nr = 5)

6.18.5 Key Variables

The descriptions of the variables are divided up into scalars, one-dimensional ar-rays indexed to the tile number, and two and three-dimensional arrays indexedto tile number and neighboring tile. This division reflects the functionality ofthese variables: The scalars are common to every part of the topology, the tile-indexed arrays to individual tiles, and the arrays indexed by tile and neighborto relationships between tiles and their neighbors.

6.18.5.1 Scalars

The number of tiles in a particular topology is set with the parameter NTILES,and the maximum number of neighbors of any tiles by MAX NEIGHBOURS. Theseparameters are used for defining the size of the various one and two dimensionalarrays that store tile parameters indexed to the tile number and are assigned inthe files generated by driver.m.

The scalar parameters exch2 domain nxt and exch2 domain nyt express thenumber of tiles in the x and y global indices. For example, the default setup ofsix tiles (Fig. 6.6) has exch2 domain nxt=6 and exch2 domain nyt=1. A topol-ogy of twenty-four square tiles, four per subdomain (as in figure 6.4), will haveexch2 domain nxt=12 and exch2 domain nyt=2. Note that these parametersexpress the tile layout in order to allow global data files that are tile-layout-neutral. They have no bearing on the internal storage of the arrays. The tilesare stored internally in a range from bi=(1:NTILES) in the x axis, and the yaxis variable bj is assumed to equal 1 throughout the package.

6.18.5.2 Arrays indexed to tile number

The following arrays are of length NTILES and are indexed to the tile number,which is indicated in the diagrams with the notation tn. The indices are omittedin the descriptions.

The arrays exch2 tnx and exch2 tny express the x and y dimensions of eachtile. At present for each tile exch2 tnx=sNx and exch2 tny=sNy, as assignedin SIZE.h and described in Section 6.18.4 exch2, SIZE.h, and Multiprocessing.


Future releases of MITgcm may allow varying tile sizes.

The arrays exch2 tbasex and exch2 tbasey determine the tiles’ Cartesianorigin within a subdomain and locate the edges of different tiles relative toeach other. As an example, in the default six-tile topology (Fig. 6.6) each in-dex in these arrays is set to 0 since a tile occupies its entire subdomain. Thetwenty-four-tile case discussed above will have values of 0 or 16, dependingon the quadrant of the tile within the subdomain. The elements of the ar-rays exch2 txglobalo and exch2 txglobalo are similar to exch2 tbasex andexch2 tbasey, but locate the tile edges within the global address space, similarto that used by global output and input files.

The array exch2 myFace contains the number of the subdomain of each tile,in a range (1:6) in the case of the standard cube topology and indicated byfn in figures 6.5 and 6.4. exch2 nNeighbours contains a count of the neighbor-ing tiles each tile has, and sets the bounds for looping over neighboring tiles.exch2 tProc holds the process rank of each tile, and is used in interprocesscommunication.

The arrays exch2 isWedge, exch2 isEedge, exch2 isSedge, and exch2 isNedgeare set to 1 if the indexed tile lies on the edge of its subdomain, 0 if not. Thevalues are used within the topology generator to determine the orientation ofneighboring tiles, and to indicate whether a tile lies on the corner of a subdo-main. The latter case requires special exchange and numerical handling for thesingularities at the eight corners of the cube.

6.18.5.3 Arrays Indexed to Tile Number and Neighbor

The following arrays have vectors of length MAX NEIGHBOURS and NTILES anddescribe the orientations between the the tiles.

The array exch2 neighbourId(a,T) holds the tile number Tn for each of thetile number T’s neighboring tiles a. The neighbor tiles are indexed (1:exch2 nNeighbours(T))in the order right to left on the north then south edges, and then top to bottomon the east then west edges.

The exch2 opposingSend record(a,T) array holds the index b of the ele-ment in exch2 neighbourId(b,Tn) that holds the tile number T, given Tn=exch2 neighborId(a,T).In other words,

exch2_neighbourId( exch2_opposingSend_record(a,T),exch2_neighbourId(a,T) ) = T

This provides a back-reference from the neighbor tiles.


The arrays exch2 pi and exch2 pj specify the transformations of indices inexchanges between the neighboring tiles. These transformations are necessaryin exchanges between subdomains because a horizontal dimension in one sub-domain may map to other horizonal dimension in an adjacent subdomain, andmay also have its indexing reversed. This swapping arises from the “folding” oftwo-dimensional arrays into a three-dimensional cube.

The dimensions of exch2 pi(t,N,T) and exch2 pj(t,N,T) are the neigh-bor ID N and the tile number T as explained above, plus a vector of length 2containing transformation factors t. The first element of the transformationvector holds the factor to multiply the index in the same dimension, and thesecond element holds the the same for the orthogonal dimension. To clarify,exch2 pi(1,N,T) holds the mapping of the x axis index of tile T to the x axisof tile T’s neighbor N, and exch2 pi(2,N,T) holds the mapping of T’s x indexto the neighbor N’s y index.

One of the two elements of exch2 pi or exch2 pj for a given tile T andneighbor N will be 0, reflecting the fact that the two axes are orthogonal. Theother element will be 1 or -1, depending on whether the axes are indexed in thesame or opposite directions. For example, the transform vector of the arraysfor all tile neighbors on the same subdomain will be (1,0), since all tiles onthe same subdomain are oriented identically. An axis that corresponds to theorthogonal dimension with the same index direction in a particular tile-neighbororientation will have (0,1). Those with the opposite index direction will have(0,-1) in order to reverse the ordering.

The arrays exch2 oi, exch2 oj, exch2 oi f, and exch2 oj f are indexed totile number and neighbor and specify the relative offset within the subdomainof the array index of a variable going from a neighboring tile N to a local tile T.Consider T=1 in the six-tile topology (Fig. 6.6), where

exch2_oi(1,1)=33exch2_oi(2,1)=0exch2_oi(3,1)=32exch2_oi(4,1)=-32

The simplest case is exch2 oi(2,1), the southern neighbor, which is Tn=6.The axes of T and Tn have the same orientation and their x axes have the sameorigin, and so an exchange between the two requires no changes to the x index.For the western neighbor (Tn=5), code oi(3,1)=32 since the x=0 vector on Tcorresponds to the y=32 vector on Tn. The eastern edge of T shows the reversecase (exch2 oi(4,1)=-32)), where x=32 on T exchanges with x=0 on Tn=2.

The most interesting case, where exch2 oi(1,1)=33 and Tn=3, involves areversal of indices. As in every case, the offset exch2 oi is added to the originalx index of T multiplied by the transformation factor exch2 pi(t,N,T). Here


exch2 pi(1,1,1)=0 since the x axis of T is orthogonal to the x axis of Tn.exch2 pi(2,1,1)=-1 since the x axis of T corresponds to the y axis of Tn, butthe index is reversed. The result is that the index of the northern edge of T,which runs (1:32), is transformed to (-1:-32). exch2 oi(1,1) is then addedto this range to get back (32:1) – the index of the y axis of Tn relative to T.This transformation may seem overly convoluted for the six-tile case, but it isnecessary to provide a general solution for various topologies.

Finally, exch2 itlo c, exch2 ithi c, exch2 jtlo c and exch2 jthi c holdthe location and index bounds of the edge segment of the neighbor tile N’s sub-domain that gets exchanged with the local tile T. To take the example of tileT=2 in the twelve-tile topology (Fig. 6.5):

exch2_itlo_c(4,2)=17exch2_ithi_c(4,2)=17exch2_jtlo_c(4,2)=0exch2_jthi_c(4,2)=33

Here N=4, indicating the western neighbor, which is Tn=1. Tn resides on thesame subdomain as T, so the tiles have the same orientation and the same xand y axes. The x axis is orthogonal to the western edge and the tile is 16points wide, so exch2 itlo c and exch2 ithi c indicate the column beyondTn’s eastern edge, in that tile’s halo region. Since the border of the tiles extendsthrough the entire height of the subdomain, the y axis bounds exch2 jtlo cto exch2 jthi c cover the height of (1:32), plus 1 in either direction to coverpart of the halo.

For the north edge of the same tile T=2 where N=1 and the neighbor tile isTn=5:

exch2_itlo_c(1,2)=0exch2_ithi_c(1,2)=0exch2_jtlo_c(1,2)=0exch2_jthi_c(1,2)=17

T’s northern edge is parallel to the x axis, but since Tn’s y axis corresponds toT’s x axis, T’s northern edge exchanges with Tn’s western edge. The western edgeof the tiles corresponds to the lower bound of the x axis, so exch2 itlo c andexch2 ithi c are 0, in the western halo region of Tn. The range of exch2 jtlo cand exch2 jthi c correspond to the width of T’s northern edge, expanded byone into the halo.

6.18.6 Key Routines

Most of the subroutines particular to exch2 handle the exchanges themselvesand are of the same format as those described in 4.3.3.3 Cube sphere communi-


cation. Like the original routines, they are written as templates which the localMakefile converts from RX into RL and RS forms.

The interfaces with the core model subroutines are EXCH UV XY RX, EXCH UV XYZ RXand EXCH XY RX. They override the standard exchange routines when genmake2is run with exch2 option. They in turn call the local exch2 subroutines EXCH2 UV XY RXand EXCH2 UV XYZ RX for two and three-dimensional vector quantities, and EXCH2 XY RXand EXCH2 XYZ RX for two and three-dimensional scalar quantities. These sub-routines set the dimensions of the area to be exchanged, call EXCH2 RX1 CUBEfor scalars and EXCH2 RX2 CUBE for vectors, and then handle the singularities atthe cube corners.

The separate scalar and vector forms of EXCH2 RX1 CUBE and EXCH2 RX2 CUBEreflect that the vector-handling subroutine needs to pass both the u and v com-ponents of the physical vectors. This swapping arises from the topological fold-ing discussed above, where the x and y axes get swapped in some cases, and isnot an issue with the scalar case. These subroutines call EXCH2 SEND RX1 andEXCH2 SEND RX2, which do most of the work using the variables discussed above.


6.19 Diagnostics–A Flexible Infrastructure

6.19.1 Introduction

This section of the documentation describes the Diagnostics package availablewithin the GCM. A large selection of model diagnostics is available for output.In addition to the diagnostic quantities pre-defined in the GCM, there exists theoption, in any experiment, to define a new diagnostic quantity and include it aspart of the diagnostic output with the addition of a single subroutine call in theroutine where the field is computed. As a matter of philosophy, no diagnosticis enabled as default, thus each user must specify the exact diagnostic informa-tion required for an experiment. This is accomplished by enabling the specificdiagnostic of interest cataloged in the Diagnostic Menu (see Section 6.19.4.1).Instructions for enabling diagnostic output and defining new diagnostic quanti-ties are found in Section 6.19.4 of this document.The Diagnostic Menu is a hard-wired enumeration of diagnostic quantities avail-able within the GCM. Once a diagnostic is enabled, the GCM will continuallyincrement an array specifically allocated for that diagnostic whenever the ap-propriate quantity is computed. A counter is defined which records how manytimes each diagnostic quantity has been incremented. Several special diagnos-tics are included in the menu. Quantities refered to as “Counter Diagnostics”,are defined for selected diagnostics which record the frequency at which a di-agnostic is incremented separately for each model grid location. Quantitiedrefered to as “User Diagnostics” are included in the menu to facilitate definingnew diagnostics for a particular experiment.

6.19.2 Equations

Not relevant.

6.19.3 Key Subroutines and Parameters

The diagnostics are computed at various times and places within the GCM.Because the MIT GCM may employ a staggered grid, diagnostics may be com-puted at grid box centers, corners, or edges, and at the middle or edge in thevertical. Some diagnostics are scalars, while others are components of vectors.An internal array is defined which contains information concerning various gridattributes of each diagnostic. The GDIAG array (in common blockdiagnostics in file diagnostics.h) is internally defined as a character*8 variable,and is equivalenced to a character*1 ”parse” array in output in order to extractthe grid-attribute information. The GDIAG array is described in Table 6.1.As an example, consider a diagnostic whose associated GDIAG parameter isequal to “UU 002”. From GDIAG we can determine that this diagnostic isa U-vector component located at the C-grid U-point. Its corresponding V-component diagnostic is located in Diagnostic # 002.

6.19. DIAGNOSTICS–A FLEXIBLE INFRASTRUCTURE 365

Table 6.1: Diagnostic Parsing Array

Diagnostic Parsing ArrayArray Value Description

parse(1) → S Scalar Diagnostic→ U U-vector component Diagnostic→ V V-vector component Diagnostic

parse(2) → U C-Grid U-Point→ V C-Grid V-Point→ M C-Grid Mass Point→ Z C-Grid Vorticity (Corner) Point

parse(3) → R Not Currently in Useparse(4) → P Positive Definite Diagnosticparse(5) → C Counter Diagnostic

→ D Disabled Diagnostic for outputparse(6-8) → C 3-digit integer corresponding to

vector or counter component mate

In this way, each Diagnostic in the model has its attributes (ie. vector orscalar, C-grid location, etc.) defined internally. The Output routines use thisinformation in order to determine what type of transformations need to beperformed. Any interpolations are done at the time of output rather thanduring each model step. In this way the User has flexibility in determining thetype of gridded data which is output.There are several utilities within the GCM available to users to enable, disable,clear, write and retrieve model diagnostics, and may be called from any routine.The available utilities and the CALL sequences are listed below.fill diagnostics: This routine will increment the specified diagnostic quantitywith a field sent through the argument list.

call fill diagnostics (myThid, chardiag, levflg, nlevs,bibjflg, bi, bj, arrayin)

where myThid = Current Process(or)chardiag = Character *8 expression for diag to filllevflg = Integer flag for vertical levels:

0 indicates multiple levels incremented in qdiagnon-0 (any integer) - WHICH single level to increment.negative integer - the input data array is single-leveledpositive integer - the input data array is multi-leveled

nlevs = indicates Number of levels to be filled (1 if levflg ¡¿ 0)positive: fill in ”nlevs” levels in the same order asthe input arraynegative: fill in -nlevs levels in reverse order.


bibjflg = Integer flag to indicate instructions for bi bj loop0 indicates that the bi-bj loop must be done here1 indicates that the bi-bj loop is done OUTSIDE2 indicates that the bi-bj loop is done OUTSIDEAND that we have been sent a local array3 indicates that the bi-bj loop is done OUTSIDEAND that we have been sent a local arrayAND that the array has the shadow regions

bi = X-direction process(or) number - used for bibjflg=1-3bj = Y-direction process(or) number - used for bibjflg=1-3arrayin = Field to increment diagnostics array

setdiag: This subroutine enables a diagnostic from the Diagnostic Menu, mean-ing that space is allocated for the diagnostic and the model routines will incre-ment the diagnostic value during execution. This routine is the underlyinginterface between the user and the desired diagnostic. The diagnostic is refer-enced by its diagnostic number from the menu, and its calling sequence is givenby:

call setdiag (num)

where num = Diagnostic number from menu

getdiag: This subroutine retrieves the value of a model diagnostic. This rou-tine is particulary useful when called from a user output routine, although itcan be called from any routine. This routine returns the time-averaged valueof the diagnostic by dividing the current accumulated diagnostic value by itscorresponding counter. This routine does not change the value of the diagnos-tic itself, that is, it does not replace the diagnostic with its time-average. Thecalling sequence for this routine is givin by:

call getdiag (lev,num,qtmp,undef)

where lev = Model Level at which the diagnostic is desirednum = Diagnostic number from menuqtmp = Time-Averaged Diagnostic Outputundef = Fill value to be used when diagnostic is undefined

clrdiag: This subroutine initializes the values of model diagnostics to zero,and is particularly useful when called from user output routines to re-initializediagnostics during the run. The calling sequence is:

call clrdiag (num)



zapdiag: This entry into subroutine SETDIAG disables model diagnostics,meaning that the diagnostic is no longer available to the user. The memorypreviously allocated to the diagnostic is released when ZAPDIAG is invoked.The calling sequence is given by:

call zapdiag (NUM)


6.19.4 Usage Notes

We begin this section with a discussion on the manner in which computer mem-ory is allocated for diagnostics. All GCM diagnostic quantities are stored in thesingle diagnostic array QDIAG which is located in the filepkg/diagnostics/diagnostics.h. and has the form:

common /diagnostics/ qdiag(1-Olx,sNx+Olx,1-Olx,sNx+Olx,numdiags,Nsx,Nsy)where numdiags is an Integer variable which should be set equal to the num-ber of enabled diagnostics, and qdiag is a three-dimensional array. The firsttwo-dimensions of qdiag correspond to the horizontal dimension of a given diag-nostic, while the third dimension of qdiag is used to identify diagnostic fields andlevels combined. In order to minimize the memory requirement of the modelfor diagnostics, the default GCM executable is compiled with room for onlyone horizontal diagnostic array, or with numdiags set to 1. In order for theUser to enable more than 1 two-dimensional diagnostic, the size of the diagnos-tics common must be expanded to accomodate the desired diagnostics. Thiscan be accomplished by manually changing the parameter numdiags in the filepkg/diagnostics/diagnostics SIZE.h. numdiags should be set greater thanor equal to the sum of all the diagnostics activated for output each multipliedby the number of levels defined for that diagnostic quantity. This is illustratedin the example below:To use the diagnostics package, other than enabling it in packages.conf andturning the usediagnostics flag in data.pkg to .TRUE., a namelist must be sup-plied in the run directory called data.diagnostics. The namelist will activate auser-defined list of diagnostics quantities to be computed, specify the frequencyof output, the number of levels, and the name of up to 10 separate output files.A sample data.diagnostics namelist file:# Diagnostic Package Choices&diagnostics listfrequency(1) = 10,levels(1,1) = 1.,2.,3.,4.,5.,fields(1,1) = ’UVEL ’,’VVEL ’,filename(1) = ’diagout1’,frequency(2) = 100,levels(1,2) = 1.,2.,3.,4.,5.,fields(1,2) = ’THETA ’,’SALT ’,


filename(2) = ’diagout2’,&end

In this example, there are two output files that will be generated for each tileand for each output time. The first set of output files has the prefix diagout1,does time averaging every 10 time steps (frequency is 10), they will write fieldswhich are multiple-level fields and output levels 1-5. The names of diagnosticsquantities are UVEL and VVEL. The second set of output files has the prefixdiagout2, does time averaging every 100 time steps, they include fields whichare multiple-level fields, levels output are 1-5, and the names of diagnosticsquantities are THETA and SALT.In order to define and include as part of the diagnostic output any field thatis desired for a particular experiment, two steps must be taken. The first isto enable the “User Diagnostic” in data.diagnostics. This is accomplished bysetting one of the fields slots to either UDIAG1 through UDIAG10, for multi-level fields, or SDIAG1 through SDIAG10 for single level fields. These are listedin the diagnostics menu. The second step is to add a call to fill diagnostics fromthe subroutine in which the quantity desired for diagnostic output is computed.


6.19.4.1 GCM Diagnostic Menu

N NAME UNITS LEVELS DESCRIPTION

1 UFLUX Newton/m2 1 Surface U-Wind Stress on the atmosphere2 VFLUX Newton/m2 1 Surface V-Wind Stress on the atmosphere3 HFLUX Watts/m2 1 Surface Flux of Sensible Heat4 EFLUX Watts/m2 1 Surface Flux of Latent Heat5 QICE Watts/m2 1 Heat Conduction through Sea-Ice6 RADLWG Watts/m2 1 Net upward LW flux at the ground7 RADSWG Watts/m2 1 Net downward SW flux at the ground8 RI dimensionless Nrphys Richardson Number9 CT dimensionless 1 Surface Drag coefficient for T and Q10 CU dimensionless 1 Surface Drag coefficient for U and V11 ET m2/sec Nrphys Diffusivity coefficient for T and Q12 EU m2/sec Nrphys Diffusivity coefficient for U and V13 TURBU m/sec/day Nrphys U-Momentum Changes due to Turbulence14 TURBV m/sec/day Nrphys V-Momentum Changes due to Turbulence15 TURBT deg/day Nrphys Temperature Changes due to Turbulence16 TURBQ g/kg/day Nrphys Specific Humidity Changes due to Turbulence17 MOISTT deg/day Nrphys Temperature Changes due to Moist Processes18 MOISTQ g/kg/day Nrphys Specific Humidity Changes due to Moist Processes19 RADLW deg/day Nrphys Net Longwave heating rate for each level20 RADSW deg/day Nrphys Net Shortwave heating rate for each level21 PREACC mm/day 1 Total Precipitation22 PRECON mm/day 1 Convective Precipitation23 TUFLUX Newton/m2 Nrphys Turbulent Flux of U-Momentum24 TVFLUX Newton/m2 Nrphys Turbulent Flux of V-Momentum25 TTFLUX Watts/m2 Nrphys Turbulent Flux of Sensible Heat



26 TQFLUX Watts/m2 Nrphys Turbulent Flux of Latent Heat27 CN dimensionless 1 Neutral Drag Coefficient28 WINDS m/sec 1 Surface Wind Speed29 DTSRF deg 1 Air/Surface virtual temperature difference30 TG deg 1 Ground temperature31 TS deg 1 Surface air temperature (Adiabatic from lowest

model layer)32 DTG deg 1 Ground temperature adjustment33 QG g/kg 1 Ground specific humidity34 QS g/kg 1 Saturation surface specific humidity35 TGRLW deg 1 Instantaneous ground temperature used as input

to the Longwave radiation subroutine36 ST4 Watts/m2 1 Upward Longwave flux at the ground (σT 4)37 OLR Watts/m2 1 Net upward Longwave flux at the top of the model38 OLRCLR Watts/m2 1 Net upward clearsky Longwave flux at the top of

the model39 LWGCLR Watts/m2 1 Net upward clearsky Longwave flux at the ground40 LWCLR deg/day Nrphys Net clearsky Longwave heating rate for each level41 TLW deg Nrphys Instantaneous temperature used as input to the

Longwave radiation subroutine42 SHLW g/g Nrphys Instantaneous specific humidity used as input to

the Longwave radiation subroutine43 OZLW g/g Nrphys Instantaneous ozone used as input to the Long-

wave radiation subroutine44 CLMOLW 0 − 1 Nrphys Maximum overlap cloud fraction used in the Long-

wave radiation subroutine45 CLDTOT 0 − 1 Nrphys Total cloud fraction used in the Longwave and

Shortwave radiation subroutines46 LWGDOWN Watts/m2 1 Downwelling Longwave radiation at the ground47 GWDT deg/day Nrphys Temperature tendency due to Gravity Wave Drag48 RADSWT Watts/m2 1 Incident Shortwave radiation at the top of the at-

mosphere49 TAUCLD per100mb Nrphys Counted Cloud Optical Depth (non-dimensional)

per 100 mb50 TAUCLDC Number Nrphys Cloud Optical Depth Counter



51 CLDLOW 0 − 1 Nrphys Low-Level ( 1000-700 hPa) Cloud Fraction (0-1)52 EVAP mm/day 1 Surface evaporation53 DPDT hPa/day 1 Surface Pressure tendency54 UAVE m/sec Nrphys Average U-Wind55 VAVE m/sec Nrphys Average V-Wind56 TAVE deg Nrphys Average Temperature57 QAVE g/kg Nrphys Average Specific Humidity58 OMEGA hPa/day Nrphys Vertical Velocity59 DUDT m/sec/day Nrphys Total U-Wind tendency60 DVDT m/sec/day Nrphys Total V-Wind tendency61 DTDT deg/day Nrphys Total Temperature tendency62 DQDT g/kg/day Nrphys Total Specific Humidity tendency63 VORT 10−4/sec Nrphys Relative Vorticity64 NOT USED65 DTLS deg/day Nrphys Temperature tendency due to Stratiform Cloud

Formation66 DQLS g/kg/day Nrphys Specific Humidity tendency due to Stratiform

Cloud Formation67 USTAR m/sec 1 Surface USTAR wind68 Z0 m 1 Surface roughness69 FRQTRB 0 − 1 Nrphys-1 Frequency of Turbulence70 PBL mb 1 Planetary Boundary Layer depth71 SWCLR deg/day Nrphys Net clearsky Shortwave heating rate for each level72 OSR Watts/m2 1 Net downward Shortwave flux at the top of the

model73 OSRCLR Watts/m2 1 Net downward clearsky Shortwave flux at the top

of the model74 CLDMAS kg/m2 Nrphys Convective cloud mass flux75 UAVE m/sec Nrphys Time-averaged u−Wind



76 VAVE m/sec Nrphys Time-averaged v −Wind77 TAVE deg Nrphys Time-averaged Temperature78 QAVE g/g Nrphys Time-averaged Specific Humidity79 RFT deg/day Nrphys Temperature tendency due Rayleigh Friction80 PS mb 1 Surface Pressure81 QQAVE (m/sec)2 Nrphys Time-averaged TurbulentKineticEnergy82 SWGCLR Watts/m2 1 Net downward clearsky Shortwave flux at the

ground83 PAVE mb 1 Time-averaged Surface Pressure84 SDIAG1 1 User-Defined Surface Diagnostic-185 SDIAG2 1 User-Defined Surface Diagnostic-286 UDIAG1 Nrphys User-Defined Upper-Air Diagnostic-187 UDIAG2 Nrphys User-Defined Upper-Air Diagnostic-288 DIABU m/sec/day Nrphys Total Diabatic forcing on u−Wind89 DIABV m/sec/day Nrphys Total Diabatic forcing on v −Wind90 DIABT deg/day Nrphys Total Diabatic forcing on Temperature91 DIABQ g/kg/day Nrphys Total Diabatic forcing on Specific Humidity92 RFU m/sec/day Nrphys U-Wind tendency due to Rayleigh Friction93 RFV m/sec/day Nrphys V-Wind tendency due to Rayleigh Friction94 GWDU m/sec/day Nrphys U-Wind tendency due to Gravity Wave Drag95 GWDU m/sec/day Nrphys V-Wind tendency due to Gravity Wave Drag96 GWDUS N/m2 1 U-Wind Gravity Wave Drag Stress at Surface97 GWDVS N/m2 1 V-Wind Gravity Wave Drag Stress at Surface98 GWDUT N/m2 1 U-Wind Gravity Wave Drag Stress at Top99 GWDVT N/m2 1 V-Wind Gravity Wave Drag Stress at Top100 LZRAD mg/kg Nrphys Estimated Cloud Liquid Water used in Radiation



101 SLP mb 1 Time-averaged Sea-level Pressure102 NOT USED103 NOT USED104 NOT USED105 NOT USED106 CLDFRC 0 − 1 1 Total Cloud Fraction107 TPW gm/cm2 1 Precipitable water108 U2M m/sec 1 U-Wind at 2 meters109 V2M m/sec 1 V-Wind at 2 meters110 T2M deg 1 Temperature at 2 meters111 Q2M g/kg 1 Specific Humidity at 2 meters112 U10M m/sec 1 U-Wind at 10 meters113 V10M m/sec 1 V-Wind at 10 meters114 T10M deg 1 Temperature at 10 meters115 Q10M g/kg 1 Specific Humidity at 10 meters116 DTRAIN kg/m2 Nrphys Detrainment Cloud Mass Flux117 QFILL g/kg/day Nrphys Filling of negative specific humidity118 NOT USED119 NOT USED120 SHAPU m/sec/day Nrphys U-Wind tendency due to Shapiro Filter121 SHAPV m/sec/day Nrphys V-Wind tendency due to Shapiro Filter122 SHAPT deg/day Nrphys Temperature tendency due Shapiro Filter123 SHAPQ g/kg/day Nrphys Specific Humidity tendency due to Shapiro Filter124 SDIAG3 1 User-Defined Surface Diagnostic-3125 SDIAG4 1 User-Defined Surface Diagnostic-4



126 SDIAG5 1 User-Defined Surface Diagnostic-5127 SDIAG6 1 User-Defined Surface Diagnostic-6128 SDIAG7 1 User-Defined Surface Diagnostic-7129 SDIAG8 1 User-Defined Surface Diagnostic-8130 SDIAG9 1 User-Defined Surface Diagnostic-9131 SDIAG10 1 User-Defined Surface Diagnostic-1-132 UDIAG3 Nrphys User-Defined Multi-Level Diagnostic-3133 UDIAG4 Nrphys User-Defined Multi-Level Diagnostic-4134 UDIAG5 Nrphys User-Defined Multi-Level Diagnostic-5135 UDIAG6 Nrphys User-Defined Multi-Level Diagnostic-6136 UDIAG7 Nrphys User-Defined Multi-Level Diagnostic-7137 UDIAG8 Nrphys User-Defined Multi-Level Diagnostic-8138 UDIAG9 Nrphys User-Defined Multi-Level Diagnostic-9139 UDIAG10 Nrphys User-Defined Multi-Level Diagnostic-10



238 ETAN (hPa,m) 1 Perturbation of Surface (pressure, height)239 ETANSQ (hPa2,m2) 1 Square of Perturbation of Surface (pressure,

height)240 THETA degK Nr Potential Temperature241 SALT g/kg Nr Salt (or Water Vapor Mixing Ratio)242 UVEL m/sec Nr U-Velocity243 VVEL m/sec Nr V-Velocity244 WVEL m/sec Nr Vertical-Velocity245 THETASQ deg2 Nr Square of Potential Temperature246 SALTSQ g2/kg2 Nr Square of Salt (or Water Vapor Mixing Ratio)247 UVELSQ m2/sec2 Nr Square of U-Velocity248 VVELSQ m2/sec2 Nr Square of V-Velocity249 WVELSQ m2/sec2 Nr Square of Vertical-Velocity250 UVELVVEL m2/sec2 Nr Meridional Transport of Zonal Momentum



251 UVELMASS m/sec Nr Zonal Mass-Weighted Component of Velocity252 VVELMASS m/sec Nr Meridional Mass-Weighted Component of Velo253 WVELMASS m/sec Nr Vertical Mass-Weighted Component of Veloci254 UTHMASS m− deg/sec Nr Zonal Mass-Weight Transp of Pot Temp255 VTHMASS m− deg/sec Nr Meridional Mass-Weight Transp of Pot Temp256 WTHMASS m− deg/sec Nr Vertical Mass-Weight Transp of Pot Temp257 USLTMASS m− kg/sec− kg Nr Zonal Mass-Weight Transp of Salt (or W.Vap

Rat.)258 VSLTMASS m− kg/sec− kg Nr Meridional Mass-Weight Transp of Salt (or W.

Mix Rat.)259 WSLTMASS m− kg/sec− kg Nr Vertical Mass-Weight Transp of Salt (or W.

Mix Rat.)260 UVELTH m− deg/sec Nr Zonal Transp of Pot Temp261 VVELTH m− deg/sec Nr Meridional Transp of Pot Temp262 WVELTH m− deg/sec Nr Vertical Transp of Pot Temp263 UVELSLT m− kg/sec− kg Nr Zonal Transp of Salt (or W.Vap Mix Rat.)264 VVELSLT m− kg/sec− kg Nr Meridional Transp of Salt (or W.Vap Mix Ra265 WVELSLT m− kg/sec− kg Nr Vertical Transp of Salt (or W.Vap Mix Rat.)266 UTRAC1 m− kg/sec− kg Nr Zonal Transp of Tracer 1267 VTRAC1 m− kg/sec− kg Nr Meridional Transp of Tracer 1268 WTRAC1 m− kg/sec− kg Nr Vertical Transp of Tracer 1269 UTRAC2 m− kg/sec− kg Nr Zonal Transp of Tracer 2270 VTRAC2 m− kg/sec− kg Nr Meridional Transp of Tracer 2271 WTRAC2 m− kg/sec− kg Nr Vertical Transp of Tracer 2272 UTRAC3 m− kg/sec− kg Nr Zonal Transp of Tracer 3273 VTRAC3 m− kg/sec− kg Nr Meridional Transp of Tracer 3274 WTRAC3 m− kg/sec− kg Nr Vertical Transp of Tracer 3275 WSLTMASS m− kg/sec− kg Nr Vertical Mass-Weight Transp of Salt (or W.

Mix Rat.)



275 UTRAC4 m− kg/sec− kg Nr Zonal Transp of Tracer 4276 VTRAC4 m− kg/sec− kg Nr Meridional Transp of Tracer 4277 WTRAC4 m− kg/sec− kg Nr Vertical Transp of Tracer 4278 UTRAC5 m− kg/sec− kg Nr Zonal Transp of Tracer 5279 VTRAC5 m− kg/sec− kg Nr Meridional Transp of Tracer 5280 WTRAC5 m− kg/sec− kg Nr Vertical Transp of Tracer 5281 TRAC1 kg/kg Nr Mass-Weight Tracer 1282 TRAC2 kg/kg Nr Mass-Weight Tracer 2283 TRAC3 kg/kg Nr Mass-Weight Tracer 3284 TRAC4 kg/kg Nr Mass-Weight Tracer 4285 TRAC5 kg/kg Nr Mass-Weight Tracer 5286 DICBIOA mol/m3/s Nr Biological Productivity287 DICCARB moleq/m3/s Nr Carbonate chg-biol prod and remin288 DICTFLX mol/m3/s 1 Tendency of DIC due to air-sea exch289 DICOFLX mol/m3/s 1 Tendency of O2 due to air-sea exch290 DICCFLX mol/m2/s 1 Flux of CO2 - air-sea exch291 DICPCO2 atm 1 Partial Pressure of CO2292 DICPHAV dimensionless 1 Average pH293 DTCONV deg/sec Nr Temp Change due to Convection294 DQCONV g/kg/sec Nr Specific Humidity Change due to Convection295 RELHUM percent Nr Relative Humidity296 PRECLS g/m2/sec 1 Large Scale Precipitation297 ENPREC J/g 1 Energy of Precipitation (snow, rain Temp)298 VISCA4 m4/sec 1 Biharmonic Viscosity Coefficient299 VISCAH m2/sec 1 Harmonic Viscosity Coefficient300 DRHODR kg/m3/r − unit Nr Stratification: d.Sigma/dr



301 DETADT2 r − unit2/s2 1 Square of Eta (Surf.P,SSH) Tendency


6.19.4.2 Diagnostic Description

In this section we list and describe the diagnostic quantities available within theGCM. The diagnostics are listed in the order that they appear in the DiagnosticMenu, Section 6.19.4.1. In all cases, each diagnostic as currently archived onthe output datasets is time-averaged over its diagnostic output frequency:

DIAGNOSTIC =1

TTOT

t=TTOT∑t=1

diag(t)

where TTOT = NQDIAGΔt , NQDIAG is the output frequency of the diagnostic,

and Δt is the timestep over which the diagnostic is updated.1) UFLUX Surface Zonal Wind Stress on the Atmosphere (Newton/m2)

The zonal wind stress is the turbulent flux of zonal momentum from thesurface. See section 3.3 for a description of the surface layer parameterization.

UFLUX = −ρCDWsu where : CD = C2u

where ρ = the atmospheric density at the surface, CD is the surface drag coef-ficient, Cu is the dimensionless surface exchange coefficient for momentum (seediagnostic number 10), Ws is the magnitude of the surface layer wind, and u isthe zonal wind in the lowest model layer.

2) VFLUX Surface Meridional Wind Stress on the Atmosphere(Newton/m2)

The meridional wind stress is the turbulent flux of meridional momentumfrom the surface. See section 3.3 for a description of the surface layer parame-terization.

VFLUX = −ρCDWsv where : CD = C2u

where ρ = the atmospheric density at the surface, CD is the surface drag coef-ficient, Cu is the dimensionless surface exchange coefficient for momentum (seediagnostic number 10), Ws is the magnitude of the surface layer wind, and v isthe meridional wind in the lowest model layer.

3) HFLUX Surface Flux of Sensible Heat (Watts/m2)The turbulent flux of sensible heat from the surface to the atmosphere is a

function of the gradient of virtual potential temperature and the eddy exchangecoefficient:

HFLUX = P κρcpCHWs(θsurface − θNrphys) where : CH = CuCt

where ρ = the atmospheric density at the surface, cp is the specific heat of air,CH is the dimensionless surface heat transfer coefficient, Ws is the magnitudeof the surface layer wind, Cu is the dimensionless surface exchange coefficientfor momentum (see diagnostic number 10), Ct is the dimensionless surface ex-change coefficient for heat and moisture (see diagnostic number 9), and θ is the


potential temperature at the surface and at the bottom model level.

4) EFLUX Surface Flux of Latent Heat (Watts/m2)The turbulent flux of latent heat from the surface to the atmosphere is a

function of the gradient of moisture, the potential evapotranspiration fractionand the eddy exchange coefficient:

EFLUX = ρβLCHWs(qsurface − qNrphys) where : CH = CuCt

where ρ = the atmospheric density at the surface, β is the fraction of the poten-tial evapotranspiration actually evaporated, L is the latent heat of evaporation,CH is the dimensionless surface heat transfer coefficient, Ws is the magnitudeof the surface layer wind, Cu is the dimensionless surface exchange coefficientfor momentum (see diagnostic number 10), Ct is the dimensionless surface ex-change coefficient for heat and moisture (see diagnostic number 9), and qsurface

and qNrphys are the specific humidity at the surface and at the bottom modellevel, respectively.

5) QICE Heat Conduction Through Sea Ice (Watts/m2)Over sea ice there is an additional source of energy at the surface due to the

heat conduction from the relatively warm ocean through the sea ice. The heatconduction through sea ice represents an additional energy source term for theground temperature equation.

QICE =Cti

Hi(Ti − Tg)

where Cti is the thermal conductivity of ice, Hi is the ice thickness, assumedto be 3 m where sea ice is present, Ti is 273 degrees Kelvin, and Tg is thetemperature of the sea ice.

NOTE: QICE is not available through model version 5.3, but is available insubsequent versions.

6) RADLWG Net upward Longwave Flux at the surface (Watts/m2)

RADLWG = FNetLW,Nrphys+1

= F ↑LW,Nrphys+1 − F ↓LW,Nrphys+1

where Nrphys+1 indicates the lowest model edge-level, or p = psurf . F ↑LW isthe upward Longwave flux and F ↓LW is the downward Longwave flux.

7) RADSWG Net downard shortwave Flux at the surface (Watts/m2)

RADSWG = FNetSW,Nrphys+1

= F ↓SW,Nrphys+1 − F ↑SW,Nrphys+1


where Nrphys+1 indicates the lowest model edge-level, or p = psurf . F ↓SW isthe downward Shortwave flux and F ↑SW is the upward Shortwave flux.

8) RI Richardson Number (dimensionless)The non-dimensional stability indicator is the ratio of the buoyancy to the shear:

RI =gθv

∂θv

∂z

(∂u∂z )2 + (∂v

∂z )2=

cp∂θv

∂z∂P κ

∂z

(∂u∂z )2 + (∂v

∂z )2

where we used the hydrostatic equation:

∂Φ∂P κ

= cpθv

Negative values indicate unstable buoyancy AND shear, small positive values(< 0.4) indicate dominantly unstable shear, and large positive values indicatedominantly stable stratification.

9) CT Surface Exchange Coefficient for Temperature and Moisture(dimensionless)The surface exchange coefficient is obtained from the similarity functions forthe stability dependant flux profile relationships:

CT = − (w′θ′)u∗Δθ

= − (w′q′)u∗Δq

=k

(ψh + ψg)

where ψh is the surface layer non-dimensional temperature change and ψg is theviscous sublayer non-dimensional temperature or moisture change:

ψh =∫ ζ

ζ0

φh

ζdζ and ψg =

0.55(Pr2/3 − 0.2)ν1/2

(h0u∗ − h0refu∗ref

)1/2

and: h0 = 30z0 with a maximum value over land of 0.01φh is the similarity function of ζ, which expresses the stability dependance ofthe temperature and moisture gradients, specified differently for stable and un-stable layers according to Helfand and Schubert, 1993. k is the Von Karmanconstant, ζ is the non-dimensional stability parameter, Pr is the Prandtl num-ber for air, ν is the molecular viscosity, z0 is the surface roughness length, u∗is the surface stress velocity (see diagnostic number 67), and the subscript refrefers to a reference value.

10) CU Surface Exchange Coefficient for Momentum (dimensionless)The surface exchange coefficient is obtained from the similarity functions forthe stability dependant flux profile relationships:

CU =u∗Ws

=k

ψm


where ψm is the surface layer non-dimensional wind shear:

ψm =∫ ζ

ζ0

φm

ζdζ

φm is the similarity function of ζ, which expresses the stability dependance ofthe temperature and moisture gradients, specified differently for stable and un-stable layers according to Helfand and Schubert, 1993. k is the Von Karmanconstant, ζ is the non-dimensional stability parameter, u∗ is the surface stressvelocity (see diagnostic number 67), and Ws is the magnitude of the surfacelayer wind.

11) ET Diffusivity Coefficient for Temperature and Moisture (m2/sec)In the level 2.5 version of the Mellor-Yamada (1974) hierarchy, the turbulentheat or moisture flux for the atmosphere above the surface layer can be expressedas a turbulent diffusion coefficient Kh times the negative of the gradient of po-tential temperature or moisture. In the Helfand and Labraga (1988) adaptationof this closure, Kh takes the form:

ET = Kh = − (w′θ′v)∂θv

∂z

=

{q � SH(GM , GH) for decaying turbulenceq2

qe� SH(GMe , GHe) for growing turbulence

where q is the turbulent velocity, or√

2 ∗ turbulent kinetic energy, qe is theturbulence velocity derived from the more simple level 2.0 model, which de-scribes equilibrium turbulence, � is the master length scale related to the layerdepth, SH is a function of GH and GM , the dimensionless buoyancy and windshear parameters, respectively, or a function of GHe and GMe , the equilibriumdimensionless buoyancy and wind shear parameters. Both GH and GM , andtheir equilibrium values GHe and GMe , are functions of the Richardson number.For the detailed equations and derivations of the modified level 2.5 closurescheme, see Helfand and Labraga, 1988.In the surface layer, ET is the exchange coefficient for heat and moisture, inunits of m/sec, given by:

ETNrphys = Ct ∗ u∗ = CHWs

where Ct is the dimensionless exchange coefficient for heat and moisture fromthe surface layer similarity functions (see diagnostic number 9), u∗ is the surfacefriction velocity (see diagnostic number 67), CH is the heat transfer coefficient,and Ws is the magnitude of the surface layer wind.

12) EU Diffusivity Coefficient for Momentum (m2/sec)In the level 2.5 version of the Mellor-Yamada (1974) hierarchy, the turbulentheat momentum flux for the atmosphere above the surface layer can be expressedas a turbulent diffusion coefficient Km times the negative of the gradient of theu-wind. In the Helfand and Labraga (1988) adaptation of this closure, Km takes


the form:

EU = Km = − (u′w′)∂U∂z

=

{q � SM (GM , GH) for decaying turbulenceq2

qe� SM (GMe , GHe) for growing turbulence

where q is the turbulent velocity, or√

2 ∗ turbulent kinetic energy, qe is theturbulence velocity derived from the more simple level 2.0 model, which de-scribes equilibrium turbulence, � is the master length scale related to the layerdepth, SM is a function of GH and GM , the dimensionless buoyancy and windshear parameters, respectively, or a function of GHe and GMe , the equilibriumdimensionless buoyancy and wind shear parameters. Both GH and GM , andtheir equilibrium values GHe and GMe , are functions of the Richardson number.For the detailed equations and derivations of the modified level 2.5 closurescheme, see Helfand and Labraga, 1988.In the surface layer, EU is the exchange coefficient for momentum, in units ofm/sec, given by:

EUNrphys = Cu ∗ u∗ = CDWs

where Cu is the dimensionless exchange coefficient for momentum from thesurface layer similarity functions (see diagnostic number 10), u∗ is the surfacefriction velocity (see diagnostic number 67), CD is the surface drag coefficient,and Ws is the magnitude of the surface layer wind.

13) TURBU Zonal U-Momentum changes due to Turbulence (m/sec/day)

The tendency of U-Momentum due to turbulence is written:

TURBU =∂u

∂t turb=

∂

∂z(−u′w′) =

∂

∂z(Km

∂u

∂z)

The Helfand and Labraga level 2.5 scheme models the turbulent flux of u-momentum in terms of Km, and the equation has the form of a diffusion equa-tion.14) TURBV Meridional V-Momentum changes due to Turbulence(m/sec/day)The tendency of V-Momentum due to turbulence is written:

TURBV =∂v

∂t turb=

∂

∂z(−v′w′) =

∂

∂z(Km

∂v

∂z)

The Helfand and Labraga level 2.5 scheme models the turbulent flux of v-momentum in terms of Km, and the equation has the form of a diffusion equa-tion.

15) TURBT Temperature changes due to Turbulence (deg/day)The tendency of temperature due to turbulence is written:

TURBT =∂T

∂t= P κ ∂θ

∂t turb= P κ ∂

∂z(−w′θ′) = P κ ∂

∂z(Kh

∂θv

∂z)


The Helfand and Labraga level 2.5 scheme models the turbulent flux of temper-ature in terms of Kh, and the equation has the form of a diffusion equation.

16) TURBQ Specific Humidity changes due to Turbulence (g/kg/day)The tendency of specific humidity due to turbulence is written:

TURBQ =∂q

∂t turb=

∂

∂z(−w′q′) =

∂

∂z(Kh

∂q

∂z)

The Helfand and Labraga level 2.5 scheme models the turbulent flux of temper-ature in terms of Kh, and the equation has the form of a diffusion equation.

17) MOISTT Temperature Changes Due to Moist Processes (deg/day)

MOISTT =∂T

∂t

∣∣∣∣c

+∂T

∂t

∣∣∣∣ls

where:∂T

∂t

∣∣∣∣c

= R∑

i

(αmB

cpΓs

)i

and∂T

∂t

∣∣∣∣ls

=L

cp(q∗ − q)

andΓs = gη

∂s

∂p

The subscript c refers to convective processes, while the subscript ls refers tolarge scale precipitation processes, or supersaturation rain. The summationrefers to contributions from each cloud type called by RAS. The dry static en-ergy is given as s, the convective cloud base mass flux is given as mB, andthe cloud entrainment is given as η, which are explicitly defined in Section ??,the description of the convective parameterization. The fractional adjustment,or relaxation parameter, for each cloud type is given as α, while R is the rainre-evaporation adjustment.

18) MOISTQ Specific Humidity Changes Due to Moist Processes(g/kg/day)

MOISTQ =∂q

∂t

∣∣∣∣c

+∂q

∂t

∣∣∣∣ls

where:

∂q

∂t

∣∣∣∣c

= R∑

i

(αmB

L(Γh − Γs)

)i

and∂q

∂t

∣∣∣∣ls

= (q∗ − q)

andΓs = gη

∂s

∂pand Γh = gη

∂h

∂p

The subscript c refers to convective processes, while the subscript ls refers tolarge scale precipitation processes, or supersaturation rain. The summationrefers to contributions from each cloud type called by RAS. The dry static en-ergy is given as s, the moist static energy is given as h, the convective cloud


base mass flux is given as mB, and the cloud entrainment is given as η, whichare explicitly defined in Section ??, the description of the convective parame-terization. The fractional adjustment, or relaxation parameter, for each cloudtype is given as α, while R is the rain re-evaporation adjustment.

19) RADLW Heating Rate due to Longwave Radiation (deg/day)The net longwave heating rate is calculated as the vertical divergence of thenet terrestrial radiative fluxes. Both the clear-sky and cloudy-sky longwavefluxes are computed within the longwave routine. The subroutine calculates theclear-sky flux, F clearsky

LW , first. For a given cloud fraction, the clear line-of-sightprobability C(p, p′) is computed from the current level pressure p to the modeltop pressure, p′ = ptop, and the model surface pressure, p′ = psurf , for theupward and downward radiative fluxes. (see Section ??). The cloudy-sky fluxis then obtained as:

FLW = C(p, p′) · F clearskyLW ,

Finally, the net longwave heating rate is calculated as the vertical divergence ofthe net terrestrial radiative fluxes:

∂ρcpT

∂t= − ∂

∂zFNET

LW ,

orRADLW =

g

cpπ

∂

∂σFNET

LW .

where g is the accelation due to gravity, cp is the heat capacity of air at constantpressure, and

FNETLW = F ↑LW − F ↓LW

20) RADSW Heating Rate due to Shortwave Radiation (deg/day)The net Shortwave heating rate is calculated as the vertical divergence of thenet solar radiative fluxes. The clear-sky and cloudy-sky shortwave fluxes arecalculated separately. For the clear-sky case, the shortwave fluxes and heat-ing rates are computed with both CLMO (maximum overlap cloud fraction)and CLRO (random overlap cloud fraction) set to zero (see Section ??). Theshortwave routine is then called a second time, for the cloudy-sky case, withthe true time-averaged cloud fractions CLMO and CLRO being used. In allcases, a normalized incident shortwave flux is used as input at the top of theatmosphere.The heating rate due to Shortwave Radiation under cloudy skies is defined as:

∂ρcpT

∂t= − ∂

∂zF (cloudy)NET

SW · RADSWT,

orRADSW =

g

cpπ

∂

∂σF (cloudy)NET

SW · RADSWT.


where g is the accelation due to gravity, cp is the heat capacity of air at constantpressure, RADSWT is the true incident shortwave radiation at the top of theatmosphere (See Diagnostic #48), and

F (cloudy)NetSW = F (cloudy)↑SW − F (cloudy)↓SW

21) PREACC Total (Large-scale + Convective) Accumulated Precip-ition (mm/day)For a change in specific humidity due to moist processes, Δqmoist, the verticalintegral or total precipitable amount is given by:

PREACC =∫ top

surf

ρΔqmoistdz = −∫ top

surf

Δqmoistdp

g=

1g

∫ 1

0

Δqmoistdp

A precipitation rate is defined as the vertically integrated moisture adjustmentper Moist Processes time step, scaled to mm/day.

22) PRECON Convective Precipition (mm/day)For a change in specific humidity due to sub-grid scale cumulus convective pro-cesses, Δqcum, the vertical integral or total precipitable amount is given by:

PRECON =∫ top

surf

ρΔqcumdz = −∫ top

surf

Δqcumdp

g=

1g

∫ 1

0

Δqcumdp

A precipitation rate is defined as the vertically integrated moisture adjustmentper Moist Processes time step, scaled to mm/day.

23) TUFLUX Turbulent Flux of U-Momentum (Newton/m2)The turbulent flux of u-momentum is calculated for diagnostic purposes onlyfrom the eddy coefficient for momentum:

TUFLUX = ρ(u′w′) = ρ(−Km∂U

∂z)

where ρ is the air density, and Km is the eddy coefficient.

24) TVFLUX Turbulent Flux of V-Momentum (Newton/m2)The turbulent flux of v-momentum is calculated for diagnostic purposes onlyfrom the eddy coefficient for momentum:

TVFLUX = ρ(v′w′) = ρ(−Km∂V

∂z)

where ρ is the air density, and Km is the eddy coefficient.


25) TTFLUX Turbulent Flux of Sensible Heat (Watts/m2)The turbulent flux of sensible heat is calculated for diagnostic purposes onlyfrom the eddy coefficient for heat and moisture:

TTFLUX = cpρPκ(w′θ′) = cpρP

κ(−Kh∂θv

∂z)

where ρ is the air density, and Kh is the eddy coefficient.

26) TQFLUX Turbulent Flux of Latent Heat (Watts/m2)The turbulent flux of latent heat is calculated for diagnostic purposes onlyfrom the eddy coefficient for heat and moisture:

TQFLUX = Lρ(w′q′) = Lρ(−Kh∂q

∂z)

where ρ is the air density, and Kh is the eddy coefficient.

27) CN Neutral Drag Coefficient (dimensionless)The drag coefficient for momentum obtained by assuming a neutrally stablesurface layer:

CN =k

ln( hz0

)

where k is the Von Karman constant, h is the height of the surface layer, andz0 is the surface roughness.NOTE: CN is not available through model version 5.3, but is available in sub-sequent versions.

28) WINDS Surface Wind Speed (meter/sec)The surface wind speed is calculated for the last internal turbulence time step:

WINDS =√u2

Nrphys + v2Nrphys

where the subscript Nrphys refers to the lowest model level.

29) DTSRF Air/Surface Virtual Temperature Difference (deg K)The air/surface virtual temperature difference measures the stability of the sur-face layer:

DTSRF = (θvNrphys+1 − θvNrphys)P κsurf

where

θvNrphys+1 =Tg

P κsurf

(1+.609qNrphys+1) and qNrphys+1 = qNrphys+β(q∗(Tg, Ps)−qNrphys)

β is the surface potential evapotranspiration coefficient (β = 1 over oceans),q∗(Tg, Ps) is the saturation specific humidity at the ground temperature and sur-face pressure, level Nrphys refers to the lowest model level and level Nrphys+1refers to the surface.


30) TG Ground Temperature (deg K)The ground temperature equation is solved as part of the turbulence packageusing a backward implicit time differencing scheme:

TG is obtained from : Cg∂Tg

∂t= Rsw −Rlw +Qice −H − LE

where Rsw is the net surface downward shortwave radiative flux, Rlw is the netsurface upward longwave radiative flux, Qice is the heat conduction throughsea ice, H is the upward sensible heat flux, LE is the upward latent heat flux,and Cg is the total heat capacity of the ground. Cg is obtained by solving aheat diffusion equation for the penetration of the diurnal cycle into the ground(Blackadar, 1977), and is given by:

Cg =

√λCs

2ω=

√(0.386 + 0.536W + 0.15W 2)2x10−3

86400.2π

.

Here, the thermal conductivity, λ, is equal to 2x10−3 lysec

cmoK , the angular ve-

locity of the earth, ω, is written as 86400 sec/day divided by 2π radians/day,and the expression for Cs, the heat capacity per unit volume at the surface, isa function of the ground wetness, W .

31) TS Surface Temperature (deg K)The surface temperature estimate is made by assuming that the model’s lowestlayer is well-mixed, and therefore that θ is constant in that layer. The surfacetemperature is therefore:

TS = θNrphysPκsurf

32) DTG Surface Temperature Adjustment (deg K)The change in surface temperature from one turbulence time step to the next,solved using the Ground Temperature Equation (see diagnostic number 30) iscalculated:

DTG = Tgn − Tg

n−1

where superscript n refers to the new, updated time level, and the superscriptn− 1 refers to the value at the previous turbulence time level.

33) QG Ground Specific Humidity (g/kg)The ground specific humidity is obtained by interpolating between the specifichumidity at the lowest model level and the specific humidity of a saturatedground. The interpolation is performed using the potential evapotranspirationfunction:

QG = qNrphys+1 = qNrphys + β(q∗(Tg, Ps) − qNrphys)


where β is the surface potential evapotranspiration coefficient (β = 1 overoceans), and q∗(Tg, Ps) is the saturation specific humidity at the ground tem-perature and surface pressure.

34) QS Saturation Surface Specific Humidity (g/kg)The surface saturation specific humidity is the saturation specific humidity atthe ground temprature and surface pressure:

QS = q∗(Tg, Ps)

35) TGRLW Instantaneous ground temperature used as input to theLongwave radiation subroutine (deg)

TGRLW = Tg(λ, φ, n)

where Tg is the model ground temperature at the current time step n.

36) ST4 Upward Longwave flux at the surface (Watts/m2)

ST4 = σT 4

where σ is the Stefan-Boltzmann constant and T is the temperature.

37) OLR Net upward Longwave flux at p = ptop (Watts/m2)

OLR = FNETLW,top

where top indicates the top of the first model layer. In the GCM, ptop = 0.0 mb.

38) OLRCLR Net upward clearsky Longwave flux at p = ptop (Watts/m2)

OLRCLR = F (clearsky)NETLW,top

where top indicates the top of the first model layer. In the GCM, ptop = 0.0 mb.

39) LWGCLR Net upward clearsky Longwave flux at the surface(Watts/m2)

LWGCLR = F (clearsky)NetLW,Nrphys+1

= F (clearsky)↑LW,Nrphys+1 − F (clearsky)↓LW,Nrphys+1

where Nrphys+1 indicates the lowest model edge-level, or p = psurf . F (clearsky)↑LW

is the upward clearsky Longwave flux and the F (clearsky)↓LW is the downwardclearsky Longwave flux.


40) LWCLR Heating Rate due to Clearsky Longwave Radiation (deg/day)

The net longwave heating rate is calculated as the vertical divergence of thenet terrestrial radiative fluxes. Both the clear-sky and cloudy-sky longwavefluxes are computed within the longwave routine. The subroutine calculates theclear-sky flux, F clearsky

LW , first. For a given cloud fraction, the clear line-of-sightprobability C(p, p′) is computed from the current level pressure p to the modeltop pressure, p′ = ptop, and the model surface pressure, p′ = psurf , for theupward and downward radiative fluxes. (see Section ??). The cloudy-sky fluxis then obtained as:

FLW = C(p, p′) · F clearskyLW ,

Thus, LWCLR is defined as the net longwave heating rate due to the verticaldivergence of the clear-sky longwave radiative flux:

∂ρcpT

∂t clearsky= − ∂

∂zF (clearsky)NET

LW ,

orLWCLR =

g

cpπ

∂

∂σF (clearsky)NET

LW .

where g is the accelation due to gravity, cp is the heat capacity of air at constantpressure, and

F (clearsky)NetLW = F (clearsky)↑LW − F (clearsky)↓LW

41) TLW Instantaneous temperature used as input to the Longwaveradiation subroutine (deg)

TLW = T (λ, φ, level, n)

where T is the model temperature at the current time step n.

42) SHLW Instantaneous specific humidity used as input to the Long-wave radiation subroutine (kg/kg)

SHLW = q(λ, φ, level, n)

where q is the model specific humidity at the current time step n.

43) OZLW Instantaneous ozone used as input to the Longwave radi-ation subroutine (kg/kg)

OZLW = OZ(λ, φ, level, n)

where OZ is the interpolated ozone data set from the climatological monthlymean zonally averaged ozone data set.


44) CLMOLW Maximum Overlap cloud fraction used in LW Radia-tion (0 − 1)CLMOLW is the time-averaged maximum overlap cloud fraction that has beenfilled by the Relaxed Arakawa/Schubert Convection scheme and will be used inthe Longwave Radiation algorithm. These are convective clouds whose radiativecharacteristics are assumed to be correlated in the vertical. For a completedescription of cloud/radiative interactions, see Section ??.

CLMOLW = CLMORAS,LW (λ, φ, level)

45) CLDTOT Total cloud fraction used in LW and SW Radiation(0 − 1)

CLDTOT is the time-averaged total cloud fraction that has been filled bythe Relaxed Arakawa/Schubert and Large-scale Convection schemes and willbe used in the Longwave and Shortwave Radiation packages. For a completedescription of cloud/radiative interactions, see Section ??.

CLDTOT = FRAS + FLS

where FRAS is the time-averaged cloud fraction due to sub-grid scale convec-tion, and FLS is the time-averaged cloud fraction due to precipitating and non-precipitating large-scale moist processes.

46) CLMOSW Maximum Overlap cloud fraction used in SW Radia-tion (0 − 1)CLMOSW is the time-averaged maximum overlap cloud fraction that has beenfilled by the Relaxed Arakawa/Schubert Convection scheme and will be used inthe Shortwave Radiation algorithm. These are convective clouds whose radiativecharacteristics are assumed to be correlated in the vertical. For a completedescription of cloud/radiative interactions, see Section ??.

CLMOSW = CLMORAS,SW (λ, φ, level)

47) CLROSW Random Overlap cloud fraction used in SW Radiation(0 − 1)CLROSW is the time-averaged random overlap cloud fraction that has beenfilled by the Relaxed Arakawa/Schubert and Large-scale Convection schemesand will be used in the Shortwave Radiation algorithm. These are convectiveand large-scale clouds whose radiative characteristics are not assumed to becorrelated in the vertical. For a complete description of cloud/radiative inter-actions, see Section ??.

CLROSW = CLRORAS,LargeScale,SW (λ, φ, level)


48) RADSWT Incident Shortwave radiation at the top of the atmo-sphere (Watts/m2)

RADSWT =S0

R2a

· cosφz

where S0, is the extra-terrestial solar contant, Ra is the earth-sun distance inAstronomical Units, and cosφz is the cosine of the zenith angle. It should benoted that RADSWT, as well as OSR and OSRCLR, are calculated at thetop of the atmosphere (p=0 mb). However, the OLR and OLRCLR diagnos-tics are currently calculated at p = ptop (0.0 mb for the GCM).

49) EVAP Surface Evaporation (mm/day)The surface evaporation is a function of the gradient of moisture, the potentialevapotranspiration fraction and the eddy exchange coefficient:

EVAP = ρβKh(qsurface − qNrphys)

where ρ = the atmospheric density at the surface, β is the fraction of the po-tential evapotranspiration actually evaporated (β = 1 over oceans), Kh is theturbulent eddy exchange coefficient for heat and moisture at the surface inm/sec and qsurface and qNrphys are the specific humidity at the surface (seediagnostic number 34) and at the bottom model level, respectively.

50) DUDT Total Zonal U-Wind Tendency (m/sec/day)DUDT is the total time-tendency of the Zonal U-Wind due to Hydrodynamic,Diabatic, and Analysis forcing.

DUDT =∂u

∂t Dynamics+∂u

∂tMoist+∂u

∂t Turbulence+∂u

∂t Analysis

51) DVDT Total Zonal V-Wind Tendency (m/sec/day)DVDT is the total time-tendency of the Meridional V-Wind due to Hydrody-namic, Diabatic, and Analysis forcing.

DVDT =∂v

∂tDynamics+∂v

∂tMoist+∂v

∂t Turbulence+∂v

∂t Analysis

52) DTDT Total Temperature Tendency (deg/day)DTDT is the total time-tendency of Temperature due to Hydrodynamic, Dia-batic, and Analysis forcing.

DTDT =∂T

∂t Dynamics+∂T

∂t MoistProcesses+∂T

∂t ShortwaveRadiation

+∂T

∂t LongwaveRadiation+∂T

∂t Turbulence+∂T

∂t Analysis


53) DQDT Total Specific Humidity Tendency (g/kg/day)DQDT is the total time-tendency of Specific Humidity due to Hydrodynamic,Diabatic, and Analysis forcing.

DQDT =∂q

∂tDynamics+∂q

∂tMoistProcesses+∂q

∂t Turbulence+∂q

∂t Analysis

54) USTAR Surface-Stress Velocity (m/sec)The surface stress velocity, or the friction velocity, is the wind speed at thesurface layer top impeded by the surface drag:

USTAR = CuWs where : Cu =k

ψm

Cu is the non-dimensional surface drag coefficient (see diagnostic number 10),and Ws is the surface wind speed (see diagnostic number 28).55) Z0 Surface Roughness Length (m)Over the land surface, the surface roughness length is interpolated to the localtime from the monthly mean data of Dorman and Sellers (1989). Over theocean, the roughness length is a function of the surface-stress velocity, u∗.

Z0 = c1u3∗ + c2u

2∗ + c3u∗ + c4 +

c5u∗

where the constants are chosen to interpolate between the reciprocal relationof Kondo(1975) for weak winds, and the piecewise linear relation of Large andPond(1981) for moderate to large winds.

56) FRQTRB Frequency of Turbulence (0 − 1)The fraction of time when turbulence is present is defined as the fraction of timewhen the turbulent kinetic energy exceeds some minimum value, defined hereto be 0.005 m2/sec2. When this criterion is met, a counter is incremented. Thefraction over the averaging interval is reported.

57) PBL Planetary Boundary Layer Depth (mb)The depth of the PBL is defined by the turbulence parameterization to be thedepth at which the turbulent kinetic energy reduces to ten percent of its surfacevalue.

PBL = PPBL − Psurface

where PPBL is the pressure in mb at which the turbulent kinetic energy reachesone tenth of its surface value, and Ps is the surface pressure.


58) SWCLR Clear sky Heating Rate due to Shortwave Radiation(deg/day)The net Shortwave heating rate is calculated as the vertical divergence of thenet solar radiative fluxes. The clear-sky and cloudy-sky shortwave fluxes arecalculated separately. For the clear-sky case, the shortwave fluxes and heat-ing rates are computed with both CLMO (maximum overlap cloud fraction)and CLRO (random overlap cloud fraction) set to zero (see Section ??). Theshortwave routine is then called a second time, for the cloudy-sky case, withthe true time-averaged cloud fractions CLMO and CLRO being used. In allcases, a normalized incident shortwave flux is used as input at the top of theatmosphere.The heating rate due to Shortwave Radiation under clear skies is defined as:

∂ρcpT

∂t= − ∂

∂zF (clear)NET

SW · RADSWT,

orSWCLR =

g

cp

∂

∂pF (clear)NET

SW · RADSWT.

where g is the accelation due to gravity, cp is the heat capacity of air at constantpressure, RADSWT is the true incident shortwave radiation at the top of theatmosphere (See Diagnostic #48), and

F (clear)NetSW = F (clear)↑SW − F (clear)↓SW

59) OSR Net upward Shortwave flux at the top of the model (Watts/m2)

OSR = FNETSW,top

where top indicates the top of the first model layer used in the shortwave radi-ation routine. In the GCM, pSWtop = 0 mb.

60) OSRCLR Net upward clearsky Shortwave flux at the top of themodel (Watts/m2)

OSRCLR = F (clearsky)NETSW,top

where top indicates the top of the first model layer used in the shortwave radi-ation routine. In the GCM, pSWtop = 0 mb.

61) CLDMAS Convective Cloud Mass Flux (kg/m2)The amount of cloud mass moved per RAS timestep from all convective cloudsis written:

CLDMAS = ηmB

where η is the entrainment, normalized by the cloud base mass flux, and mB

is the cloud base mass flux. mB and η are defined explicitly in Section ??, the


description of the convective parameterization.

62) UAVE Time-Averaged Zonal U-Wind (m/sec)The diagnostic UAVE is simply the time-averaged Zonal U-Wind over theNUAVE output frequency. This is contrasted to the instantaneous Zonal U-Wind which is archived on the Prognostic Output data stream.

UAVE = u(λ, φ, level, t)

Note, UAVE is computed and stored on the staggered C-grid.

63) VAVE Time-Averaged Meridional V-Wind (m/sec)The diagnostic VAVE is simply the time-averaged Meridional V-Wind over theNVAVE output frequency. This is contrasted to the instantaneous MeridionalV-Wind which is archived on the Prognostic Output data stream.

VAVE = v(λ, φ, level, t)

Note, VAVE is computed and stored on the staggered C-grid.

64) TAVE Time-Averaged Temperature (Kelvin)The diagnostic TAVE is simply the time-averaged Temperature over the NTAVEoutput frequency. This is contrasted to the instantaneous Temperature whichis archived on the Prognostic Output data stream.

TAVE = T (λ, φ, level, t)

65) QAVE Time-Averaged Specific Humidity (g/kg)The diagnostic QAVE is simply the time-averaged Specific Humidity over theNQAVE output frequency. This is contrasted to the instantaneous SpecificHumidity which is archived on the Prognostic Output data stream.

QAVE = q(λ, φ, level, t)

66) PAVE Time-Averaged Surface Pressure - PTOP (mb)The diagnostic PAVE is simply the time-averaged Surface Pressure - PTOPover the NPAVE output frequency. This is contrasted to the instantaneousSurface Pressure - PTOP which is archived on the Prognostic Output datastream.

PAVE = π(λ, φ, level, t)= ps(λ, φ, level, t)− pT


67) QQAVE Time-Averaged Turbulent Kinetic Energy (m/sec)2

The diagnostic QQAVE is simply the time-averaged prognostic Turbulent Ki-netic Energy produced by the GCM Turbulence parameterization over the NQQAVEoutput frequency. This is contrasted to the instantaneous Turbulent Kinetic En-ergy which is archived on the Prognostic Output data stream.

QQAVE = qq(λ, φ, level, t)

Note, QQAVE is computed and stored at the “mass-point” locations on thestaggered C-grid.

68) SWGCLR Net downward clearsky Shortwave flux at the surface(Watts/m2)

SWGCLR = F (clearsky)NetSW,Nrphys+1

= F (clearsky)↓SW,Nrphys+1 − F (clearsky)↑SW,Nrphys+1

where Nrphys+1 indicates the lowest model edge-level, or p = psurf . F (clearsky)SW ↓

is the downward clearsky Shortwave flux and F (clearsky)↑SW is the upwardclearsky Shortwave flux.

69) SDIAG1 User-Defined Surface Diagnostic-1The GCM provides Users with a built-in mechanism for archiving user-defineddiagnostics. The generic diagnostic array QDIAG located in COMMON /DIAG/,and the associated diagnostic counters and pointers located in COMMON /DI-AGP/, must be accessable in order to use the user-defined diagnostics (see Sec-tion 6.19.3). A convenient method for incorporating all necessary COMMONfiles is to include the GCM vstate.com file in the routine which employs theuser-defined diagnostics.In addition to enabling the user-defined diagnostic (ie., CALL SETDIAG(84)),the User must fill the QDIAG array with the desired quantity within the User’sapplication program or within modified GCM subroutines, as well as incrementthe diagnostic counter at the time when the diagnostic is updated. The QDIAGlocation index for SDIAG1 and its corresponding counter is automatically de-fined as ISDIAG1 and NSDIAG1, respectively, after the diagnostic has beenenabled. The syntax for its use is given by

do j=1,jmdo i=1,imqdiag(i,j,ISDIAG1) = qdiag(i,j,ISDIAG1) + ...enddoenddo

NSDIAG1 = NSDIAG1 + 1


The diagnostics defined in this manner will automatically be archived by theoutput routines.

70) SDIAG2 User-Defined Surface Diagnostic-2The GCM provides Users with a built-in mechanism for archiving user-defineddiagnostics. For a complete description refer to Diagnostic #84. The syntax forusing the surface SDIAG2 diagnostic is given by

do j=1,jmdo i=1,imqdiag(i,j,ISDIAG2) = qdiag(i,j,ISDIAG2) + ...enddoenddo

NSDIAG2 = NSDIAG2 + 1

The diagnostics defined in this manner will automatically be archived by theoutput routines.

71) UDIAG1 User-Defined Upper-Air Diagnostic-1The GCM provides Users with a built-in mechanism for archiving user-defineddiagnostics. For a complete description refer to Diagnostic #84. The syntax forusing the upper-air UDIAG1 diagnostic is given by

do L=1,Nrphysdo j=1,jmdo i=1,imqdiag(i,j,IUDIAG1+L-1) = qdiag(i,j,IUDIAG1+L-1) + ...enddoenddoenddo

NUDIAG1 = NUDIAG1 + 1

The diagnostics defined in this manner will automatically be archived by theoutput programs.

72) UDIAG2 User-Defined Upper-Air Diagnostic-2The GCM provides Users with a built-in mechanism for archiving user-defineddiagnostics. For a complete description refer to Diagnostic #84. The syntax forusing the upper-air UDIAG2 diagnostic is given by

do L=1,Nrphysdo j=1,jmdo i=1,imqdiag(i,j,IUDIAG2+L-1) = qdiag(i,j,IUDIAG2+L-1) + ...enddoenddo


enddo

NUDIAG2 = NUDIAG2 + 1

The diagnostics defined in this manner will automatically be archived by theoutput programs.

73) DIABU Total Diabatic Zonal U-Wind Tendency (m/sec/day)DIABU is the total time-tendency of the Zonal U-Wind due to Diabatic pro-cesses and the Analysis forcing.

DIABU =∂u

∂tMoist+∂u

∂t Turbulence+∂u

∂t Analysis

74) DIABV Total Diabatic Meridional V-Wind Tendency (m/sec/day)DIABV is the total time-tendency of the Meridional V-Wind due to Diabaticprocesses and the Analysis forcing.

DIABV =∂v

∂tMoist+∂v

∂t Turbulence+∂v

∂t Analysis

75) DIABT Total Diabatic Temperature Tendency (deg/day)DIABT is the total time-tendency of Temperature due to Diabatic processesand the Analysis forcing.

DIABT =∂T



+∂T


∂t Turbulence+∂T

∂t Analysis

If we define the time-tendency of Temperature due to Diabatic processes as

∂T

∂t Diabatic=

∂T



+∂T


∂t Turbulence

then, since there are no surface pressure changes due to Diabatic processes, wemay write

∂T

∂t Diabatic=pκ

π

∂πθ

∂t Diabatic

where θ = T/pκ. Thus, DIABT may be written as

DIABT =pκ

π

(∂πθ

∂t Diabatic+∂πθ

∂t Analysis

)


76) DIABQ Total Diabatic Specific Humidity Tendency (g/kg/day)DIABQ is the total time-tendency of Specific Humidity due to Diabatic pro-cesses and the Analysis forcing.

DIABQ =∂q


∂t Turbulence+∂q

∂t Analysis

If we define the time-tendency of Specific Humidity due to Diabatic processesas

∂q

∂tDiabatic=∂q


∂t Turbulence

then, since there are no surface pressure changes due to Diabatic processes, wemay write

∂q

∂tDiabatic=

1π

∂πq

∂t Diabatic

Thus, DIABQ may be written as

DIABQ =1π

(∂πq

∂t Diabatic+∂πq

∂t Analysis

)

77) VINTUQ Vertically Integrated Moisture Flux (m/sec · g/kg)The vertically integrated moisture flux due to the zonal u-wind is obtained byintegrating uq over the depth of the atmosphere at each model timestep, anddividing by the total mass of the column.

VINTUQ =

∫ top

surf uqρdz∫ top

surf ρdz

Using ρδz = − δpg = − 1

g δp, we have

VINTUQ =∫ 1

0

uqdp

78) VINTVQ Vertically Integrated Moisture Flux (m/sec · g/kg)The vertically integrated moisture flux due to the meridional v-wind is obtainedby integrating vq over the depth of the atmosphere at each model timestep, anddividing by the total mass of the column.

VINTVQ =

∫ top

surfvqρdz∫ top

surfρdz


Using ρδz = − δpg = − 1

g δp, we have

VINTVQ =∫ 1

0

vqdp

79) VINTUT Vertically Integrated Heat Flux (m/sec · deg)The vertically integrated heat flux due to the zonal u-wind is obtained by in-tegrating uT over the depth of the atmosphere at each model timestep, anddividing by the total mass of the column.

VINTUT =

∫ top

surfuTρdz∫ top

surfρdz

Or,

VINTUT =∫ 1

0

uTdp

80) VINTVT Vertically Integrated Heat Flux (m/sec · deg)The vertically integrated heat flux due to the meridional v-wind is obtained byintegrating vT over the depth of the atmosphere at each model timestep, anddividing by the total mass of the column.

VINTVT =

∫ top

surfvTρdz∫ top

surfρdz

Using ρδz = − δpg , we have

VINTVT =∫ 1

0

vTdp

81 CLDFRC Total 2-Dimensional Cloud Fracton (0 − 1)If we define the time-averaged random and maximum overlapped cloudiness

as CLRO and CLMO respectively, then the probability of clear sky associatedwith random overlapped clouds at any level is (1-CLRO) while the probability ofclear sky associated with maximum overlapped clouds at any level is (1-CLMO).The total clear sky probability is given by (1-CLRO)*(1-CLMO), thus the totalcloud fraction at each level may be obtained by 1-(1-CLRO)*(1-CLMO).

At any given level, we may define the clear line-of-site probability by appro-priately accounting for the maximum and random overlap cloudiness. The clearline-of-site probability is defined to be equal to the product of the clear line-of-site probabilities associated with random and maximum overlap cloudiness. The


clear line-of-site probability C(p, p′) associated with maximum overlap clouds,from the current pressure p to the model top pressure, p′ = ptop, or the modelsurface pressure, p′ = psurf , is simply 1.0 minus the largest maximum overlapcloud value along the line-of-site, ie.

1 −MAXp′p (CLMOp)

Thus, even in the time-averaged sense it is assumed that the maximumoverlap clouds are correlated in the vertical. The clear line-of-site probabilityassociated with random overlap clouds is defined to be the product of the clearsky probabilities at each level along the line-of-site, ie.

p′∏p

(1 − CLROp)

The total cloud fraction at a given level associated with a line- of-site calcu-lation is given by

1 −(1 −MAXp′

p [CLMOp]) p′∏

p

(1 − CLROp)

The 2-dimensional net cloud fraction as seen from the top of the atmosphere isgiven by

CLDFRC = 1 −(1 −MAXNrphys

l=l1[CLMOl]

)Nrphys∏l=l1

(1 − CLROl)

For a complete description of cloud/radiative interactions, see Section ??.82) QINT Total Precipitable Water (gm/cm2)The Total Precipitable Water is defined as the vertical integral of the specifichumidity, given by:

QINT =∫ top

surf

ρqdz

=π

g

∫ 1

0

qdp

where we have used the hydrostatic relation ρδz = − δpg .

83) U2M Zonal U-Wind at 2 Meter Depth (m/sec)The u-wind at the 2-meter depth is determined from the similarity theory:

U2M =u∗kψm2m

usl

Ws=ψm2m

ψmsl

usl

where ψm(2m) is the non-dimensional wind shear at two meters, and the sub-script sl refers to the height of the top of the surface layer. If the roughness


height is above two meters, U2M is undefined.

84) V2M Meridional V-Wind at 2 Meter Depth (m/sec)The v-wind at the 2-meter depth is a determined from the similarity theory:

V2M =u∗kψm2m

vsl

Ws=ψm2m

ψmsl

vsl

where ψm(2m) is the non-dimensional wind shear at two meters, and the sub-script sl refers to the height of the top of the surface layer. If the roughnessheight is above two meters, V2M is undefined.

85) T2M Temperature at 2 Meter Depth (deg K)The temperature at the 2-meter depth is a determined from the similarity the-ory:

T2M = P κ(θ∗k

(ψh2m + ψg) + θsurf ) = P κ(θsurf +ψh2m + ψg

ψhsl+ ψg

(θsl − θsurf ))

where:

θ∗ = − (w′θ′)u∗

where ψh(2m) is the non-dimensional temperature gradient at two meters, ψg

is the non-dimensional temperature gradient in the viscous sublayer, and thesubscript sl refers to the height of the top of the surface layer. If the roughnessheight is above two meters, T2M is undefined.

86) Q2M Specific Humidity at 2 Meter Depth (g/kg)The specific humidity at the 2-meter depth is determined from the similaritytheory:

Q2M = P κ(q∗k

(ψh2m + ψg) + qsurf ) = P κ(qsurf +ψh2m + ψg

ψhsl+ ψg

(qsl − qsurf ))

where:

q∗ = − (w′q′)u∗


is the non-dimensional temperature gradient in the viscous sublayer, and thesubscript sl refers to the height of the top of the surface layer. If the roughnessheight is above two meters, Q2M is undefined.

87) U10M Zonal U-Wind at 10 Meter Depth (m/sec)The u-wind at the 10-meter depth is an interpolation between the surface windand the model lowest level wind using the ratio of the non-dimensional windshear at the two levels:

U10M =u∗kψm10m

usl

Ws=ψm10m

ψmsl

usl


where ψm(10m) is the non-dimensional wind shear at ten meters, and the sub-script sl refers to the height of the top of the surface layer.

88) V10M Meridional V-Wind at 10 Meter Depth (m/sec)The v-wind at the 10-meter depth is an interpolation between the surface windand the model lowest level wind using the ratio of the non-dimensional windshear at the two levels:

V10M =u∗kψm10m

vsl

Ws=ψm10m

ψmsl

vsl

where ψm(10m) is the non-dimensional wind shear at ten meters, and the sub-script sl refers to the height of the top of the surface layer.

89) T10M Temperature at 10 Meter Depth (deg K)The temperature at the 10-meter depth is an interpolation between the surfacepotential temperature and the model lowest level potential temperature usingthe ratio of the non-dimensional temperature gradient at the two levels:

T10M = P κ(θ∗k

(ψh10m + ψg) + θsurf ) = P κ(θsurf +ψh10m + ψg

ψhsl+ ψg

(θsl − θsurf ))

where:

θ∗ = − (w′θ′)u∗


is the non-dimensional temperature gradient in the viscous sublayer, and thesubscript sl refers to the height of the top of the surface layer.

90) Q10M Specific Humidity at 10 Meter Depth (g/kg)The specific humidity at the 10-meter depth is an interpolation between thesurface specific humidity and the model lowest level specific humidity using theratio of the non-dimensional temperature gradient at the two levels:

Q10M = P κ(q∗k

(ψh10m + ψg) + qsurf ) = P κ(qsurf +ψh10m + ψg

ψhsl+ ψg

(qsl − qsurf ))

where:

q∗ = − (w′q′)u∗


is the non-dimensional temperature gradient in the viscous sublayer, and thesubscript sl refers to the height of the top of the surface layer.

91) DTRAIN Cloud Detrainment Mass Flux (kg/m2)The amount of cloud mass moved per RAS timestep at the cloud detrainment

level is written:DTRAIN = ηrDmB


where rD is the detrainment level, mB is the cloud base mass flux, and η is theentrainment, defined in Section ??.

92) QFILL Filling of negative Specific Humidity (g/kg/day)Due to computational errors associated with the numerical scheme used forthe advection of moisture, negative values of specific humidity may be gener-ated. The specific humidity is checked for negative values after every dynamicstimestep. If negative values have been produced, a filling algorithm is invokedwhich redistributes moisture from below. Diagnostic QFILL is equal to the netfilling needed to eliminate negative specific humidity, scaled to a per-day rate:

QFILL = qn+1final − qn+1

initial

whereqn+1 = (πq)n+1/πn+1

6.19.5 Dos and Donts

6.19.6 Diagnostics Reference

6.20. RW BASIC BINARY I/O UTILITIES 405

6.20 RW Basic binary I/O utilities

The rw package provides a very rudimentary binary I/O capability for quicklywriting single record direct-access Fortran binary files. It is primarily used forwriting diagnostic output.

6.20.1 Introduction

Package rw is an interface to the more general mdsio package. The rw pack-age can be used to write or read direct-access Fortran binary files for two-dimensional XY and three-dimensional XYZ arrays. The arrays are assumedto have been declared according to the standard MITgcm two-dimensional orthree-dimensional floating point array type:

C Example of declaring a standard two dimensional "long"C floating point type array (the _RL macro is usuallyC mapped to 64-bit floats in most configurations)

_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)

Each call to an rw read or write routine will read (or write) to the firstrecord of a file. To write direct access Fortran files with multiple records usethe package mdsio (see section 6.16). To write self-describing files that containembedded information describing the variables being written and the spatialand temporal locations of those variables use the package mnc (see section 6.15)which produces netCDF1 [44] based output.

1http://www.unidata.ucar.edu/packages/netcdf


6.21 FFT Filtering Code


6.22. GCHEM PACKAGE 407

6.22 GCHEM Package

6.22.1 Introduction

This package has been developed as interface to the PTRACERS package. Thepurpose is to provide a structure where various (any) tracer experiments canbe added to the code. For instance there are placeholders for routines to readin parameters needed for any tracer experiments, a routine to read in extrafields required for the tracer code, routines for either external forcing or internalinteractions between tracers and routines for additional diagnostics relating tothe tracers. Note that the gchem package itself is only a means to call thesubroutines used by specific biogeochemical experiments, and does not ”do”anything on its own.

There are two examples: cfc which looks at 2 tracers with a simple externalforcing and dic with 5 tracers whose tendency terms are related to one another.We will discuss these here only as how they provide examples to use this package.

6.22.2 Key subroutines and parameters

FRAMEWORKGCHEM OPTIONS.h includes the compiler options to be used in any experi-ment. For instance #define ALLOW CFC allows the CFC code to be run. Animportant compiler option is #define GCHEM SEPARATE FORCING whichdetermined how and when the tracer forcing is applied (see discussion on Forc-ing below). There are further runtime parameters set in data.gchem and keptin common block GCHEM.h. These runtime options include:• tIter0 which is the integer timestep when the tracer experiment is initialized.If nIter0 = tIter0 then the tracers are initialized to zero or from initial files.If nIter0 > tIter0 then tracers (and previous timestep tendency terms) areread in from a the ptracers pickup file. Note that tracers of zeros will be carriedaround if nIter0 < tIter0.• nsubtime is the integer number of extra timesteps required by the tracerexperiment. This will give a timestep of deltaTtracer/nsubtime for the de-pendencies between tracers. The default is one.• File names - these are several filenames than can be read in for external fieldsneeded in the tracer forcing - for instance wind speed is needed in both DICand CFC packages to calculate the air-sea exchange of gases. Not all file nameswill be used for every tracer experiment.

INITIALIZATIONThe values set at runtime in data.gchem are read in using gchem readparms.Fwhich is called from packages readparms.F. This will include any external forc-ing files that will be needed by the tracer experiment.

There are two routine used to initialize parameters and fields needed by theexperiment packages. These are gchem init fixed.F which is called from pack-ages init fixed.F, and gchem init vari.F called from packages init variable.F. The


first should be used to call a subroutine specific to the tracer experiment whichsets fixed parameters, the second should call a subroutine specific to the tracerexperiment which sets (or initializes) time fields that will vary with time.

LOADING FIELDSExternal forcing fields used by the tracer experiment are read in by a subroutine(specific to the tracer experiment) called from gchem fields load.F. This latteris called from forward step.F.

FORCINGTracer fields are advected-and-diffused by the ptracer package. Additionalchanges (e.g. surface forcing or interactions between tracers) to these fieldsare taken care of by the gchem interface. For tracers that are essentially passive(e.g. CFC’s) but may have some surface boundary conditions this can easilybe done within the regular tracer timestep. In this case gchem calc tendency.Fis called from forward step.F, where the reactive (as opposed to the advec-tive diffusive) tendencies are computed. These tendencies, stored on the 3Dfield gchemTendency, are added to the passive tracer tendencies gPtr ingchem add tendency.F, which is called from ptracers forcing.F. For tracers withmore complicated dependencies on each other, and especially tracers which re-quire a smaller timestep than deltaTtracer, it will be easier to use gchem forcing sep.Fwhich is called from forward step.F. There is a compiler option set in GCHEM OPTIONS.hthat determines which method is used: #define GCHEM SEPARATE FORCINGdoes the latter where tracers are forced separately from the advection-diffusioncode, and #undef GCHEM SEPARATE FORCING includes the forcing in theregular timestepping.

DIAGNOSTICSThis package also also used the passive tracer routine ptracers monitor.F whichprints out tracer statistics as often as the model dynamic statistic diagnostics(dynsys) are written (or as prescribed by the runtime flag PTRACERS monitorFreq,set in data.ptracers). There is also a placeholder for any tracer experiment spe-cific diagnostics to be calculated and printed to files. This is done in gchem diags.F.For instance the time average CO2 air-sea fluxes, and sea surface pH (amongothers) are written out by dic biotic diags.F which is called from gchem diags.F.

6.22.3 Do’s and Don’ts

The pkg ptracer is required with use with this pkg. Also, as usual, the runtimeflag useGCHEM must be set to .TRUE. in data.pkg. By itself, gchem pkgwill read in data.gchem and will write out gchem diagnostics. It requires tracerexperiment specific calls to do anything else (for instance the calls to dic andcfc pkgs).

6.22. GCHEM PACKAGE 409

6.22.4 Reference Material

Chapter 7

Diagnostics and tools

There are numerous tools for pre-processing data, converting model output andanalysis written in Matlab, fortran (f77 and f90) and perl. As yet they re-main undocumented although many are self-documenting (Matlab routines have”help” written into them).

Here we’ll summarize what is available but this is an ever growing resourceso this may not cover everything that is out there:

7.1 Utilities supplied with the model

We supply some basic scripts with the model to facilitate conversion or readingof data into analysis software.

7.1.1 utils/scripts

In the directory utils/scripts you will find joinds and joinmds: these are perlscripts used from joining the multi-part files created by MITgcm. Use joinmdsalways. You will only need joinds if you are working with output older thantwo years (prior to c23).

7.1.2 utils/matlab

In the directory utils/matlab you will find several Matlab scripts (.m or dot-emfiles). The priniciple script is rdmds.m used for reading the multi-part modeloutput files in to matlab. Place the scripts in your matlab path or change thepath appropriately, then at the matlab prompt type:

>> help rdmds

to get help on how to use rdmds.Another useful script scans the terminal output file for ”monitor” informa-

tion.

411

412 CHAPTER 7. DIAGNOSTICS AND TOOLS

Most other scripts are for working in the curvilinear coordinate systemswhich as yet are unpublished and undocumented.

7.2 Pre-processing software

There is a suite of pre-processing software for intepolating bathymetry andforcing data, written by Adcroft and Biastoch. At some point, these will bemade available for download. If you are in need of such software, contact oneof them.

Chapter 8

Interface with ECCO

This chapter describes the interface between MITgcm and the large scale opti-mization schemes employed in the ECCO project. The interface provides theso-called “ECCO environment”, which is maintained alongside the kernel MIT-gcm code.

8.1 The ECCO state estimation cost function

DRAFT!!!

The current ECCO state estimation covers an nY ears = 11 year model trajec-tory. A variety of data sets enter a least squares cost function, in addition topenalty terms which constrain deviations of control variables beyound their apriori errors.

8.1.1 Sea surface height from TOPEX/Poseidon and ERS-1/2 altimetry

Altimetric SSH contributions from T/P and ERS-1/2 are four-fold:

1. an nY ears time mean SSH misfit between model and T/P

2. daily SSH anomaly misfits between T/P and model

3. daily SSH anomaly misfits between ERS-1/2 and model

4. daily absolute SSH misfit between T/P and model, weighted by the fullgeoid error covariance.

8.1.1.1 Input fields

413

414 CHAPTER 8. INTERFACE WITH ECCO

field file name deccription unit

psbar psbarfile daily model mean SSH fields [m]tpmean topexmeanfile nY ears T/P mean [cm]tpobs topexfile daily T/P SSH anomalies [cm]erspobs ersfile daily ERS-1/2 SSH anomalies [cm]wp geoid errfile diagonal of geoid error covariance [m]wtp, wers ssh errfile rms of SSH anomalies [cm]

8.1.1.2 nYears time mean SSH misfit

1. Compute nY ears model mean spatial distribution

psmean(i, j) =1

nDaysRec

nDaysRec∑i=1

psbar(i, j) (8.1)

2. Compute global offset between nY ears model and T/P mean:

offset = tpmean − psmean

=1

normaliz.

∑i,j

{tpmean(i, j) − psmean(i, j)} · cosphi(i, j) · tpmeanmask(i, j)

(8.2)

3. Misfits are computed w.r.t. global offset.First spatial distribution:

cost ssh mean(i, j) =1wp2

{[ psmean(i, j) − psmean ] − [ tpmean(i, j) − tpmean

] }2=

1wp2

{ psmean(i, j) − tpmean(i, j) + offset }2

(8.3)

Finally, sum over all spatial entries:

cost ssh mean =∑i,j

cost ssh mean(i, j) (8.4)

8.1.1.3 Misfit of daily SSH anomalies

Computation is same for T/P and ERS-1/2. Here we write out computation forT/P.

8.1. THE ECCO STATE ESTIMATION COST FUNCTION DRAFT!!! 415

1. Compute difference in anomalies:

cost ssh anom(i, j, t) =1

wtp2{ [ psbar(i, j, t) − psmean(i, j) ] − [ tpobs(i, j, t) ] }2

(8.5)

where t denotes time (day) index, and where it is assumed that nY earsmean T/P spatial distribution tpmean(i, j) has already been removed fromdata tpobs(i, j)!

2. Sum over all spatial points and all times

cost ssh anom =∑

t

∑i,j

cost ssh anom(i, j, t) (8.6)

8.1.1.4 Flow chart

cost_ssh||- < compute nYears model mean >||- < read nYears T/P mean >| CALL COST_READTOPEXMEAN||- < compute global T/P vs. model offset >||- < compute cost_hmean >| CALL COST_SSH_MEAN||- < ... >

8.1.1.5 Weights and notes

• All data are currently masked to zero where less than 13 depth levels,mimicing no contribution for depth less than 1000m.

• cosphi term in weights is set to 1.

• bad T/P and ERS-1/2 values are flagged ≤ −9990.

• T/P and ERS-1/2 data ≤ 1. exp−8 cm are flagged as bad values

• wp is read from geoid errfile and 1/wp2 is pre-computed in ecco cost weights


wp for SSH mean misfit1/wp2 is pre-computed in ecco cost weights;

wp is read from geoid errfile;

wtp and wers for SSH anomaly misfit1/wtp2, 1/wers2 are pre-computed in ecco cost weights;

• wtp, wers are read from single ssh errfile

• both are converted to meters and halvedwtp −→ wtp · 0.01 · 0.5

• ERS error is set to T/P error + 5cmwers = wtp + 0.5cm

8.1.1.6 Cost diagnostics

• Map out cost ssh mean(i, j)

• Map out cost ssh anom(i, j, t) averaged over 1 month, i.e.

1monthly entries

monthly∑t

cost ssh anom(i, j, t)

• sum over daily entries and plot daily average as function of time. i.e.

1daily entries

∑i,j

cost ssh anom(i, j, t)

8.1.2 Hydrographic constraints

Observation of temperature and salinity from various sources are used to con-strain the model. These are:

1. CTD obs. for T , S from various WOCE sections

2. XBT obs. for T

3. Sea surface temperature (SST) and salinity (SSS) from Reynolds et al.(???)

4. T , S from ARGO floats

5. T , S from fields from Levitus (???)

8.1.2.1 Input fields


field file name deccription unit

tbar tbarfile monthly model mean pot. temperature [◦C]sbar sbarfile monthly model mean salinity [ppt]tdat tdatfile monthly mean Levitus pot. temperature [◦C]sdat sdatfile monthly mean Levitus salinity [ppt]ctdtobs ctdtfile monthly WOCE CTD pot. temperature [◦C]ctdsobs ctdsfile monthly WOCE CTD salinity [ppt]xbtobs xbtfile monthly XBT in-situ(!) temperature [◦C]sstdat sstdatfile monthly Reynolds pot. SST [◦C]sssdat sssdatfile monthly Reynolds SSS [ppt]argotobs argotfile monthly ARGO in-situ(!) temperature [◦C]argosobs argosfile monthly ARGO salinity [ppt]wti, wsi data errfile vert. stdev. profile for T , Swtvar temperrfile spatially varying stdev. [◦C]wsvar salterrfile spatially varying stdev. [ppt]

8.1.2.2 XBT data

cost xbt t(i, j, k) =

[fac · ratio

wti2 + wtvar2

nMonsRec∑τ=1

{Tbar(τ) − T 2θ[xbtobs(τ)]}2

](i, j, k)

(8.7)

8.1.2.3 WOCE CTD data

cost ctd t(i, j, k) =

[fac · ratio

wti2 + wtvar2

nMonsRec∑τ=1

{Tbar(τ) − ctdTobs(τ)}2

](i, j, k)

cost ctd s(i, j, k) =

[fac · ratio

wsi2 + wsvar2

nMonsRec∑τ=1

{Sbar(τ) − ctdSobs(τ)}2

](i, j, k)

(8.8)

8.1.2.4 ARGO float data

cost argo t(i, j, k) =

[fac · ratio

wti2 + wvar2

nMonsRec∑τ=1

{Tbar(τ) − T 2θ[argoTobs(τ)]}2

](i, j, k)

cost argo s(i, j, k) =

[fac · ratio

wsi2 + wsvar2

nMonsRec∑τ=1

{Sbar(τ) − argoSobs(τ)}2

](i, j, k)

(8.9)


8.1.2.5 Reynolds sea surface T, S data

cost sst(i, j) =

[wsst

nMonsRec∑τ=1

{Tbar(τ) − sstDat(τ)}2

](i, j)

cost sss(i, j) =

[wsss

nMonsRec∑τ=1

{Sbar(τ) − sssDat(τ)}2

](i, j)

(8.10)

8.1.2.6 Levitus montly T, S climatological data

Model vs. data misfits are taken from nY earsmonthly model means vs. Levitusmonthly data. The description below is for potential temperature. Procedurefor salinity is fully analogous. Spatial indices (i, j, k) are omitted throughout.

1. Compute nY ears monthly model means for each month imon:

Tbar(imon) =1

nY ears

nY ears∑iyear=1

Tbar(iyear, imon)

2. Compute misfit:

cost theta(i, j, k) =

[fac · ratiowti2

12∑imon=1

{Tbar(imon) − Tdat(imon)

}2](i, j, k)

8.1.2.7 Weights and notes

• T 2θ is an operator mapping in-situ to potential temperatures

• Latitudinal weight not used:

cosphi(i, j) = 1

• fac = cosphi ·mask• Spatially constant weights:

1. Read standard deviation vertical profiles for T , Sdata errfile −→ wti(k), wsi(k)data errfile −→ ratio = 0.25 =

(12

)22. Take inverse squares:

wtheta(k) =ratio

wti(k)2

wsalt(k) =ratio

wsi(k)2

• Spatially varying weights:


1. Read standard deviation fieldstemperrfile −→ wtvar(i, j, k)salterrfile −→ wsvar(i, j, k)

2. Weights are combination of spatially constant and varying parts:

wtheta2(i, j, k) =ratio

wti(k)2 + wtvar(i, j, k)2

wsalt2(i, j, k) =ratio

wsi(k)2 + wsvar(i, j, k)2

• Sea surface T , S weights:

– SST: wsst = fac · wtheta(1): horizontally constant

– SSS: wsss = fac · wsalt2(i, j, 1): horizontally varying

(Why this difference? I don’t know.)

8.1.2.8 Diagnostics

• Map out wtheta2(i, j, k), wsalt2(i, j, k).


8.2 The external forcing package exf

8.2.1 Summary

c Field definitions, units, and sign conventions:

c ===============================================

c

c ustress :: Zonal surface wind stress in N/m^2

c > 0 for increase in uVel, which is west to

c east for cartesian and spherical polar grids

c Typical range: -0.5 < ustress < 0.5

c Southwest C-grid U point

c Input field

c

c vstress :: Meridional surface wind stress in N/m^2

c > 0 for increase in vVel, which is south to

c north for cartesian and spherical polar grids

c Typical range: -0.5 < vstress < 0.5

c Southwest C-grid V point

c Input field

c

c hflux :: Net upward surface heat flux excluding shortwave in W/m^2

c hflux = latent + sensible + lwflux

c > 0 for decrease in theta (ocean cooling)

c Typical range: -250 < hflux < 600

c Southwest C-grid tracer point

c Input field

c

c sflux :: Net upward freshwater flux in m/s

c sflux = evap - precip - runoff

c > 0 for increase in salt (ocean salinity)

c Typical range: -1e-7 < sflux < 1e-7


c Input field

c

c swflux :: Net upward shortwave radiation in W/m^2

c swflux = - ( swdown - ice and snow absorption - reflected )


c Typical range: -350 < swflux < 0


c Input field

c

c uwind :: Surface (10-m) zonal wind velocity in m/s

c > 0 for increase in uVel, which is west to

c east for cartesian and spherical polar grids

c Typical range: -10 < uwind < 10

c Southwest C-grid U point

c Input or input/output field

c

c vwind :: Surface (10-m) meridional wind velocity in m/s

c > 0 for increase in vVel, which is south to

c north for cartesian and spherical polar grids

c Typical range: -10 < vwind < 10

c Southwest C-grid V point


c

c atemp :: Surface (2-m) air temperature in deg K

8.2. THE EXTERNAL FORCING PACKAGE EXF 421

c Typical range: 200 < atemp < 300



c

c aqh :: Surface (2m) specific humidity in kg/kg

c Typical range: 0 < aqh < 0.02



c

c lwflux :: Net upward longwave radiation in W/m^2

c lwflux = - ( lwdown - ice and snow absorption - emitted )


c Typical range: -20 < lwflux < 170


c Input field

c

c evap :: Evaporation in m/s

c > 0 for increase in salt (ocean salinity)

c Typical range: 0 < evap < 2.5e-7


c Input, input/output, or output field

c

c precip :: Precipitation in m/s

c > 0 for decrease in salt (ocean salinity)

c Typical range: 0 < precip < 5e-7



c

c runoff :: River and glacier runoff in m/s

c > 0 for decrease in salt (ocean salinity)

c Typical range: 0 < runoff < ????



c !!! WATCH OUT: Default exf_inscal_runoff !!!

c !!! in exf_readparms.F is not 1.0 !!!

c

c swdown :: Downward shortwave radiation in W/m^2

c > 0 for increase in theta (ocean warming)

c Typical range: 0 < swdown < 450


c Input/output field

c

c lwdown :: Downward longwave radiation in W/m^2

c > 0 for increase in theta (ocean warming)

c Typical range: 50 < lwdown < 450


c Input/output field

c

c apressure :: Atmospheric pressure field in N/m^2

c > 0 for ????

c Typical range: ???? < apressure < ????


c Input field

C

C

c NOTES:

c ======


c

c Input and output units and sign conventions can be customized

c using variables exf_inscal_* and exf_outscal_*, which are set

c by exf_readparms.F

c

c Output fields fu, fv, Qnet, Qsw, and EmPmR are

c defined in FFIELDS.h

c

c #ifndef SHORTWAVE_HEATING, hflux includes shortwave,

c that is, hflux = latent + sensible + lwflux +swflux

c

c If (EXFwindOnBgrid .EQ. .TRUE.), uwind and vwind are

c defined on northeast B-grid U and V points, respectively.

c

c Arrays *0 and *1 below are used for temporal interpolation.

8.3. THE CALENDAR PACKAGE CAL 423

8.3 The calendar package cal

Christian Eckert, MIT/EAPS, May-2000This calendar tool was originally intended to enable the use of absolute dates

(Gregorian Calendar dates) in the ocean general circulation model MITgcmuv.There is, however, a fair amount of routines that can be used independently ofthe MITgcmuv. After some minor modifications the whole package can be usedeither as a stand-alone calendar or in connection with any dynamical model thatneeds calendar dates. Some straightforward extensions are still pending e.g. theavailability of the Julian Calendar, to be able to resolve fractions of a second,and to have a timestep that is longer than one day.

8.3.1 Basic assumptions for the calendar tool

It is assumed that the SMALLEST TIME INTERVAL to be resolved is ONESECOND.

Further assumptions are that there is an INTEGER NUMBER OF MODELSTEPS EACH DAY, and that AT LEAST ONE STEP EACH DAY is made.

Not each individual routine depends on these assumptions; there are only afew places where they enter.

8.3.2 Format of calendar dates

In this calendar tool a complete date specification is defined as the followinginteger array:

c integer date(4)cc ( yyyymmdd, hhmmss, leap_year, dayofweek )cc date(1) = yyyymmdd <-- Year-Month-Dayc date(2) = hhmmss <-- Hours-Minutes-Secondsc date(3) = leap_year <-- Leap Year/No Leap Yearc date(4) = dayofweek <-- Day of the Weekcc leap_year is either equal to 1 (normal year)c or equal to 2 (leap year)cc dayofweek has a range of 1 to 7.

In case the Gregorian Calendar is used, the first day of the week is Friday,since day of the Gregorian Calendar was Friday, 15 Oct. 1582. As a date arraythis date would be specified as

c refdate(1) = 15821015c refdate(2) = 0c refdate(3) = 1c refdate(4) = 1


8.3.3 Calendar dates and time intervals

Subtracting calendar dates yields time intervals. Time intervals have the fol-lowing format:

c integer datediff(4)cc datediff(1) = # Daysc datediff(2) = hhmmssc datediff(3) = 0c datediff(4) = -1

Such time intervals can be added to or can be subtracted from calendardates. Time intervals can be added to and be subtracted from each other.

8.3.4 Using the calendar together with MITgcm

Each routine has as an argument the thread number that it is belonging to,even if this number is not used in the routine itself.

In order to include the calendar tool into the MITgcm setup the MIT-gcm subroutine ”initialise.F” or the routine ”initilise fixed.F”, depending onthe MITgcm release, has to be modified in the following way:

c #ifdef ALLOW_CALENDAR

c C-- Initialise the calendar package.

c #ifdef USE_CAL_NENDITER

c CALL cal_Init(

c I startTime,

c I endTime,

c I deltaTclock,

c I nIter0,

c I nEndIter,

c I nTimeSteps,

c I myThid

c & )

c #else

c CALL cal_Init(

c I startTime,

c I endTime,

c I deltaTclock,

c I nIter0,

c I nTimeSteps,

c I myThid

c & )

c #endif

c _BARRIER

c #endif

It is useful to have the CPP flag ALLOW CALENDAR in order to switchfrom the usual MITgcm setup to the one that includes the calendar tool. TheCPP flag USE CAL NENDITER has been introduced in order to enable theuse of the calendar for MITgcm releases earlier than checkpoint 25 which donot have the global variable *nEndIter*.


8.3.5 The individual calendars

8.3.5.1 Simple model calendar

This calendar can be used by defining

c TheCalendar=’model’

in the calendar’s data file ”data.cal”.In this case a year is assumed to have 360 days. The model year is divided

into 12 months with 30 days each.

8.3.5.2 Gregorian Calendar

This calendar can be used by defining

c TheCalendar=’gregorian’

in the calendar’s data file ”data.cal”.

8.3.6 Short routine description

c o cal_Init - Initialise the calendar. This is the interface

c to the MITgcm.

c

c o cal_Set - Sets the calendar according to the user

c specifications.

c

c o cal_GetDate - Given the model’s current timestep or the

c model’s current time return the corresponding

c calendar date.

c

c o cal_FullDate - Complete a date specification (leap year and

c day of the week).

c

c o cal_IsLeap - Determine whether a given year is a leap year.

c

c o cal_TimePassed - Determine the time passed between two dates.

c

c o cal_AddTime - Add a time interval either to a time interval

c or to a date.

c

c o cal_TimeInterval - Given a time interval return the corresponding

c date array.

c

c o cal_SubDates - Determine the time interval between two dates

c or between two time intervals.

c

c o cal_ConvDate - Decompose a date array or a time interval

c array into its components.

c

c o cal_CopyDate - Copy a date array or a time interval array to

c another array.

c

c o cal_CompDates - Compare two calendar dates or time intervals.


c

c o cal_ToSeconds - Given a time interval array return the number

c of seconds.

c

c o cal_WeekDay - Return the weekday as a string given the

c calendar date.

c

c o cal_NumInts - Return the number of time intervals between two

c given dates.

c

c o cal_StepsPerDay - Given an iteration number or the current

c integration time return the number of time

c steps to integrate in the current calendar day.

c

c o cal_DaysPerMonth - Given an iteration number or the current

c integration time return the number of days

c to integrate in this calendar month.

c

c o cal_MonthsPerYear - Given an iteration number or the current

c integration time return the number of months

c to integrate in the current calendar year.

c

c o cal_StepsForDay - Given the integration day return the number

c of steps to be integrated, the first step,

c and the last step in the day specified. The

c first and the last step refer to the total

c number of steps (1, ... , cal_IntSteps).

c

c o cal_DaysForMonth - Given the integration month return the number

c of days to be integrated, the first day,

c and the last day in the month specified. The

c first and the last day refer to the total

c number of steps (1, ... , cal_IntDays).

c

c o cal_MonthsForYear - Given the integration year return the number

c of months to be integrated, the first month,

c and the last month in the year specified. The

c first and the last step refer to the total

c number of steps (1, ... , cal_IntMonths).

c

c o cal_Intsteps - Return the number of calendar years that are

c affected by the current integration.

c

c o cal_IntDays - Return the number of calendar days that are


c

c o cal_IntMonths - Return the number of calendar months that are


c

c o cal_IntYears - Return the number of calendar years that are


c

c o cal_nStepDay - Return the number of time steps that can be

c performed during one calendar day.

c

c o cal_CheckDate - Do some simple checks on a date array or on a

c time interval array.


c

c o cal_PrintError - Print error messages according to the flags

c raised by the calendar routines.

c

c o cal_PrintDate - Print a date array in some format suitable for

c the MITgcmuv’s protocol output.

c

c o cal_TimeStamp - Given the time and the iteration number return

c the date and print all the above numbers.

c

c o cal_Summary - List all the setttings of the calendar tool.


8.4 The line search optimisation algorithm

8.4.1 General features

The line search algorithm is based on a quasi-Newton variable storage methodwhich was implemented by [21].

TO BE CONTINUED...

8.4.2 The online vs. offline version

• Online versionEvery call to simul refers to an execution of the forward and adjoint model.Several iterations of optimization may thus be performed within a singlerun of the main program (lsopt top). The following cases may occur:

– cold start only (no optimization)

– cold start, followed by one or several iterations of optimization

– warm start from previous cold start with one or several iterations

– warm start from previous warm start with one or several iterations

• Offline versionEvery call to simul refers to a read procedure which reads the result of aforward and adjoint run Therefore, only one call to simul is allowed, itmax= 0, for cold start itmax = 1, for warm start Also, at the end, x(i+1)needs to be computed and saved to be available for the offline model andadjoint run

In order to achieve minimum difference between the online and offline codexdiff(i+1) is stored to file at the end of an (offline) iteration, but recomputedidentically at the beginning of the next iteration.

8.4.3 Number of iterations vs. number of simulations

- itmax: controls the max. number of iterations- nfunc: controls the max. number of simulations within one iteration

SummaryFrom one iteration to the next the descent direction changes. Within one iter-ation more than one forward and adjoint run may be performed. The updatedcontrol used as input for these simulations uses the same descent direction, butdifferent step sizes.

DescriptionFrom one iteration to the next the descent direction dd changes using the resultfor the adjoint vector gg of the previous iteration. In lsline the updated control

xdiff(i, 1) = xx(i− 1) + tact(i− 1, 1) ∗ dd(i− 1)

8.4. THE LINE SEARCH OPTIMISATION ALGORITHM 429

serves as input for a forward and adjoint model run yielding a new gg(i,1).In general, the new solution passes the 1st and 2nd Wolfe tests so xdiff(i,1)represents the solution sought:

xx(i) = xdiff(i, 1)

If one of the two tests fails, an inter- or extrapolation is invoked to determinea new step size tact(i-1,2). If more than one function call is permitted, thenew step size is used together with the ”old” descent direction dd(i-1) (i.e. ddis not updated using the new gg(i)), to compute a new

xdiff(i, 2) = xx(i− 1) + tact(i− 1, 2) ∗ dd(i− 1)

that serves as input in a new forward and adjoint run, yielding gg(i,2). If now,both Wolfe tests are successful, the updated solution is given by

xx(i) = xdiff(i, 2) = xx(i− 1) + tact(i− 1, 2) ∗ dd(i− 1)

In order to save memory both the fields dd and xdiff have a double usage.

xdiff- in lsopt top: used as x(i) - x(i-1) for Hessian update- in lsline: intermediate result for control update x = x + tact*dd

dd- in lsopt top, lsline: descent vector, dd = -gg and hessupd- in dgscale: intermediate result to compute new preconditioner

The parameter file lsopt.par

• NUPDATE max. no. of update pairs (gg(i)-gg(i-1), xx(i)-xx(i-1))to be stored in OPWARMD to estimate Hessian [pair of current iter. isstored in (2*jmax+2, 2*jmax+3) jmax must be ¿ 0 to access these en-tries] Presently NUPDATE must be ¿ 0 (i.e. iteration without reference toprevious iterations through OPWARMD has not been tested)

• EPSX relative precision on xx bellow which xx should not be improved

• EPSG relative precision on gg below which optimization is consideredsuccessful

• IPRINT controls verbose (¿=1) or non-verbose output

• NUMITER max. number of iterations of optimisation; NUMTER = 0:cold start only, no optimization

• ITER NUM index of new restart file to be created (not necessarily =NUMITER!)


• NFUNC max. no. of simulations per iteration (must be ¿ 0); is used ifstep size tact is inter-/extrapolated; in this case, if NFUNC ¿ 1, a newsimulation is performed with same gradient but ”improved” step size

• FMIN first guess cost function value (only used as long as first iterationnot completed, i.e. for jmax ¡= 0)

OPWARMI, OPWARMD files Two files retain values of previous itera-tions which are used in latest iteration to update Hessian:

• OPWARMI: contains index settings and scalar variablesn = nn no. of control variablesfc = ff cost value of last iterationisize no. of bytes per record in OPWARMDm = nupdate max. no. of updates for Hessianjmin, jmax pointer indices for OPWARMD file (cf. below)gnorm0 norm of first (cold start) gradient ggiabsiter total number of iterations with respect to cold start

• OPWARMD: contains vectors (control and gradient)entry name description

1 xx(i) control vector of latest iteration2 gg(i) gradient of latest iteration3 xdiff(i),diag preconditioning vector; (1,...,1) for cold start

2*jmax+2 gold=g(i)-g(i-1) for last update (jmax)2*jmax+3 xdiff=tact*d=xx(i)-xx(i-1) for last update (jmax)

Error handling

Example 1: jmin = 1, jmax = 3, mupd = 5

1 2 3 | 4 5 6 7 8 9 empty empty

|___|___|___| | |___|___| |___|___| |___|___| |___|___| |___|___|

0 | 1 2 3

Example 2: jmin = 3, jmax = 7, mupd = 5 ---> jmax = 2

1 2 3 |

|___|___|___| | |___|___| |___|___| |___|___| |___|___| |___|___|

| 6 7 3 4 5

Figure 8.1: Examples of OPWARM file handling


lsopt_top||---- check arguments|---- CALL INSTORE| || |---- determine whether OPWARMI available:| * if no: cold start: create OPWARMI| * if yes: warm start: read from OPWARMI| create or open OPWARMD||---- check consistency between OPWARMI and model parameters||---- >>> if COLD start: <<<| | first simulation with f.g. xx_0; output: first ff_0, gg_0| | set first preconditioner value xdiff_0 to 1| | store xx(0), gg(0), xdiff(0) to OPWARMD (first 3 entries)| || >>> else: WARM start: <<<| read xx(i), gg(i) from OPWARMD (first 2 entries)| for first warm start after cold start, i=0||||---- /// if ITMAX > 0: perform optimization (increment loop index i)| (| )---- save current values of gg(i-1) -> gold(i-1), ff -> fold(i-1)| (---- CALL LSUPDXX| ) || ( |---- >>> if jmax=0 <<<| ) | | first optimization after cold start:| ( | | preconditioner estimated via ff_0 - ff_(first guess)| ) | | dd(i-1) = -gg(i-1)*preco| ( | || ) | >>> if jmax > 0 <<<| ( | dd(i-1) = -gg(i-1)| ) | CALL HESSUPD| ( | || ) | |---- dd(i-1) modified via Hessian approx.| ( || ) |---- >>> if <dd,gg> >= 0 <<<| ( | ifail = 4| ) || ( |---- compute step size: tact(i-1)| ) |---- compute update: xdiff(i) = xx(i-1) + tact(i-1)*dd(i-1)| (| )---- >>> if ifail = 4 <<<| ( goto 1000| )| (---- CALL OPTLINE / LSLINE| ) |

... ... ...

Figure 8.2: Flow chart (part 1 of 3)


... ...| )| (---- CALL OPTLINE / LSLINE| ) || ( |---- /// loop over simulations| ) (| ( )---- CALL SIMUL| ) ( || ( ) |---- input: xdiff(i)| ) ( |---- output: ff(i), gg(i)| ( ) |---- >>> if ONLINE <<<| ) ( runs model and adjoint| ( ) >>> if OFFLINE <<<| ) ( reads those values from file| ( )| ) (---- 1st Wolfe test:| ( ) ff(i) <= tact*xpara1*<gg(i-1),dd(i-1)>| ) (| ( )---- 2nd Wolfe test:| ) ( <gg(i),dd(i-1)> >= xpara2*<gg(i-1),dd(i-1)>| ( )| ) (---- >>> if 1st and 2nd Wolfe tests ok <<<| ( ) | 320: update xx: xx(i) = xdiff(i)| ) ( || ( ) >>> else if 1st Wolfe test not ok <<<| ) ( | 500: INTERpolate new tact:| ( ) | barr*tact < tact < (1-barr)*tact| ) ( | CALL CUBIC| ( ) || ) ( >>> else if 2nd Wolfe test not ok <<<| ( ) 350: EXTRApolate new tact:| ) ( (1+barmin)*tact < tact < 10*tact| ( ) CALL CUBIC| ) (| ( )---- >>> if new tact > tmax <<<| ) ( | ifail = 7| ( ) || ) (---- >>> if new tact < tmin OR tact*dd < machine precision <<<| ( ) | ifail = 8| ) ( || ( )---- >>> else <<<| ) ( update xdiff for new simulation| ( )| ) \\\ if nfunc > 1: use inter-/extrapolated tact and xdiff| ( for new simulation| ) N.B.: new xx is thus not based on new gg, but| ( rather on new step size tact| )| (---- store new values xx(i), gg(i) to OPWARMD (first 2 entries)| )---- >>> if ifail = 7,8,9 <<<| ( goto 1000| )

... ...



... ...| )| (---- store new values xx(i), gg(i) to OPWARMD (first 2 entries)| )---- >>> if ifail = 7,8,9 <<<| ( goto 1000| )| (---- compute new pointers jmin, jmax to include latest values| ) gg(i)-gg(i-1), xx(i)-xx(i-1) to Hessian matrix estimate| (---- store gg(i)-gg(i-1), xx(i)-xx(i-1) to OPWARMD| ) (entries 2*jmax+2, 2*jmax+3)| (| )---- CALL DGSCALE| ( || ) |---- call dostore| ( | || ) | |---- read preconditioner of previous iteration diag(i-1)| ( | from OPWARMD (3rd entry)| ) || ( |---- compute new preconditioner diag(i), based upon diag(i-1),| ) | gg(i)-gg(i-1), xx(i)-xx(i-1)| ( || ) |---- call dostore| ( || ) |---- write new preconditioner diag(i) to OPWARMD (3rd entry)| (|---- \\\ end of optimization iteration loop||||---- CALL OUTSTORE| || |---- store gnorm0, ff(i), current pointers jmin, jmax, iterabs to OPWARMI||---- >>> if OFFLINE version <<<| xx(i+1) needs to be computed as input for offline optimization| || |---- CALL LSUPDXX| | || | |---- compute dd(i), tact(i) -> xdiff(i+1) = x(i) + tact(i)*dd(i)| || |---- CALL WRITE_CONTROL| | || | |---- write xdiff(i+1) to special file for offline optim.||---- print final information|O


Chapter 9

Model Uses

To date, the MITgcm model has been used for number of purposes. The follow-ing papers are an incomplete yet fairly broad sample of MITgcm applications:

• R.S. Gross and I. Fukumori and D. Menemenlis (2003). Atmospheric andoceanic excitation of the Earth’s wobbles during 1980-2000. Journal ofGeophysical Research-Solid Earth, vol.108.

• M.A. Spall and R.S. Pickart (2003). Wind-driven recirculations and ex-change in the Labrador and Irminger Seas. Journal of Physical Oceanog-raphy, vol.33, pp.1829–1845.

• B. Ferron and J. Marotzke (2003). Impact of 4D-variational assimilationof WOCE hydrography on the meridional circulation of the Indian Ocean.Deep-Sea Research Part II-Topical Studies in Oceanography, vol.50, pp.2005–2021.

• G.A. McKinley and M.J. Follows and J. Marshall and S.M. Fan (2003).Interannual variability of air-sea O-2 fluxes and the determination of CO2sinks using atmospheric O-2/N-2. Geophysical Research Letters, vol.30.

• B.Y. Huang and P.H. Stone and A.P. Sokolov and I.V. Kamenkovich(2003). The deep-ocean heat uptake in transient climate change. Journalof Climate, vol.16, pp.1352–1363.

• G. de Coetlogon and C. Frankignoul (2003). The persistence of wintersea surface temperature in the North Atlantic. Journal of Climate, vol.16,pp.1364–1377.

• K.H. Nisancioglu and M.E. Raymo and P.H. Stone (2003). Reorganizationof Miocene deep water circulation in response to the shoaling of the CentralAmerican Seaway. Paleoceanography, vol.18.

• T. Radko and J. Marshall (2003). Equilibration of a Warm Pumped Lenson a beta plane. Journal of Physical Oceanography, vol.33, pp.885–899.

435

436 CHAPTER 9. MODEL USES

• T. Stipa (2002). The dynamics of the N/P ratio and stratification in alarge nitrogen-limited estuary as a result of upwelling: a tendency foroffshore Nodularia blooms. Hydrobiologia, vol.487, pp.219–227.

• D. Stammer and C. Wunsch and R. Giering and C. Eckert and P. He-imbach and J. Marotzke and A. Adcroft and C.N. Hill and J. Marshall(2003). Volume, heat, and freshwater transports of the global ocean circu-lation 1993-2000, estimated from a general circulation model constrainedby World Ocean Circulation Experiment (WOCE) data. Journal of Geo-physical Research-Oceans, vol.108.

• P. Heimbach and C. Hill and R. Giering (2002). Automatic generation ofefficient adjoint code for a parallel Navier-Stokes solver. ComputationalScience-ICCS 2002, PT II, Proceedings, vol.2330, pp.1019–1028.

• D. Stammer and C. Wunsch and R. Giering and C. Eckert and P. Heim-bach and J. Marotzke and A. Adcroft and C.N. Hill and J. Marshall (2002).Global ocean circulation during 1992-1997, estimated from ocean observa-tions and a general circulation model. Journal of Geophysical Research-Oceans, vol.107.

• M. Solovev and P.H. Stone and P. Malanotte-Rizzoli (2002). Assessmentof mesoscale eddy parameterizations for a single-basin coarse-resolutionocean model. Journal of Geophysical Research-Oceans, vol.107.

• C. Herbaut and J. Sirven and S. Fevrier (2002). Response of a simplifiedoceanic general circulation model to idealized NAO-like stochastic forcing.Journal of Physical Oceanography, vol.32, pp.3182–3192.

• C. Wunsch (2002). Oceanic age and transient tracers: Analytical andnumerical solutions. Journal of Geophysical Research-Oceans, vol.107.

• J. Sirven and C. Frankignoul and D. de Coetlogon and V. Taillandier(2002). Spectrum of wind-driven baroclinic fluctuations of the ocean inthe midlatitudes. Journal of Physical Oceanography, vol.32, pp.2405–2417.

• R. Zhang and M. Follows and J. Marshall (2002). Mechanisms of thermo-haline mode switching with application to warm equable climates. Journalof Climate, vol.15, pp.2056–2072.

• G. Brostrom (2002). On advection and diffusion of plankton in coarseresolution ocean models. Journal of Marine Systems, vol.35, pp.99–110.

• T. Lee and I. Fukumori and D. Menemenlis and Z.F. Xing and L.L. Fu(2002). Effects of the Indonesian Throughflow on the Pacific and Indianoceans. Journal of Physical Oceanography, vol.32, pp.1404–1429.

• C. Herbaut and J. Marshall (2002). Mechanisms of buoyancy transportthrough mixed layers and statistical signatures from isobaric floats. Jour-nal of Physical Oceanography, vol.32, pp.545–557.

437

• J. Marshall and H. Jones and R. Karsten and R. Wardle (2002). Caneddies set ocean stratification?, Journal of Physical Oceanography, vol.32,pp.26–38.

• C. Herbaut and J. Sirven and A. Czaja (2001). An idealized model studyof the mass and heat transports between the subpolar and subtropicalgyres. Journal of Physical Oceanography, vol.31, pp.2903–2916.

• R.H. Kase and A. Biastoch and D.B. Stammer (2001). On the Mid-DepthCirculation in the Labrador and Irminger Seas. Geophysical ResearchLetters, vol.28, pp.3433–3436.

• R. Zhang and M.J. Follows and J.P. Grotzinger and J. Marshall (2001).Could the Late Permian deep ocean have been anoxic?. Paleoceanography,vol.16, pp.317–329.

• A. Adcroft and J.R. Scott and J. Marotzke (2001). Impact of geothermalheating on the global ocean circulation. Geophysical Research Letters,vol.28, pp.1735–1738.

• J. Marshall and D. Jamous and J. Nilsson (2001). Entry, flux, and exit ofpotential vorticity in ocean circulation. Journal of Physical Oceanography,vol.31, pp.777–789.

• A. Mahadevan (2001). An analysis of bomb radiocarbon trends in thePacific. Marine Chemistry, vol.73, pp.273–290.

• G. Brostrom (2000). The role of the annual cycles for the air-sea exchangeof CO2. Marine Chemistry, vol.72, pp.151–169.

• Y.H. Zhou and H.Q. Wu and N.H. Yu and D.W. Zheng (2000). Excitationof seasonal polar motion by atmospheric and oceanic angular momentums.Progress in Natural Science, vol.10, pp.931–936.

• R. Wardle and J. Marshall (2000). Representation of eddies in primitiveequation models by a PV flux. Journal of Physical Oceanography, vol.30,pp.2481–2503.

• P.S. Polito and O.T. Sato and W.T. Liu (2000). Characterization andvalidation of the heat storage variability from TOPEX/Poseidon at fouroceanographic sites. Journal of Geophysical Research-Oceans, vol.105,pp.16911–16921.

• R.M. Ponte and D. Stammer (2000). Global and regional axial ocean angu-lar momentum signals and length-of-day variations (1985-1996). Journalof Geophysical Research-Oceans, vol.105, pp.17161–17171.

• J. Wunsch (2000). Oceanic influence on the annual polar motion. Journalof GEODYNAMICS, vol.30, pp.389–399.


• D. Menemenlis and M. Chechelnitsky (2000). Error estimates for an oceangeneral circulation model from altimeter and acoustic tomography data.Monthly Weather Review, vol.128, pp.763–778.

• Y.H. Zhou and D.W. Zheng and N.H. Yu and H.Q. Wu (2000). Excita-tion of annual polar motion by atmosphere and ocean. Chinese ScienceBulletin, vol.45, pp.139–142.

• J. Marotzke and R. Giering and K.Q. Zhang and D. Stammer and C.Hill and T. Lee (1999). Construction of the adjoint MIT ocean generalcirculation model and application to Atlantic heat transport sensitivity.Journal of Geophysical Research-Oceans, vol.104, pp.29529–29547.

• R.M. Ponte and D. Stammer (1999). Role of ocean currents and bot-tom pressure variability on seasonal polar motion. Journal of GeophysicalResearch-Oceans, vol.104, pp.23393–23409.

• J. Nastula and R.M. Ponte (1999). Further evidence for oceanic excitationof polar motion. Geophysical Journal International, vol.139, pp.123–130.

• K.Q. Zhang and J. Marotzke (1999). The importance of open-boundaryestimation for an Indian Ocean GCM-data synthesis. Journal of MarineResearch, vol.57, pp.305–334.

• J. Marshall and D. Jamous and J. Nilsson (1999). Reconciling thermody-namic and dynamic methods of computation of water-mass transformationrates. Deep-Sea Research PART I-Oceanographic Research Papers, vol.46,pp.545–572.

• J. Marshall and F. Schott (1999). Open-ocean convection: Observations,theory, and models. Reviews of Geophysics, vol.37, pp.1–64.

• J. Marshall and H. Jones and C. Hill (1998). Efficient ocean modelingusing non-hydrostatic algorithms. Journal of Marine Systems, vol.18,pp.115–134.

• A.B. Baggeroer and T.G. Birdsall and C. Clark and J.A. Colosi and B.D.Cornuelle and D. Costa and B.D. Dushaw and M. Dzieciuch and A.M.G.Forbes and C. Hill and B.M. Howe and J. Marshall and D. Menemenlisand J.A. Mercer and K. Metzger and W. Munk and R.C. Spindel andD. Stammer and P.F. Worcester and C. Wunsch (1998). Ocean climatechange: Comparison of acoustic tomography, satellite altimetry, and mod-eling. Science, vol.281, pp.1327–1332.

• C. Wunsch and D. Stammer (1998). Satellite altimetry, the marine geoid,and the oceanic general circulation. Annual Review of Earth and Plane-tary Sciences, vol.26, pp.219–253.

439

• A. Shaw and Arvind and K.C. Cho and C. Hill and R.P. Johnson and J.Marshall (1998). A comparison of implicitly parallel multithreaded anddata-parallel implementations of an ocean model. Journal of Parallel andDistributed Computing, vol.48, pp.1–51.

• T.W.N. Haine and J. Marshall (1998). Gravitational, symmetric, andbaroclinic instability of the ocean mixed layer. Journal of Physical Oceanog-raphy, vol.28, pp.634–658.

• R.M. Ponte and D. Stammer and J. Marshall (1998). Oceanic signals inobserved motions of the Earth’s pole of rotation. Nature, vol.391, pp.476–479.

• D. Menemenlis and C. Wunsch (1997). Linearization of an oceanic generalcirculation model for data assimilation and climate studies. Journal ofAtmospheric and Oceanic Technology, vol.14, pp.1420–1443.

• H. Jones and J. Marshall (1997). Restratification after deep convection.Journal of Physical Oceanography, vol.27, pp.2276–2287.

• S.R. Jayne and R. Tokmakian (1997). Forcing and sampling of oceangeneral circulation models: Impact of high-frequency motions. Journal ofPhysical Oceanography, vol.27, pp.1173–1179.

Bibliography

[1] A. Adcroft. Numerical Algorithms for use in a Dynamical Model of theOcean. PhD thesis, Imperial College, London, 1995.

[2] A. Adcroft and J.-M. Campin. Comparison of finite volume schemes anddirect-space-time methods for ocean circulation models. Ocean Modelling,2002. in preparation.

[3] A. Adcroft and D. Marshall. How slippery are piecewise-constant coastlinesin numerical ocean models? Tellus, 50(1):95–108, 1998.

[4] A.J. Adcroft and J.-M. Campin. Re-scaled height coordinates for accuraterepresentation of free-surface flows in ocean circulation models. OceanModelling, 2004. in press.

[5] A.J. Adcroft, C.N. Hill, and J. Marshall. Representation of topography byshaved cells in a height coordinate ocean model. MWR, 125:2293–2315,1997.

[6] Hill C. Adcroft, A. and J. Marshall. A new treatment of the coriolis termsin c-grid models at both high and low resolutions. MWR, 127:1928–1936,1999.

[7] A. Arakawa and V. Lamb. Computational design of the basic dynamicalprocesses of the ucla general circulation model. In Methods in Computa-tional Physics, volume 17, pages 174–267. Academic Press, 1977.

[8] J.-M. Campin, A. Adcroft, C. Hill, and J. Marshall. Conservation of prop-erties in a free-surface model. Ocean Modelling, 6:221–244, 2004.

[9] Daniel Jamous Chris Hill, Alistair Adcroft and John Marshall. A strategyfor terascale climate modeling. In In Proceedings of the Eighth ECMWFWorkshop on the Use of Parallel Processors in Meteorology, pages 406–425.World Scientific, 1999.

[10] A. da Silva, A. C. Young, and S. Levitus. Atlas of surface marine data 1994,volume 1: Algorithms and procedures. NOAA Atlas NESDIS, 6, 1994.http:////ingrid.ldeo.columbia.edu/SOURCES/.DASILVA/.SMD94/.

441

442 BIBLIOGRAPHY

[11] G. Danabasoglu and McWilliams J.C. Sensitivity of the global ocean circu-lation to parameterizations of mesoscale tracer transports. JC, 8(12):2967–2987, 1995.

[12] Roland A. de Szoeke and Roger M. Samelson. The duality betweenthe Boussinesq and Non-Boussinesq hydrostatic equations of motion.J. Phys. Oceanogr., 32(8):2194–2203, 2002.

[13] Adcroft et al. Energy conversion in discrete numerical models. OceanModelling, 2002. in preparation.

[14] G. M. Flato and W. D. Hibler, III. Modeling pack ice as a cavitating fluid.J. Phys. Oceanogr., 22:626–651, 1992.

[15] P. Fofonoff and R.C. Millard, Jr. Algorithms for computation of fundamen-tal properties of seawater. Unesco Technical Papers in Marine Science 44,Unesco, 1983.

[16] P.R. Gent and J.C. McWilliams. Isopycnal mixing in ocean circulationmodels. J. Phys. Oceanogr., 20:150–155, 1990.

[17] P.R. Gent, J. Willebrand, T.J. McDougall, and J.C. McWilliams. Pa-rameterizing eddy-induced tracer transports in ocean circulation models.J. Phys. Oceanogr., 25:463–474, 1995.

[18] R. Giering. Tangent linear and adjoint model compiler. users manual1.4 (tamc version 5.2). Report , Massachusetts Institute of Technology,MIT/EAPS; 77 Massachusetts Ave.; Cambridge (MA) 02139; USA, 1999.http://puddle.mit.edu/∼ralf/tamc/tamc.html.

[19] R. Giering. Tangent linear and adjoint biogeochemical models. In P. Ka-sibhatla, M. Heimann, P. Rayner, N. Mahowald, R.G. Prinn, and D.E.Hartley, editors, Inverse methods in global biogeochemical cycles, volume114 of Geophysical Monograph, pages 33–48. American Geophysical Union,Washington, DC, 2000.

[20] R. Giering and T. Kaminski. Recipes for adjoint code construction. ACMTransactions on Mathematical Software, 24:437–474, 1998.

[21] J.C. Gilbert and C. Lemarechal. Some numerical experiments with variable-storage quasi-newton algorithms. Math. Programming, 45:407–435, 1989.

[22] A. Griewank. Achieving logarithmic growth of temporal and spatial com-plexity in reverse Automatic Differentiation. Optimization Methods andSoftware, 1:35–54, 1992.

[23] A. Griewank. Evaluating Derivatives. Principles and Techniques of Algo-rithmic Differentiation, volume 19 of Frontiers in Applied Mathematics.SIAM, Philadelphia, 2000.

BIBLIOGRAPHY 443

[24] M. Griffies, S. and W. Hallberg, R. Biharmonic friction with a smagorinsky-like viscosity for use in large-scale eddy-permitting ocean models. MWR,128(8):2935–2946, 2000.

[25] R.L. Haney. Surface thermal boundary conditions for ocean circulationmodels. J. Phys. Oceanogr., 1:241–248, 1971.

[26] I.M. Held and M.J. Suarez. A proposal for the intercomparison of thedynamical cores of atmospheric general circulation models. Bulletin of theAmerican Meteorological Society, 75(10):1825–1830, 1994.

[27] W. D. Hibler, III. A dynamic thermodynamic sea ice model.J. Phys. Oceanogr., 9:815–846, 1979.

[28] W. D. Hibler, III. Modeling a variable thickness sea ice cover.Mon. Wea. Rev., 1:1943–1973, 1980.

[29] C. Hill, M. Follows, V. Bugnion, and J. Marshall. Spatial and tempo-ral impacts of ocean general circulation on carbon sequestration. GlobalBiogeochemical Cycles, 2002. submitted.

[30] C. Hill and J. Marshall. Application of a parallel navier-stokes model toocean circulation in parallel computational fluid dynamics. In N. SatofukaA. Ecer, J. Periaux and S. Taylor, editors, Implementations and ResultsUsing Parallel Computers, pages 545–552. Elsevier Science B.V.: New York,1995.

[31] W.R. Holland and L. B. Lin. On the origin of mesoscale eddies and theircontribution to the general circulation of the ocean. i. a preliminary nu-merical experiment. J. Phys. Oceanogr., 5:642–657, 1975a.

[32] E. C. Hunke and J. K. Dukowicz. An elastic-viscous-plastic model for seaice dynamics. J. Phys. Oceanogr., 27:1849–1867, 1997.

[33] David R. Jackett and Trevor J. McDougall. Minimal adjustment of hy-drographic profiles to achieve static stability. J. Atmos. Ocean. Technol.,12(4):381–389, 1995.

[34] Shi Jiang, Peter H. Stone, and Paola Malanotte-Rizzoli. An assessmentof the Geophysical Fluid Dynamics Laboratory ocean model with coarseresolution: Annual-mean climatology. J. Geophys. Res., 104(C11):25,623–25,645, 1999.

[35] W.G. Large, G. Danabasoglu, S.C. Doney, and J.C. McWilliams. Sensitiv-ity to surface forcing and boundary layer mixing in a global ocean model:Annual-mean climatology. JPO, 27(11):2418–2447, 1997.

[36] S. Levitus and T.P.Boyer. World Ocean Atlas 1994 Volume 4: Temperature.Technical report, NOAA Atlas NESDIS 4, 1994.

444 BIBLIOGRAPHY

[37] J. Marotzke, R. Giering, K.Q. Zhang, D. Stammer, C. Hill, and T. Lee.Construction of the adjoint mit ocean general circulation model and ap-plication to atlantic heat transport variability. J. Geophys. Res., 104,C12:29,529–29,547, 1999.

[38] J. Marshall, A. Adcroft, C. Hill, L. Perelman, and C. Heisey. A finite-volume, incompressible navier stokes model for studies of the ocean onparallel computers. JGR, 102(C3):5753–5766, 1997.

[39] J. Marshall, C. Hill, L. Perelman, and A. Adcroft. Hydrostatic, quasi-hydrostatic, and nonhydrostatic ocean modeling. JGR, 102(C3):5733–5752,1997.

[40] J. Marshall, H. Jones, and C. Hill. Efficient ocean modeling using non-hydrostatic algorithms. JMR, 18:115–134, 1998.

[41] Trevor J. McDougall, David R. Jackett, Daniel G. Wright, and RainerFeistel. Accurate and computationally efficient algorithms for potentialtemperature and density of seawater. J. Atmos. Ocean. Technol., 2003.

[42] Gerdes R., Koberle C., Beckmann A., Herrmann P., and Willebrand J.Mechanisms for spreading of mediterranean water in coarse-resolution nu-merical models. JPO, 29(8):1682–1700, 1999.

[43] J.M. Restrepo, G.K. Leaf, and A. Griewank. Circumventing storage lim-itations in variational data assimilation studies. SIAM J. Sci. Comput.,19:1586–1605, 1998.

[44] R. K. Rew, G. P. Davis, S. Emmerson, and H. Davies. NetCDF User’sGuide for C, FORTRAN 77, and FORTRAN 90, an interface for dataaccess, version 3. Report, Unidata Program Center, Boulder, Colorado,1997. http://www.unidata.ucar.edu/packages/netcdf/.

[45] P.L. Roe. Some contributions to the modelling of discontinuous flows.In B.E. Engquist, S. Osher, and R.C.J. Somerville, editors, Large-ScaleComputations in Fluid Mechanics, volume 22 of Lectures in Applied Math-ematics, pages 163–193. American Mathematical Society, Providence, RI,1985.

[46] Albert J. Semtner, Jr. A model for the thermodynamic growth of sea ice innumerical investigations of climate. J. Phys. Oceanogr., 6:379–389, 1976.

[47] D. Stammer, C. Wunsch, R. Giering, C. Eckert, P. Heimbach, J. Marotzke,A. Adcroft, C. Hill, and J. Marshall. The global ocean circulation during1992 – 1997, estimated from ocean observations and a general circulationmodel. J. Geophys. Res., 2001. in press.

[48] D. Stammer, C. Wunsch, R. Giering, Q. Zhang, J. Marotzke, J. Marshall,and C. Hill. The global ocean circulation estimated from topex/poseidonaltimetry and a general circulation model. Technical Report 49, Center for

BIBLIOGRAPHY 445

Global Change Science, Massachusetts Institute of Technology, Cambridge(MA), USA, 1997.

[49] H. Stommel. The western intensification of wind-driven ocean currents.Trans. Am. Geophys. Union, 29:206, 1948.

[50] K. M. Trenberth, J. Olson, and W. G. Large. The mean annual cycle inGlobal Ocean wind stress. J. Phys. Oceanogr., 20:1742–1760, 1990.

[51] K. M. Trenberth, J. Olson, and W. G. Large. Atmospheric simulationsusing a gcm with simplified physical parametrization, i model climatologyand variability in multidecadal experiments. Clim. Dynamics, 20:175–191,2003.

[52] R. Wajsowicz. A consistent formulation of the anisotropic stress tensor foruse in models of the large-scale ocean circulation. JCP, 105(2):333–338,1993.

[53] M. Winton. A reformulated three-layer sea ice model. J. At-mos. Ocean. Technol., 17:525–531, 2000.

[54] J. Zhang, W. D. Hibler, III, M. Steele, and D. A. Rothrock. Arctic ice-ocean modeling with and without climate restoring. J. Phys. Oceanogr.,28:191–217, 1998.

[55] Jinlun Zhang and Drew Rothrock. Modeling arctic sea ice with an efficientplastic solution. J. Geophys. Res., 105:3325–3338, 2000.

manual cfd

Documents