TheStructureoftheFV3 Solver

LucasHarris

andtheGFDLFV3 Team

FV3WorkshopinTaiwan

4 December2017

FV3 DesignPhilosophy

• Discretizationshouldbeguidedbyphysicalprinciplesasmuchas

possible

• Finite-volume,integratedformofconservationlaws

• Upstream-biasedfluxes

• Operators“reverseengineered”toachievedesiredproperties• Computationalefficiencyiscrucial.Fastmodelscanbegoodmodels!

• Solvershouldbebuiltwithvectorizationandparallelisminmind

• Dynamicsisn’tthewholestory!Couplingtophysicsandtheoceanis

important.

Finite-volumemethodology

• InFV3,allvariablesare3Dcell- orface-means…notgridpoint values

• WesolvenotthedifferentialEulerequationsbuttheircell-integrated

formsusingintegraltheorems

• Everythingisaflux,includingthemomentumequation

• Massconservationisensured,toroundingerror

• C-Dgrid:Vorticitycomputedexactly;accuratedivergencecomputation

• Mimetic:Physicalpropertiesrecoveredbydiscretization,particularly

Newton’s3rd law

OutlineoftheFV3 solver

AbstractofFV3

• Gnomoniccubed-spheregridforscalabilityanduniformity

• Fully-compressiblevector-invariantEulerequations

• Vertically-Lagrangian dynamics

• C-Dgriddiscretization• Forward-in-time2DLin-RoodadvectionusingPPMoperators

• Fullynonhydrostatic withsemi-implicitsolver

• Runtimehydrostaticswitch

FV3 timeintegrationsequence

• FV3 isaforward-in-timesolver

• Flux-divergencetermsandphysicstendenciesevaluatedforward-in-time

• Pressure-gradientandsound-wavetermsevaluatedbackward-in-timefor

stability

• Lagrangian verticalcoordinate:flowconstrainedalongtime-evolving

Lagrangian surfaces

• Tracersaresub-cycled sincetheirstabilityconditionismuchless

restrictivethanthesound- orgravity-wavemodes

fv_dynamics()

FV3 solver

dyn_core()

Lagrangian dynamics

fv_tracer2d()

Sub-cycledtracertransport

OpenMP onk

Lagrangian_to_Eulerian()

VerticalRemapping

(i,k)OpenMP onj

c_sw(),etc.

C-gridsolver

d_sw()

ForwardLagrangian dyn.

OpenMP onk

update_dz_d()

Forwardδz evaluation

OpenMP onk

one_grad_p()/nh_p_grad()

BackwardshorizontalPGF

OpenMP onk

riem_solver()

BackwardsverticalPGF,

soundwaveprocesses

(i,k)OpenMP onj

[physics]

fv_update_phys()

Consistentfieldupdate

fv_dynamics()

FV3 solver

dyn_core()

Lagrangian dynamics

fv_tracer2d()

Sub-cycledtracertransport

OpenMP onk

Lagrangian_to_Eulerian()

VerticalRemapping

(i,k)OpenMP onj

c_sw(),etc.

C-gridsolver

d_sw()

ForwardLagrangian dyn.

OpenMP onk

update_dz_d()

Forwardδz evaluation

OpenMP onk

one_grad_p()/nh_p_grad()

BackwardshorizontalPGF

OpenMP onk

riem_solver()

BackwardsverticalPGF,

soundwaveprocesses

(i,k)OpenMP onj

[physics]

fv_update_phys()

Consistentfieldupdate

dt_atmosk_split

“remapping”loop

n_split

“acoustic”loop

PrognosticVariablesδp Totalairmass(includingvaporandcondensates)

Equal to hydrostatic pressuredepthoflayerθv Virtualpotentialtemperature

u,v HorizontalD-gridwindsinlocalcoordinate

(defined oncellfaces)

w Verticalwinds

δz Geometriclayerdepth

qi Passive tracers

Cell-meanpressure,density,divergence,andspecificheatarealldiagnostic quantitiesAllvariablesarelayer-meansinthevertical:Noverticalstaggering

Lagrangian DynamicsinFV3

Whattheyare

Whattheequationsare

Howtheyaresolved

WhatareLagrangian Dynamics?

• TheEulerequationscanbewritteninLagrangian orEulerianforms…orEulerianinthehorizontal,andLagrangian inthevertical

• Thisconstrainstheflowalongquasi-horizontalsurfaces

• Surfacesdeformduringtheintegration,representingverticalmotionandadvection“forfree”

• Doesrequirelayerthicknesstobeaprognosticvariable

5.2. Dependent variables and governing equations

Variable Description�p

⇤ Vertical difference in hydrostatic pressure, proportional to massu D-grid face-mean horizontal x-direction windv D-grid face-mean horizontal y-direction wind

⇥

v

Cell-mean virtual potential temperaturew Cell-mean vertical velocity�z Geometric layer height

Table 5.1: Solution variables in FV3

The continuous Lagrangian equations of motion, in a layer of finite depth�z and mass �p

⇤, are then given as

D

L

�p

⇤+ r · (V�p

⇤) = 0 (5.3)

D

L

�p

⇤⇥

v

+ r · (V�p

⇤⇥

v

) = 0

D

L

�p

⇤w + r · (V�p

⇤w) = �g�z

@p

0

@z

D

L

u = ⌦v � @

@x

� 1

⇢

@p

⇤

@x

���z

D

L

v = �⌦u � @

@y

� 1

⇢

@p

⇤

@x

���z

The operator D

L

is the “vertically-Lagrangian” derivative, formally equalto @�

@t

+

@

@z

(w�) for an arbitrary scalar �. The flow is entirely along theLagrangian surfaces, including the vertical motion (which deforms thesurfaces as appropriate, an effect included in the semi-implicit solver).The vertical component of absolute vorticity is given as ⌦ and the ki-netic energy is given as =

12(euu + evv), and p is the full nonhydrostatic

pressure. The nonhydrostatic pressure gradient term in the w equationis computed by the semi-implicit solver described Chapter 7. There is noprojection of the vertical pressure gradient force into the horizontal; sim-ilarly, there is no projection of the horizontal winds u, v into the vertical,despite the slopes of the Lagrangian surfaces.

The vertically-Lagrangian discretization requires expressions for ei-ther the mass or for z, as only one can be used as an independent variable;

33

DL� =@�

@t+

@

@z(w�)

Lagrangian Dynamics:Flux-formadvection

• Advectionisnotjustforpassivetracers!NearlyeverythinginFV3 isaflux

• Eventhemomentumadvectiontermscanbeexpressedasscalarfluxes

• Lin-Rood(1996)2Dadvectionscheme:

• Reverse-engineeredschemedevisedfrom1DPPMoperators

• Allowsmonotonicityandpositivityfromsubgrid reconstructions

• Monotonicityis“smart”diffusion

• Again:advectionisalongLagrangian surfaces

Lagrangian Dynamics:Traceradvectionandsub-cycling• Tracerscanbeadvected withalongertimestep thanthedynamics

• FV3 permitsaccumulationofmassfluxesduringLagrangian dynamics.

Thesefluxesarethenusedtocomputetheadvectionoftracers

beforetheverticalremapping

• Typicallyoneortwotracertimesteps issufficientforstability.

• Traceradvectionisalwaysmonotone(oratleastpositivedefinite)to

avoidnewextrema.Explicitdiffusionisnotused.

• Question: whataboutphysicaleddydiffusion(cf.PBLscheme?)

Lagrangian Dynamics:C-Dgridsolver• FV3 solvesforthe(purelyhorizontal)

D-gridstaggeredwinds.Butsolver

requiresface-normalfluxes

• Tocomputetimestep-meanfluxes,the

C-gridwindsareinterpolatedandthenadvancedahalf-timestep.

• AsortofsimplifiedRiemannsolver

• TheC-gridsolveristhesameastheD-grid,butuseslower-orderfluxesfor

efficiency

• Two-griddiscretizationandtime-centeredfluxesavoidcomputational

modes

�

D-grid winds

C-grid winds

Fluxes

Fig. 2. Geometry of the wind staggerings and fluxes for a cell on a non-orthogonal grid.The angle � is that between the covariant and contravariant components; in orthogonalcoordinates � = ⇥/2.

45

Lagrangian Dynamics:Momentumequation• FV3solvestheflux-formvectorinvariant

equations

• Nonlinearvorticityfluxterminmomentum

equation,confoundinglinearanalyses

• D-gridallowsexactcomputationof

absolutevorticity—noaveraging!

• Vorticityusessamefluxasδp:consistency

improvesgeostrophicbalance,andSW-PV

advected asascalar!

2. The Nested Grid Model126

a. Finite-Volume Dynamical Core and cubed-sphere grid127

The FV core is a hydrostatic, 3D dynamical core using the vertically-Lagrangian dis-128

cretization of L04 and the horizontal discretization of Lin and Rood (1996, 1997, hence-129

forth LR96 and LR97, respectively), using the cubed-sphere geometry of PL07 and Putman130

(2007). This solver discretizes a hydrostatic atmosphere into a number of vertical layers, each131

of which is then integrated by treating the pressure thickness and potential temperature as132

scalars. Each layer is advanced independently, except that the pressure gradient force is133

computed using the geopotential and the pressure at the interface of each layer (Lin 1997).134

The interface geopotential is the cumulative sum of the thickness of each underlying layer,135

counted from the surface elevation upwards, and the interface pressure is the cumulative136

sum of the pressure thickness of each overlying layer, counted from the constant-pressure137

top of the model domain downward. Vertical transport occurs implicitly from horizontal138

transport along Lagrangian surfaces. The layers are allowed to deform freely during the139

horizontal integration. To prevent the layers from becoming infinitesimally thin, and to ver-140

tically re-distribute mass, momentum, and energy, the layers are periodically remapped to141

a pre-defined Eulerian coordinate system.142

The governing equations in each horizontal layer are the vector-invariant equations:143

⇧�p

⇧t+⌅ · (V�p) = 0144

⇧�p�

⇧t+⌅ · (V�p�) = 0145

146

⇧V

⇧t= �⇥k̂ ⇤V �⌅

�⇥ + ⇤⌅2D

⇥� 1

⌅⌅p

⇤⇤⇤z

147

where the prognostic variables are the hydrostatic pressure thickness �p of a layer bounded148

by two adjacent Lagrangian surfaces, which is proportional to the mass of the layer; the149

potential temperature �; and the vector wind V. Here, k̂ is the vertical unit vector. The150

6

Lagrangian Dynamics:Momentumequation• FV3solvestheflux-formvectorinvariant

equations

• Nonlinearvorticityfluxterminmomentum

equation,confoundinglinearanalyses

• D-gridallowsexactcomputationof

absolutevorticity—noaveraging!

• Vorticityusessamefluxasw:consistencyimprovesnonlinearbalance,andupdraft helicityadvectedasascalar!

2. The Nested Grid Model126

a. Finite-Volume Dynamical Core and cubed-sphere grid127

The FV core is a hydrostatic, 3D dynamical core using the vertically-Lagrangian dis-128

cretization of L04 and the horizontal discretization of Lin and Rood (1996, 1997, hence-129

forth LR96 and LR97, respectively), using the cubed-sphere geometry of PL07 and Putman130

(2007). This solver discretizes a hydrostatic atmosphere into a number of vertical layers, each131

of which is then integrated by treating the pressure thickness and potential temperature as132

scalars. Each layer is advanced independently, except that the pressure gradient force is133

computed using the geopotential and the pressure at the interface of each layer (Lin 1997).134

The interface geopotential is the cumulative sum of the thickness of each underlying layer,135

counted from the surface elevation upwards, and the interface pressure is the cumulative136

sum of the pressure thickness of each overlying layer, counted from the constant-pressure137

top of the model domain downward. Vertical transport occurs implicitly from horizontal138

transport along Lagrangian surfaces. The layers are allowed to deform freely during the139

horizontal integration. To prevent the layers from becoming infinitesimally thin, and to ver-140

tically re-distribute mass, momentum, and energy, the layers are periodically remapped to141

a pre-defined Eulerian coordinate system.142

The governing equations in each horizontal layer are the vector-invariant equations:143

⇧�p

⇧t+⌅ · (V�p) = 0144

⇧�p�

⇧t+⌅ · (V�p�) = 0145

146

⇧V

⇧t= �⇥k̂ ⇤V �⌅

�⇥ + ⇤⌅2D

⇥� 1

⌅⌅p

⇤⇤⇤z

147

where the prognostic variables are the hydrostatic pressure thickness �p of a layer bounded148

by two adjacent Lagrangian surfaces, which is proportional to the mass of the layer; the149

potential temperature �; and the vector wind V. Here, k̂ is the vertical unit vector. The150

6

Manyflowsarevortical

Notjustlarge-scaleflows

NASAGoddard

MayItalktoyouabout…diffusion?

• Alldynamicalcoresrequireartificialdiffusion(implicitorexplicit)

toremoveenergycascadingtogridscale

• FV3 hasimplicitdiffusionfrommonotoneadvection,forward-

backwardtimestepping,andverticalremapping

• Exceptforverticalremapping…allexplicitandimplicitdiffusionis

alongtheLagrangian surfaces

MayItalktoyouabout…diffusion?

• C-Dgriddoesnotexplicitlyseedivergence—soexplicitdivergence

dampingisanintrinsicpartofthesolver

• Non-monotoneadvectionisusefulinnonhydrostatic dynamics

Noisecanbecontrolledbyaddingdiffusiontothevorticityfluxes

• Explicitly-dissipatedkineticenergycanbeconvertedtoheat• Divergenceandvorticitydampingcanbecontrolledseparately

• Asimplifiedsecond-orderSmagorinsky damping,withanonlinear

flow-dependentcoefficient,isalsoavailable

Backwardhorizontalpressuregradientforce• ComputedfromNewton’ssecondandthirdlaws,andGreen’sTheorem

• Errorslower,withmuchlessnoise,comparedtotraditionalevaluations

• Purelyhorizontalno along-coordinateprojection

• PGFequalandopposite—3rd law!Momentumconserved

• Curl-freeintheabsenceofdensitygradients• Nonhydrostatic andhydrostaticcomponentscanbecomputedseparately

• log(phyd)PGFmoreaccurate

Backwardhorizontalpressuregradientforce• ComputedfromNewton’ssecondandthirdlaws,andGreen’sTheorem

• Errorslower,withmuchlessnoise,comparedtotraditionalevaluations

• Purelyhorizontalno along-coordinateprojection

• PGFequalandopposite—3rd law!Momentumconserved

• Curl-freeintheabsenceofdensitygradients• Nonhydrostatic andhydrostaticcomponentscanbecomputedseparately

• log(phyd)PGFmoreaccurate

Backwardhorizontalpressuregradientforce• ComputedfromNewton’ssecondandthirdlaws,andGreen’sTheorem

• Errorslower,withmuchlessnoise,comparedtotraditionalevaluations

• Purelyhorizontalno along-coordinateprojection

• PGFequalandopposite—3rd law!Momentumconserved

• Curl-freeintheabsenceofdensitygradients• Nonhydrostatic andhydrostaticcomponentscanbecomputedseparately

• log(phyd)PGFmoreaccurate

Verticalprocesses:VerticalelastictermsandLagrangian verticalcoordinate

Semi-implicitsolver

• Verticalpressuregradientandlayerdepthδzissolvedbysemi-

implicitsolver

• Vertically-propagatingsoundwavesweaklydamped

• ThisisallthatisneededtomaketheclassicFVhydrostaticalgorithm

nonhydrostatic

• Fullycompressibleandnonhydrostatic!FullEulerequationssolved

• w,δzadvected asothervariables—consistent!

• Nonhydrostatic horizontalPGFevaluatedsamewayashydrostatic

TheLagrangian VerticalCoordinate

• Thedomainisseparatedintoanumberofquasi-horizontalLagrangian

layers (kindex)

• Allflowiswithinthelayers(“logically”horizontal)• No cross-layerlayerflowordiffusionVerticalmotiondeformsthelayersinstead

• Periodically,ahighly-accurateconservativeremappingisdoneto

avoidlayersbecominginfinitismally thin(δp→0)

• RemappingintervallikeLagrangian advection’stimestep:

longertimesteps yieldlessoverallartificialdiffusion

• Thisistheonly waycross-layerdiffusionisintroduced!!

TheLagrangian VerticalCoordinate

• Themassδpandheightδzareprognosticvariables.Heightandpressureofeachlayeraredynamicallycomputed.

• Lagrangian coordinateworksforany basecoordinate• Hybrid-pressure,hybrid-height,andhybrid-isentropiccoordinateshavebeensuccessfullyused

• Verticaladvectionisimplicitthroughtheverticalmovementoflayers

• ThereisnoCourantnumberrestrictionortime-splitting!

• Verticaladvectiondoesnotneedaseparatecomputation.Computingδpandδzissufficient.

• Implicitadvectionnotonlysavestimebutalsopermitsverythinlayerswithoutrequiringasmallertimestep

TheCubed-SphereGridThe3inFV3

TheCubed-SphereGrid

• Gnomoniccubed-spheregrid:

coordinatesaregreatcircles

• Widestcellonly√2widerthan

narrowest

• Moreuniformthanconformal,elliptic,

orspring-dynamicscubedspheres

• Tradeoff:coordinateisnon-orthogonal,andspecialhandlingneedstobedoneat

theedgesandcorners.

�

D-grid winds

C-grid winds

Fluxes

Fig. 2. Geometry of the wind staggerings and fluxes for a cell on a non-orthogonal grid.The angle � is that between the covariant and contravariant components; in orthogonalcoordinates � = ⇥/2.

45

TheCubed-SphereGrid

• Gnomoniccubed-sphereisnon-

orthogonal

• Insteadofusingnumerousmetricterms,

usecovariantandcontravariantwinds

• Solutionwindsarecovariant,advectionisbycontravariantwinds

• KEishalftheproductofthetwo• Windsuandvaredefinedinthelocal

coordinate:rotationneededtogetzonal

andmeridionalcomponents

�

D-grid winds

C-grid winds

Fluxes

Fig. 2. Geometry of the wind staggerings and fluxes for a cell on a non-orthogonal grid.The angle � is that between the covariant and contravariant components; in orthogonalcoordinates � = ⇥/2.

45

Alittlebitaboutinitialization

FV3 hasahostofutilitiestogenerategrids,orography,andinitial

conditionsfromcoldstart

• Anygridcanbegeneratedinseconds• InitializefromNCEPorECanalyses

• Remapfromdifferentverticalspacings

• Comprehensivetopographygeneration,includingsubgrid orography

• AdvancedFCTorographyfilterallowspreservationoftotaltopography,peaks,andvalleys;limitsslopesteepness;andpreventsnonzerotopographyinthe

ocean

• Atmosphericnudgingtoanalysesforsimpleassimilatedinitialization,orfor

aerosolandchemistrystudies

EssentialRuntimeOptions• dt_atmos:physicstimestep

• k_split:Numberofverticalremappings perdt_atmos• n_split:Numberof“acoustic”timesteps perk_split• hord_xx:Horizonal advectionalgorithms

• kord_yy:Verticalremappingoptions

• d4_bg,vtdm4:explicitdampingoptionsfordivergenceandfor

fluxes(exceptscalars),oforder2*(nord+1)• dddmp:Smagorinsky dampingcoefficient

• d_con:Convertsexplicitly-dampedKEtoheat

Gridrefinement(preview)FV3 supportsbothstretchingandnestingforgridrefinement

Gridstretchingissimpleandsmooth Gridnestingisefficientandflexible

• FV3isabletomimicmanyphysicalproperties,particularlyNewton’s

laws,massconservation,andexcellentvorticitydynamics

• Lagrangian dynamicsisverypowerful!

• Increasedparallelism• Implicitverticaladvection,withoutcomputation

• Muchreducedimplicitverticaldiffusion

• ImprovesPBLandsurfaceinteraction

• Solverisfullycompressible,andsohorizontallylocal

• Fullynonhydrostatic solvermaintainsexcellenthydrostaticflowwhile

consistentlyimplementingnonhydrostatic elasticterms

• Flexibleadvection,diffusion,andgridstructureoptions