Recent advances in DA at NCEP

“Where America’s Climate, Weather, Ocean and Space Weather Services Begin”

December 8, 1016

Presented by John Derber

National Centers for Environmental Prediction

Recent Highlight

• Upgrade of global DA system (12 May 2016)

– 4D-hybrid en-var system

– Use of All-Sky AMSU-A radiances

– Upgrade of CRTM

– Bias correction of Aircraft

– Additional observations

– Some model changes

2

Future

• Improved forecast model– FV3 implementation in global/regional

• Coupled Data Assimilation• Improved techniques in assimilation system.

– Improved representation of model error using ensembles– Improved balance in initial state (especially with moisture variables)

• JEDI framework – long term direction• Improved use of observing system

– Addition of new data sources• GOES-R• JPSS-1• Additional aircraft observations• COSMIC-2• Etc.

– Observational handling and database– Forward models for observations

3

Talk focus

• Lots of presentations on techniques here. Fewer on

observations, so I will focus more on observations, but all

aspects important.

• All types of observational data can be used better.

• Details, Details, Details.

4

JEDI

• Joint Effort for Data assimilation Integration

(JEDI)

• Similar to idea of ECMWF OOPS system.

• Longer term direction – but parts will be included as they

are ready

• Phase I

– Unified forward operators

– Observational data base

5

Joint Effort for Data assimilation Integration(JEDI)

STRATEGY1. Collective path to unification, while allowing multiple levels of engagement

2. Modular, Object-Oriented code for flexibility, robustness and optimization

3. Mutualize model-agnostic components across

• Applications (atmosphere, ocean, land, aerosols, strongly coupled, etc.)

• Models & Grids (operational/research, regional/global models)

• Observations (past, current and future)

OBJECTIVES1. Facilitate innovative developments to address DA grand challenges

2. Increase R2O transition rate from community

3. Increase science productivity and code performance

JEDI + Academia

Obs. Pre-processor• Reading

• Data selection

• Basic QC

Solver• Variational/EnKF

• Hybrid

Background &

Background Error

Observations

Model

Unified

Forward

Operator

(UFO)

• Model Initial Conditions

DATA ASSIMILATION COMPONENTS for Atmosphere, Ocean, Waves, Sea-ice,

Land, Aerosols, Chemistry, Hydrology,

Ionosphere

DATA ASSIMILATION COMPONENTS for Atmosphere, Ocean, Waves, Sea-ice,

Land, Aerosols, Chemistry, Hydrology,

Ionosphere

Analysis Increments

Read

Model

Interpolate

ObserverCRTM, Bias Correction,

QC, Cloud Detection, etc.

Write

Obs. TypeObs.

Locations

(4D)

Model(s)

NEMS /

ESMF

Couple

r (model values @obs. locations)

[Jacobian, Revised QC, Obs. Error, Bias, …]H(xk)

Model

Options

Observer

Options

Model / Obs. Type

MatchingLook-up table

locstreams

Unified

Forward

Operator

NEMS/ESMF

Atm Dycore(TBD)

Wave(WW3/SWAN)

Sea Ice

(CICE/SIS2/KISS)

Aerosols(GOCART)

Ocean(HYCOM/MOM)

Land Surface(NOAH)

Atm Physics(GFS)

Atm DA(GSI)

(model equivalent)

(observations)

Observational database

• Preprocessing - everything that happens to data before it gets to the DA system

• Communications

– Volume issues with some data sources

• Metadata

• Station history for monitoring and quality control

• Restricted data

• Transition to BUFR for conventional observations

• Radiation correction with Radiosondes

• Specification of observation error (instrument and representativeness)

– Preliminary quality control – reject lists

– Situational dependence

– Correlated error

9

Transition to BUFR reporting

• Ingleby, B., P. Pauley, A. Kats, J. Ator, D. Keyser, A. Doerenbecher, E. Fucile, J. Hasegawa, E. Toyoda, T. Kleinert, W. Qu, J. St James, W. Tennant, and R. Weedon, 2016: Progress towards high-resolution, real-time radiosonde reports. Bull. Amer. Meteor. Soc. doi:10.1175/BAMS-D-15-00169.1, in press.

• In long term, more accurate – complete observations, but many difficulties.

– Inclusion of balloon drift, station location, all levels in one report.

– Change in paradigm from significant levels to frequent reports.

• NCEP behind some centers in using BUFR data.

10

Transition to BUFR reporting

11

500hPa Height

Transition to BUFR reporting

12

Black – standard levels

Blue – significant levels

Red – BUFR levels

14 German stations

using Vaisala RS92

Rawinsonde bias correction

• More automation and bias corrections by the

instrument producers make the need for bias

correction less.

• However, many radiosondes still need bias correction.

• The NCEP bias correction tables have not been

updated for many years. Many new types.

• Project underway to use collocated GPS retrievals of

temperature (above 100hPa) to create new bias

correction.

13

Rawinsonde bias correction

14

Quality control & data monitoring.

• Improved non-linear (variational) quality control techniques in analysis can reduce impact of bad observations.

• Observations rejected by setting observation error to ∞, non-linear QC will down-weight observations

• Still need better monitoring (and feedback to source) to ensure that known bad observations are not used and eventually corrected.

• Improved reporting techniques can eventually reduce the number of bad observations.

• Have to do in way that allows input from desk meteorologist, does not add risk to system, and still allows needed flexibility.

15

Observational error specification

• Moving towards station by station and situation dependent observation error specification

• Separate instrument and representativeness error (including bias)

• Representativeness error will be modeled and dependent on forecast model/resolution/etc.

• Inclusion of correlated error

– Or using correlated error to chose what data to use.

– Impact on convergence is an issue.

– Situation dependent correlated error.

16

Improvements to observation error

specification and bias correction

• Increased granularity of observational errors

– Initially by instrument type

– Eventually observation by observation

• Separate modeling of instrument and

representativeness error

• Inclusion of correlated errors for satellite radiances

(GMAO, Bathmann and Collard)

– Reduction in specified observational error variance

– Regularization of covariance matrix

Observation error correlation matrices for AIRS over sea, before (left) and after reconditioning R (right). R is reconditioned by first setting the smallest eigenvalues equal to λmax/K1 and then inflating the diagonal. Here K1=150.

Observation error correlation Matrices for AIRS over land, before (left) and after reconditioning R (right). R is reconditioned by first setting the smallest eigenvalues equal to λmax/K1 and then inflating the diagonal. Here K1=150.

Observation errors for AIRS over sea, before and after reconditioning R. R is reconditioned by first setting the smallest eigenvalues equal to λmax/K1 and then inflating the diagonal. Here K1=150.

The cost function (left) and log of the gradient (right) during minimization. Green data points represent the result of using full R for AIRS and IASI globally, while black data points represent the result of using diagonal R.

Analysis increment RMS differences after using a full R globally for AIRS and IASI in a 2 month parallel GFS experiment

Temperature fit to observations (O minus F) for the 2 month experiment. Dotted lines indicate full R results,

while solid lines indicate the results of using a diagonal R.

Humidity fit to observations (O minus F) for the 2 month experiment. Dotted lines indicate full R

results, while solid lines indicate the results of using a diagonal R.

Fit to a passive IASI water vapor channel, in the 2 month GFS experiment

The 500 mb geopotential height anomaly correlation in the northern

hemisphere (left), southern hemisphere (right) after using a full R

globally for AIRS and IASI in a 2 month parallel GFS experiment.

Forward models

• Transforms analysis or model variables to the observations.

• Satellite observations

– Radiative transfer - RTTOV/CRTM – brightness temperatures/radiances

• Clouds and precipitation

• Surface emissivity

• Atmospheric composition (gases and aerosols)

• FOV size and path

– GPS-RO – bending angles

27

Surface emissivity issues under

scattering conditions – reflection of diffuse radiation and restricting to < 60 degrees

CRTM CRTM

RTTOV

Original Work-around

AMSU-A

Channel 3 Observation minus First-Guess

Forward model

• Conventional observations

– Wind components to radial winds

– Near surface effects on observations

– Model layers vs. point observations

• Improvements in the forward model should allow

reduction in representativeness error (σ and/or

bias)

29

Final comments

• All parts of a data assimilation system are important –all aspects can be improved.

• Our DA system is being redesigned based on JEDI concept to allow the increased use of information in the observations.

• Data handling, use of metadata, improved quality control and improved specification of observational errors in addition to development of better forward models for all observation types are essential to improve the extraction of information from observations.

30