verification of daily cfs forecasts huug van den dool & suranjana saha cfs was designed as...
TRANSCRIPT
![Page 1: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/1.jpg)
Verification of Daily CFS forecastsHuug van den Dool & Suranjana Saha
• CFS was designed as ‘seasonal’ prediction system• Hindcasts 1981-2005, 15 ‘members’ per month• Here we look at CFS(T62L64) as NWP (never mind the delayed
ocean analysis)
• 25 years of forecasts (4500 forecasts out to 9 months) by a ‘constant’ T62L64 model !!!!!.
website http://cfs.ncep.noaa.gov/• Reference: S. Saha, S. Nadiga, C. Thiaw, J. Wang, W. Wang, Q. Zhang, H. M. van den Dool, H.-L. Pan,
S. Moorthi, D. Behringer, D. Stokes, M. Pena, S. Lord, G. White, W. Ebisuzaki, P. Peng, P. Xie , 2006 :
The NCEP Climate Forecast System. Journal of Climate, Vol. 19, No. 15, pages 3483.3517 • Today: No ensemble averages, hardly time averages• One could compare to similarly large data sets, such as: CDAS & CDC’s MRF &
ECMWF ‘monthly’ system
Acknowledgement: Cathy Thiaw (reruns) and on CDAS info Bob Kistler/Fanglin Yang/Pete Caplan
![Page 2: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/2.jpg)
Issues• Day-5 scores• Climatology of scores• Bias Correction (mean, sd)• NH/SH/TR; Z500, PSI200, CHI200• wk3, wk4, MJO• Lorenzian Error growth equation• The perfect forecast• Scores as function of a) space, b) EOF mode• Steps to improve signal to noise ratio• FUTURE (CFSRR)
![Page 3: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/3.jpg)
NCEP’s NEW CFS Components for S/I Climate
• T62/64-layer version of the ~2003 NCEP atmospheric GFS (Global Forecast System) model– Model top 0.2 mb– Simplified Arakawa-Schubert convection (Pan)– Non-local PBL (Pan & Hong)– SW radiation (Chou, modifications by Y. Hou)– Prognostic cloud water (Moorthi, Hou & Zhao)– LW radiation (GFDL, AER in operational wx model)– R2 Initial conditions for Atmosphere&Land
• GFDL Modular Ocean Model, version 3 (MOM-3)– 40 levels– 1 degree resolution, 1/3 degree on equator
• Global Ocean Data Assimilation (GODAS)• Fully coupled atmosphere-ocean (no flux correction)
![Page 4: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/4.jpg)
Day 5 AC-scores, using the harmonically smoothed model and observed climatologies
(which are more ‘competitive’ than the old-old-old monthly climo used on Pete Caplan’s page
Variables: Extra-tropical Z500 (NH, SH)
PSI200 and CHI200
![Page 5: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/5.jpg)
[Hrs]
ForecastLead time
6480
o o o o oo o o o o
Time
[Days]30 31 1 2 3 9 10 11 12 13 19 20 21 22 23
168144120
96724824
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o oo o o o o
o o o o o
![Page 6: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/6.jpg)
Fig. 2. The climatological annual cycle of the mean value of the geopotential height at a grid point close to Washington, DC, is shown as a smooth red curve. Shown are both the observed climatology (left panel) and the forecast climatology at a lead of 6480 hours (~ 9 months) (right panel). The 24-yr mean values as calculated directly from the data are shown by the blue curves. The reason for the discontinuities in the right panel is that these values are only available during roughly half of the days in a year. The time on the abscissa in the right panel is the initial time of the forecast, not the actual time when the forecast is valid. In this case this implies an offset by roughly nine months. Unit is m. ACKNOWLEDGEMENT: Ake Johansson
![Page 7: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/7.jpg)
Day 5 AC-scores, using the harmonically smoothed model and observed climatologies
(which are more ‘competitive’ than the old-old-old monthly climo used on Pete Caplan’s page
Variables: Extra-tropical Z500 (NH, SH)
PSI200 and CHI200
![Page 8: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/8.jpg)
![Page 9: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/9.jpg)
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresZ500 - NH 1981-2005
Z500: 72.3 (4500 cases, grand mean)Congratulations with a constant system!
180 forecasts per dot
![Page 10: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/10.jpg)
0.7
0.7
Grand Mean NH over 1984-2005: 70.5 (CDAS1)
((CDAS statistics has several problems))
From Pete Caplan’s EMC website:
![Page 11: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/11.jpg)
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresZ500 - NH 1981-2005 (CDAS)
CDAS
![Page 12: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/12.jpg)
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresZ500 - NH 1981-2005
CFS
![Page 13: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/13.jpg)
Next Two Slides: Excursion to SH Z500
![Page 14: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/14.jpg)
Z500: 62.9 (4500 cases, grand mean)
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresZ500 - SH 1981-2005
Scores go UP and UP! “Congratulations” to whom ???
SH (CFS) Scores ‘volatile’ in SH (still 180 per dot)
![Page 15: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/15.jpg)
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresZ500 - SH 1981-2005 (CDAS)
SH (CDAS)
![Page 16: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/16.jpg)
Doing the best we can:Comparing CFS to CDAS day 5 Z500 scores)
NH SH
CFS 1981-2005
T62L64, coupled, R2, harmonic daily climo
0.723 0.629
CDAS 1984-2005
T62L28, R1, ‘old’ monthly climo
0.705 0.623
Warning: Number of forecasts per month differ. Climos differ!!! and change in ’96 for CDAS
Flawed
![Page 17: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/17.jpg)
Prelim Conclusion
• R2/CDAS2/CFS 5-day forecasts are (slightly) better than R1/CDAS.
• NH much better than SH (typical for pre-2000 technology) – this will change largely in next CFSRR (Jan2010??)
• Trend-issues and non-constancy of system (will get worse)
![Page 18: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/18.jpg)
Next Two Slides: Excursion to TROPICS
![Page 19: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/19.jpg)
PSI200: 63.0 (4500 cases grand mean)
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresPSI200 - TROPICS 1981-2005
No increase???
180 forecasts per dot
ENSO?
![Page 20: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/20.jpg)
CHI200: 45.9 (4500 cases grand mean)This may be lowish for MJO type prediction.
30
40
50
60
70
80
Anom
aly
Corr
ela
tion
1981198319851987198919911993199519971999200120032005YEAR
Annual Mean CFS day 5 scoresCHI200 - TROPICS 1981-2005
Definitely no increase over time. Perhaps a decrease!!!
180 forecasts per dot
ENSO?
![Page 21: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/21.jpg)
Climatological Annual Cycle of day-5 scores
![Page 22: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/22.jpg)
60
70
80A
nom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthZ500 - NH 1981-2005
50
60
70A
nom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthZ500 - SH 1981-2005
Forecasts verifying in month shown
NH: Best month is Feb (76.4), Worst month is July (67.7). Near sinusoidal variation. Range=8.7pnts
375 forecasts per month
SH: Best month is Aug (64.9), Worst month is March (59.0). Typically 62-64, except Feb-Apr. Range=5.9pnts
Curious: March is best/worst in NH/SH. Aug is best/worst in SH/NH.
NH
SH
Should we be surprised that the winter-summer difference in NH is NOT larger!??
![Page 23: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/23.jpg)
The NH has an annual cycle in skill, every year.
(Each color curve is a different year.)
The SH does NOT have a clear annual variation in skill each year.
15 forecasts per month
55
65
75
85
Anom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthZ500 - NH 1981-2005
45
55
65
75
Anom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthZ500 - SH 1981-2005
![Page 24: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/24.jpg)
60
70
80A
nom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthZ500 - NH 1981-2005
50
60
70
Anom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthZ500 - SH 1981-2005
55
65
75
Anom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthPSI200 - TROPICS 1981-2005
35
45
55
Anom
aly
Correla
tion
1 2 3 4 5 6 7 8 9 10 11 12Month
Climo Day 5 Scores by MonthCHI200 - TROPICS 1981-2005
forecasts verifying in month shown
375 forecast per dot(month) Large annual variation in the tropics! But volatile.
![Page 25: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/25.jpg)
How about Bias Correction???One of the claimed usages of
hindcasts
![Page 26: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/26.jpg)
0
20
40
60
80
100
Anom
aly
Corr
ela
tion
0 2 4 6 8 10 12 14forecast day
raw bias corrected
Daily CFS ScoresZ500 1981-2005 NH July
Indisputable but very small improvement.
Is Z500 incorrigeable?
Based on 375 forecasts.
The looks of a die-off curve
![Page 27: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/27.jpg)
The gain due to bias correction in a few selected months. Day 5 scores Z500 1981-2005
Raw Bias Corrected
NH DEC 73.0 74.5 (+1.5)
NH JUL 65.8 68.3 (+2.5)
SH FEB 61.8 63.5 (+1.7)
SH AUG 62.2 63.7 (+1.5)
![Page 28: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/28.jpg)
Is Z500 incorrigeable? Largely Yes, because the systematic
error is small.Wait till you see CHI200
![Page 29: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/29.jpg)
0
20
40
60
80
100
Anom
aly
Corr
ela
tion
0 2 4 6 8 10 12 14forecast day
raw bias corrected
Daily CFS ScoresCHI200 1981-2005 TR December
Chi200 improves tremendously from cleaning up the bias, especially early on.
375 forecasts
Can we do MJO forecasts ??
![Page 30: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/30.jpg)
0
20
40
60
80
100
Anom
aly
Corr
ela
tion
0 2 4 6 8 10 12 14forecast day
raw bias corrected
Daily CFS ScoresPSI200 1981-2005 TR February
PSI200 in Tropics does not improve very much from bias removal. !!!!
The looks of a die-off curve
![Page 31: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/31.jpg)
Day 5 scores
• AC (obviously; already shown),
But also:
• SD
• eDOF
![Page 32: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/32.jpg)
60
70
80
90
100
110
120sta
ndard
devia
tion (
gpm
)
d j f m a m j j a s o n d j
sd (day 5)sd (OBS)
CFS daily scores 1981-2005 Z500 NH
CFS misses 5-20 % of variability
Loss of variance does not increase beyond 10-15 days and is never more than 20%
![Page 33: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/33.jpg)
26
28
30
32
34
36
38
40
42
44effective d
egre
es o
f fr
eedom
d j f m a m j j a s o n d j
deg of freedom (OBS)degrees of freedom (day 5)
CFS daily scores 1981-2005 Z500 NH
CFS misses several degrees of freedom
Loss of dof does not increase beyond 10-15 days and is never more than shown above
![Page 34: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/34.jpg)
60
70
80
90
100
110
120sta
ndard
devia
tion (
gpm
)
d j f m a m j j a s o n d j
sd (day 5)sd (OBS)
CFS daily scores 1981-2005 Z500 NH
60
70
80
90
100
110
120
sta
ndard
devia
tion (
gpm
)
d j f m a m j j a s o n d j
sd (day 5)sd (OBS)
CFS daily scores 1981-2005 Z500 SH
Compare the SD reduction in the
NH
To that in the
SH
![Page 35: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/35.jpg)
26
28
30
32
34
36
38
40
42
44effective d
egre
es o
f fr
eedom
d j f m a m j j a s o n d j
deg of freedom (OBS)degrees of freedom (day 5)
CFS daily scores 1981-2005 Z500 NH
26
28
30
32
34
36
effective d
egre
es o
f fr
eedom
d j f m a m j j a s o n d j
deg of freedom (OBS)degrees of freedom (day 5)
CFS daily scores 1981-2005 Z500 SH
Compare the DOF reduction in the
NH
To that in the
SH
![Page 36: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/36.jpg)
64
66
68
70
72
74
76
78
26
28
30
32
34
36
38
40
0 1 2 3 4 5 6 7 8 9 10111213
AC-NH at day 5 degrees of freedom
CFS daily scores 1981-2005 Z500 NH
The climatological AC variation can be explained by eDOF and sd: low dof > high score, high sd > high score.
![Page 37: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/37.jpg)
Distribution of Skill in space is important
and largely unexplored and unexplained
(because we never had enough data)
![Page 38: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/38.jpg)
![Page 39: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/39.jpg)
![Page 40: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/40.jpg)
![Page 41: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/41.jpg)
![Page 42: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/42.jpg)
Skill as a function of (EOF) mode is interesting
![Page 43: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/43.jpg)
-20
0
20
40
60
80
100
Anom
aly
Corr
ela
tion
-2
0
2
4
6
8
10
EV
by M
ode (
%)
0 10 20 30 40 50 60 70 80 90100EOF mode #
skill_day-5 EV
CFS Skill in Daily Z500 predictionFebruary 1981-2005 NH
![Page 44: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/44.jpg)
-20
0
20
40
60
80
100
Anom
aly
Corr
ela
tion
-2
0
2
4
6
8
10
EV
by M
ode (
%)
0 10 20 30 40 50 60 70 80 90100EOF mode #
skill_day-5 EV skill_day-10
CFS Skill in Daily Z500 predictionFebruary 1981-2005 NH
![Page 45: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/45.jpg)
-200
20406080
100
Anom
aly
Corr
ela
tion
-20246810
EV
by M
ode (
%)
0 10 20 30 40 50 60 70 80 90100EOF mode #
skill_day-5 EV skill_day-10
skill_day15 skill_day28
CFS Skill in Daily Z500 predictionFebruary 1981-2005 NH
![Page 46: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/46.jpg)
-200
20406080
100
Anom
aly
Corr
ela
tion
-20246810
EV
by M
ode (
%)
0 10 20 30 40 50 60 70 80 90100EOF mode #
skill_day-5 EV
Regular AC persistence
persistence's regular AC
CFS Skill in Daily Z500 predictionFebruary 1981-2005 NH
![Page 47: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/47.jpg)
Now: OUT TO 270 DAYS !!
![Page 48: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/48.jpg)
Die-off curve, Z500, NH, 1981-2005, 4500 forecasts, all seasons aggregated
0
20
40
60
80
100
0 15 30 45 60 75 90105120135150165180195210225240255270
0
20
40
60
80
100
0 5 10 15 20 25 30-0.5
0
0.5
1
1.5
30 45 60 75 90105120135150165180195210225240255270
Detail, first month.
Detail: Month 2 to Month 8
NH
![Page 49: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/49.jpg)
Die-off curve, Z500, SH, 1981-2005, 4500 forecasts, all seasons aggregated
0
20
40
60
80
100
0 15 30 45 60 75 90105120135150165180195210225240255270
0
20
40
60
80
100
0 5 10 15 20 25 300
0.5
1
1.5
2
30 45 60 75 90 105120135150165180195210225240255270
SH
![Page 50: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/50.jpg)
FOCUS• Week 3
• Week 4
• = days 15-21 and 22-28
• Physical basis of wk3/wk4
• Ocean interaction is as much a liability as a promise at this point in time.
![Page 51: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/51.jpg)
CFS Pers CFS Pers
NH 15.2 4.2 6.1 3.9
TR 36.2 29.3 29.6 27.5
SH 9.6 6.2 2.0 3.4
DAY 15 DAY 28
Anomaly correlation of CFS and ‘Persistence’ Z500 prediction in Feb 1981-2005
![Page 52: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/52.jpg)
signal
______________________ ratio
NOISE
Improving signal to noise ratio: (often cosmetic) by reducing noise by applying some operator and, hopefully, not hurting the signal by this operator
1) Take a time mean (7 days)
2) Ensemble mean (not done here)
3) EOF filters (or better (maximally predictable modes))
![Page 53: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/53.jpg)
Day 15 Wk3 Wk4 Day 28
NH 15.2 15.7 9.2 6.1
TR 36.2 46.7 42.9 29.6
SH 9.6 10.6 5.2 2.0
Anomaly correlation of CFS Z500 prediction, daily as well as weekly, in Feb 1981-2005
Effect of a 7 day mean………
![Page 54: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/54.jpg)
Wk3 Wk3
EOF1
WK3EOF1-10
WK4 WK4
EOF1
WK4EOF1-10
15.7 22.0 18.7 9.2 23.9 12.0
Effect of EOF truncation……
So, overall, we went from daily scores (15.2….6.1) to weekly scores (15.7 and 9.2) to filtered weekly scores (22.0 and 23.9 at best)
![Page 55: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/55.jpg)
15.7
![Page 56: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/56.jpg)
9.2
![Page 57: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/57.jpg)
From Wanqiu Wang’s page.
Constructed Analogue
Longer time average, and N member ensemble average helps.
![Page 58: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/58.jpg)
020406080
100120140160
RM
SE
(gpm
)
0 1 2 3 4 5 6 7 8 910111213141516lead
Savijarvi Eq(5)
CFS-jan
RMSE growth CFS NHNOSEC
Initial error=6.1 gpm ; Systematic error growth s=13.5 gpm/day ; Small error amplification a=0.181/day ;
e-infinity=157.7gpm ; NOSEC
![Page 59: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/59.jpg)
Savijarvi’s equation 5:
Lorenz Tellus 1982
![Page 60: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/60.jpg)
020406080
100120140160
RM
SE
(gpm
)
0 1 2 3 4 5 6 7 8 910111213141516lead
Savijarvi Eq(5)
CFS-jan
RMSE growth CFS NHSEC
Initial error=4.7 gpm ; Systematic error growth s=13.1 gpm/day ; Small error amplification a=0.181/day ;
e-infinity=155.7gpm ; SEC
![Page 61: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/61.jpg)
020406080
100120140160
RM
SE
(gpm
)
0 1 2 3 4 5 6 7 8 910111213141516lead
Savijarvi Eq(5)
CFS-jan
RMSE growth CFS NHSECSD
Initial error=4.7 gpm ; Systematic error growth s=12.6 gpm/day ; Small error amplification a=0.200/day ;
e-infinity=158.3gpm ; SECSD
![Page 62: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/62.jpg)
15
25
35
45
55
65
75
-3
-2
-1
0
1
2
3
0 1 2 3 4
raw sec secsd sec-secsd
RMSE growth CFS NH 'jan' (Z500)
![Page 63: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/63.jpg)
![Page 64: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/64.jpg)
Among 4500 CFS daily forecasts:
• Was the RMSE at day 4 ever smaller than at day 3????? (Model vs Analysis)
• Yes, in one single case: the forecast from Dec, 31, 2003. Raise the champagne???
![Page 65: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/65.jpg)
![Page 66: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/66.jpg)
![Page 67: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/67.jpg)
1 2 3 4 5 6 9
AC 98.8 98.6 97.9 98.5 97.0 91.6
RMSE 17.1 20.2 24.6 20.5 27.4 45.2 87.9
Jan 1
Jan 5
An exceptionally “good” forecast, especially for T62 (CDAS2).
The best forecast (for day 5) ever seen in this building, by any model
Flow Situation is NOT particularly persistent
Forecasts from Dec 30 or Jan 1, 2 are not that good. Only Dec 31, 0Z.
Other models not that good for this case
Too good to be true? The date raises some suspicion
![Page 68: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/68.jpg)
Due to Fanglin Yang
![Page 69: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/69.jpg)
Another piece of evidence:
• Scores in the SH for 2003123100 were also exceptionally good (0.92, 2nd highest is 0.77).
• It is unlikely that both hemispheres produce ‘hit-the-jackpot’ type forecasts.
• Could something be the matter with the R2 analyses.
![Page 70: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/70.jpg)
What could be wrong in the (R2 or any) analyses verifying the
CFS forecast from 31 Dec, 0Z??• Drunken observers around the planet in a
wave-like pattern (New Years Eve distracts!
• Satellites flying in and out of the year (in local time), i.e. time problems. Indeed much satellite data is rejected (for days in early January), sometimes ‘wholesale’.
• But can this derail the analysis enough to explain a ‘very good’ forecast?
![Page 71: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/71.jpg)
ABOUT THE FUTURE
![Page 72: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/72.jpg)
Reanalysis NH SH
OPR 85 80
CFSRR-lite 74 72
CFS (R2) 72 63
R1 71 62
5 day forecast anomaly correlation
Annual Mean
![Page 73: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/73.jpg)
Conclusions• CFS is (slightly) better than CDAS (Z500).• Over time (1981-2005), the CFS “system”
appears quite constant with various qualifiers• If we need an as-constant-as possible in-house
system, look no further than CFS (better than CDAS)
• Bias correction: small +ve impact on Z500 and Ψ200 (1-2pnts)
• Bias correction: huge +ve impact on CHI-200 (15 pnts), but this is worrisome
• CFS loses 5-20% in terms of SD and eDOF in the first few days, then, admirably, stays nearly constant out to 270 days
![Page 74: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/74.jpg)
More conclusions
• Scores in SH/TR more volatile than in NH. • SH lacks a proven annual cycle in scores• Spatial distribution of scores not easy to understand• Chi200 scores at day 1 point to serious problems in R2
and ‘consistency’• Scores as a function of EOF non-surprising• Error growth equation fits (Lorenz/Savijarvi) indicates
large (mainly random) error growth due to systematic model error. Surprising. Needs more study. The good news about it is …..
• We have one perfect forecast.
![Page 75: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/75.jpg)
More Conclusions
• (Very) modest skill in wk3 and wk4
• Even with 2 of 3 steps for signal to noise improvements in place, the AC is only 0.20-0.25. (No ensemble average here)
• Waiting for the next CFS and CFSRR (higher Res, consistent IC)
![Page 76: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/76.jpg)
Conclusions:
• The day 1 – 3 forecasts appear to be too damped, and damp faster than a regression would. Increasing anomaly amplitude as a postprocessor (undoing the sd decrease) actually improves the rms error early on. A curiosity?
• Probably: initial conditions are damped as well.
![Page 77: Verification of Daily CFS forecasts Huug van den Dool & Suranjana Saha CFS was designed as ‘seasonal’ prediction system Hindcasts 1981-2005, 15 ‘members’](https://reader036.vdocuments.us/reader036/viewer/2022070414/5697c0281a28abf838cd6938/html5/thumbnails/77.jpg)
Example:
Systematic errors for mid-January at day 5.