satellite analysis branch 2008 year in review anthony e. salemi – satellite analysis branch (sab)...

Satellite Analysis Branch Satellite Analysis Branch 2008 Year in Review2008 Year in Review

Anthony E. Salemi – Satellite Analysis Branch (SAB)

Gregory Gallina, Keith Liddick, Alan Schwartz, Matthew Seybold, Michelle Spampata, and Michael Turk

METSAT Conference - 28 April 2009

Milestones• Dec 2007 – SSD Management affirms findings of

ATET—ADT unsuitable replacement for SAB subjective estimates

• Dec 2007 – Bulletin format changed to reflect that of JTWC and CPHC

• Summer 2008 – SAB disseminates near RT microwave (MW) fixes as experimental product

• Fall 2008 – SAB incorporates MW fixes in text bulletins for Eastern and Southern Hemispheres

• Jan 2009 – SAB offers MW analysis on case-by-case basis to JTWC, TCWCs and RSMCs by request

• Mar 2009 – SAB providing Dvorak and MW fixes in ATCF format to TPC – Awaiting Testing by TPC

Milestones: Microwave FixesMilestones: Microwave Fixeshttp://www.ssd.noaa.gov/PS/TROP/2008/midata/archive.html

Sample MW History FileSample MW History File

Milestones: New Bulletin FormatMilestones: New Bulletin Format

WWPN20 KNES 160945 A. 25W (HAISHEN) B. 16/0857Z C. 27.5N D. 153.5E E. THREE/MTSAT F. T3.0/3.0/D1.0/24HRS G. IR/EIR/VIS/AMSU/SSMIS/WINDSAT H. REMARKS...CONVECTION MEASURES 4/10 RESULTING IN A DT OF

2.5. PT AGREES WHILE MET IS 3.0. FT BASED ON MET...ADT CI OF 2.9 AND 0739Z WINDSAT IMAGE DEPICTING A PARTIAL EYEWALL. ~TURK

I. ADDL POSITIONS 16/0324Z 26.9N 151.3E AMSU 16/0654Z 27.4N 152.4E SSMIS 16/0739Z 27.5N 153.2E WINDSAT

695

6

459

147

449

229

652

612

37

597

147

650

270

781

515

15

481

123

440

192

843

507

146

514

129

518

183

854

659

12

580

109

648

226

815

673

6

527

173

530

137

1054

912

76

463

200

535

247

799

511

112

716

186

544

273

992

594

60

443

281

707

290

832

756

63

624

236

714

177

812

643

53

540

173

574

222

843

0

500

1000

1500

2000

2500

3000

3500N

um

ber

of

Cla

ssif

cati

on

s

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 10-yrAvg

SAB Dvorak Classifications

Atlantic CPAC EPAC NIO SIO SPAC WPACG.Galllina SAB 01/01/09

3207

3382

2637

3094

2609

2851

30493100

32323334

3050

Validation Methodology•Intensity

•Use Atkinson-Holliday wind/pressure relationship

•Ground Truth consists of, in order of importance:

•Recon

•Other in situ observations (ship/surface reports)

•SAB CI number (when validating experimental techniques)

•Ground Truth used to stratify results by intensity

•Position

•Use recon in Atlantic

•Microwave Analysis (within 90 minutes) elsewhere

SAB Performance 1995-2008 As Compared to Recon

35

54

40 26

35

21

46

44

30 33

42

33

22

33

8481

74

8987

6983

86

74

77

68

80

87

78

0

10

20

30

40

50

60

70

80

90

100

95 96 97 98 99 2000 01 02 03 04 05 06 07 08

Year

Per

cen

tag

e

Within 1/4 T-no Within 3/4 T-no

1995-2007 Average: 35% within 1/4 T-no; 80% within 3/4 T-no

M Turk SAB 9/20/08

E Hem Intensity Errors 2004-2008Based on ground truth observations within 3 hrs

1

5

9

26

23

9

7

1 1

0

5

10

15

20

25

30

< -2.0 -2.0 -1.5 -1.0 -0.5 0 0.5 1.0 1.5 2.0 > 2.0

Difference in T-number

Nu

mb

er o

f O

ccu

rren

ces

N=82A Schwartz 3/22/09

32% within 1/4 T-no 71% within 3/4 T-no

NIntensity Parameter % w/in 1/4 T-no % w/in 3/4 T-no

Tropical Depression CI number 3 0.0 100.0Pressure 0.0 100.0

Tropical Storm CI number 31 32.3 67.7Pressure 19.4 54.8

Category1 & 2 CI number 37 32.4 62.2Pressure 18.9 45.9

Category 3+ CI number 11 36.4 100.0Pressure 0.0 36.4

Overall CI number 82 31.7 70.7Pressure 17.1 50.0

SAB-Observed

E Hem Intensity Errors 2004-2008

SAB performance based on Intensity

CIMSS SATCON vs SAB in 2008: A Homogeneous Comparison

Closest observation within 3 hours of each other

0

36

83

28

7

25

45

29

2 1 00220 00

42

0

10

20

30

40

50

60

70

80

90

-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0

Differences in T-no.

Nu

mb

er

of

Occu

rren

ces

CIMSS SATCON - Recon

SAB - Recon

N = 151M.Spampata 12/08

Underestimated intensity Overestimated intensity

CIMSS SATCON 55% accuracy

97% within .7 T-no.

SAB Classifications30% accuracy

77% within .7 T-no.

RSMC ADT Declared Operational Apr ‘09

• RSMC Nadi• TCWC in Perth, Darwin and Brisbane• RSMC La Reunion• TCWC Jakarta & Port Moresby (presumably, but

untested)

Northern Hemisphere• RSMC Tokyo (untested) and New Delhi (work in

progress)

http://www.ssd.noaa.gov/PS/TROP/adt.html***ADT using JTWC forecasts are on same page***

JT ADT Agrees Better with SABRSMC vs JTWC ADT (v7.2.3)

A Homogeneous Comparison of CI Number Differences With SAB for 2008-2009 SH Systems

2

11

3735

39

1

6

10

20

46 46

10

1

3

13

33

0

5

10

15

20

25

30

35

40

45

50

< -2.0 -2.0 -1.5 -1.0 -0.5 0 0.5 1.0 1.5 2.0 > 2.0

Differences (ADT-SAB) in T-number

Nu

mb

er

of

Occ

urr

ence

s

RSMC JTWC

M Turk SAB N=143

RSMC

25% within 1/4 T-num of SAB78% within 3/4 T-num of SAB

JTWC

32% within 1/4 T-num of SAB78% within 3/4 T-num of SAB

RSMC ADT Posits Somewhat BetterRSMC vs JTWC ADT (v7.2.3)

A Homogeneous Comparison Against MW Data for 2008-2009 S Hem Systems

42

10

45

2 2 2

15

30

9

45

7

3

13

23

13

3

2323

0

5

10

15

20

25

30

35

40

45

0s 10s 20s 30s 40s 50s 60s 70s 80s 90s 100+

Position Difference (nmi)

Nu

mb

er

of

Oc

cu

rre

nc

es

RSMC JTWC

M Turk SAB 4/2/09 N=143

RSMC

Average: 31 nmi66% LT 30 nmi (PCN 1)77% LT 40 nmi (PCN 3)

JTWC

Average: 32 nmi57% LT 30 nmi (PCN 1)77% LT 40 nmi (PCN 3)

Results by Intensity%Agreement %Desirable %Agreement %Desirable

CI number 14.3 71.4 0.0 66.7Pressure 23.8 85.7 19.0 85.7

18 Position (nmi)CI number 37.7 91.3 53.6 97.1Pressure 27.5 71.0 33.3 81.2



20 Position (nmi)


119 Position (nmi)

Position differences based on MW analysis NOT SAB position. Since MWdata not always available, N is less than that used in intensity estimates

22.0 32.0

36.8 36.5

16.7 16.5

JTWC ADT-SAB

143

69

27

31.2 32.4

42.1 37.4

21

Overall

26MajorHcn

TropicalStorm

Category1 & 2 Hcn

%Agreement: % of estimates within 1/4 T-no or 2.5 mb of SAB estimate%Desirable: % of estimates within 3/4 T-no or 7.5 mb of SAB estimate

NRSMC ADT-SAB

TropicalDepressi

on

Intensity Parameter

Results by RSMC

%Agreement %Desirable %Agreement %Desirable


31 Position (nmi)



57 Position (nmi)



31 Position (nmi)

%Agreement: % of estimates within 1/4 T-no or 2.5 mb of SAB estimate%Desirable: % of estimates within 3/4 T-no or 7.5 mb of SAB estimatePosition differences based on MW analysis NOT SAB position. Since MWdata not always available, N is less than that used in intensity estimates

JTWC ADT-SAB

Overall40

21.1 20.5

Intensity N ParameterReunion ADT-SAB

ParameterAustralia ADT-SAB JTWC ADT-SAB

Overall66

25.6 33.6

Intensity N ParameterFiji ADT-SAB JTWC ADT-SAB

Overall37

51.7 41.9

Intensity N

NESDIS Enhancements NESDIS Enhancements on Operational Productson Operational Products

Gregory M. Gallina – Satellite Analysis Branch (SAB)

Keith Liddick, Anthony E. Salemi, Alan Schwartz, Matthew Seybold,

Michelle Spampata, and Michael Turk

METSAT Conference - 28 April 2009

Product System Development & Implementation (PSDI) Program

• Operational Implementation of the CIMSS Advanced Dvorak Technique & New Development phase (ADT v8.0)

• Operational Implementation of an Ensemble Tropical Rainfall Technique (eTRaP)

• Global Tropical Cyclone Formation Probability Product

• GOES-Based Multi-Platform Tropical Cyclone Surface Wind Product

• AMSU-Based Tropical Cyclone Intensity and Structure Estimation from Aqua and MetOp

Ike (09L): ADT Unable to Resolve Small EyeIke (09L): ADT Unable to Resolve Small Eye

• Small eye barely discernible in conventional and microwave imagery

• ADT used CDO cloud scene, resulting in falling T-numbers and underestimates in intensity

Ike (09L) Current Intensity Time Series

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

1715

Z18

45Z

0645

Z14

15Z

1715

Z19

15Z

2115

Z23

15Z

0645

Z07

45Z

0945

Z11

15Z

1815

Z20

15Z

1815

Z20

15Z

2115

Z23

15Z

0645

Z07

15Z

0745

Z08

45Z

0945

Z11

15Z

1145

Z19

45Z

2115

Z23

15Z

0315

Z06

45Z

0715

Z08

15Z

0945

Z11

45Z

1345

Z17

15Z

1915

Z21

15Z

2315

Z02

15Z

0645

Z07

15Z

0915

Z11

15Z

1445

Z17

15Z

1915

Z20

45Z

2315

Z02

15Z

0345

Z06

45Z

0715

Z08

15Z

1145

Z13

15Z

1515

Z17

15Z

2045

Z22

15Z

2315

Z00

45Z

0145

Z02

45Z

0345

Z06

45Z

55666666777777888899999999991010101010101010101010111111111111111111111212121212121212121212121313131313

ADT CI Recon CI SAB CI

First 6 hr PeriodValid @ 0600 UTC

Third 6 hr PeriodValid @ 1800 UTC

Fourth 6 hr PeriodValid @ 0000 UTC

How To: Make an Ensemble TRaP A Combination Approach

Example using Single-Orbit TRaP Ingredients for Rita eTRaP:

Multiple Sensor RainRates, Forecast Periods, and Forecast Agencies

# ensemble members = 7 x 9 x 6 x 3 = 1134Culled to 200 members (this is configurable)

24 h eTRaP forecast for Ritavalid 0000 UTC 25 Sept 2005

AVG QPF PM QPF P => 50 mm

P => 100 mm P => 150 mm P => 200 mm

Smoothes the maxima out &Increases areal coverage Maintains Max values

eTRaP Validation Statistics for 2008eTRaP Validation Statistics for 2008

0

0.5

1

1.5

2

0 50 100 150 200

Rain threshold (mm)

Fre

qu

ency

bia

s

eTRaP

TRaP

0

0.1

0.2

0.3

0.4

0.5

0 50 100 150 200

Rain threshold (mm)

Eq

uit

ab

le t

hre

at

sc

ore

eTRaP

TRaP

0

0.5

1

1.5

2

0 50 100 150 200 250

Rain threshold (mm)

Fre

qu

en

cy

bia

s

eTRaP

TRaP

0

0.1

0.2

0.3

0.4

0.5

0 50 100 150 200 250

Rain threshold (mm)

Eq

uit

ab

le t

hre

at

sco

re

eTRaP

TRaP

6 h

24 h

eTRaP Verification of Probability ForecastseTRaP Verification of Probability Forecasts

ReliabilityReceiver Operating

Characteristic (ROC)

6 hr

24 hr

20082008

Update on CIRA GOES-PSDI Project Extension5.2bFY03 Tropical Cyclone Formation Probability (TCFP) Product

Collection of developmental datasets needed toexpand current product to global domain has begun

Satellite water vapor imagery for IndianOcean & S. Hemisphere collected from • Meteosat 5 & 7 (centered at 63 E)

Nov 2005 – present• Meteosat 9 (centered at 0 E):

June 2007 – present

Starting collection and quality control of best tracks for Indian Ocean and Southern Hemisphere

Global GFS analyses already archived at CIRA back to 1982

Tropical Cyclone Forecast Probability Product Tropical Cyclone Forecast Probability Product Verification of Probability Forecasts for WPAC/CPAC Verification of Probability Forecasts for WPAC/CPAC

in 2008in 2008Reliability

Receiver Operating Characteristic (ROC)

Entire Domain

0

5

10

15

20

25

30

35

40

45

50

0 5 10 15 20 25 30 35 40 45 50

Expected Genesis Occurance Rate (%)

Ob

se

rve

d G

en

es

is O

cc

ura

nc

e R

ate

(%

)

Product GOR

Clim GOR

Perfect

0

5

10

15

20

25

30

35

40

45

50

0 5 10 15 20 25 30 35 40 45 50

Expected Genesis Occurance Rate (%)

Ob

se

rve

d G

en

es

is O

cc

ura

nc

e R

ate

(%

)

Product GOR

Clim GOR

Perfect

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

False Alarm Rate

Pro

bab

ilit

y o

f D

etec

tio

n

Clim

Product

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

False Alarm Rate

Pro

ba

bil

ity

of

De

tec

tio

n

Clim

Product

Over 5x5 Subgrids

No skill l

ine

No skill l

ine

Over Confident

Over Confident

Under Confident

Under Confident

Skillful

Skillful

Project recently funded; will enjoy 24/7 support in 2 yrs:•May 2008: Development Phase Underway•May 2009: Pre-Operational Phase•June 2010: Operational Phase Begins

Multiplatform Surface WindMultiplatform Surface Wind

Product Description• Wind Analysis is produced 6-hourly (~ 25 past

synoptic) and contains– IR flight-level proxy winds– AMSU 2-d non-linear balance winds– Scatterometery (A-SCAT, QuikSCAT)– Cloud/Feature track winds below 600 mb.

• Fixes come from the wind analysis– MSLP dynamically estimated from the axi-symmetric

winds – Wind Radii to the nearest 5 n. mi

• Request for transition to operations by JTWC

MSLP VerificationMSLP Verification

Avg. Error 1.73

MAE 7.29

RMSE 9.92

R^2 80.68

Mean 980.6

Validated against 2008 Recon in Atlantic (within 2-hrs)

MAE - Mean Absolute ErrorPOD - Probability of DetectionFAR - False Alarm RateRMSE - Root Mean Square ErrorR^2 - Variance Explained

Statistical Terms

NENW

SE

SW G.Gallina SAB 04/15/09

Avg. Error: 0.1nmMAE: 14nmN: 56POD: 98%FAR: 7%Variance Explained or R^2 33%

All Quadrants TotaledAvg. Error: - 13nmMAE: 20nmN: 81POD: 98%FAR: 5%Variance Explained or R^2 49%

Avg. Error: - 20nmMAE: 35nmN: 325POD: 99%FAR: 9%Variance Explained or R^2 33% Avg. Error: 5nm

MAE: 16nmN: 81POD: 97% FAR: 6%Variance Explained or R^2 40%

Avg. Error: 14nmMAE: 30nmN: 128POD: 98% FAR: 11%Variance Explained or R^2 52%


Avg. Error: - 17nmMAE: 29nmN: 81POD: 97% FAR: 3%Variance Explained or R^2 33%





Avg. Error: - 21 nmMAE: 43 nmN: 128POD: 100% FAR: 4.5%Variance Explained or R^2 30%


Avg. Error: 1.5nmMAE: 36nmN: 128POD: 99% FAR: 12%Variance Explained or R^2 43%

Avg. Error: - .5nmMAE: 20nmN: 81POD: 97% FAR: 6%Variance Explained or R^2 56%

34 Kts(Black)

50 Kts(Green)

64 Kts(Red)

Multiplatform Satellite Derived Tropical Cyclone Surface Wind Field

Validation Statistics

Wind Radius by color

Validated against 2008 Recon in Atlantic (within 2-hrs)

Update on CIRA POES-PSDI Project( Request FY99-38) FY08 AMSU-Based Tropical Cyclone Intensity

and Structure Estimation from Aqua and MetOp

• Retrieval codes for Aqua and MetOp AMSUs were obtained from SMCD

• Operational code has been modified to incorporate retrieval subroutines.

• The operational scripts are being modified by CIRA with coordination with NCO and TPC. Once completed the BUFR scripts will be modified by NCO

• Operational code will be finalized and tested once the operational data stream is finalized.

Brief Statistics OverviewForecast Event

and Occurs (POD)

‘a’

Type 1 Error

False Alarm Rate (FAR)

‘b’

Type 2 Error

Failure to Detect (FOD)

‘c’

NO Forecast where Event

did NOT Occur

‘d’

Observed

Forecast

T

T F

F

Equitable Threat Score (ETS)

ETS a ar

a b c ar

ar a c a b

Relative Operating Characteristic

Hit Rate or Prob. of Detection (POD) = FAR=

ROC Plot is Hit Rate vs. FAR

_____

b + d

____

a + b

Bias – comparison of areal coverage between forecast and observed. Ideal ratio is 1.0Reliability – the ratio of the variation of the true score and the variation of the observed score OR Observed Frequency vs. Forecast Probability

satellite analysis branch 2008 year in review anthony e. salemi – satellite analysis branch (sab)...

Documents