evaluation

aerosol CCI retrievals

Reading 2012

the participants / the task

• AATSR F v142• AATSR O v202/v2q2• AATSR S v040/v031• MERIS A v21• MERIS B v11• MERIS E 802• PARASOL v30

• MODIS_aqua c5.1

• MODIS_terrac5.1

• SEAWIFS• MISR v31

• evaluate daily global

‘CCI-aerosol’ retrievals

– for the entire year 2008• AOD 550nm• Angstrom (if offered)

• versus AERONET / MAN

• performance in context

of established retrievals

ECMWF AOD

FBOV (single assim.)FMNG (dual assim.)

less aerosol

ECMWF Angstrom

FBOV (single assim.)FMNG (dual assim.)

larger sizes

data content AOD year 2008

MISR

MODIS TERRA

PARASOL

MERIS-ESA

MERIS-BAER

MERIS-ALAMO

ATSR-SU

ATSR ORAC

ATSR ADV

data volume AOD 1/1/2008

MODIS AQUA

MISR

PARASOL

MERIS-ESA

MERIS-BAER

MERIS-ALAMO

ATSR-SU

ATSR ORAC

ATSR ADV

MODIS TERRA

the scoring concept

• establish 1x1 gridded AERONET daily data

– default: use +/- 30min of overpass of satellite

– option: use daily averages

• identify local data pairs to level 3 daily 1x1

satellite retrievals (test data & reference data)

• require at least 10 events to perform a

statistical analysis (e.g. correlation, bias)

the scoring concept

• establish 1x1 gridded AERONET daily data

– default: use +/- 30min of overpass of satellite

– option: use daily averages

• identify local data pairs to level 3 daily 1x1

satellite retrievals (test data & reference data)

• require at least 10 events to perform a

statistical analysis (e.g. correlation, bias)

scoring steps

• establish regional scores (1: best, 0: poor)

– Step1 regional spatial score for each day• (rank) correlation and (rank) bias

– Step2 regional temporal score for each 1x1 • (rank) correlation and (rank) biast

– Step3 combined regional score• total score = bias * spat.corr * temp.corr

– Step4 combine regional to one global score• ‘single score’ from regional total scores

• investigated property: AOD550

MODIS TERRA AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– albsolute

AOD bias versus AERONET

smaller larger

MODIS TERRA

MISR

PARASOL

MERIS-ESA

MERIS-BAER

MERIS-ALAMO

ATSR-SU

ATSR-ADV

ATSR-ORAC

AOD difference to MODIS

smallerthan MODIS

largerthan MODIS

PARASOL

ATSR-SU

ATSR-ORAC

ATSR-ADV

AOD total error comparison

Increasingly larger ERROR

MODIS TERRA

MISR

PARASOL

MERIS-ESA

MERIS-BAER

MERIS-ALAMO

ATSR-SU

ATSR-ORAC

ATSR-ADV

overall AOD scores

global land ocean• MODIS A/T .63 .59 .66• AATSR S v40 -.60 .63 -.61• AATSR O v202 -.56 -.56 .58• MISR .53 .58 .53• AATSR F v142 -.52 -.56 -.53• AATSR S v31 -.48 -.56 -.47• MERIS B v11 .42 .35 .46• SEAWIFS -.41 -.37• PARASOL -.23 -.25• MERIS ESA 802 -.20 -.22• MERIS A 21 -.20 -.21

based on a10+ sample statistics

absolutelarger scoreis better

+/- sign for overall bias

let us go regional (1)

• boreal– MODIS (.47), MERIS-BAER(.22)

• N.America temp– ATSR-S40 (-.62), ATSR-O202 (.58), ATSR-F142 (-.57),

MISR (.56), MODIS (.44)• S.America trop

– Modis (-.67)• S.America temp

– MODIS (.44), Seawifs (.34), MERIS-BEAR (.31)• N.Africa

– ARSR-S31 (-.72), ATSR-S40(.70), MOD (.67), BEAR (.37)• S.Africa

– MODIS (.63)

10 + events

let us go regional (2)

• EU-Asia boreal– MODIS (-.58)

• EU-Asia temp– MODIS (.63), MISR (-.62), ATSR-S40 (-.58), ATSR-F142

(-.54), ATSR-O202 (-.53), ATSR-S31 (-.52), BAER (-.41)• Asia trop

– Modis (-.62), MERIS-BEAR (.47)• Europe

– MISR (-.63), ATSR-S40 (-.61), ATSR-F142 (-.59), ATSR-O202 (.59), MODIS (.58), ATSR-S31 (-.53), BEAR (.31)

• N.Pacific temp– MODIS (.61), ARSR-F142 (-.54), MISR (.53), BEAR (.47)

• N.Atlantic temperate– MODIS (.61), SEAWIFS (.47), BEAR (.45)

10 + events

what regional scores tell us

• ATSR data coverage is poor compared to MODIS•

• comparisons in regions with sufficient data statistics indicated that all three ATSR AOD data-sets in quality are at/near the level of MODIS/MISR

• current overall ATSR AOD ranking: – 1. SU4.0– 2. O202– 3. F142

• MERIS potentially offers data similar to MODIS

• MERIS BEAR 1.1 demonstrates data-volume processing … but data-quality is relatively poor

overall Angstrom scores

global land ocean• AATSR S v40 .60 .56 .65• AATSR S v31 .52 .47 .57• AATSR F v142 .47 .51 .47• AATSR O v202 .41 .38 .43• MISR .38 .43 .37• MERIS ESA 802 . 27 .28• PARASOL .26 .27• MERIS A 21 .21 .22

all Angstrom data are biased HIGH!land scores are poorer than ocean scores

based on a10+ sample statistics

absolutelarger scoreis better

+/- sign for overall bias

final thoughts

• some of the current CCI retrievals for AOD have

reached the maturity of US products

• while ATSR has become competitive, MERIS

products lag behind

• ocean reference data are needed (MAN data are

too sparse) to statistically evaluate retrieval

quality over oceans, also in order to get reliable

scores for PARASOL and MERIS data

extra slides

further thoughts

• the combination of ‘sufficient’ coverage and of

‘sufficient’ accuracy matters

• temporal correlation (seasonality) is generally

poorer than spatial correlation (patterns)

issues

more solid (statistical) evaluations require …

• better spatial coverage

–how to get references for non AERONET loc ?

• better temporal coverage

–4 months are not enough for limited swath

• better accuracy

–more confidence in AERONET reference

–fewer sites, less variability, several per hour

2ass … minus … 1 ass

improvement deteriation

2 ass … minus … 1 ass

improvement deteriation

O202 … minus … SU40

improvement deteriation

O202 … minus … SU40

improvement deteriation

O202 … minus … SU40

improvement deteriation

SU40 … minus … SU31

improvement deteriation

F142 … minus … SU40

improvement deteriation

extrasthis

this

Thnis

this

Thnis

Thnis

Thnis

MODIS AQUA AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MISR AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

POLDER AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MERIS ESA AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MERIS BEAR AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MERIS ALAMO AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

AATSR SU40 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

AATSR SU31 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

AATSR O202 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

AATSR O202iq AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

effectively ‘identical to O202

AATSR ADV142 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

AATSR SU40 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MODIS terra

AATSR O202 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MODIS terra

AATSR ADV142 AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MODIS terra

POLDER AOD performance

• total error

– bias

– spatial

– temporal

• bias

– tendency

– absolute

MODIS terra