land product validation update
DESCRIPTION
F. Baret, J. Nightingale, S. Garrigues, J. Nickeson Missoula, 17 June 2009. LAND PRODUCT VALIDATION update. Outline. CEOS LPV context and structure Guidelines for best practices The way forward …. CEOS/WGCV/LPV. - PowerPoint PPT PresentationTRANSCRIPT
LAND PRODUCT VALIDATIONupdate
F. Baret, J. Nightingale, S. Garrigues, J. Nickeson
Missoula, 17 June 2009
1/29
OutlineOutline
• CEOS LPV context and structure
• Guidelines for best practices
• The way forward …
2/29
CEOS/WGCV/LPVCEOS/WGCV/LPV
3/29
•CEOS (committee on Earth Observation Satellite): group of space agencies aiming at harmonizing their activities at the international level.
•Composed of several working groups (WG) including WG on Calibration and Validation (WGCV).
•WGCV is made of several subgroups:- IVOS (Infrared and Visible Optical Sensors)- Microwave- SAR- Terrain Mapping- Atmospheric Chemistry- Land Product Validation (LPV)
Mission Statement & GoalsMission Statement & Goals• To foster quantitative validation of higher level global land
products derived from remote sensing data
• To relay results to users
• To develop and promote international standards and protocols for field sampling, scaling, error budgeting, data exchange
• To provide feed-back to international structures (GEO/GEOSS, GCOS, GTOS, IGBP …) for :• product definition, accuracy and quality assurance• Requirements for future missions
4/29
Role of LPV within ECVsRole of LPV within ECVs
• Essential Climate Variables (ECVs) are recognized to play a key role:– within international scientific structures (GCOS, GTOS)– international convention for verification (UNFCCC) to
complement/compare with national (official) figures
• Strong requirements on the evaluation of product uncertainties: need coordinated and consensus validation efforts mandatory for all the ECVs
5/29
Implementation toolsImplementation tools
• CEOS has no proper funding mechanisms
• Actions based on best efforts by space agencies– CEOS provides recommendations to space agencies…– Possible direct actions on satellite data access
• Direct actions within the community– Synergizing existing projects– Initiating new projects
• Mostly a bottom-up approach
6/29
Products targetedProducts targeted
• Land cover with GOFC-GOLD• Fire (mainly burnt area) with GOFC-GOLD• Biophysical variables (LAI, fAPAR)• Albedo, BRDF, surface reflectance• Land Surface Temperature and emissivity (with IVOS)• Soil moisture• Biomass? In discussion with GOFC-GOLD• Snow?
7/29
LPV proposed new structureLPV proposed new structure
sub-groups by product family with leads from the community if possible from different continents
– Albedo: G. Schaepman & C. Schaaf– Land-Cover: M. Herold & M. Friedl– Fire: K. Tansey & L. Boschetti– Vegetation: R. Fernandes, S. Plummer, J. Nightingale– Land Surface Temperature & emissivity: S. Hook & J. Sobrino– Soil: W. Wagner & T. Jackson
8/29
Role of sub-group leadsRole of sub-group leads
• Coordinate the validation activities at the global level:– Guideline for best practices
• Lead the writing of a guideline document for best practices on validation• Ensures community consensus and publication/distribution• Ensures that the document is updated when new data / methods are available
– Validation activities: promote their development/implementation• Data sharing• Data compilation• Implementation of validation exercise• Publication/distribution of results
• Convey information to and from the community– Plans/Status/Results of the validation towards CEOS/international
organizations/community
9/29
+ LPV Web site
http://lpvs.gsfc.nasa.gov+ LPV Wiki http://lpvs.pbworks.com/+ LPV Listservs
10/29
Communication tools
OutlineOutline
• CEOS LPV context and structure
• Guidelines for best practices
• The way forward …
11/29
Best practices guidelines documentBest practices guidelines document
•The “best practices guidelines” should be:•Based on current knowledge, tools, data and methods
• Tested and easily repeatable
•The Best Practices guidelines document should:• Define the best practices including data and methods to conduct validation of a satellite-derived land product. • Be a “living” document that is updated as tools, data, methods are improved through scientific endeavour
•Process for Endorsement by CEOS/ community• Peer review process
11/29
Proposed common structureProposed common structure
13/29
1 Introduction
2 Validation 2.1 Data sets 2.2 Global validation
3 Intercomparison 3.1 The satellite data 3.2 Global intercomparison
4 Recommendations / Conclusions
5 References
Validation and intercomparisonValidation and intercomparison
14/29
‘validation’ is the process of quantitatively defining the systemresponse to known, controlled signal inputs.
‘Validation’ refers to assessing the uncertainty of higher level, satellite sensor derived products by analytical comparison to reference data, which is presumed to represent the target value.
‘Intercomparison’ of data products provides an initial indication of gross differences and possibly insights into the reasons for the differences.
(Justice et al. 2000)
Validation and intercomparison are mandatory and complementary
Validation allows ‘absolute’ quantification of uncertaintiesbut often limited by the number and quality of available reference data
Intercomparison provides a more exhaustive evaluation of consistencies/differences
Required by usersrequired when combining several products
Products definition and uncertaintiesProducts definition and uncertainties
• Products definition: ECVs not always very clearly defined (GCOS/GTOS documents).
• Need more process model related defintions• Feedback to GCOS/GTOS
• Required uncertainties attached to ECVs not defined in a traceable way
• Need more process related uncertainties evaluation• Threshold / Optimal / Target ?• Feedback to GCOS/GTOS
15/29
Data sets for validation: sitesData sets for validation: sites
16/29
Existing data sets
Albedo: 19 BSRN sitesLAI/fAPAR: 80-100 sitesLand cover: 4300 points
Distribution of sites: must be representative of surface types, state/conditions - systematic (FRA 2010)- stratified
Need to capitalize the information
Data sets for validation: measurementsData sets for validation: measurements
17/29
Reference measurements - interpretation by experts based on HSR images (land-cover, fire)
- quantitative measurements (albedo, LAI-fAPAR, LST&E , moisture)
- definition of the variable measured
- footprint and scaling
Empirical Transfer Function
--
Point Measurement (PM)(1-50 m)
Elementary SamplingUnit (ESU)(20-100 m)
10-20 PM/ESU
Site(3 km)
30-50 ESU/Site
Variable extraction and averaging at the ESU level
High spatialresolutionbiophysicalvariable map
DHP, LAI2000 …
GPS located
Need for high spatial resolution images (Landsat/SPOT)
Global validationGlobal validation
18/29
Metrics used:Land cover : accuracy /user-producer Confusion matrix …LAI-fAPAR: RMSE, weighed RMSE, biases …
Users need more information on the structure of uncertainties
Satellite data for intercomparisonSatellite data for intercomparison
19/29
- Spatial samplingexhaustive (Land cover)systematicstratified (LAI, fAPAR, Albedo …)
- Spatial support areaneed to get same projection / resolution
- Temporal support periodneed to get synchronous / same resolution
Degradation oforiginal
characteristics
- Definition of variables: LCCS …
Global intercomparisonGlobal intercomparison
20/29
Land cover
• Temporal continuity
• Temporal consistency
Global intercomparison: LAIGlobal intercomparison: LAI
• Smoothness of temporal evolution
• Statistical distributions
• Scatterplots
Current statusCurrent status
24/29
Land cover
Level 2
Level 3
Level 4: operational validation
Level 1
Fire
Biophysical
Albedo
Moisture
LST & E
Operational Validation: land coverOperational Validation: land cover
25/29
Operational lc validation framework
Deg
ree
of u
sabi
lity
and
flexi
bilit
y
Updated valid./change
Validation of new products
Design based sample of reference sites
In-s
itu
glob
al
Primary validation
LCCS-based Interpretation(Regional Networks)
Reference database:statistically robust, consistent, harmonized, updated, and accessible
Updated interpretations
Time
Comparative validation
Existing global
LC products
Lege
nd t
rans
latio
nsProductsynergy
Data reprocessingLink to regional datasets
Operational Validation: BiophysicalOperational Validation: Biophysical
26/29
OLIVE INFORMATIONUser communityResultsObjectives
References & Contributors
1. Information to thecommunity
CandidateContribution
To DIRECTEvaluationcriterions
4. Addition of sites fordirect validation byindividual contributors
EVALUATION
Histograms/PFTScatterplotsTemporal continuityTemporal consistencySpatial consistency
Direct validationCandidateProduct
2. Stand-alone comparisonto existing productsby potential productproducer: Test Mode
Stand-aloneReport
InternalDATABASE
Existing productsBELMANIP2-Test
Database descr.How to proceedInput Formats
Existing productsBELMANIP2-Valid
3. Actual validation withresults visible by the community & additionof the product to the community database:Validation Mode
DIRECT
Concluding remarksConcluding remarks• Very strong ECV context
• Need funding mechanisms for sustainable validation activities: validation costs!!
• Importance of reference measurements: – data sharing– improved cooperation with existing/developing networks– Availability of high spatial resolution images
• A lot of new products … and not all at the ultimate validation level …. very challenging
• Mandatory for product improvement / combination27/29