l3mto qc and accuracy
DESCRIPTION
L3MTO QC and Accuracy. Marina Timofeyeva Contributors: Annette Hollingshead, Dave Unger and Andrea Bair. General QC Flow. CPC: CD forecast. LFPS: Make forecast Primary QC Fix troubles. RHQ: Collecting trouble tickets. Archive. WS: Post product on web Fix Troubles. WFO: - PowerPoint PPT PresentationTRANSCRIPT
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
L3MTO QC and AccuracyL3MTO QC and Accuracy
Marina Timofeyeva
Contributors: Annette Hollingshead, Dave Unger and Andrea Bair
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
CPC: CD forecast
RHQ:Collecting trouble ticketsCollecting trouble tickets
WFO:1.1. Ensure link to the productEnsure link to the product2.2. Secondary QC of the productSecondary QC of the product3.3. Local value addedLocal value added4.4. Customer services and outreachCustomer services and outreach
LFPS:1.Make forecast2.Primary QC3.3.Fix troublesFix troubles
WS:1. Post product on web
2.2. Fix TroublesFix Troubles
General QC FlowGeneral QC Flow
Archive
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
L3MTO Primary QCInterrogate CPC site and
check for new forecast data1. National 2. Alaska & Hawaii
Parse data: extract mean & standard deviation
Import data into Splus
Run Splus scripts
Export data
1. Parse Splus output 2. Create XML formatted files
Run QC scripts
Push data to web team
Pass
Troubleshoot and fix problems
Fail
Early am ET(~ 7am- Noon)
6-8 hours
45 minutes
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Secondary QC (WFOs)Secondary QC (WFOs)
Erroneous Data Check
Approve data
Web & Functionality Error Check
Conspicuous Data Check
Send trouble ticket to WS
Pass
Pass
Pass
Fail
FailSend trouble ticket to Regional HQ
Start
Fail
Fail
Pass
Section A
Section B
Section CSend trouble ticket to
Regional HQ
Send trouble ticket to Regional HQ
Other Discrepancy:Detailed QC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
WFO Technical Support Team
Who are they?Regional representatives available to help answer any L3MTO questions
What is their role?Answer customer and partner questions and inquiries that NWS personnel are unable to address (see Guideline doc 3.2.3)
Provide technical expertise & guidance:- interpreting L3MTO - comparing L3MTO to CPC 3-Month Temperature
Outlook
How to find your regional rep?Page 13 in the L3MTO Guidelines document
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Web problem QC failureStation
metadata
L3MTO Frustration
Content and methodology
Contact Regional Technical
Support Team
WFO Quick Guide on Getting Help with L3MTO
Check CSSA, if error report to
Regional CSPM and RCPM
[email protected] Trouble Ticket
Procedure
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
WFOL3MTO Problem
Reporting Protocol ( )
Detailed QCSecondary QCWeb QCAdd
Stations Metadata Methodology
Web Team
Annette Hollingshead
1) Add Station (if not in CMS): report to CSPM;2) Metadata Errors: - If CSSA is up-to-date, then submit change to CSPM - If CSSA is NOT up-to-date submit change to CSPM and RCPM who will expedite CSSA change
TT collectors
Marina Timofeyeva
Issue Trouble Ticket
Report to CSPM / RCPMTechnical Support
Team
Technical Support Teams
Protocol in Guidelines document
Compare to CSSA Discussion
Inconsistencies, MethodologyTechnical Issues
Buffer issues
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
http://ww2.wrh.noaa.gov/hsd/climate/L3MTO/ARC/index.phphttp://ww2.wrh.noaa.gov/hsd/climate/L3MTO/ARC/index.php
• Section A allows checking short leads L3MTO “sanity” with current climate trends• Section B allows checking L3MTO agreement with some of CPC tools• Section C allows checking L3MTO presentation and methodology correctness• Section D presents L3MTO long term performance evaluation
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
Season: MJJ
A CPC forecast of 49% (or greater) BELOW NORMAL probability is required to produce a significant* forecast for BELOW Normal category at the station. This automatically reduces the forecast for ABOVE Normal category to 28.3% or less.
A CPC forecast of 36% (or greater) ABOVE NORMAL probability is required to produce a significant* forecast for ABOVE Normal category at the station. This automatically reduces the forecast for BELOW Normal category to 28.3% or less.
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
L3MTO Verification
• L3MTO forecast archive
• Additional scores to coordinate with NWS verification practices
• Extended selection choices for years and leads
Univ. of AZ verification tool expanded to include:
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
ARC section D verification intended for WFO use, it shows results of 3 verification methods:
• Heidke Skill Score verifies 3 categorical forecast (pie chart format), indicates % improvement over climatology
• Reliability Diagram verifies long term performance of individual %-tile of POE
• Continuous Rank Probability Skill Score verifies accuracy, skill and resolution of POE, the most stringent score of the 3
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARCDJF JJA
• Most stations have correct forecasts 30% (or more) of the time, and have little variability between the 13 leads
• For all years analyzed together, shorter leads in the winter seasons have higher scores than the same in the warm seasons
• Larger spread between stations’ performance in the warm seasons than cold
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
Bars show 95%Confidence Limits
0.0 0.2 0.4 0.6 0.8 1.0POE Percentile
0.0
0.2
0.4
0.6
0.8
1.0
Ob
serv
ed F
req
uen
cy
Reliability Diagram for San Jose for the Years 1994-2005 for the Season DJF
Perfect ReliabilityForecastClimatology
Bars show 95%Confidence Limits
0.0 0.2 0.4 0.6 0.8 1.0POE Percentile
0.0
0.2
0.4
0.6
0.8
1.0
Ob
serv
ed F
req
uen
cy
Reliability Diagram for San Jose for the Years 1994-2005 for the Season JJA
Perfect ReliabilityForecastClimatology
• For each station and forecast target season a reliability diagram was created to analyze each percentile’s performance.
• Confidence limits around the perfect forecast line, were drawn from a binomial distribution at 95% confidence.
• The forecast was compared with 1971-2000 climatology performance
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
QC Tool: L3MTO Analysis Resource Center - ARC
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Verification Summary
• Using multiple verification scores in climate forecast evaluation provides the most comprehensive assessment of long term forecast performance
• Verification analysis identified locations and 3-month periods with satisfactory forecast performance, and their spatial and temporal variability
• Overall, for the majority of stations and seasons, the L3MTO indicates higher skill than the 1971-2000 climatology
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Verification Summary
• Although L3MTO in general proves to be more skillful than 1971-2000 climatology, it is still important to identify station/season combinations where L3MTO does not have long term skill. At such stations/seasons a Verification Filter will be applied.
• The verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level that depends on number of independent forecast made for the stations. If the HSS is greater than passing, then the forecast is reported, if less, then 1971-2000 climatology is reported.
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Verification Filter
5. CPC produces CD forecasts monthly
6. CD outlook together with bi-variate statistics reported from the regression analysis, station forecast is produced
7. Verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level.
8. Data distribution and display are provided at the L3MTO web site
(6) Station Forecast Produced
(7) Verification Filter
(5) CPC Producesthe CD Forecast
Monthly Routine:
SufficientScore?
ProduceForecast Data
(8) Distribution and Display
Yes
ProvideClimatology Data
No
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Additional Verification Information
Additional Verification InformationAdditional Verification Information
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Additional Verification Information
Additional Verification Information
CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS
Additional Verification Information