forecast text editing rates and analysis
DESCRIPTION
Jaime Clark and Bianca Gibson Brianna Laugher Tennessee Leeuwenburg December 2012. Forecast text editing rates and analysis. www.cawcr.gov.au. Graphical Forecast Editor. GFE Text formatters. GFE Text formatters - editing. GFE Forecasting. Guidance. Grid Editing & Smart Tools. - PowerPoint PPT PresentationTRANSCRIPT
The Centre for Australian Weather and Climate ResearchA partnership between CSIRO and the Bureau of Meteorology
Forecast text editing rates and analysis
Jaime Clark and Bianca GibsonBrianna LaugherTennessee LeeuwenburgDecember 2012
www.cawcr.gov.au
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Graphical Forecast Editor
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Text formatters
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Text formatters - editing
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Grid Initialisation
Forecast Database
GFEGFESmart Tools
GFE Grid Editing
Guidance Grid Editing & Smart Tools
Text Formatting
Forecast Production
Numerical Modelling
LAPS, GASP, ECMWFJapan, UK, US
Observations
Operational Consensus
Forecast (OCF)
Direct Model Output
Model Output Statistics
Automatic Forecast Generation System
Python Modules
Text
Graphics
Product Generator Forecast Database
Forecast Products
Text Formatters
GFE Forecasting
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Text Formatting
Forecast Production
Automatic Forecast Generation System
Python Modules
Text
Graphics
Product Generator Forecast Database
Forecast Products
Text Formatters
GFE Forecasting
Grids
editedtext
original
gfemetrics
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Metrics
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Metrics
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Metrics
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
GFE Metricshttp://tardis.bom.gov.au/gfemetricsWeb front-end to “FDB” database created/maintained by ITBInitially developed by Brianna, further developed by Anthony Briggs
STATS/SUMMARY Can view graphs/tables of total edits: by time, by state, by formatter
VIEW Can view any edits made to any GFE forecasts Can query edits by state, date, formatter, period, issuance, locale Can search for edited/unedited/either containing particular words
TOOLS Can view graphs/tables of most used tools: by time, by state
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Metric?
We initially hoped that improvements to the formatters would be reflected in lower edit rates... this appears not to be the case... but it's hard to tell...
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Dimensions of edits
<—————— REPRESENTATIVENESS ——————> Is the generated text consistent with grid contents? “Wrong” results (genuine error) vs stylistic changes (subjective) “Possible shower” -> “Mostly dry”
<—————— POLICY ——————> Is the edited text consistent with Services policy? Policy may need resolving with Services “Fine”, “moderating”, “freshening”
<—————— MISSING DATA ——————> Is the info required to produce the edited text present in the GFE? Formatter missing rules - “Humid”, “Hot”, sea breeze GFE missing grids – climatology, uncertainty! Other info in forecaster brain (local expertise)
Can potentially
fix
Won't fix
Can't fix(for everyone)
Should fixASAP
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Limitations
Formatters are mainly just a group of phrases – Sky, Wx, Wind, Temp, Seas, Swell. Effort is directed at improving phrases – but there's no way to check edit rates of phrases
Hard to get a sense of any general trend when a query returns hundreds of results
Little/no time to devote to exploring regional differences
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Monash Talented Students program
Science-Industry Student Research Placement Program“short-term, research-based industry placements”
Rationale(i) facilitating industry experience for upper undergraduate students; (ii) building collaborations in work integrated learning between Monash and
industry partners; (iii) furthering science and its broader communication.
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
About Bianca
Second year in undergraduate Computer Science
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Analysing edits by formatter
Per forecast periodPer statePercentages
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Marine edits by forecast period - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Marine edits by forecast period - totals
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Precis edits by forecast period - percentages
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Town, metro and district edits by forecast period - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
NSW edits by formatter - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
NSW edits by formatter - totals
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
SA edits by formatter - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
TAS edits by formatter - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
VIC edits by formatter - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
WA edits by formatter - percentage
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Two-Town comparisons
Hypothesis: Bigger towns (capital cities) get more attention than smaller towns
Does “the gap” represent editing that would be done if forecasters had infinite time? Or does it represent stylistic editing that could be done without?
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Two-Town comparisons - VIC
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Two-Town comparisons - TAS
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Go-Live comparisons
Hypothesis: initial edits very low, increase as forecasters learn to use the GFE and have more time for text editing
“Month 0” is the month the GFE went live in that regionVIC: 5/2009NSW: 9/2010TAS: 6/2011SA: 11/2011WA: 10/2012 Each state has a different version of the GFE and different weather
conditionsThe dot and percentage on the far right represents edits in 11/2012
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Go-Live comparison – all formatters (incl. warning)
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Go-Live comparison - Precis
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Go-Live comparison - Metropolitan
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Go-Live comparison - marine
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
About Jaime
4th year in double undergraduate degree:
Science Major - Pure Mathematics Minor - Developmental Psychology Minor – Computational Mathematics
Computer Science Major - Intelligent Systems Major – Software Engineering
5 year course
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Edits per formatter by phrase type edited
How do we do this?Formatter
Unedited forecast Edited forecast
Break into sentences
Identify the sentence type
Examining key known words within
the sentence to determine type
Unedited categorized sentences Edited categorized sentences
Compare categories: Identify if a change is present
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC WA
0%10%20%30%40%50%60%70%80%90%
100%
wxwindwx windwind skyskywx wind skywx sky
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC WA
0100200300400500600700800900
wxwindwx windwind skyskywx wind skywx sky
Phrase type edited - Town
01-NOV-2012 --- 30-NOV-2012 :: TOWN_FORECAST
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC WA
0100200300400500600700800
wxwindwx windwind skyskywx sky
Phrase type edited - District
01-NOV-2012 --- 30-NOV-2012 :: DISTRICT_FORECAST
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC WA
0%10%20%30%40%50%60%70%80%90%
100%
wxwindwx windwind skyskywx sky
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC WA
050
100150200250300350400
wxwindwx windwind skyskywx wind skywx sky
Phrase type edited - Metropolitan
01-NOV-2012 --- 30-NOV-2012 :: METROPOLITAN_FORECAST
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC WA
0%10%20%30%40%50%60%70%80%90%
100%
wxwindwx windwind skyskywx wind skywx sky
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC
0500
10001500200025003000
wxwindwx windseawind skyswellwx skywind sea
Phrase type edited - Coastal
01-NOV-2012 --- 30-NOV-2012 :: COASTAL_FORECAST
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC
0%10%20%30%40%50%60%70%80%90%
100%
wxwindwx windseawind skyswellwx skywind sea
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC
0100200300400500600700800
wxwindwx windseawind skyswellwind sea
Phrase type edited – Local Waters
01-NOV-2012 --- 30-NOV-2012 :: LOCAL_WATERS_FORECAST
day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+NSW SA TAS VIC
0%10%20%30%40%50%60%70%80%90%
100%
wxwindwx windseawind skyswellwind sea
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Phrase type edited - Marine
01-NOV-2012 --- 30-NOV-2012 :: MARINE_FORECAST
day0-1 day3+WA
0
10
20
30
40
50
60
wxseawindswell
day0-1 day3+WA
0%
20%
40%
60%
80%
100%
wxseawindswell
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Precis: Investigating additions/removals
How do we do this?Formatter
Unedited forecast Edited forecast
Identify if the given key is present
Unedited keys identified Edited keys identified
Compare keys: Identify if a key has been added/removed
Ratio to un-edited Text
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Morning – strongly removed
01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT
morning morning morning morning morning morning morning morning morning morningday0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+
NSW SA TAS VIC WA
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IntersectedRemovedAdded
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Afternoon – strongly added
01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT
afternoon afternoon afternoon afternoon afternoon afternoon afternoon afternoon afternoonday0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1
NSW SA TAS VIC WA
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IntersectedRemovedAdded
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Late – strongly added
01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT
late late late late late late late late late lateday0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+
NSW SA TAS VIC WA
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IntersectedRemovedAdded
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Developing – strongly removed
01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT
developing developing developing developing developing developing developing developing developing developingday0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+
NSW SA TAS VIC WA
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IntersectedRemovedAdded
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Clearing – strongly removed
01-JAN-2012 --- 30-NOV-2012 :: PRECIS_TEXT
clearing clearing clearing clearing clearing clearing clearing clearing clearing clearingday0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+ day0-1 day3+
NSW SA TAS VIC WA
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
IntersectedRemovedAdded
The Centre for Australian Weather and Climate Research A partnership between CSIRO and the Bureau of Meteorology
Towns - Haze/Humid?
haze humidday3+
VIC
0
50
100
150
200
250
300
IntersectedRemovedAdded
01-FEB-2012 --- 29-FEB-2012 :: PRECIS_TEXT
The forecaster trick to get humid into the forecast:
– Add haze into the grids.– Generate forecasts.– Remove haze from the grids.– Complete a search and replace
all: haze to humid.
The Centre for Australian Weather and Climate ResearchA partnership between CSIRO and the Bureau of Meteorology
Bianca Gibson: [email protected] Clark: [email protected] Laugher: [email protected], x4690Tennessee Leeuwenburg: [email protected], x4310
Web: www.cawcr.gov.au
Thank youwww.cawcr.gov.au