kelvin k. droegemeier university of oklahoma internet2 spring meeting 20 april 2004
Post on 30-Jan-2016
19 Views
Preview:
DESCRIPTION
TRANSCRIPT
The The Collaborative Collaborative
Radar Acquisition Field Radar Acquisition Field Test (CRAFT): A Unique Public-Test (CRAFT): A Unique Public-Private Partnership in Mission-Private Partnership in Mission-
Critical Data DistributionCritical Data DistributionKelvin K. DroegemeierKelvin K. DroegemeierUniversity of OklahomaUniversity of Oklahoma
Internet2 Spring MeetingInternet2 Spring Meeting20 April 200420 April 2004
NCDC
NEXRAD Doppler Radar NetworkNEXRAD Doppler Radar Network
The Moore, Oklahoma TornadoThe Moore, Oklahoma Tornadoof 3 May 1999of 3 May 1999
Storm A
Storm B
Storm D
Storm E
Storm C
Storm H
Storm G
100 kmrange ring
Damage Paths Ground Truth - 53 Tornadoes
Courtesy D. Zittel
Trimmed Detections and Ground Truth Damage Paths
Hits (142)
Misses (25)
FAs (21) Courtesy D. Zittel
USA Today
Doppler Weather Radar: Doppler Weather Radar: Measuring the Wind SpeedMeasuring the Wind Speed
A Unique Tool for A Unique Tool for Sampling the 3D Sampling the 3D
Atmosphere in DetailAtmosphere in Detail The radar scans a 3-D conical volume of the The radar scans a 3-D conical volume of the
atmosphere every 5-6 minutes through a series atmosphere every 5-6 minutes through a series of 360 degree sweeps…of 360 degree sweeps…
… … at several elevation anglesat several elevation angles
A Unique Tool for A Unique Tool for Sampling the 3D Sampling the 3D
Atmosphere in DetailAtmosphere in Detail
NEXRAD Facts and NEXRAD Facts and FiguresFigures
159 radars (142 in the Continental US)159 radars (142 in the Continental US)– 121 National Weather Service radars121 National Weather Service radars– 26 Department of Defense radars26 Department of Defense radars– 12 Federal Aviation Administration radars12 Federal Aviation Administration radars
NEXRAD Data TypesNEXRAD Data Types Archive Level IArchive Level I
– analog, time domain outputanalog, time domain output Level II dataLevel II data
– digital base data at full precision and all elevation anglesdigital base data at full precision and all elevation angles Archive Level IIIArchive Level III
– digital products + lowest four elevation anglesdigital products + lowest four elevation angles– reduced precision: data are binnedreduced precision: data are binned
Archive Level IVArchive Level IV
Archive Level IArchive Level I– analog, time domain outputanalog, time domain output
Level II dataLevel II data– digital base data at full precision and all elevation anglesdigital base data at full precision and all elevation angles
Archive Level IIIArchive Level III– digital products + lowest four elevation anglesdigital products + lowest four elevation angles– reduced precision: data are binnedreduced precision: data are binned
Archive Level IVArchive Level IV
NEXRAD Data TypesNEXRAD Data Types
The Value – Hazard DetectionThe Value – Hazard Detection
The Value – Real Time 3D The Value – Real Time 3D AnalysesAnalyses
http://caps.ou.edu
Carbone et al. (2002)
The Value – Fundamental PredictabilityThe Value – Fundamental Predictability
The Value – Storm ClimatologiesThe Value – Storm Climatologies
The Value – Computer The Value – Computer Prediction ModelsPrediction Models
6 pm 7 pm 8 pmR
adar
Hourly Radar Observations(Fort Worth Shown by the Pink Star)
6 pm 7 pm 8 pmR
adar
Com
pu
ter
For
ecas
t
2 hr 3 hr 4 hr
Fcs
t w
/o R
adar 2 hr 3 hr 4 hr
6 pm 7 pm 8 pmR
adar
Space Shuttle Columbia Debris TrailSpace Shuttle Columbia Debris Trail
Bird Migration StudiesBird Migration Studies
NEXRAD data are *THE* mostvaluable data for mesoscale meteorology…
but they were not available in real timeto the academic and corporate communities
The Collaborative Radar The Collaborative Radar Acquisition Field Test Acquisition Field Test
(CRAFT)(CRAFT) Begun in fall, 1998 as a Begun in fall, 1998 as a collaborative R&D projectcollaborative R&D project
to demonstrate the real time compression and to demonstrate the real time compression and Internet-based transmission of WSR-88D base Internet-based transmission of WSR-88D base (Level II) data(Level II) data
PhilosophyPhilosophy: Leverage existing infrastructures : Leverage existing infrastructures (Internet), public-domain software (Unidata LDM), (Internet), public-domain software (Unidata LDM), and secure, low-cost services (56K phone lines)and secure, low-cost services (56K phone lines)
Short-term goalsShort-term goals: Storm-scale research and : Storm-scale research and improvement of NCDC archive capabilityimprovement of NCDC archive capability
Long-term goalLong-term goal: National real-time capability for : National real-time capability for operations, research, and private industryoperations, research, and private industry
Initiated and led by the OU Center for Initiated and led by the OU Center for Analysis and Prediction of Storms (CAPS) Analysis and Prediction of Storms (CAPS)
National Severe Storms Laboratory (NSSL)National Severe Storms Laboratory (NSSL) NOAA WSR-88D Radar Operations Center NOAA WSR-88D Radar Operations Center
(ROC)(ROC) UCAR Unidata ProgramUCAR Unidata Program National Climatic Data Center (NCDC)National Climatic Data Center (NCDC)
University of WashingtonUniversity of Washington
TREMENDOUS support from the US WeatherTREMENDOUS support from the US WeatherResearch ProgramResearch Program
Coordinating Coordinating OrganizationsOrganizations
Foundation #1: NSSL’s Foundation #1: NSSL’s RIDDSRIDDS
Courtesy NSSL
Foundation #2: Unidata Local Foundation #2: Unidata Local Data Manager (LDM) Data Manager (LDM)
Source
LDM
Source
Source
LDM LDM
LDMLDM
LDM LDM
LDM
LDM
Internet
Courtesy D. Fulker, Unidata
Foundation #3: Foundation #3: Data Compression – BZIP2Data Compression – BZIP2
The Internet
Adaptation to LDM by Harry Edmon, U. of Washington
Foundation #4: The Foundation #4: The InternetInternet
TheInternet
Invention courtesy A. Gore
CommodityNGI
Internet2Abilene
Foundation #5Foundation #5 Incredible cooperation, collegiality, and Incredible cooperation, collegiality, and
agency supportagency support
NCDC
Original Technical Original Technical StrategyStrategy
RIDDS
WSR-88D
At the radar site
Original Technical Original Technical StrategyStrategy
RIDDS Linux PC + LDM + Compression
($1000)
WSR-88D
At the radar site
Original Technical Original Technical StrategyStrategy
RIDDS Linux PC + LDM + Compression
($1000)
Router ($2000)
WSR-88D
At the radar site
Original Technical Original Technical StrategyStrategy
RIDDS Linux PC + LDM + Compression
($1000)
Dedicated 56K line, T1, ($2000 - $6000/year)
or NWS LANRouter ($2000)
WSR-88D
At the radar site
LDM Server (OU)
Original Technical Original Technical StrategyStrategy
RIDDS Linux PC + LDM + Compression
($1000)
LDM Server (OU)
Dedicated 56K line, T1, ($2000 - $6000/year)
or NWS LANRouter ($2000)
WSR-88D
At the radar site
Users (LDM)
Internet or Abilene
Slow But Sure GrowthSlow But Sure Growth
February, 2000
BZIP2 off-the-shelf compression algorithmBZIP2 off-the-shelf compression algorithm– maximum 16:1 in clear air; 8:1 in the worst case; maximum 16:1 in clear air; 8:1 in the worst case;
average of about 12:1average of about 12:1 Maximum aggregate data rate is 15 Maximum aggregate data rate is 15
Mbytes/radar/5 min uncompressed = 48 Mbits/sec Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for compressed for entire national networkentire national network– bandwidth isn’t the main issue, but rather bandwidth isn’t the main issue, but rather
topology, quality of service, and data storagetopology, quality of service, and data storage 56K lines found to be inadequate in some cases 56K lines found to be inadequate in some cases
(squall lines) – maybe 10% of time(squall lines) – maybe 10% of time– Exceed available bandwidth by 10-20%Exceed available bandwidth by 10-20%– Can create latencies of 10s of minutes (more later)Can create latencies of 10s of minutes (more later)– Up to 26% improvement in compression possible in Up to 26% improvement in compression possible in
BZIP2, but requires changing code at user endBZIP2, but requires changing code at user end– A pre-processing algorithm (NSSL) provides 10% A pre-processing algorithm (NSSL) provides 10%
improvement – no changes in BZIP requiredimprovement – no changes in BZIP required
Technical HighlightsTechnical Highlights
Humble Beginnings…Humble Beginnings…
January, 1999
Eight More in 1 Year Eight More in 1 Year January, 2001
14 Radars
s fn
0
20
40
60
80
100
120
0 5 10 15 20 25 30
Radar Data LatencySEAAMACMBFLGFTWSRPTULLBBFSMOKCTCS
Hour
Latency DataLatency Data
The Idea Catches HoldThe Idea Catches HoldJanuary, 2002
The NWS Makes a Bold The NWS Makes a Bold DecisionDecision
Fall 2004
Regional Collection Concept Regional Collection Concept Proposed 5 Yrs AgoProposed 5 Yrs Ago
NCEP
NCDC
Reiterated Fall 2002Reiterated Fall 2002
NWS is implementing a data delivery system NWS is implementing a data delivery system based upon the CRAFT concept (regional hubs, based upon the CRAFT concept (regional hubs, Abilene as the backbone) Abilene as the backbone)
This infrastructure could become This infrastructure could become THETHE NWS data NWS data and model grid delivery system of the futureand model grid delivery system of the future
Current StatusCurrent Status
We’ve Come a Long Way!We’ve Come a Long Way!Today – 123 Radars!!
– Initiated after 2001 Stakeholders WorkshopInitiated after 2001 Stakeholders Workshop– NWS agreed to let OU provide real time Level II data to NWS agreed to let OU provide real time Level II data to
private companies and others (academia), private companies and others (academia), as R&D as R&D partnerspartners, in exchange for latency and other information , in exchange for latency and other information to help evaluate system performanceto help evaluate system performance
– Any group could participateAny group could participate– Provided an opportunity for private sector partners to Provided an opportunity for private sector partners to
gain experience using Level II data and develop new gain experience using Level II data and develop new capabilities with themcapabilities with them
– Had 6 private sector partnersHad 6 private sector partners– Companies agreed to the collaborative nature of the Companies agreed to the collaborative nature of the
project, i.e., project, i.e., everyone wins when everyone contributeseveryone wins when everyone contributes
Private Sector Joins the Private Sector Joins the EffortEffort
The Benefits – Industry The Benefits – Industry DevelopmentDevelopment
Baron Services, Inc.
A True Public-Private A True Public-Private PartnershipPartnership
Recent NOAA + Pvt Sector Money $3.0M
123 NEXRAD radars are delivering Level II data to University of 123 NEXRAD radars are delivering Level II data to University of Oklahoma in real timeOklahoma in real time– Subsequent distribution to more than 30 users including NCDC, NWS HQ, UCAR Subsequent distribution to more than 30 users including NCDC, NWS HQ, UCAR
Unidata, numerous universities and NOAA laboratories, and several (paying) private Unidata, numerous universities and NOAA laboratories, and several (paying) private companiescompanies
Extremely high reliability (99+%) with latency less than 10 secExtremely high reliability (99+%) with latency less than 10 sec Top-tier nodes announced during week of 12 April 2004Top-tier nodes announced during week of 12 April 2004
– University of OklahomaUniversity of Oklahoma– Purdue UniversityPurdue University– ERCERC
Current StatusCurrent Status
IRaDS will continue to collect and analyzenetwork information for R&D
Data Ingest is Sized to Handle Data Ingest is Sized to Handle Future NEXARD ImprovementsFuture NEXARD Improvements
EnhancementEnhancement Factor of Data Factor of Data Flow Increase Flow Increase (Relative to (Relative to
August 2002)August 2002)
Possible Possible ImplementatioImplementatio
n Daten Date
Cumulative Cumulative Data Flow Data Flow Increase Increase
(Relative to (Relative to August 2002)August 2002)
4.1 min VCP4.1 min VCP 1.241.24 Jan 2003Jan 2003 1.241.24
TCP/IP TCP/IP Wideband Wideband InterfaceInterface
1.001.00 July 2004July 2004 1.241.24
0.25 km 0.25 km Reflectivity Reflectivity DataData
1.561.56 October 2004October 2004 1.931.93
0.5 deg 0.5 deg Azimuth Azimuth SamplingSampling
2.002.00 January 2005January 2005 3.863.86
Doppler Data Doppler Data to End of 2to End of 2ndnd TripTrip
1.491.49 October 2005October 2005 5.765.76
Clutter Filters Clutter Filters and SNR and SNR ThrshldThrshld
1.971.97 January 2005January 2005 11.3511.35
Dual Dual PolarizationPolarization
2.322.32 January 2008January 2008 26.3526.35
OU and NSSL have worked with FAA Mike Monroney OU and NSSL have worked with FAA Mike Monroney Aeronautical Center in OKC to transmit TDWR data in real Aeronautical Center in OKC to transmit TDWR data in real timetime
Can be extended to all TDWR, ASR and ARSR systemsCan be extended to all TDWR, ASR and ARSR systems The idea of using Abilene to transmit data was suggested The idea of using Abilene to transmit data was suggested
by Lincoln Laboratory 2 years after CRAFT was underwayby Lincoln Laboratory 2 years after CRAFT was underway
Weber (2000)
Addition of FAA Air Traffic Addition of FAA Air Traffic RadarsRadars
Baron Services centrally collects data now (Huntsville, AL)Baron Services centrally collects data now (Huntsville, AL) TV stations only scan lowest elevation angle, which would TV stations only scan lowest elevation angle, which would
be very useful in biological target ID and trackingbe very useful in biological target ID and tracking
Possible Addition of TV Station Possible Addition of TV Station RadarsRadars
Baron Services TV Radar Sites
Linked Environments for Linked Environments for Atmospheric Discovery (LEAD)Atmospheric Discovery (LEAD)
A Cyberinfrastucture for Mesoscale Meteorology A Cyberinfrastucture for Mesoscale Meteorology Research and EducationResearch and Education
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
Traditional PredictionTraditional Prediction
OBSERVATIONSOBSERVATIONS
Radar DataRadar DataMobile MesonetsMobile Mesonets
Surface ObservationsSurface ObservationsUpper-Air BalloonsUpper-Air BalloonsCommercial AircraftCommercial Aircraft
Geostationary and Polar Geostationary and Polar Orbiting SatelliteOrbiting Satellite
Wind ProfilersWind ProfilersGPS SatellitesGPS Satellites
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
Traditional PredictionTraditional Prediction
OBSERVATIONSOBSERVATIONS
Radar DataRadar DataMobile MesonetsMobile Mesonets
Surface ObservationsSurface ObservationsUpper-Air BalloonsUpper-Air BalloonsCommercial AircraftCommercial Aircraft
Geostationary and Polar Geostationary and Polar Orbiting SatelliteOrbiting Satellite
Wind ProfilersWind ProfilersGPS SatellitesGPS Satellites
The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!
The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static
OBSERVATIONSOBSERVATIONS
Radar DataRadar DataMobile MesonetsMobile Mesonets
Surface ObservationsSurface ObservationsUpper-Air BalloonsUpper-Air BalloonsCommercial AircraftCommercial Aircraft
Geostationary and Polar Geostationary and Polar Orbiting SatelliteOrbiting Satellite
Wind ProfilersWind ProfilersGPS SatellitesGPS Satellites
Analysis/Assimilation
Quality ControlRetrieval of Unobserved
QuantitiesCreation of Gridded Fields
Prediction
PCs to Teraflop Systems
Product Generation, Display,
Dissemination
End Users
NWSPrivate Companies
Students
OBSERVATIONSOBSERVATIONS
Radar DataRadar DataMobile MesonetsMobile Mesonets
Surface ObservationsSurface ObservationsUpper-Air BalloonsUpper-Air BalloonsCommercial AircraftCommercial Aircraft
Geostationary and Polar Geostationary and Polar Orbiting SatelliteOrbiting Satellite
Wind ProfilersWind ProfilersGPS SatellitesGPS Satellites
The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static
MesoscaleWeather
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
NWS National Static Observations & Grids
MesoscaleWeather
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
NWS National Static Observations & Grids
MesoscaleWeather
Local Observations
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
UsersADaM ADAS Tools
NWS National Static Observations & Grids
MesoscaleWeather
Local Observations
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
UsersADaM ADAS Tools
NWS National Static Observations & Grids
MesoscaleWeather
Local Observations
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
MyLEADPortal
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Interaction Level I
UsersADaM ADAS Tools
NWS National Static Observations & Grids
Experimental DynamicObservations
MesoscaleWeather
Local Observations
MyLEADPortal
Local Physical Resources
Remote Physical (Grid) Resources
Virtual/Digital Resources and Services
LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather
Interaction Level II
New NSF Engineering Research Center New NSF Engineering Research Center for Adaptive Sensing of the Atmosphere for Adaptive Sensing of the Atmosphere
(CASA)(CASA) UMass/Amherst and OU are lead partnersUMass/Amherst and OU are lead partners Total funding over 10 years: Total funding over 10 years: $100M$100M Concept: inexpensive, phased array Doppler Concept: inexpensive, phased array Doppler
radars on cell towers and buildingsradars on cell towers and buildings Adaptive dynamic sensing of multiple targets Adaptive dynamic sensing of multiple targets
including aircraftincluding aircraft
top related