extracting meaningful results from the national ... · extracting meaningful results from the...
TRANSCRIPT
![Page 1: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/1.jpg)
Wisconsin Traffic Operations and Safety Laboratory
Department of Civil and Environmental EngineeringUniversity of Wisconsin-Madison
Extracting Meaningful Results from
the National Performance
Management Research Data Set
Glenn Vorhes, PE, GISP
2018 AASHTO GIS for Transportation Symposium
March 20, 2018
![Page 2: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/2.jpg)
What is the NPMRDS?
• Fine grained vehicle travel time / speed information for all* of North America• Aggregated in 5 minute bins for passenger, freight, and combined.• Formerly originated from Navteq/HERE (V1) and later from INRIX (V2)• Derived from probe vehicle data. Passive, No fixed/moving sensors, “floating car”
• Cellular• Reidentification (Bluetooth)• Toll tags, license plates, MAC IDs, etc.• GPS breadcrumbs
• Time, location, speed, heading• Processing done internally; results from the black box. Typically consistent and
clean.
• Unit of analysis is the TMC (Traffic Message Channel)• Directional segments** with lengths from 0.4 to 3.4 miles, ~1 mile typically
• Free!*** GPS probe data costs hundreds, up to thousands for real time
* Limited to the NHS** Structure and definition of TMC segments changed with the current version, still changing*** No free lunch, data is available but comes with usage restrictions
![Page 3: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/3.jpg)
What’s it Good For?• Performance Metrics, MAP-21 reporting requirements
• Travel Time Index peak-period travel time / free-flow travel time*• Planning Time Index 95th percentile travel time / free-flow travel time*
• Congestion Analysis• Delay Calculations• Work zone performance, impacts
• Let the data determine the baseline for each segment• *FHWA free flow – 85th percentile off-peak times, M-R: 9-4, 7-10, SU: 6am-10pm• Make your own! Ex. Before, during, after construction
![Page 4: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/4.jpg)
Pros and Cons (challenges)
+ -
• Spatially and temporally granular• Compares favorably with other data sources
• No volume information• Other sources needed for delay
• NoData• No readings or too slow
• Low sample size• One vehicle?
• May have sampling bias• Seems high on freight vehicles
• Not useable as a navigable network for routing applications
• Version 2• Naming inconsistencies
• Ex. Name: “I-39 N”, Direction: “N”• Moving target
• Addition, removal, and modification of TMCs without formal release schedule
![Page 5: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/5.jpg)
Project GoalsStorage, Access, and Visualization of NPMRDS
• NPMRDS is a valuable resource albeit with some processing and storage difficulties
• Received as a collection of shapefiles and CSVs• Quickly exceeds limits of Excel• Joins with spatial data for visualizations inefficient at best
• Two NPMRDS Versions with widely differing structures increase complexity• A database solution is necessary
• Efficient joins• Indexing on TMC id and time representation• In database stored procedures/functions for aggregation, statistics
• Data is right there• Beats any attempt at a full query with for subsequent analysis with
Python (NumPy/SciPy), R, Stata, MATLAB, etc.• Spatial indices for efficient queries by extent (PostgreSQL/PostGIS)• Power and storage for billions of travel time rows (Oracle)
![Page 6: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/6.jpg)
Version Transition – Data AcquisitionVersion 1 Version 2
• FTP Site managed by HERE (formerly Navteq)
• Handy method for bulk downloads for whole of North America
• Huge files requiring considerable processing just to open to say nothing of database storage; gzipped tarball with gzipped CSVs
• Regional Integrated Transportation Information System (RITIS)
• NPMRDS component managed by Center for Advanced Transportation Technology (CATT) Laboratory at the University of Maryland
• Massive Data Downloader• Nice query tool, harder for bulk download
• Other analytics tools in dashboard, not free
8 GB+
![Page 7: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/7.jpg)
RITIS NPMRDS Interface
npmrds.ritis.org
![Page 8: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/8.jpg)
Version Transition – Spatial DataVersion 1 Version 2
• “Static” Table • Road Name, Direction• Complementary Information
• Links (Shapefile)• Geometries• Can be shared between TMCs• Digitization inconsistent with travel
• Lookup Table• Link Id, TMC, Digitization/Travel Flag
• Database procedure to join TMCs and links• Copy if necessary• Flip if necessary – travel and digitization
direction match• Dissolve by TMC• Simple spatial analysis and mapping
• Flat Structure – No more lookup table• Geometry and road information in the
same table• Travel and digitization consistent• No shared geometries
• Introduction of inner and outer TMCs
• Example 110_01235 where _ is:• P (N/E) or N (S/W), Internal Paths• + (N/E) or - (S/W), External Paths
![Page 9: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/9.jpg)
Version Transition – Times/SpeedsVersion 1 Version 2
• Date-Time as separate columns• Date• Epoch, 0-287 5 minute bins per day
• Travel times for passenger, freight, and all/combined in one row
• TMC lengths can change• Typically small changes but some
considerably larger ones• Updated once or twice a year• Need mechanism to reference a specific
version
• All of North America loaded• 21,621,017,302 Travel Time Rows
• True datetime representation in CSV• mm/dd/yyyy HH:MM
• Selectable metrics in download interface• Speed• Historic Average Speed• Reference Speed• Travel Time• Data Density?
• Can choose any or all of passenger, freight, or all vehicles
• Each vehicle classification in one row• Requires additional indexing
• Only loaded for Wisconsin so far,• Plans for more, Surrounding States, beyond
![Page 10: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/10.jpg)
Coverage Changes
•Version 2 is mostly a subset of Version 1
•Very few additions
•Version 2 has been a moving target –Coverage updates but no release schedule
•Both versions were/are intended to cover the Federal Highway Administration (FHWA) National Highway System (NHS) and nothing more
•Depends on State input
![Page 11: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/11.jpg)
Coverage
NHS vs
NPMRDS
•Good but certainly not perfect
• Excellent resource but causes a lot of frustration for performance metrics
![Page 12: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/12.jpg)
NPMRDS API
• RESTful interface to retrieve desired NPMRDS information• To date, focused on consumption by web applications, JSON format• Planned to allow an additional format request parameter to trigger
download of a zip file containing spatial and tabular information
• Eliminates need for version specific application code
• Only requirement is that the request doesn’t span the version break• Underlying geometries are different• Database queries against separate tables
• Development Efforts• Smooth the transition, TMC Matching• Improved query performance• Spatial and date range limits• API key and request logging
![Page 13: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/13.jpg)
Architecture
Database
Server
Time/Speed Data Spatial Data
Version 1< 1/31/2017
Version 2> 2/1/2017
Dependent ApplicationsAPI Request
(Geo)JSON
Desktop GISArcMap Query LayerQGIS native support
npm install github:glennvorhes/webmapsjsExample: import {quickMap} from 'webmapsjs/dist/olHelpers/quickMap’;Documentation: https://transportal.cee.wisc.edu/gis/webmaps/webmapsstatic/docs/js/index.html
Statistics
![Page 14: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/14.jpg)
Example Applications – API GUI
![Page 15: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/15.jpg)
Heatmap GeneratorMajor Incident occurred on March 27, 2015 near Janesville, WIOverturned semi, chemical spillHeatmap provides visualization of travel speed relative to TMC free flow speed
![Page 16: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/16.jpg)
Historic Travel TimesShow progression of delay for all segments in a region over the course of a dayMoving window average with user selectable range (20 minutes down to raw 5 minute bins
![Page 17: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/17.jpg)
August 21, 2017 - Eclipse
![Page 18: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/18.jpg)
Year ComparisonChange in a variety of metrics from one year to the next.Example: Measure of travel time reliability (80th / 50th percentile) speedLower is better; more consistent travel times
![Page 19: Extracting Meaningful Results from the National ... · Extracting Meaningful Results from the National Performance Management Research Data Set Glenn Vorhes, PE, GISP ... Heatmap](https://reader035.vdocuments.us/reader035/viewer/2022062918/5ede1711ad6a402d66695ed2/html5/thumbnails/19.jpg)
Conclusion
• Development work yet to be done• Improved query performance, smarter indices• Smooth transition between NPMRDS versions• More retrieval formats• New visualizations
• Reduced barrier to entry for web application and advanced desktop analytics
Thank you for coming!Questions?