Using FME to Convert TIGER Spatial Data From Oracle Spatial to ESRI Shapefiles
Mei-Leng Freeman, Tim Puzycki, Jay E. Spurlin Spatial Products Software Branch Geography Division, Bureau of the Census
2
Introduction
• Public products – TIGER/Line Shapefiles – Partnership Program Shapefiles
• Internal products – Data Access and Dissemination Staff
(DADS) shapefiles for use in the American Fact Finder websites (internal and external)
– Live Individual Shapefile Request and Delivery System (LISRDS)
– Quality Control (QC) shapefile for various projects
3
Shapefile Creation: Legacy Approach
• Legacy TIGER – Utilized for 1990 and 2000 Decennial Censuses – In-house database format – Binary files – Partitioned by county
• Legacy Shapefile Generation (shpGen) – Java shapefile classes – Perl driver scripts – Directly read from a TIGER partition – Legacy TIGER C API – Directly wrote to a shapefile – ASCII Job configuration files (user defined tags)
4
Modernized Approach
• MAF/TIGER – The Legacy Master Address File (MAF) and
Topologically Integrated Geographic Encoding and Reference (TIGER) datasets were combined.
– Oracle Spatial and Oracle Topology Data Model v10.2
– Seamless, national data set • Current shpGen
– Perl driver scripts – Pre-generated FME mapping files (.FME) – FME Engine handles reading from Oracle Spatial
and writing to ESRI shapefiles
5
Initial Process
• Considerations for automation – Automate everything possible. – Use COTS software, wherever possible. – Ensure consistency of data after conversion. – Provide ease of maintenance. – Make source code reusable. – Fairly stable source data structure. – Fairly stable target shapefile formats.
6
FME Mapping File Usage
• Manually create FME mapping files (.FME) – Previous FME mapping files used as
templates. – A standardized set of macros is used.
• The FME mapping files are released as part of the software installation.
• Automated production control system drives the process and provides any dynamic inputs.
7
Oracle Wallet Support
• Stores database credentials: – Username – Password – Database
• Avoids providing Username and Password to command line interface
8
Example FME Mapping File
9
Example FME Mapping File
Manually Created FME Mapping Files
• TIGER/Line Shapefiles – Produced from “Benchmarks” (snapshots) of
the MAF/TIGER Database. – Public TIGER/Line shapefiles are posted on
the world wide web. • Partnership Shapefiles
– MAF/TIGER Partnership Software (MTPS)
• LISRDS Shapefiles – On-demand access to MAF/TIGER data – Data source may be “live” or a snapshot
10
Current process
• A wider set of shapefile requirements – Mapping performance issues
• Chained line representation of area boundaries • Un-duplication of lines in partitioned layers
– Visual representation issues • Partial features • ‘Balance of’ features
– Connect with Oracle Wallet
11
FME Mapping Files from FME Workbench
• DADS/AFF Shapefiles – Both .FME and .FMW files are archived with
the source code. – The .FME files are released to production,
and used by FME Engine. – A similar procedure (though not yet as
automated, and not making use of the geoprocessing tools) is used by another branch for the generalized and projected shapefiles.
– Samples on the following slides:
12
13
14
15
16
17
18
19
20
Example: DADS/AFF Line Shapefiles
21
State 09
State 05
State 02
State 13
Example: DADS/AFF Line Shapefiles
22
State 09
State 05
State 02
State 13
23
24
Minnesota (State 27)
Iowa (State 19)
25
Workbench File Example: County Subdivisions
26
Example: Connect with Oracle Wallet
Thank You!
• Questions? • For more information:
– Jay E. Spurlin • [email protected]
– Bureau of the Census • 4600 Silver Hill Road, Stop 7400 • Washington, DC 20233
• Products made using FME Engine – TIGER/Line Shapefiles
• http://www.census.gov/geo/www/tiger/index.html – American FactFinder (map base data)
• http://www.factfinder.census.gov/home/saff/main.html?_lang=en
