Download - FME and FEMA's National Flood Hazard Layer
![Page 1: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/1.jpg)
FME & FEMA’s National Flood Hazard Layer (NFHL) Rob Gaines and Eliza Ledwell
![Page 2: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/2.jpg)
About Us
Eliza Ledwell Rob Gaines
IBM – FEMA Risk MAP Customer and Data Services,
Managing Consultant [email protected]
IBM – FEMA Risk MAP Customer and Data Services,
Data Services Manager [email protected]
![Page 3: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/3.jpg)
FEMA Risk MAP
Risk MAP Vision: • Work collaboratively with state & local entities • Deliver quality data • Increase public awareness • Reduce risk to life & property
Mapping, Analysis, and Planning
![Page 4: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/4.jpg)
FEMA Risk MAP
Customer & Data Services (CDS): • IT Hosting • Application development • Communications & user support • Consulting
Mapping, Analysis, and Planning
![Page 5: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/5.jpg)
FEMA’s National Flood Hazard Layer (NFHL)
• Widespread: all effective Flood Insurance Rate Maps for areas covered by digital data
• Updated daily: the single most up-to-date source of FEMA regulatory flood hazard information
• Public-facing: NFHL web services receive over 20 million hits per month
![Page 6: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/6.jpg)
Organization and Display The NFHL is organized into more than 50 data layers
The NFHL layers include:
• Flood hazard zones and labels • Base Flood Elevations (BFEs) • Cross-sections and coastal transects • Revision information such as LOMR and
FIRM Panel boundaries • Community boundaries and names • Structures such as levees, hydraulic, and
others involved in flood control
![Page 7: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/7.jpg)
Ways to Access NFHL Data
WFS
NFHL Status Page
Google Earth TM
WMS
REST
FEMA GeoPlatform
MSC
![Page 8: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/8.jpg)
NFHL on FEMA GeoPlatform
![Page 9: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/9.jpg)
Data Flow
Flood Study
Revisions
PTS CDS NFHL Users
Data Revision (PTS)
Data Production (Mapping Partners)
Application and Data Hosting (CDS)
![Page 10: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/10.jpg)
Data Flow
Flood Study
Revisions
PTS CDS NFHL Users
Application and Data Hosting (CDS)
![Page 11: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/11.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate
Data Extract Data
![Page 12: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/12.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate
Data Extract Data
FME
![Page 13: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/13.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data
Manual data movement • Data is received through download links in an email • Must be manually transferred to secure DHS hosting environment
![Page 14: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/14.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data
FME
Tracking – the data undergoes basic integrity checks and is logged into the NFHL Tracking database
![Page 15: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/15.jpg)
NFHL Tracking Database
![Page 16: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/16.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data
FME
QC – the data is checked for quality issues and rejected if issues are found
![Page 17: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/17.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data
FME
Staging – data is loaded to the NFHL Staging Database (an offline file geodatabase) once it has passed QC
![Page 18: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/18.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data
FME
Incorporation – data is published to the live NFHL database once it becomes effective
![Page 19: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/19.jpg)
How Does the NFHL Get Updated?
Receive Data Track Data QC Data Stage Data Incorporate Data Extract Data
FME
Extraction – data is extracted out of the live NFHL database to jurisdictional and state datasets and made available on www.msc.fema.gov
![Page 20: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/20.jpg)
Current Process
![Page 21: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/21.jpg)
Job Management
Current Process Overview
Detect new submission
files
File Repository
Track submission information
Tracking Database
QC submission
data
Stage submission
data
Publish datasets
Extract datasets
File Repository
NFHL DB
![Page 22: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/22.jpg)
Old System vs. Current Old • Manual input of
parameters • Manual running of
workspaces
Current • Automatic detection of
new submission files • Automatic job triggering
and population of parameters
• Increased tracking visibility
![Page 23: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/23.jpg)
Automatic File Detection
![Page 24: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/24.jpg)
Requirements Predefined file structure: L:\NFHL_Staging\TransferData\<BatchID>\<Submission Data>
![Page 25: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/25.jpg)
Requirements Standardized file formats: • ESRI Personal Geodatabase (.mdb) containing submission data • Excel sheet containing submission manifest • Matching filenames
![Page 26: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/26.jpg)
Automatic File Detection 1. Read in relevant files 2. Query log to see if files have been encountered 3. If new, record in log
Tracking DB FME Read-In
Model File Repository Downstream
Models
![Page 27: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/27.jpg)
Automatic File Detection Step 1: Read in relevant files using Directory and File Pathnames Reader pointed to base file repository
• Identify relevant filetypes
![Page 28: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/28.jpg)
Automatic File Detection Step 2: Query Tracking database to identify any datasets that have already been ‘seen’ by the system
• Disregard any seen datasets • Record any new datasets
![Page 29: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/29.jpg)
Automatic File Detection Step 3: File information can now be used to run downstream processes and associated with tracking entries
![Page 30: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/30.jpg)
Job Management Framework
![Page 31: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/31.jpg)
Job Management Basics • Very useful for automating multi-stage data processes
• Tracking database captures information about each submission and records the runtimes and results of each processing phase
• Parent jobs query tracking database to determine what work to perform, then spawn child jobs to handle each dataset
• Child jobs perform work and write results back to tracking database
Tracking DB
Parent Job Child Jobs Child Jobs Child Jobs
read
run
write
Submission Data
![Page 32: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/32.jpg)
Job Management
Current Process Overview
Detect new submission
files
File Repository
Track submission information
Tracking Database
QC submission
data
Stage submission
data
Publish datasets
Extract datasets
File Repository
NFHL DB
![Page 33: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/33.jpg)
Job Management Example: QC Step 1: Parent job queries tracking DB to determine datasets in need of QC
• Can use in-model logic or DB view/query
Query for datasets that have been tracked but not QCed
![Page 34: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/34.jpg)
Job Management Example: QC Step 2: Parent job runs child job for each dataset via WorkspaceRunner
• Use data elements from Tracking DB to set parameters for each job
Use data elements to set parameters
![Page 35: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/35.jpg)
Job Management Example: QC Step 3: Write results back to Tracking DB with child job
• Write timestamped status to master table so that future QC jobs will not re-check the same dataset
QC timestamp
![Page 36: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/36.jpg)
New Process: FME Server
![Page 37: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/37.jpg)
Current System vs. New Current (FME Desktop) • Manual file download and
upload • Transmission by email /
http download • Manual resolution of QC
issues
New (FME Server) • Web-based file upload • Realtime QC results • Fully automated
![Page 38: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/38.jpg)
Job Management
Current Process Overview
Detect new submission
files
File Repository
Track submission information
Tracking Database
QC submission
data
Stage submission
data
Publish datasets
Extract datasets
File Repository
NFHL DB
![Page 39: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/39.jpg)
Extraction Manager
Replication Manager
Submission Manager (FME Server)
New Process Overview
Log Submission
User Upload
Tracking Database
Publish datasets
Extract datasets
File Repository
NFHL DB
Track submission information
QC submission
data
QC Results
File Repository
![Page 40: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/40.jpg)
New Process Lessons Learned • FME Server plays very nice with parent/child job approach.
• It’s difficult to pass information between FME Server workspaces.
• Tracking Database is more important than ever with FME Server.
• It can be difficult to provide output reporting with the out-of-the-box FME Server UI.
• Error handling is important.
• The Schema Reader is awesome for quality control.
![Page 41: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/41.jpg)
SubmissionManager.fmw
![Page 42: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/42.jpg)
![Page 43: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/43.jpg)
The Future • Expanded self-serve reporting options
• Additional data submission / QC processes
• Data integration web services
![Page 44: FME and FEMA's National Flood Hazard Layer](https://reader036.vdocuments.us/reader036/viewer/2022082218/589d6af01a28abd91d8b5e65/html5/thumbnails/44.jpg)
Thank you! Robert Gaines IBM – FEMA Risk MAP Customer and Data Services, Data Services Manager [email protected]
Eliza Ledwell IBM – FEMA Risk MAP Customer and Data Services, Managing Consultant [email protected]