a first look at san francisco’s new etl job platform
TRANSCRIPT
A First Look at San Francisco’s New ETL Job Platform
Samuel ValdezJanine Heiser
Agenda
• About Us• Some Context• A Very Brief History• The ETL Job Platform• Workflow• Looking Ahead• Questions
About Us
City and County of San Francisco
Department of Technologyhttp://sfgov.org/dt/
Samuel ValdezEnterprise GIS
EngineerSan Francisco Enterprise
GIS Program (SFGIS)
Janine Heiser
Open Data Engineer
DataSF
Some Context
DataSF (open data program)Our mission is to empower use of the City’s data. Our vision is that the City’s data is understood, documented, and of high quality.
(With some help from the Department of Technology.)
Learn more at http://datasf.org/
Automated Data Publishing“... goal of increasing the number and timeliness of datasets on [SF OpenData]...”
SF OpenData (Socrata)
A Very Brief History
Why is SFGIS involved?• Well-established legacy• Traditionally shared (spatial) data• Most open-data were spatial• Enterprise perspective and relationships• Technical skills, tools, and experience
Generation 0• Predecessor platform• FME Server-based• Deployed around 2011 to support the EAS• Several key ETL jobs• Project champion• Organic evolution• Learning experience
The ETL Job Platform
Platform Architecture I
“ETL Job” Services
Platform Architecture II
Platform Architecture III
Workflow
“Requirements”• Few workspace authors• Need a safe place to try out workspaces• A few hundred ETL jobs (100-500?)• “Simple” ETL jobs• Use best-practices• And more
Workflow I
Workflow II
Workflow III
Looking Ahead
Looking Ahead x 2
Possible future talks• Version control• Supporting services, including geo• Job scheduling• Business continuity• Operations and maintenance• And more
Possible platform evolution• Scale out• More value-added services• Support streaming data• Third-party scheduler (not FME-centric)• Asynchronous ETL job execution• And more?
Questions?
Thank you!
Jeff Johnson – SFGIS & OD ManagerJoy Bonaguro – Chief Data OfficerMiguel Gamiño – City CIOAaron Koning – FME Server Product Mgr
