baking an fme dashboard: two cups of python & a dash of frustration
TRANSCRIPT
Baking an FME Dashboard: Two cups of Python with a pinch of frustration
Jubal HarpsterPrincipal and Co-Founder Spatial Development International
Thank You!
A little about Me:
Regular FME User Since 1998
Started translating to ArcSDE at version 3.
FME Server Beta User since ~2007
I Use FME to..
Get Features & Checkins from Fousquare
Detect Changes to Open Street Map
Creating Regional Basemaps
The “Dashboard”
The Business Problem in a nutshell Port of Seattle runs multiple FME translations
Need the ability to audit transformations
Need to verify if people are running their workspaces
Need to know what was updated when and by who.
Need to know if any features failed Be able to ‘see’ the results enterprise wide on
Dashboard
The “Dashboard”
Piece of Cake!
Decided early to use TCL script to get windows user name TCLCaller: set x $tcl_platform(user)
Create a simple custom transformer FMX that can easily be ‘dropped’ into each workspace
With some Python on Shutdown Invoke pyfme
The “Dashboard”
The Approach Custom Transformer for flexibility across
workspaces TCL for minor functions
Python Statup and Shutdown Scripts Start counts End counts Status messages Errors etc
The “Dashboard”: Challenge One
Start Up only have access to limited information
Problem: No way to tell FME the destination before it gets there (to get start counts)
Solution: Published Parameters written to the Log
The “Dashboard”: Challenge Two
Shutdown Scripts only have access to limited information
Problem: FME loses generated values (like User Name)
Solution: Track values in the logs and in the DB using SQL
The “Dashboard”
Our elegant & simple solution becomes a dreaded log file parse
The “Dashboard”:Challenge Four
But what about errors? How do we test? How do we record problems or know if FME
crashes?
Problem: If things really go wrong Shutdown won’t execute
Solution: The Terminator
The “Dashboard”:Challenge Five
More Errors Sometimes 3rd party formats kill processes
(i.e. illegal CAD features in ArcSDE)
The Problem: Sometimes bad things happen
The Solution: Set FME To Ignore Errors
The “Dashboard”: The Process
• Insert failures
• Grab Parameters
Startup
• Log Features
• Log Derived Parameters
Mid-way
• Update SuccessShut
down
The “Dashboard”: Using pyfme
Pyfme in Python callers treats input as features Create Geometry Get & Set Attributes
Can Access Published Paramters param_buffwidth = pyfme.macros.BufferWidth
Access Transformers and Piplines
Tips: The integrated IDE is OK. But use something
else. I use Aptana Studio or Eclipse & pydev
Create a class that invokes the pyfme modules
class MyPythonFactory(object): def __init__(self):
pass def input(self,feature):
self.pyoutput(feature) def close(self):
pass
The “Dashboard”: Using pyfme
Using pyfme
Tips: Use Custom Transformers
Embedding PythonCallers in a custom transformer is an easy way to install and distribute functions
There are lots of Python developers out there Python samples at FMEpeida Olivers Python Corner on FMEpedia Python.org
@FMEDoctors on twitter can help in a pinch But only if the problem can be solved in 140 chars
Thank You!
Questions?
For more information: Jubal Harpster
[email protected] @jharpster
Spatial Development International http://www.spatialdev.com