how to optimize gis workflows

How to Optimize GIS Workflows

With Mark Stoakes & Steve MacCabe

Who are we anyway?

Connect Virtually Any Format with FME

FME & GIS can do amazing things

Here is a taste of what’s possible.

Manage your attributes

Validate your attributes

Distribute a Validation Report

Automate the process

Managing Attributes

Attribute ManagementWhy another attribute handling transformer?

➢ Because there are already too many… plus we needed a couple other operations

Goal: consolidate the attribute management transformers

Doomed transformers:AttributeCreatorAttributeCopierAttributeRenamerExpressionEvaluatorStringConcatenatorAttributeKeeperAttributeRemover

Plus… …many string operations:case changetrim/padsubstring/regex

One transformer to rule all your attributes

● Create attributes

● Rename attributes

● Copy attributes

● Remove attributes

● Set attribute values

● Re-order attributes

● Concatenate attributes (Text Editor)

● Arithmetic expressions (Arithmetic Editor)

● Set conditional values (Conditional Value)

● Reference earlier attributes

AttributeManager

AttributeManager creates an opportunity for more focused and simplified attribute management.

#ThanksFME2016

Let’s Go - Demo!

AttributeManger:

Lets you organize your attributes in

your workflows

Attribute Validation

Attribute Validation – why?

If you distribute poor quality data then you've amplified the data issue.

If you distribute good quality data you've amplified the benefits.

Data validation is key to many FME workflows…

Attribute Validation – Standards

Bunch of ISO standards – ISO19xxxMostly replaced by - ISO 19157

Also CAD & Data Model Standards

Attribute Validation – FME ToolsAttribute Validation – Key Tools

● Tester and TestFilter● AttributeCreator (conditional tests)● AttributeClassifier● AttributeRangeFilter & AttributeFilter● NullAttributeMapper● DuplicateRemover● StringSearcher● StatisticsCalculator

AttributeValidator...

Goals:Simplify validation workflows (Reduce number of transformers)

Be more confident about the data you’re loading into a destination dataset

...say goodbye to bad data

#ThanksFME2016

Attribute ValidationSome of the validation tests available:

➢ Attribute type (integer, float, char, xml, json, etc.)➢ In – either a list or range – good for domain validation➢ Regular Expressions➢ Unique➢ Not Null➢ …and more…

Attribute Validation Output Output is:

➢ Summary Attribute:_fme_validation_message

➢ List of all messages: _fme_validation_message_list{}

Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression '[ABCD]{3}'{0} Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression '[ABCD]{3}'{1} Attribute 'num_measures' with value '12' fails check for in Range '[0,10]'{2} Attribute 'CodePrefix' with value 'ABE' fails check for in 'ABC,ABD,TXU,TXV'

Let’s Go - Demo!

AttributeValidator:

Lets you scrutinize your attributes and

simplify your data validation

workflows

Q&AChat your questions to us

Writing Features...a different approach

Traditional vs. state-of-the-art...Write data with a writer

Write data with a FeatureWriter

Write the conclusion to your data process

Write data in the middle of workflows to avoid chaining workspaces.

Write the summary of your data output.

#ThanksFME2016

Use the FeatureWriter to allow post-processing freedom.

#FreeYourData

Some Easy Wins:• Create summary reports

– Email an attachment• Copy data after writing

– FTP results– Load into Dropbox

Automation Workflows• Quality checking• Database operations• Database validation• FME Server Notification

Transformation

...transform your data

A new pattern for database workflows

Transform your database workflows

SQLCreator

...drop constraints

FeatureReader

...read your data

Transformation

...transform your data

FeatureWriter

...write to database

SQLExecutor

...enable constraints

SQLExecutor

...validate/index/unique

A new pattern for FME Server notification workflows

FeatureReader & FeatureWriter simplies you workspace

Let’s Go - Demo!

FeatureWriter:

Lets you post-process your

output data

Automation

FME Server OverviewFME Engine workhorse and runs FME Workspaces

Automate Deliver Data Download Data

Love

FME!

ScaleReal Time Data

Let FME Server do the work

Examples of things we can automate:● Directory watcher● Run the workspace through a notification ● Data Delivery

○ Dropbox○ email

Let’s Go - Demo!

Summary

Take advantage of our new transformers.

Free StuffThere are too many to list so be sure to check out www.safe.com

for more.

Knowledge Articles● AttributeManager● AttributeValidator

Blog Posts● FeatureWriter● AttributeManager

Free FME Training● FME Desktop Basic: March 8-9● FME Server: March 15-16● FME Desktop Advanced: March 22-23

60+ CITIESApril - J une

Poll Question

Which of today's tasks do you see yourself performing on your GIS data?

Q&AChat your questions into us

Thank you!

Get a free trial of FME at www.safe.com

Resources: Attribute ValidationOriginal PR: 8945Knowledge Center: Validate your Data's Attributes with the AttributeValidator Transformer

Webinar 2015: Data Validation Victories: Tips for Better Data Quality:

Data Validation & Quality Assurance

WWTour:FME & Data Validation: http://www.slideshare.net/SafeSoftware/fme-data-validation

WWTour 2014: Validation and Quality Control with FME & FME Server and also at SlideShare

Validating your data before loading it into a database: video by Robyn

FME 2014 UC - Colonial Pipeline: CAD Data Validation using FME

Automated Quality Control: ISo 9000:2000 Quality Measures ISO 19157/19131 Metria 2014

Berk Bayer 2014 Application of FME to Maintain Quality Control (QC) over Standardized Spatial Data

FME WWTour 2013 – Introducing Geometry Validator

Resources: AttributeManagerKnowledge Center: Control Your Data's Attributes with the AttributeManager Transformer