magical methods for batch data processing
TRANSCRIPT
![Page 1: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/1.jpg)
Potent Potions for Batch Data Processing
![Page 2: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/2.jpg)
250,000 CAD files & rasters on mobile devices
![Page 3: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/3.jpg)
![Page 4: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/4.jpg)
Tip: Know Your Potions and Choose Wisely
![Page 5: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/5.jpg)
Today’s Potions
1. Wildcards
2. Batch Deploy
3. Parent/child Workspaces
4. Parent/child Server Workspaces
![Page 6: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/6.jpg)
Potion 1: One Wild<card>
Dataset
![Page 7: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/7.jpg)
Multi-Dataset Picker
![Page 8: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/8.jpg)
Multi-Dataset Picker
![Page 9: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/9.jpg)
Multi-Dataset Picker
![Page 10: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/10.jpg)
Multi-Dataset Picker
Shapefile MapInfo
Most rasters
DWG DGN SQLite
![Page 11: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/11.jpg)
Dataset Wildcards
Extended glob syntax:
Symbol Matches
? Any single character
* Any sequence of zero or more characters
[chars] Any single character in chars.
[a-d] Any character between a and d inclusive
{a,b,...} Any of the sub-patterns a, b
/**/ 0 or more subdirectories
![Page 12: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/12.jpg)
Time to brew Potion 1
![Page 13: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/13.jpg)
Potion 1: Enticements
![Page 14: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/14.jpg)
Wildcard Bulk Data Processing
Enticements ü Simple to set up
ü Can transform across file boundaries
- Needs memory & time
![Page 15: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/15.jpg)
Potion 1: Pitfalls
![Page 16: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/16.jpg)
Wildcard Bulk Data Processing Pitfalls
x Recovery from data errors difficult
x Feature Type vs File vs Format Issues
x No granular log x No ability to
parallelize
![Page 17: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/17.jpg)
Potion 2: Batch Deploy
![Page 18: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/18.jpg)
Batch Deploy Script Writer
![Page 19: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/19.jpg)
Batch Deploy Script Writer
![Page 20: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/20.jpg)
Batch Deploy Script Writer
![Page 21: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/21.jpg)
Batch Deploy Script Writer
![Page 22: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/22.jpg)
Batch Deploy Script Writer
![Page 23: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/23.jpg)
Batch Deploy Script Writer
![Page 24: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/24.jpg)
Time to brew Potion 2
![Page 25: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/25.jpg)
Potion 2: Enticements
![Page 26: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/26.jpg)
Batch Deploy Enticements
ü Simple to set up ü Runs quickly ü Can script via
command line ü Run on demand
![Page 27: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/27.jpg)
Potion 2: Pitfalls
![Page 28: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/28.jpg)
Batch Deploy Pitfalls
x Recovery from data errors difficult
x No granular log x Destination dataset
naming can be tricky
![Page 29: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/29.jpg)
Potion 3: Parent/Child Workspaces
![Page 30: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/30.jpg)
Parent/Child Workspace Ingredients
• Parent Workspace: – PathReader – WorkspaceRunner
• Child Workspace:
– FeatureWriter
![Page 31: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/31.jpg)
Parent Ingredients
![Page 32: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/32.jpg)
Parent Ingredients
![Page 33: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/33.jpg)
Child Ingredients
![Page 34: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/34.jpg)
Time to brew Potion 3
![Page 35: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/35.jpg)
Potion 3: Enticements
![Page 36: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/36.jpg)
Parent/Child Workspace Enticements
ü Separate transformation from workflow
ü Generate audit logs ü All authored within
Workbench
![Page 37: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/37.jpg)
Potion 3: Pitfalls
![Page 38: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/38.jpg)
Parent/Child Workspace
Pitfalls
x Not all writers can be used concurrently
x Slow to run each child workspace separately
x Recovery from data errors not easy if concurrent runs used
![Page 39: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/39.jpg)
Potion 4: Parent/Child Server
Workspaces
![Page 40: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/40.jpg)
Parent/Child Server
Workspace Ingredients
• Parent Workspace: – PathReader – FMEServerJobSubmitter – FMEServerJobWaiter
• Child Workspace:
– FeatureWriter
![Page 41: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/41.jpg)
Parent Ingredients
![Page 42: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/42.jpg)
Parent Ingredients
![Page 43: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/43.jpg)
Parent Ingredients
![Page 44: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/44.jpg)
Child Ingredients
![Page 45: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/45.jpg)
Time to brew Potion 4
![Page 46: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/46.jpg)
Potion 4: Enticements
![Page 47: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/47.jpg)
Parent/Child Server
Workspace Enticements
ü Separate transformation from workflow
ü Generate audit logs ü All authored within
Workbench ü Make full use of
parallelism = FAST
![Page 48: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/48.jpg)
Potion 4: Pitfalls
![Page 49: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/49.jpg)
Parent/Child Server
Workspace Pitfalls
x Not all writers can be used concurrently
x Data needs to be accessible to Server Engines - Consider using Server Data Resources
x Craft your reload/audit plan
![Page 50: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/50.jpg)
Summary ● Many ways to handle bulk data moves
● Choose your potion wisely - each has pluses and minuses
● FME Server is the most robust automation choice
![Page 51: Magical Methods for Batch Data Processing](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ce84611a28ab210a8b6417/html5/thumbnails/51.jpg)
Questions?
Batch processing tutorial: fme.ly/b59