status of geoserver wps
TRANSCRIPT
Status of GeoServer WPS
Ing. Andrea Aime, GeoSolutions
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Introductions
Your host:
Andrea Aime (geowolf on Twitter)
Long time developer and project steering committee member of GeoTools and GeoServer
Map rendering, spatial analysis, data access, reprojection, security, …
His company
GeoSolutions
Specialized in open source solutions based on GeoTols/GeoServer/GeoNetwork/MapStore, strong focus on image processing, spatial analysis, catalogs, web and mobile front ends for mapping and data analysis
WPS: five minute intro
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Web Processing Service
Wikipedia introduces OGC WPS as:
[A service] designed to standardize the way that GIS calculations are made available to the Internet.
WPS can describe any calculation including all of its inputs and outputs, and trigger its execution
The specific processes served up by a WPS implementation are defined by the owner of that implementation.
Although WPS was designed to work with spatially referenced data, it can be used with any kind of data.
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
WPS 1.0 operations
Standard operations:
GetCapabilites
DescribeProcess
Execute
GeoServer extended operations:
GetExecutionStatus
GetExecutionResult
Dismiss (new, borrowed from WPS 2.0)
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
GetCapabilities
Some metadata about the server and its owner
The list of available processes, with a description
Let’s say we’re interested in the JTS:buffer process
FOSS4G 2011, Denver 12th-16th September 2011
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
DescribeProcess
List of inputs and outputs
Descriptions
List of accepted formats (not shown in the xml)
FOSS4G 2011, Denver 12th-16th September 2011
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Execute
Buffer a L shaped geometry with distance “2”
Get the result back as GML
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Synchronous vs asynchronous
WPS client
WPS
Launch process
Send back results
Simple Suitable for fast executions
Synchronous
WPS client
WPS
Launch process
Status URL
Check progress GetExecutionStatus
50%
Check progress GetExecutionStatus
100% Full results
More complex Suitable for longer computations
Asynchronous
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Inline vs reference outputs
WPS client
WPS
Launch process
a) Get back XML with results embedded
Simple Suitable for smallish outputs
Inline XML/raw
WPS client
WPS
Launch process
Get back XML with links to results
Download each result
GetExecutionResult
More complex Suitable for large downloads
(allows for retry/resume)
Reference outputs
b) Get back result directly
GeoServer: Integrated WPS
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Common WPS setup
WPS
Remote WCS
Remote WFS
HTTP server
WPS client
Request +
data or links to data
Result data
Fetch data
Local filesystem
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
GeoServer WPS integration
WPS
Remote WCS
Remote WFS
HTTP server
WPS client
GeoServer Layers
WMS client WMS
GeoServer UI
Local filesystem
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Demo request builder
List processes
Describe
Set parameters and execute
All in one form
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Local layers and chaining
FOSS4G 2011, Denver 12th-16th September 2011
WFS
WMS
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Rendering transformations
On-the-fly data transformations inside rendering
chain
Calling WPS processes from SLD docs
Optimized for performance
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Rendering transformations
FOSS4G 2011, Denver 12th-16th September 2011
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Rendering transformations
FOSS4G 2011, Denver 12th-16th September 2011
Writing processes
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
First, do you need to?
• 80+ built-in processes
• JTS geom manipulation
• Compound WFS query
• Raster to vector
• Vector to raster
• Stats analysis processes
• Designed for chaining
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
But if you really need to …
Python Groovy
JavaScript Ruby
JSR-223
http://geoscript.org
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• ProcessFactory
• Enumerates processes it can generate
• Describes them
• Creates instances of them
• Suitable for wrapping another set of existing processes (e.g., a GRASS, a Sextante, a GDAL command line utils wrapper)
Low level Java API
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• You write the javabean, annotate it to advertise and describe inputs and outputs, finally declare the bean in the Spring application context:
Java annotations
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• Community modules to write processes in scripting languages for the JVM: Jython, Javascript, Groovy, …
• Just drop a script file in the right directory and the process is ready to be used
GeoScript scripting languages
Python Groovy JavaScript Ruby
http://geoscript.org
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Sample scripting application
http://localhost:8080/geoserver/www/wps.html
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Land property distrib. in the polygon
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
A look at the data dir
1. Data
2. WPS process scripted in Jython
3. The GUI
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
The Jython script
http://geoscript.org/py/index.html
What’s new in GeoServer 2.7.x
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• So far, only the node that was running the async process knew about it
Asynchronous process clustering
Client Get Process
Status
Load balancer
WPS1
WPS2
WPS3
??? Get Process Status
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• ProcessStatusStore, shared state of the currently running processes
• ProcessArtifactsStore, shared outputs store
Asynchronous process clustering
Client Get Process
Status
Load balancer
WPS1
WPS2
WPS3
Get Process Status
Hazelc
ast
sh
are
d s
tatu
s
!!
Sh
are
d f
ilesyste
m
for
larg
e o
utp
uts
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• GUI to ProcessStatusStore
• Also allows admin to dismiss process executions
Status page available to admins
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• Allows the client to dismiss the execution of an asynchronous process
• Borrowed from WPS 2.0 (as a new vendor op in WPS 1.0)
Dismiss
http://host:port/geoserver/ows?service=WPS&version=1.0.0&
request=GetExecutionStatus
&executionId=397e8cbd-7d51-48c5-ad72-b0fcbe7cfbdb
http://host:port/geoserver/ows?service=WPS&version=1.0.0&
request=Dismiss
&executionId=397e8cbd-7d51-48c5-ad72-b0fcbe7cfbdb
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
• All processes get immediately dismissed if they are in the following states
• Queued
• Reading inputs (poison pill on inputs)
• Writing outputs (poison pill on outputs)
• A process that’s running needs to give up willingly (it gets a dismiss notification). We cannot just kill a a thread in Java ;-)
Dismiss
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Process security
• GeoServer comes with 80+ built-in processes
• It was already possible to selectively disable them
• Now, each can be associated to one or more user roles
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Global process limits
• Make sure each Execute request does not chew too much resources
• Limit how many processes run in parallel
• Limit how much time a process can run
• Automatic dismiss past the limit
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
Local process limits
• Add limits on input sizes, range of acceptable values, max number of values for repeated inputs
• All advertised in DescribeProcess
• Pluggable framework to add more input validators
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
New output formats: GPX and KML
• Java processes have java objects as input and outputs
• ProcessParameterIO instances translate common java Object into external formats
• E.g., FeatureCollection GML, GeoJson, CSV
• In GeoServer 2.7.x there are two new community modules to transform FeatureCollection also into GPX and KML
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
New «Download» process
• Single process for «advanced clip and ship»
• Filter, reproject, cut on mask both raster and vector data
• WPS async support key for large data extractions (WFS and WCS not really suitable for this case)
• Evaluate in advance download size to enforce max extraction size limits
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
New «Download» process
• A GUI to the process, as a MapStore customization
What’s cooking
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
ogr2ogr WPS output formats
• Extension to the ogr2ogr WFS output format
• Can turn any FeatureCollection output type into any of the ogr2ogr supported output formats
• Can be already downloaded in 2.8.x nightly builds
WPS GML ogr2ogr Desired format
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
WPS-remote
• You have a bunch of data processing servers, with native/command line «processes»
• Want to discover them, expose as WPS, invoke, track progress, and publish results
WPSAgent.py
Ge
oS
erv
er
WPS remote module
XMPP WPSAgent.py
WPSAgent.py
Flooding model
Fire propagation
model
Snow levels forecast
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
WPS-remote
• Asset Allocator example: on demand counter-piracy ship positioning wrt wheather & other conditions
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
WPS-remote
• Automatic publishing of result as new layers via the importer module
Ge
oS
erv
er
WPS remote module
XMPP WPSAgent.py
Snow levels forecast
Importer
Shared filesystem
1) Execute 2) Get results location
3) Call importer
4) Side effect: publish result as new layer
5) Return OWS context
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
More WPS integration
• Mind, still on the wish list, but with good chance of getting funding
• «Automate» module, would allow periodic operations of new data in GeoServer. Focus on data ingestion, temporary layer management, it would integrate internal WPS process calls
• Allow calling WPS processes from Importer module, for more complex input data transformations before publication
FOSS4G-NA 2015, San Francisco March 9th-12th 2015
WPS 2.0
• Still looking for sponsors!
• Could it be you?