state of geoserver 2 - inspire · open source projects geotools geoserver geowebcache osgeo...
Post on 09-Nov-2018
225 Views
Preview:
TRANSCRIPT
State ofGeoServer 2.12
Andrea AimeTechnical Leadandrea.aime@geo-solutions.it@geowolf
GeoSolutionsInnovative, robust and cost-effective solutions leveraging best-of-breed Open Source products.
WelcomeOpen Source ProjectsGeoToolsGeoServerImageIO-ExtJAI-ExtJAI Tools
OSGeo FoundationCharter MemberGeoServer PSCGeoTools PSC
Jody GarnettCommunity Leadjgarnett@boundlessgeo.com@jodygarnett
BoundlessProvides geospatial tools and services for managing data and building applications.
WelcomeOpen Source ProjectsGeoToolsGeoServeruDig
OSGeo FoundationBoard MemberOSGeo Incubation ChairGeoTools Project Officer
Eclipse FoundationLocationTech Steering CommitteeLocationTech Technology Project
Open Source ProjectsGeoToolsGeoServerGeoWebCache
OSGeo FoundationGeoServer PSCGeoWebCache Lead
Kevin SmithSoftware Engineerksmith@boundlessgeo.com smithkm
BoundlessProvides geospatial tools and services for managing data and building applications.
Welcome
GeoServer 2017
GeoServer at a GlanceJava Web Application to share and edit geospatial data.
Publish data from any major spatial data source using open standards.
Core Protocols
WMS – mapsWFS – vector WFS-T – editing WCS – coverageWPS* – processCSW* – search
Tile Protocols
WMTS – tilesTMS – tilesWMS-C – tiles
28 → 37 members of team geoserver (to be updated624 → 619 No clue this year (check with Kevin)500+ → 525 pull requestsNew contributors being added(others stepping back)Smaller base of active contributors this year.Pull requests still going strong, very active codebase.Simone replaces Andrea as OSGeo Officer.13 releases this year
GeoServer Project StatusHow about team user-list?
2005 → 1400 (SF now checking on people not active on list)
Also geoserver-italia, espanol and opengeo.cn mailing lists
Release Schedule
stable maintenance
July 2.12.5
June
May 2.12.4
April
March 2.12.3
February 2.11.5
January 2.12.2
December 2.11.4
November 2.12.1
October 2.11.3
September 2.12.0
August 2.10.5 2.11.2
User List Participation● Answering users questions relies on a low number of people
Action:
● We are sending reminder on effective use of the user list on subscription and at the bottom of each mail
● Community building activity - remind developers about the definition of “supported”(not only reviewing pull requests, also answering user questions, used to be obvious, but it’s not obvious to devs hired into open source)
● Bug stomp introduced in 2016○ Relying on volunteer time not working out○ allowed developers to reserve time
● Moved to fixed schedule in 2017○ Arranging day each month too random
● Last Friday of each month○ Check blog post for tips on participation
Standing “last friday of the month” bug stomp
Held “sprint” to update from Restlet to Spring MVC for our REST configuration API.
Thanks to GeoSolutions for hosting, to Boundless and Astun for sending developers, and to all the sponsors who helped fund this important and badly needed, but boring work.
REST API Migration
2.12Viareggio Code SprintCommunity
REST API Migration
● Migration complete● REST API documented
○ Available online now!
2.12Viareggio Code SprintCommunity
Vector data sources
GeoPackage:
● Provides an sqlite database containing vector or raster data○ Initial idea from spatialite project○ Now an OGC Standard to replace shapefile and personal file geodatabase
● GeoServer module○ GeoPackage is a supported plugin in GeoTools○ We can now promote it to a GeoPackage extension ○ Needs some work, splitting into 4 GeoServer packages, store, wfs output format, wms output
format, wps output format○ Bug fix - raster mosaic geopackage had y-axis inverted.
GeoPackage (community)
2.12GeoSolutionsBoundless
CommunityBoundless Suite
● GeoServer importer supports GeoJSON files○ Allows GeoJSON to be imported into PostGIS database and published in one step○ Based on unsupported module in GeoTools
● Wish to package as a community module for GeoServer○ Allow GeoServer to directly publish GeoJSON files○ Text based, non indexed format, only suitable for small files
GeoJSON store
2.12Ian TurtonCommunity
● GeoServer importer supports CSV files○ Allows CSV to be imported into PostGIS database and published in one step○ Based on unsupported module in GeoTools (used as a tutorial for programmers)
● Package as a community module for GeoServer○ Allow GeoServer to directly publish CSV files○ Text based, non indexed format, only suitable for small files
CSV store
2.12BoundlessCommunity
CSV input/output for WPS● Allows the input and output of CSV for use in WPS
processes● This will lead to GeoTools CSV Datastore being
upgraded to community (and hopefully to supported)● Come see more at “Building a Table Joining Like
Service with Web Processing Services” Friday Afternoon
2.12Ian TurtonAstunRnD
Raster data andprocessing
● Initial version in 2.11● Several fixes in 2.12, e.g.
accuracy, performance, dateline crossing handling
● Being tested against global mosaics of Sentinel2 (each image in a different UTM zone), more improvements/fixes incoming
Heterogeneous CRS image mosaics
2.11/2.12Boundless/GeoSolutionsRnD/DLR
UTM60 image footprints UTM1 image footprints
Dateline wrapped image mosaic
● Reprojected images could have shown border effects
Better handling of reprojection boundaries
2.10/2.11GeoSolutionsCommunity
● Higher order interpolation did not play well with nodata, causing border effects:
○ Bilinear○ Bicubic
● Requires jai-ext enabled
Improved no-data handling in interpolation
2.10/2.11GeoSolutionsCommunity
Before
After
● Respect scale/offset attributes● Improved support for related dimensions (e.g. time/runtime in forecast),
automatically finding the best related value when just one dimension is provided
NetCDF reader improvements
2.11Niels/Devon/Ben/DanieleScitus/Boundless/Transient/GeoSolutionsCustomer
Runtime
S3 GeoTIFF Community ModuleSupport for GeoTiffs hosted on Amazon S3
● Reader based off of the GeoTiff reader.● Very basic caching of images from S3 based off of EhCache● InputStream implementation from JAI for reading imagery from S3
2.12DevonBoundlessBoundless
s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC80010022016230LGN00_B1.TIF?useAnon=true&awsRegion=US_WEST_2
Styling
Style Page
2.10BoundlessCommunity
Refactored CSS Style Page into the main application.
Works withSLD, CSS, YSLD, Mapbox
Improved experience for everybody!
New features for CSS (2.10)● Rendering transformations supported (feature parity with SLD!)● Rule nesting, helps with selector duplication and makes styles more readable
/* @title Levels */* { transform: ras:Contour(levels: 1100 1200 1300 1400 1500 1600 1700); stroke: black; z-index: 0;}
/* @title Values */[@scale < 3000] { transform: ras:RasterAsPointCollection(); mark: symbol('square'); :mark { size: 2; fill: black; } label: [GRAY_INDEX]; label-anchor: 0 0.5; label-offset: 5 0; font-family: Arial; font-fill: black; z-index: 1;}
Extract contours at the desired levels
When zoomed-in enough, extract point values, display center with a mark, and value on top of it
2.10Andrea AimeCommunity
New features for CSS (2.12)● Code completion in the editor● LessCSS compatible color functions
○ Darken, lighten, saturate, ….
● More readable scale denominator rules○ E.g. @sd < 100k
● Impressive CSS->SLD translation speedup○ 50 times faster on large CSS sheets
● Easier env variable integration (in progress, not yet landed)
○ @varname or @varname(defaultValue)
2.10Andrea AimeCommunity
YSLD extension module● Style Layer Descriptor represented in
YAML○ Uses indentation rather than XML○ Easier to read○ More compact○ More flexible syntax○ Contains variables for reusable code○ Compatible with SLD
● REST API can convert formats○ Preview your existing styles as YSLD
using REST API○ View the SLD representation of YSLD
2.11Boundless
symbolizers:- polygon: stroke-color: 'blue' stroke-width: 1 fill-color: '#7EB5D3'- text: label: ${name} fill-color: 'black' anchor: [0.5, 0.5] x-maxDisplacement: 40 x-autoWrap: 70
Community
● JSON representation of styling○ More limited than SLD/YSLD/CSS
● Works well with vector tiles○ Cross platform styling (OpenLayers, MapBoxGL)
● Work in progress:○ Currently a GeoTools community module ○ Should package and go to extension if we get time!
MapBox Style Community Module
2.12David VickBoundless Suite
MapBox Style in GeoServer andOpenLayers
Labelling improvements
● Kerning on by default(adjusts spacing between chars, e.g.Check space between T and r in Trimble)
● Underline and Strikethrough● Char and word spacing control
● Work in progress:○ Labelling border of polygon
(instead of the centroid)○ Filtering labels based on
polygon screen size in pixels
2.112.12
Andrea AimeNuno Oliveria
Community/Geoinfo/OpenGeoGroep
Improved QGIS Compatibility ● Additional custom “Well Known” marks● Support for SLD 1.1 External Marks (e.g. TTF font based)● SVG Parameters 1.0 spec
<PointSymbolizer> <Graphic> <ExternalGraphic> <OnlineResource xlink:type="simple" xlink:href="firestation.svg?fill=#FF0000" /> <Format>image/svg</Format> </ExternalGraphic> <size>128</size> </Graphic> </PointSymbolizer>
2.11Andrea AimeCommunity
● Label export, most shared configurations exported● Some in the works (word/char spacing, strikethrough, polygon border labelling)● Some changes needed in GeoServer SE 1.1 parser too
QGIS 3.0 style export improvements
2.12Andrea AimeGeoSolutionsOpenGeoGroep
WMS
Vector Tiles ExtensionExtension for publishing vector tiles
- Great solution for modern High Definition Screens
- Use for GeoWebCache for “MapBox without MapBox”
- Couple with MapBox style for consistent styling
2.11Gabriel RoldanBoundless Suite
Better centroid / label placement in “vector” outputsKML
● generate placemarks that are always inside polygon(for concave polygons)
Vector Tiles
● Pregenerate label points in tiles
○ Reduce effort on client
2.12Kevin SmithJust DeOliveraRnD
Label Labelvs
WMS sortBy vendor param● Sort by recency, cloud cover, and so on● Supported by vector data and image mosaic
2.12Andrea AimeGeoSolutionsCommunity
Sort by recencySort by cloud cover
ncWMS like extensions to GetMap (community)● A new styling language designed for dynamic colormaps● Applies to all rasters based on their statistics● Use can control min/max, log scale, animation● Part of OGC TestBed 12 activities
2.10GeoSolutionsOGC
WFS
“null” prefix bug finally solved
● Occasionally GeoServer could return WFS GML with “null” as the feature prefix
● Solved for simple features● Solved for complex features
on global services
2.10Nuno OliveiraGeoSolutionsIGEA/NLS
App-schema improvements
● Improvements to connection pool usage (only one connection used per request now)
● Some improvements helping while using MongoDB as a source for app-schema records
● Hale mapping support being extended to support MongoDB
2.12Nuno OliveiraGeoSolutionsNPRA
GeoJSON output for complex features● Complex features not limited to GML anymore
2.10Nuno OliveiraGeoSolutionsNPRA
{ "type":"FeatureCollection", "totalFeatures":"unknown", "features":[ { "type":"Feature", "id":"gu.93", "geometry":{ "type":"Polygon", "coordinates":[ ... ] }, "properties":{ "description":"Olivine basalt, tuff, microgabbro, minor sedimentary rocks", "occurrence":[ { "shape":{ "type":"Polygon", "coordinates":[ ... ] }, "name":"urn:x-test:GeologicUnit:16777549126931093", "observationMethod":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "positionalAccuracy":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } },
{ "shape":{ "type":"Polygon", "coordinates":[ ... ] }, "name":"urn:x-test:GeologicUnit:16777549126931093", "observationMethod":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "positionalAccuracy":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "specification":{
} } ], "name":"urn:x-test:GeologicUnit:16777549126931093" } // properties ends here}
WCS
Vendor param for mosaics: cqlFilter and sortBy ● Sort by recency, cloud cover,
and so on● Filter on the same● Maps to image mosaic read
parameters● Uses attributes in the mosaic
index● Consistent with WMS now,
first see and then download
2.12Andrea AimeGeoSolutionsCommunity
NetCDF input/output improvements
2.12NielsScitusCommunity
● Can configure NCML files as sources● Support variable's attributes definition in
NetCDF output● Preserve scalars from input
(NetCDF in/NetCDF out)
Tiling
● GeoServer supports “cascading” remote WMTS services
○ GeoServer acts as a client, rather than just rerouting each request
○ Same as WMS cascading
● WMTS cascade is a work in progress
○ Should land in 2.12
WMTS cascading
2.12Ian Turton/Emanuele TajariolGeoSolutionsCustomer
Parameter filter cache drop improvements● GeoWebCache can cache based on parameters
like style, time, or elevation.○ Could not truncate (remove cached tiles) across all
parameter values, each must be known and truncated separately.
○ Tiles could be orphaned if the allowed values changed.○ GeoServer can truncate tiles when the data within them
changes, but only caught the default values for parameters other than style
● Added operations for○ Cross parameter truncate○ Orphan purge
● Enabled cross parameter truncate on data updateGWC 1.12
GS2.12
Kevin SmithBoundlessMapStory
WMS/WMTS multidim domain discovery module (community)● Data with N dimensions (time, elevation, custom ones)● Dimensions can be related (forecasts, time and run time), the data can be
scattered and found in pockets, hard to locate data in the ND cube● Add extra calls to query the domains● Developed as part of OGC Testbed 12● See the engineering report and give us feedback:
2.10Andrea/NunoGeoSolutionsOGC
http://docs.opengeospatial.org/per/16-042r1.html#WMTS_Multidimensional_GeoSolutions
Configuration and management
Security Consideration - Disabled writing by default● WFS-T allows editing using
Transaction requests○ May not be what you expect?
● Default configuration includes security restrictions on WFS-T functionality
● Restricting editing of data to the administrator account.
● Makes service read-only(while still advertising wfs-t)
2.11Nuno OliveiraCommunity
● Great performance improvement!● Test with 10000 layers
Faster startup times for large data dirs
2.11AndreaCommunity
Version COLD(Startup time seconds)
HOT(Startup time seconds)
2.10.1 428 39
2.10.1 + JDCBConfig 62 49
GSIP 155 68 29
● Great performance improvement!● Tested on topp:states to make catalog lookup times more evident (assume diff is a constant)
Faster runtime times for large data dirs
2.11AndreaCommunity
Version Throughput req/sec Avg resp. time ms
2.10.1 169 47
2.10.1 + JDCBConfig 68 117
GSIP 155 233 34
● When loading a shapefile the PRJ file only matched on AUTHORITY code
● Faster lookup implementation allows GeoServer to search the EPSG database and match far more often
Better/Faster EPSG code lookups
2.11AndreaCommunity
This reduces a common frustration when configuring new layers in GeoServer
Module “status” REST API (and GUI)● Verifying what modules are installed was a
challenge● REST API access to the “status page”● Easy to add additional entries● Used to check environment for difficult to
configure modules that use native code
2.11Morgan ThompsonBoundlessBoundless Suite
geoserver/rest/about/status
Parametric configuration● Stick parameters in store configurations and
some security bits ● Helps migrating a data dir between
environments (e.g., integration/test/beta/production)
● Set the -DALLOW_ENV_PARAMETRIZATION=true system var
● Create a property file with variable values
2.10Alessio FabianiGeoSolutions
EumetsatWorld Bank
store_url = file:///var/geoserver/store/teststore
Backup/restore (community)● Save the configuration to a zip file (not the
data!)● Restore later, in the same or different
environment● UI to run backup and restore● Asynch operations
○ During backup the configuration is forced read only○ During restore it’s fully locked
2.10GeoSolutionsEumetsatWorld Bank
Web interface in Tools / Resource Browser
● Manage and edit resources
● Data Directory or JDBCStore
● Community module
● Packaged plugin coming soon
● REST API sporting equivalent functionality
found in core
Resource Browser (community)
2.10Scitus DevelopmentBoundless/NGA
Security
OAuth (community)● Allows GeoServer authentication to be
provided by OAuth● Allows users to login with:
○ GeoNode○ Google○ Github○ … add yours here, it’s pluggable
● Developed for integrationwith GeoNode
2.10Alessio FabianiGeoSolutionsBoundless
SecurityConsiderations
Take care to report the issue in a responsible fashion:
● Keep exploit details out of issue report● Mark the issue as a vulnerability.● Be prepared to work with Project Steering Committee (PSC) on a solution● Keep in mind PSC members are volunteers
and an extensive fix may require fundraising / resources
If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at info@osgeo.org.
Security Vulnerabilities - Responsible Disclosure
Responsible disclosure policy is working:
● Most users contacted us in a responsible fashion● One user shared confidential customer report on the user list
Where do reports come from?
● Majority reported from security audits○ Usually performed under contract○ Automated tools tend to focus on web administration ui, rather than web services
● GeoServer is subject to several security audits a year
Security Vulnerabilities
Upgrade to get security fixes● Security fixes being added to supported versions only● Please upgrade your GeoServer installations!
Looking Ahead
Java 9 CompatibilityJava 9 has been delayed until 2017, but we would like to ensure GeoServer will function. (Oracle stops free security updates to Java 8)
We have a couple incompatibilities with the GeoTools “service provider interface” plugin system that we will look at resolving during the code sprint!
The Java Advanced Imaging library enables GeoServer to do all kinds of great image processing! The JAI-EXT project extends this library with replacement no-data / footprint aware operations.
Sadly Oracle has not maintained the project, and has not released it as open source.
(see Replace-JAI on GeoTools wiki)
Raster Processing EngineJoint OSGeo / LocationTech game plan:
● Raster Processing Engine APIdefinition of raster-processing-api
● Raster Processing Enginereference pure-java implementation
● Raster Processing Wrapperbenchmark jai-wrapper implementation
● GeoTools Replace JAImigrate to raster processing engine api and coordinate transition affected projects (GeoServer, GeoNode, uDig, GeoScript, etc...)
Thanks!
Earth observation for EO
OpenSearch for Earth Observations● Recent addition to OGC protocol for search of earth imagery● Simpler alternative to CSW with ebRIM● Two steps search
○ First find collection of similar products (by satellite/sensor)○ Then search products in it (by other attributes, e.g., cloud cover, off nadir)
● Keeps database of searchable metadata● Links to OGC services providing said data● Allows downloading ISO/O&M metadata
2.12GeoSolutionsDLR/EVO-ODAS
OpenSearch for Earth Observations● Result is RSS with human readable descriptions and
○ Links to OGC services providing said data○ Links to download ISO/O&M metadata○ Links to download the raw data
● Exposes one feature type per collection, allowing easy mosaic setup with same filtering and ordering
2.12GeoSolutionsDLR/EVO-ODAS
OpenSearch for Earth Observations
http://cloudsdi.geo-solutions.it/geoserver/oseo/search?parentId=SENTINEL2&cloudCover=30]
● E.g., search all SENTINEL2- images with cloud cover less than 30%
2.12GeoSolutionsDLR/EVO-ODAS
OpenSearch for EO: admin REST API● Create/Update/Delete the collections and its information● Create/Update/Delete the product and its information● Either one shot (throw a zip file with all the info) or granular (edit
searchables first, then thumbnail, then metadata, links, …)
Swagger documentation
2.12GeoSolutionsDLR/EVO-ODAS
top related