wms performance shootout 2009
TRANSCRIPT
OSGeo Presentation
LabelGeoServerMapServer FCGI
16.68
1014.318.6
2014.919.4
4014.719.1
???Pagina ??? (???)10/22/2009, 13:51:24Pagina / RasterMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40
bigtiff cgi 564.813.31515.2
bigtiff fcgi 548.22425.224.3
bigtiff fcgi 5611.126.928.127.3
ecw cgi 56412.612.812.6
ecw fcgi 543.39.69.48.9
ecw fcgi 567.42019.819.1
tiled tiff cgi 562.71111.411.3
tiled tiff fcgi 546.616.116.717
tiled tiff fcgi 566.514.916.416.9
Polygon DrawingMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40
shape6.515.11515.5
Shape 54 fcgis12.727.227.928.2
shape 56 fcgi1225.826.226.8
pg5.914.71515.2
pg fcgi10.123.22424.3
oracle3.511.912.512.8
oracle fcgi9.121.121.621.8
oracle ogr0.71.71.71.7
sde1.19.410.110
sde fcgi8.822.422.923.3
Point DrawingMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40
shape6.614.815.215.6
shape 54 fcgi12.927.928.328.7
shape 56 fcgi12.226.526.627.1
pg6.115.115.315.5
pg fcgi10.72424.324.8
oracle3.29.910.711.3
oracle fcgi9.823.223.323.7
oracle ogr0.71.71.71.7
sde1.19.710.49.9
sde fcgi9.623.323.824.2
Andrea Aime (GeoServer)
Jeff McKenna (MapServer)WMS Performance Shootout
Executive summary
Compare the performance of WMS serversGeoServer
MapServer
In a number of different workloads: Vector: plain polygon, point with graphical icons, road network with cased lines, road network with labels
Raster: global imagery
Against different data backends: Vector: shapefiles, PostGIS, Oracle spatial, SDE over Oracle spatial
Raster: ECW and GeoTiff mosaics
Benchmarking History
3rd FOSS4G benchmarking exercise. Past exercises included:FOSS4G 2007: Refractions Research run and published the first comparison with the help of GeoServer and MapServer developers. Focus on big shapefiles, postgis, minimal styling
FOSS4G 2008: OpenGeo run and published the second comparison with some review from the MapServer developers. Focus on simple thematic mapping, raster data access, WFS and tile caching
Friendly competition: goal is to improve all software
Past MapServer improvements
improvements in large shapefile indexing
raster read optimization (single pass for multiple bands)
enhancing polygon labelling placement
EPSG codes caching
PostGIS improvements
Oracle improvements
Past GeoServer improvements
Overall rendering pipeline improvements
GML writing speed-ups
Raster data access improvements
PostGIS data store optimizations
Rules of engagement
Each server is tested in its latest version
Each server performs exactly the same workloadSame set of WMS requests
Same data backends
Same image output format
All modifications made to improve performance are to be included in a future release of the software
Data used cannot be modified for display, other than indexing
All testing to be done on the same benchmarking machines
Hardware Configuration
JMeterGeoServerMapServerShapefilesECWGeoTIFFOraclePostGISSDE
Bench
WMS
Database
Software Specs
MapServer:5.6.0-beta3, GDAL SVN
Apache 2.2
FastCGI - 8 static processes
GeoServer:2.0-RC2, Java 6 update18, GDAL 1.4.5
Tomcat 6.0.20 - 8 threads
Database:Oracle 11.1.0.7
SDE 9.3.1
PostGIS 1.4.0
Hardware specs
Bench (2003):Dell PowerEdge 17501 Xeon Nocona 3Ghz, 1 GB RAM, OS: RedHat Enterprise 5.3
WMS (2004):Dell PowerwEdge 2850
4 Xeon Nocona 3.4Ghz, 8 GB RAM
6 73 GB, 15K RPM hard drives, OS: RedHat Enterprise 5.3
Database (2007):Dell Optiplex 755 tower
1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core, 4Gb RAM
100Gb SATA 3.0Gb/s 7200 rpm HD, OS: RedHat Enterprise 5.3
Hot vs Cold Benchmarks
Hot benchmarkThe file system caches are fully loaded
The software has had the occasion to perform whatever pre-processing is necessary (e.g., open connections to the data source)
Cold benchmarkThe system has been just started
There is no existing cache
Both are unrealistic, production is usually a mix of both
Hot benchmarks are more practical to run
Methodology
Each test run performs requests with 1, 10, 20 and 40 parallel clients (for a total of 1500 requests)
Each test uses a random set of requests stored in a CSV file: no two requests in the same run are equal, but all servers perform the same workload
For each request the random factors are:The image size (between 640x480 and 1024x768)
The geographic envelope (extent)
Each test is run three times in a row, the results of the third run are used for the comparison: this benchmark assumes full file system caches (hot benchmark)
The other GIS server is shut down while the tests are run
Datasets Used
Polygon layer: areawater_merge: the TIGER 2008 set of polygons describing water surfaces for the state of Texas. 380000 polygons, 195MB shapefile, EPSG:4269
Point layer: gnis_names09: all location and points of interest names for the state of Texas in the GNIS database. 103000 points, 2.8 MB shapefile, EPSG:4269
Line layer, edges_merge: all line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. over 5M lines, 1.2GB shapefile, 1.4GB dbf, EPSG:4269
Raster layer: Bluemarble TNG, 86400 x 43200 pixels, 7 overview layers (TIF, ECW, tiled 512&8, GeoRaster)
Equivalent traffic table
The presentation charts results as throughput in requests per second.
This table translates hits per second at sustained peak load in terms of requests per hour and per day
T1: plain polygon rendering
TIGER 2008 areawater layer, merged over Texas
Simply fill the polygon in blue, no outline
Image scales roughly between 1:10.000 and 1:100.000
T1: Shapefiles
T1: PostGIS
T1: Oracle
T1: SDE
T1: observations
GeoServer is between 25% and 50% faster reading shapefiles out of the box, MapServer with FastCGI is slightly faster
PostGIS performs roughly 5% better than Oracle on higher loads
SDE seems to introduce no overhead compared to direct Oracle access for GeoServer, and a slight overhead for MapServer
GeoServer and MapServer have exactly the same performance when the backend is PostGIS and Oracle
T2: point rendering
GNIS names dataset over Texas
Each point is rendered using a little GIF/PNG graphics
T2: shapefiles
T2: PostGIS
T2: Oracle
T2: SDE
T3: roads map
TIGER 2008 edges, just roads (rivers filtered out), three different line styles based on the road class
T3: shapefiles
T3: PostGIS
T3: Oracle
T3: SDE
T4: labelled roads
Same as T3, but higher scale (between 1:10k and 1:3k) and with labels following lines
T4: shapefiles
T4: PostGIS
T4: Oracle
T4: SDE
T5: raster
Bluemarble TNG, as single ECW file, 512 mosaic TIFF, and BigTIFF (MapServer) vs 8 tiles mosaic TIFF (GeoServer)
ECW file is 260MB
TIFFs are uncompressed, tiled, with overviews. Around 16GB for each mosaic
T5: ECW
T5: 512 tiles mosaic
T5: BigTiff vs 8 tiles tiff
GeoServer cannot do BigTiff, so a 8 tiles mosaic was used (almost 2G B per tile)
FOSS4G 2009 improvements
Several code enhancements were made during this year's exercise:MapServer Fixed problem with labels following features
Fixed problem with 5.6.0-beta for SDE layers
Found problem with 5.6.0-beta for Oracle fastcgi and memory leak
GeoServerFixed problem with Oracle connection pooling and index scanning
Fixed problem with ECW crashes under high load
Hopeful Changes for Next Shootout
Involvement of more WMS servers
Put the servers under a faster internet connection for quicker data transfers and tests setups
Modify the number of parallel client requests performed so that the 2-4-8 parallel clients case gets exposed
Phases/cycles for benchmarking to allow for teams to interpret results and make software enhancements
Reprojection in both vector and raster data
Audience suggestions?
More Information
Scripts, configuration files, results stored in OSGeo SVN:
http://svn.osgeo.org/osgeo/foss4g/benchmarking/
Wiki home: http://wiki.osgeo.org/wiki/Benchmarking_2009
Mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking
Additional MapServer Results (not presented)
MapServer Roads/Lines Test Results
MapServer Labelled Roads/Lines Test Results
MapServer Polygon Test Results
MapServer Point Test Results
MapServer Raster Test Results
FastCGI Configuration Used for MapServer Tests
FastCgiServer /var/www/cgi-bin/mapserv.fcgi -processes 8 -initial_env LD_LIBRARY_PATH=/usr/local/lib:/opt/build/instant_client_11_1/lib:/opt/build/arcsde_c_sdk/lib
GeoServerMapServer FCGI
110.19.6
1024.523.6
2024.623.9
4023.924.5
ThreadsGeoServerMapServer FCGI
11110.8
1025.124.3
2025.324.6
4025.825
???Pagina ??? (???)10/22/2009, 13:28:43Pagina / ThreadsGeoServerMapServer FCGI
110.19.6
1024.523.6
2024.623.9
4023.924.5
???Pagina ??? (???)10/22/2009, 13:41:12Pagina / GeoServerMapServer FCGI
19.58.9
1023.322.3
2023.623.2
4023.323.6
Line DrawingMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40
shape5.111.311.111.5
shape 54 fcgi8.818.518.319.1
shape 56 fcgi8.217.116.817.6
pg4.411.21111.7
pg fcgi6.515.515.115.2
oracle2.47.77.78.7
oracle fcgi5.612.812.212.9
oracle ogr0.61.61.61.6
sde16.77.27.3
sde fcgi4.710.99.811
GeoServerMapServer FCGI
17.26.6
1016.315.8
2014.815.1
4016.616.4
LabelGeoServerMapServer FCGI
17.26.6
1016.315.8
2014.815.1
4016.616.4
???Pagina ??? (???)10/22/2009, 13:38:42Pagina / GeoServerMapServer FCGI
15.95.6
1012.512.6
2011.511.6
4012.712.8
LabelGeoServerMapServer FCGI
15.95.6
1012.512.6
2011.511.6
4012.712.8
???Pagina ??? (???)10/22/2009, 13:45:01Pagina / GeoServerMapServer FCGI
16.14.7
1013.311.1
2011.69.7
401311
LabelGeoServerMapServer FCGI
16.14.7
1013.311.1
2011.69.7
401311
???Pagina ??? (???)10/22/2009, 13:47:32Pagina / GeoServerMapServer CGIMapServer FCGI
16.35.46.6
1013.411.914.5
2013.912.715.5
4013.212.615.3
LabelGeoServerMapServer CGIMapServer FCGI
16.35.46.6
1013.411.914.5
2013.912.715.5
4013.212.615.3
???Pagina ??? (???)10/22/2009, 13:49:25Pagina / GeoServerMapServer FCGI
16.68
1014.318.6
2014.919.4
4014.719.1
GeoServerMapServer FCGI
166.8
1013.216.8
2013.917.5
4013.517.3
LabelGeoServerMapServer FCGI
166.8
1013.216.8
2013.917.5
4013.517.3
???Pagina ??? (???)24/10/2009, 12:21:12Pagina / GeoServerMapServer FCGI
15.66.4
1012.816.4
2013.317.4
4013.117.2
LabelGeoServerMapServer FCGI
15.66.4
1012.816.4
2013.317.4
4013.117.2
???Pagina ??? (???)10/22/2009, 14:04:00Pagina / GeoServerMapServer CGIMapServer FCGI
1547.4
1011.112.620
2011.212.819.8
4010.512.619.1
ThreadsGeoServerMapServer CGIMapServer FCGI
1547.4
1011.112.620
2011.212.819.8
4010.512.619.1
???Pagina ??? (???)10/22/2009, 14:04:00Pagina / GeoServerMapServer CGIMapServer FCGI
1116.512
102514.925.8
2025.215.326.2
4025.115.526.8
GeoServerMapServer CGIMapServer FCGI
19.36.612.2
1024.314.926.5
202515.326.6
4025.215.727.1
GeoServerMapServer FCGI
110.710.1
1024.423.7
2025.224.2
4025.724.5
GeoServerMapServer FCGI
110.29.2
1023.121.1
2023.821.7
402322
ThreadsGeoServerMapServer FCGI
110.29.2
1023.121.1
2023.821.7
402322
???Pagina ??? (???)10/23/2009, 12:03:51Pagina / ThreadsGeoServerMapServer FCGI
19.58.9
1023.322.3
2023.623.2
4023.323.6
???Pagina ??? (???)10/23/2009, 12:05:48Pagina / GeoServerMapServer FCGI
110.59.8
102523.3
2025.223.4
4024.623.9
GeoServerMapServer FCGI
11110.8
1025.124.3
2025.324.6
4025.825
ThreadsGeoServerMapServer FCGI
110.59.8
102523.3
2025.223.4
4024.623.9
???Pagina ??? (???)10/22/2009, 13:31:12Pagina / GeoServerMapServer CGIMapServer FCGI
14.82.76.5
109.71114.9
209.611.416.4
409.911.316.9
ThreadsGeoServerMapServer CGIMapServer FCGI
14.82.76.5
109.71114.9
209.611.416.4
409.911.316.9
???Pagina ??? (???)10/22/2009, 14:04:01Pagina / GeoServerMapServer CGIMapServer FCGI
16.44.811.1
1013.713.326.9
2013.31528.1
4013.615.227.3
ThreadsGeoServerMapServer CGIMapServer FCGI
16.44.811.1
1013.713.326.9
2013.31528.1
4013.615.227.3
???Pagina ??? (???)24/10/2009, 12:33:20Pagina / ThreadsGeoServerMapServer CGIMapServer FCGI
1116.512
102514.925.8
2025.215.326.2
4025.115.526.8
???Pagina ??? (???)10/22/2009, 13:09:28Pagina / ThreadsGeoServerMapServer FCGI
110.710.1
1024.423.7
2025.224.2
4025.724.5
???Pagina ??? (???)10/23/2009, 12:01:33Pagina / Lines with LabelsMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40
shape4.410.210.510.4
shape 54 fcgi9.220.52120.8
shape 56 fcgi6.614.515.515.3
pg4.210.510.610.7
pg fcgi614.114.614.4
oracle2.88.99.39.4
oracle fcgi5.412.913.513.3
oracle ogr0.71.#NAN1.#NAN1.#NAN
sde17.79.28.8
sde fcgi512.913.413.3
ThreadsGeoServerMapServer CGIMapServer FCGI
19.36.612.2
1024.314.926.5
202515.326.6
4025.215.727.1
???Pagina ??? (???)10/22/2009, 13:41:56Pagina / GeoServerMapServer CGIMapServer FCGI
17.658.2
10151117.1
2013.11116.8
4015.111.517.6
LabelGeoServerMapServer CGIMapServer FCGI
17.658.2
10151117.1
2013.11116.8
4015.111.517.6
???Pagina ??? (???)10/22/2009, 13:36:47Pagina /