geocompressor - bynder.hexagon.com · hexagon geocompressor 2018 v16.5.0.1417 march 2018 competitor...

41
White Paper GeoCompressor Compression Performance Comparison The Power Portfolio from Hexagon Geospatial combines the best photogrammetry, remote sensing, GIS and cartography technologies available. Flowing seamlessly from the desktop to server-based solutions, these technologies specialize in data organization, automated geoprocessing, spatial data infrastructure, workflow optimization, web editing, and web mapping. The Provider Suite gives you the power to organize all your geospatial and business data into one centralized library and deliver it to others easily. 12 July 2019

Upload: others

Post on 09-May-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

White Paper

GeoCompressor

Compression Performance Comparison

The Power Portfolio from Hexagon Geospatial combines the best photogrammetry, remote sensing, GIS and cartography technologies available. Flowing seamlessly from the desktop to server-based solutions, these technologies specialize in data organization, automated geoprocessing, spatial data infrastructure, workflow optimization, web editing, and web mapping.

The Provider Suite gives you the power to organize all your geospatial and business data into one centralized library and deliver it to others easily.

12 July 2019

Page 2: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 2

Contents Summary .................................................................................................................................... 4

Disclaimer ................................................................................................................................ 4

Changelog ............................................................................................................................... 4

Project Setup ............................................................................................................................. 5

Data ......................................................................................................................................... 5

Preview .................................................................................................................................... 6

Test Hardware ......................................................................................................................... 7

Test Software ........................................................................................................................... 7

Output Test Formats ................................................................................................................ 8

Support Matrix ......................................................................................................................... 8

Metholodogy ............................................................................................................................ 9

Results ..................................................................................................................................... 10

Lossy Compression ............................................................................................................... 10

Creation Time ..................................................................................................................... 10

File Size.............................................................................................................................. 11

Lossless Compression ........................................................................................................... 12

Creation Time ..................................................................................................................... 12

File Size.............................................................................................................................. 13

Output Discrepencies ............................................................................................................. 14

Pyramid Generation ............................................................................................................... 15

Pyramid Example ............................................................................................................... 16

Final Results - Lossy .............................................................................................................. 18

Creation Time ..................................................................................................................... 18

File Size.............................................................................................................................. 19

Final Results - Lossless ......................................................................................................... 20

Creation Time ..................................................................................................................... 20

File Size.............................................................................................................................. 21

Conclusion ............................................................................................................................... 22

Page 3: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 3

Appendix A – Data Investigation ............................................................................................ 22

Output Variations ................................................................................................................... 22

Lossless JPEG2000 Profiles .................................................................................................. 23

Appendix B – Software Workflow .......................................................................................... 24

Esri Arcgis Desktop ................................................................................................................ 24

Global Mapper ....................................................................................................................... 28

FME ....................................................................................................................................... 31

ERDAS IMAGINE .................................................................................................................. 33

GDAL ..................................................................................................................................... 34

GeoCompressor..................................................................................................................... 38

Source Material Reference ..................................................................................................... 40

Technical Reference .............................................................................................................. 40

Contact us................................................................................................................................ 41

About Hexagon ........................................................................................................................ 41

Page 4: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 4

Summary Recognizing a cross-industry struggle to efficiently compress large, high-resolution imagery and point clouds, Hexagon Geospatial leveraged our own Enhanced Compression Wavelet (ECW) compression technology to design and deliver a targeted application that can compress and mosaic imagery to as small as five percent of their original size while retaining the image’s full visual quality.

GeoCompressor continues to garner a loyal following of Data Provider customers seeking to use every competitive edge to ensure their data processing pipeline is providing the best quality and providing the fastest turnaround time possible getting their data into the hands of their customers.

In contrast to the related white paper that focuses on hardware requirements and peak throughput expectations of GeoCompressor, this white paper is focused on:

• How do you compare how fast <insert software package here> is in producing <insert file format here>?

Due to the abundant variety of possible geospatial image storage schemes, we contrast Hexagon’s own ECW format against other implementations and industry storage strategies. Although comparing ECW seems unusual, for about two decades Hexagon has licensed our technology to third parties and indeed our competitors. Given that our compression ERDAS ECW/JP2 SDK is unencumbered in any way, how much of a difference, really, is the implementation? How much faster is GeoCompressor anyway?

Let’s find out –

Disclaimer

Comparing against any other software is always challenging, let alone when processes can take days or weeks to complete. In most software packages, a myriad of options are available to tune or tweak, not to mention a continual series of version updates and bug fixes are applied that alter the test bed. For the scope of this paper however, all software listed is out of the box with default settings on the latest software version available as of May 2018.

While it is possible that setting an obscure option may improve things dramatically, it simply was not feasible to spend exhaustive amounts of time with each software package. Should an obvious oversight be made in error, we will update the changelog in the interest of transparency. Please contact us if you believe this to be the case.

Changelog

The following changelog will be updated when corrections or updated information is added.

V1.0 Initial release

Page 5: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 5

Project Setup

Data

Our colleagues from the HxGN Content Program at Hexagon’s Geosystems division provided a representative dataset as they also had the same question regarding production speed.

See the overview of the mosaic characteristics below, covering a region in Washington State, USA.

Mosaic Info

Dimensions: 423,460 x 183,720 px 77.798 gigapixel Structure: 4 Band, Multiband UINT8 Mosaic images: 141 images Estimated Filesize: 289.82 GB uncompressed Actual Filesize: 149.40 GB deflate compressed Projection: EPSG:6339 Resolution: 0.3 meters

Each tile has marginal overlap, but was otherwise fully orthorectified and corrected, ready for viewing and exploitation. A representative tile’s metadata reported by gdalinfo software is shown below.

Driver: GTiff/GeoTIFF Files: 4512001_ne_10_30.tif Size is 16900, 23720 Pixel Size = (0.300000000000000,-0.300000000000000) Metadata: AREA_OR_POINT=Area Image Structure Metadata: COMPRESSION=DEFLATE INTERLEAVE=PIXEL Band 1 Block=256x256 Type=Byte, ColorInterp=Red Band 2 Block=256x256 Type=Byte, ColorInterp=Green Band 3 Block=256x256 Type=Byte, ColorInterp=Blue Band 4 Block=256x256 Type=Byte, ColorInterp=Undefined

The overlapping region is identical data so the z-order or stacking yields identical output in the final mosaic.

Page 6: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 6

Preview

Figure 1 - Mosaic overview, Band display order 1,2,3

Figure 2 - Mosaic overview, Band display order 4,3,2

Figure 3 - tile overlap example

Page 7: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 7

Test Hardware

• HP Z820 Workstation

• 2x Intel Xeon E5 2690 CPU

• 128gb DDR3 RAM

• 15tb RAID-1

• 6x 7200rpm 3tb HDD

• LSI MR9270-8i with 1gb cache

• 1tb ZTurbo SSD drive

Although the hardware is now several years old, Z-Workstation is reflective of the hardware often used by data vendors. Although this means the results produced did not reflect the best performance possible on newer hardware, the more important figures to review are the comparison figures. If newer hardware, faster disks were to be used, it is expected that the relative performance would remain the same in most cases across all formats.

Test Software

Name Version Release Date

Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018

Competitor “A” (Unable to disclose)

Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

OSGEO GDAL v2.2.3 March 2018

Blue Marble Global Mapper v19.1.0 February 2018

Safe Software FME® v2018.0.0.2 January 2018

Esri ArcGIS® Desktop v10.6.0.8321 February 2018

Page 8: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 8

Output Test Formats

The following formats are not intended to be exhaustive, however do present a large cross-section of industry formats used today. MRF and GeoPackage in particular are “newer” than some of the more traditional formats and while they bring some functional differences, that is outside the scope of this paper. The focus remains on creation time and by extension storage requirements, and presumes the audience is aware of these functional differences to narrow down possible formats to address their requirements.

Lossy Lossless

ECW MrSID

MrSID JPEG2000

JPEG2000 Meta Raster Format (PNG)

Meta Raster Format (JPEG) Meta Raster Format (LERC)

GeoPackage (JPG PNG) GeoPackage (Deflate)

GeoTIFF (JPEG) GeoTIFF (LZW)

IMG HFA

Support Matrix

Highlighted green matrix cells denote Write, Compression, or Encoder support. This may mean that the software supports reading the format, just not creating it in the software version tested.

Software / Format ECW MrSID JPEG2000 MRF GeoPackage GeoTIFF (JPG)

GeoTIFF (Deflate)

GeoCompressor

ERDAS IMAGINE

Competitor A

Global Mapper

GDAL 1

ArcGIS

FME

ER Mapper

1 Supported, however ESDK was unavailable to test this scenario

Page 9: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 9

Only ECW and MrSID encoders supported multi-threaded encoding. All others showed minimal to no multi-threaded usage despite being configured to do so.

Methodology

Each software package has its own workflow for defining a mosaic project and writing output. Generally, all software followed the same two-step process:

• First, inputting the 141 data files to define a virtual “mosaic” project. Given the tiles were already processed, no further enhancements were needed to correct the data.

• Second, use the project to output the required format, while monitoring execution time.

Sometimes customers choose to split the first phase into a separate “mosaic” to write out a very large, uncompressed output. Then they execute a secondary “compress” step that takes the single, large input to write compressed output. In almost all cases, the only reason this would be done is to work around software limitations. Therefore, we followed the “mosaic and compress” flow.

For formats that do not have inherent pyramid layers or overview layers generated2 , a third phase was also run to generate pyramids to better reflect the full workflow.

This pyramid phase is often overlooked; however, it is important to discuss as this can seriously impact time to market and general applicability of the output dataset. While pyramids can be considered optional in very specific use cases, in the majority, pyramids are required and should form part of any head-to-head comparison. After all, no end customer enjoys opening a file only to then wait hours for pyramids to be generated.

2 See Appendix for how to make this determination

Page 10: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 10

Results

Lossy Compression

Creation Time

Initial Output Generation Time (Days:Hours:Minutes)3

Software ECW 50:1 ECW 15:1 MrSID 50:1

MrSID 15:1

JPEG2000 15:1

MRF (JPG)

GPKG (JPG)

GeoTIFF (JPG)

GeoCompressor 0:00:50 0:00:52 0:05:04

ERDAS IMAGINE 0:03:20 0:03:25 0:05:25

Competitor A 0:12:09 0:12:19 0:13:20

Global Mapper 0:04:13 0:04:30 Crash4 0:04:44

GDAL 0:03:32 0:03:46 0:15:28 0:03:07 0:06:44 0:02:29

ArcGIS Hang5 6

FME 0:02:27 0:02:35 0:06:03 0:03:13 0:09:14

ER Mapper 0:02:26 0:02:32 0:07:09

In the interest of time, not all possible outputs were run. Note the timing numbers reflect time to create the “initial” output. As discussed above, some file formats are not in a usable state without pyramids being generated. See the Final Results Lossy table below.

3 Shorter execution time means better performance and output was created faster. For brevity, results are listed rounded to nearest minute. 4 See Appendix A. Reproducible always at 88% complete. 5 See Appendix B. Esri JPEG2000 writer was unusably slow writing out the mosaic project. Although no specific crash occurred, progress would stall and never completed after 7 days. 6 Esri depends on the GDAL library for MRF generation. Refer to GDAL for indicative results, even though mosaic input type would differ.

Page 11: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 11

File Size

Initial Output Filesize (Kilobytes)7

Software ECW 50:1 ECW 15:1 MrSID 50:1 MrSID 15:1 JPEG2000 15:1

MRF (JPG) GPKG (JPG)

GeoTIFF (JPG)

GeoCompressor 4,472,770 17,991,104 22,779,366

ERDAS IMAGINE 4,472,770 17,991,104 22,779,366

Competitor A 7,794,547 21,986,679 18,658,534

ERDAS IMAGINE 4,472,7708 17,991,104 22,779,366

Competitor A 7,794,547 21,986,679 18,658,534

Global Mapper 2,496,314 9,413,039 Crash9 10,004,137

GDAL 1,899,816 9,590,664 12,628,090 65,676,969 131,105,484 34,744,868

ArcGIS Hang10

FME 1,425,309 5,945,204 5,000,239 5,852,59211

ER Mapper 2,579,554 9,270,706 15,116,214

No conclusions should be drawn on the output sizes because any lossy analysis should only be done in conjunction with an image quality assessment (outside the scope of this paper).

The other important consideration is that despite best efforts of the author, the outputs from each software were frustratingly difficult to match to ensure apples-to-apples comparison. Some software did not warn at the beginning that it was unable to create 4-band output, but instead, the software changed the number of the bands in the output to 3 bands without the author explicitly setting it.

Some Software created alpha or a fifth band opacity channel, others did not. These issues, in particular, are why ECW and JP2 outputs varied wildly, even with the same target compression rate specified. Again, this is a useful data point only.

7 Lower file size is generally the most desirable, however image quality is outside the scope of this paper. 15:1 for wavelet formats and 85% for JPEG is generally considered visually lossless. 8 ERDAS IMAGINE uses identical ECWSDK libraries and so file sizes are expected to be identical. 9 See Appendix A. Reproducible always at 88% complete 10 See Appendix B. Esri JPEG2000 writer was unusably slow writing out the mosaic project. Although no specific crash occurred, progress would stall and never completed after 7 days. 11 Another unfortunate disparity was that FME GeoPackage created by default a Google Maps Compatible output which warps and loses some of the pyramid levels. GDAL output does not do this and retains a native^2 tilematrix retaining full resolution and thus much larger size. These should not be directly compared.

Page 12: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 12

Lossless Compression

Creation Time

Initial Output Generation Time (Days:Hours:Minutes)12

Software MrSID JPEG2000 MRF (PNG)

MRF (LERC)13

GeoTIFF (Deflate)

GeoTIFF (LZW)

IMG HFA

GeoCompressor 0:08:27

ERDAS IMAGINE 14 0:10:11 0:06:47

Competitor A 0:08:40 0:16:24

Global Mapper Crash15 0:11:18 0:08:52

GDAL 16OpenJPEG v2.1 - 0:14:29 ECWJP2 v3.3 - 0:11:56

0:08:27 0:04:15 0:01:06 0:21:24

ArcGIS 11:02:0017

FME 0:11:19

ER Mapper 0:11:02

In the interest of time, not all possible outputs were run. Omitted numbers are not necessarily due to software constraints but rather this paper’s limited scope.

12 Shorter execution time means better performance and output was created faster. For brevity, results are listed rounded to nearest minute. 13 LERC supports both lossy and lossless modes. Only the latter was tested here. 14 ERDAS IMAGINE is unable to write 4-band Lossless MrSID output. 15 See Appendix A. Reproducible always. 16 For the lossless JP2 case only, both the OpenJPEG and ECWJP2 3.3 driver was used. In all other areas, the JP2 output was via OpenJPEG driver only. 17 ArcGIS reported the file completed after an extraordinary 11 days of processing time, however see Appendix B where the output is clearly incomplete, despite the file being valid.

Page 13: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 13

File Size

Initial Output Filesize (Kilobytes)18

Software MrSID JPEG2000 MRF (PNG) MRF (LERC)19 GeoTIFF (Deflate)

GeoTIFF (LZW)

IMG HFA

GeoCompressor 126,626,142

ERDAS IMAGINE 20 126,626,887 405,497,88721

Competitor A 129,859,089 128,524,942

Global Mapper Crash22 147,893,340 177,842,559 246,994,792

GDAL

OpenJPEG v2.1: 126,557,717 ECWJP2 v3.3: 126,601,306

131,304,941 140,806,988 146,769,868

ArcGIS Invalid23

FME 126,601,313

ER Mapper 92,326,33024

Unlike lossy output, image quality is not a concern given all outputs are reversible. Since some formats require pyramids, defer to Final Results – Lossless – File Size table below before drawing any storage conclusions from this initial output. The fact that all wavelet-based formats already require less storage than MRF, GeoTIFF without pyramids is an important metric that cannot be stressed highly enough.

18 Lower file size is generally the most desirable, however image quality is outside the scope of this paper. 15:1 for wavelet formats and 85% for JPEG is generally considered visually lossless. 19 LERC supports both lossy and lossless modes. Only the latter was tested here. MRF filesize calculation includes the MRF, IDX and PJG auxiliary files. 20 ERDAS IMAGINE is unable to write 4-band Lossless MrSID output. 21 IMG output from both ERDAS IMAGINE and Global Mapper produces embedded pyramids by default. File size shown in final table was unchanged for this reason. 22 See Appendix A. Reproducible always. 23 ArcGIS reported the file completed after an extraordinary 11 days of processing time, however see Appendix B where the output is clearly incomplete, despite the file being valid. 24 3-band output, which accounts of file size difference, despite being lossless.

Page 14: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 14

Output Discrepancies

Due to the size and duration of running processes, often limitations, discrepancies or user error would only surface after the output was created. Some of them are noted below:

• GeoPackage (JPEG) and MRF with JPEG compression does not support 4-band input. Output from both FME and GDAL was 3-band only.

• ER Mapper with ECW and JPEG2000 output writes 3-band output. Although it is possible to correct this, these numbers were retained to represent the default case like all other software. It also explains the size difference.

• Esri JPEG2000 output using the “Mosaic to New Raster” toolbox was incredibly slow and produced invalid output.

• The output was clearly corrupt or truncated, despite no errors or warnings being returned.

• Esri also attempted to write .OVR pyramid files even though this should not be required for the format. See Appendix B.

• Timings included here were based on cancelling the process on the presumption that since it was generating an OVR, the parent JP2 must be considered complete.

• FME GeoPackage output does support other Tiling Schemes but defaulted to Google Maps Compatible.

• Global Mapper IMG output was 3-band only with no discernible option to enable 4-band.

• GeoTIFF JPG output differed between GDAL and Global Mapper. The former created 3-band with alpha mask bands, the latter created 3-band only with YCBCR color space for improved compression. GDAL does support the same color space however is not enabled by default.

Kakadu Kdu_show

Page 15: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 15

Pyramid Generation

Pyramid Generation Time (Days:Hours:Minutes)25

Software ECW 50:1

ECW 15:1

MrSID 50:1

MrSID 15:1

JPEG2000 15:1

MRF (JPG)

GPKG (JPG)

GeoTIFF (JPG)

GeoTIFF (Deflate)

MRF (LERC)

MRF (PNG)

IMG HFA

GDAL N/A N/A N/A N/A N/A 1:19:59 0:05:15 1:15:01 1:20:31 1:22:39 2:01:09

ERDAS IMAGINE

0:02:4426

Not all formats require pyramid generation as a secondary processing phase but can be a substantial overhead that should not be ignored.

To save time, only a subset of software was used to generate embedded pyramid files. Although this is a large assumption that the pyramid generation time would be somewhat similar across software, it was another necessary shortcut. These pyramid times were added as an approximation to generate the total processing across all software in Final Results – Lossless – Creation Time table.

The GDAL pyramid utility gdaladdo was executed as follows, with the only variable being the input file and compression options where needed to retain the compression type. Deflate input has deflate overviews. JPEG has jpeg overviews etc. 10 resolution levels were created in all examples and a timer placed on execution.

> gdaladdo -r average --config NUM_THREADS_OVERVIEW ALL_CPUS27 --config COMPRESS_OVERVIEW DEFLATE e:\results\2017_hxip_wa_2a_z10_gdal_deflate.tif 2 4 8 16 32 64 128 256 512 1024 0...10...20...30...40...50...60...70...80...90...100 - done. Execution time: 160310.072 s

The “average” resampling algorithm was used as the generally preferred option for continuous geospatial imagery to obtain reasonable image quality. Nearest resampling would have decreased the time, however, it would not have mirrored real-world usage, which is the intent of this paper.

25 Shorter execution time means better performance and output was created faster. For brevity, results are listed rounded to nearest minute. 26 ERDAS IMAGINE used as the de facto implementation of its own IMG format. It also performs RRD generation in a single pass, but the generation time is reported separately. 27 Despite the ALL_CPUS flag, no output took advantage of the 32 cores when generating pyramids.

Page 16: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 16

Pyramid Example

GeoPackage

Initial Output Without Pyramids Post Pyramid Generation

> gdalinfo 2017_hxip_wa_2a_z10_gdal_geopackage_jpgpng.gpkg Driver: GPKG/GeoPackage Size is 423460, 183720

[sic...] Origin = (606324.000000000000000,5102016.000000000000000) Pixel Size = (0.300000000000000,-0.300000000000000) Metadata: IDENTIFIER=2017_hxip_wa_2a_z10_gdal_geopackage_jpgpng ZOOM_LEVEL=11 Image Structure Metadata: INTERLEAVE=PIXEL Corner Coordinates: Upper Left ( 606324.000, 5102016.000) (121d37'31.08"W, 46d 3'48.42"N) Lower Left ( 606324.000, 5046900.000) (121d38'14.83"W, 45d34' 3.05"N) Upper Right ( 733362.000, 5102016.000) (119d59' 4.06"W, 46d 1'54.91"N) Lower Right ( 733362.000, 5046900.000) (120d 0'39.88"W, 45d32'11.47"N) Center ( 669843.000, 5074458.000) (120d48'51.48"W, 45d48' 9.99"N) Band 1 Block=256x256 Type=Byte, ColorInterp=Red Mask Flags: PER_DATASET ALPHA Band 2 Block=256x256 Type=Byte, ColorInterp=Green Mask Flags: PER_DATASET ALPHA Band 3 Block=256x256 Type=Byte, ColorInterp=Blue Mask Flags: PER_DATASET ALPHA Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha

The absence of Overviews for each band mean any region of interest outside of native resolution would require substantial amount of IO to be performed and performance would suffer. The lack of a 4th non-alpha band also shows the discrepancy between other formats that correctly retain the 4th CIR band.

> gdalinfo 2017_hxip_wa_2a_z10_gdal_geopackage_jpgpng.gpkg Driver: GPKG/GeoPackage Size is 423460, 183720 Band 1 Block=256x256 Type=Byte, ColorInterp=Red Overviews: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359,414x179 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359, 414x179 Band 2 Block=256x256 Type=Byte, ColorInterp=Green Overviews: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359, 414x179 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359, 414x179 Band 3 Block=256x256 Type=Byte, ColorInterp=Blue Overviews: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359, 414x179 Mask Flags: PER_DATASET ALPHA Overviews of mask band: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359, 414x179 Band 4 Block=256x256 Type=Byte, ColorInterp=Alpha Overviews: 211730x91860, 105865x45930, 52933x22965, 26466x11483, 13233x5741, 6617x2871, 3308x1435, 1654x718, 827x359, 414x179

Page 17: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 17

ECW

Initial Output Post Pyramid generation

> gdalinfo 2017_hxip_wa_2a_z10_15x.ecw Driver: ECW/ERDAS Compressed Wavelets (SDK 5.4) Size is 423460, 183720

[sic...] Origin = (606324.000000000000000,5102016.000000000000000) Pixel Size = (0.300000000000000,-0.300000000000000) Metadata: COLORSPACE=MULTIBAND COMPRESSION_RATE_TARGET=15 VERSION=3 Corner Coordinates: Upper Left ( 606324.000, 5102016.000) Lower Left ( 606324.000, 5046900.000) Upper Right ( 733362.000, 5102016.000) Lower Right ( 733362.000, 5046900.000) Center ( 669843.000, 5074458.000) Band 1 Block=256x256 Type=Byte, ColorInterp=Red Description = Red Min=0.000 Max=254.000 Minimum=0.000, Maximum=254.000, Mean=107.771, StdDev=50.144 Overviews: 211730x91860, 105865x45930, 52932x22965, 26466x11482, 13233x5741, 6616x2870, 3308x1435, 1654x717, 827x358, 413x179 Band 2 Block=256x256 Type=Byte, ColorInterp=Green Description = Green Min=0.000 Max=254.000 Minimum=0.000, Maximum=254.000, Mean=105.517, StdDev=37.401 Overviews: 211730x91860, 105865x45930, 52932x22965, 26466x11482, 13233x5741, 6616x2870, 3308x1435, 1654x717, 827x358, 413x179 Band 3 Block=256x256 Type=Byte, ColorInterp=Blue Description = Blue Min=0.000 Max=254.000 Minimum=0.000, Maximum=254.000, Mean=93.635, StdDev=26.066 Overviews: 211730x91860, 105865x45930, 52932x22965, 26466x11482, 13233x5741, 6616x2870, 3308x1435, 1654x717, 827x358, 413x179 Band 4 Block=256x256 Type=Byte, ColorInterp=Undefined Description = Band #4 Min=0.000 Max=254.000 Minimum=0.000, Maximum=254.000, Mean=158.327, StdDev=43.259 Overviews: 211730x91860, 105865x45930, 52932x22965, 26466x11482, 13233x5741, 6616x2870, 3308x1435, 1654x717, 827x358, 413x179 Band 5 Block=256x256 Type=Byte, ColorInterp=Alpha Description = AllOpacity Min=0.000 Max=0.000 Overviews: 211730x91860, 105865x45930, 52932x22965, 26466x11482, 13233x5741, 6616x2870, 3308x1435, 1654x717, 827x358, 413x179

Not required. GDAL and other libraries correctly advertise “virtual” overview levels for wavelet based formats including ECW, MrSID and JPEG2000. Although they are not exactly the same as pyramids, they achieve the same result of rapid area of interest extraction regardless of resolution. The important thing to note is that the number of advertised Overview levels match those above with the smallest level being 413x179. On JPEG2000, it is possible to create files without these overviews or more specifically, resolution levels and it too will degrade decoding performance at arbitrary scales if this is done. Refer to the Lossless JPEG2000 deep-dive section for examples.

Page 18: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 18

Final Results - Lossy

Creation Time

Total Output Generation Time (Days:Hours:Minutes)28

Software ECW 50:1 ECW 15:1 MrSID 50:1 MrSID 15:1 JPEG2000 15:1

MRF (JPG) GPKG (JPG)

GeoTIFF (JPG)

GeoCompressor 0:00:50 0:00:52 0:05:04

ERDAS IMAGINE 0:03:20 0:03:25 0:05:25

Competitor A 0:12:09 0:12:19 0:13:20

Global Mapper 0:04:13 0:04:30 Crash29 1:19:45

GDAL 0:03:32 0:03:46 0:15:28 1:23:06 0:11:59 1:17:30

ArcGIS Hang30 31

FME 0:02:27 0:02:35 0:06:03 1:23:12 0:14:29

ER Mapper 0:02:26 0:02:32 0:07:09

28 Shorter execution time means better performance and output was created faster. For brevity, results are listed rounded to nearest minute. 29 See Appendix A. Reproducible always at 88% complete. 30 See Appendix B. Esri JPEG2000 writer was unusably slow writing out the mosaic project. Although no specific crash occurred, progress would stall and never completed after 7 days. 31 Esri depends on the GDAL library for MRF generation. See GDAL for indicative results, even though mosaic input type would dif fer.

Page 19: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 19

File Size

Total Output Filesize (Kilobytes)32

Software ECW 50:1 ECW 15:1 MrSID 50:1 MrSID 15:1 JPEG2000 15:1

MRF (JPG) GPKG (JPG)

GeoTIFF (JPG)

GeoCompressor 4,472,770 17,991,104 22,779,366

ERDAS IMAGINE 4,472,77033 17,991,104 22,779,366

Competitor A 7,794,547 21,986,679 18,658,534

Global Mapper 2,496,314 9,413,039 Crash34 10,004,137

GDAL 1,899,816 9,590,664 12,628,090 67,293,160 175,991,650

46,585,578

ArcGIS Hang35

FME 1,425,309 5,945,204 5,000,239 5,852,59236

ER Mapper 2,579,554 9,270,706

32 Lower file size is generally the most desirable, however image quality is outside the scope of this paper. 15:1 for wavelet formats and 85% for JPEG is generally considered visually lossless. The values expressed include embedded pyramids, where needed. 33 ERDAS IMAGINE uses identical ECWSDK libraries and so file sizes are expected to be identical. 34 See Appendix A. Reproducible always at 88% complete. 35 See Appendix B. Esri JPEG2000 writer was unusably slow writing out the mosaic project. Although no specific crash occurred, progress would stall and never completed after 7 days. 36 Due to the disparity in outputs, the FME output sample is not considered, only that of GDAL.

Page 20: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 20

Final Results - Lossless

Creation Time

Total Output Generation Time (Days:Hours:Minutes)37

Software MrSID JPEG2000 MRF (PNG)

MRF (LERC)38

GeoTIFF (Deflate)

GeoTIFF (LZW)

IMG HFA

GeoCompressor 0:08:27

ERDAS IMAGINE 39 0:10:11 0:06:47

Competitor A 0:08:40 0:16:24

Global Mapper Crash40 2:07:49 0:08:52 0:21:24

GDAL 41OpenJPEG v2.1 - 0:14:29 ECWJP2 v3.3 - 0:11:56

2:09:36 1:26:54 1:21:37

ArcGIS 11:02:0042

FME 0:11:19

ER Mapper 0:11:02

In the interest of time, not all possible outputs were run. Omitted numbers are not necessarily due to software constraints but rather this paper’s limited scope.

37 Shorter execution time means better performance and output was created faster. For brevity, results are listed rounded to nearest minute. 38 LERC supports both lossy and lossless modes. Only the latter was tested here. 39 ERDAS IMAGINE is unable to write 4-band Lossless MrSID output. 40 See Appendix A. Reproducible always. 41 For the lossless JP2 case only, both the OpenJPEG and ECWJP2 3.3 GDAL driver was used. In all other areas, the JP2 output was via OpenJPEG only. 42 ArcGIS reported the file completed after an extraordinary 11 days of processing time, however, see Appendix B where the output is clearly incomplete, despite the file being valid.

Page 21: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 21

File Size

Total Output Filesize (Kilobytes)43

Software MrSID JPEG2000 MRF (PNG) MRF (LERC)44 GeoTIFF (Deflate)

GeoTIFF (LZW)

IMG HFA

GeoCompressor 126,626,142

ERDAS IMAGINE 45 126,626,887 405,497,88746

Competitor A 129,859,089 128,524,942

Global Mapper Crash47 48196,698,142 236,530,603 246,994,792

GDAL

OpenJPEG v2.1: 126,557,717 ECWJP2 v3.3: 126,601,306

176,250,570 184,799,472 195,927,416

ArcGIS Invalid49

FME 126,601,313

ER Mapper 92,326,33050

43 Lower file size is generally the most desirable, however image quality is outside the scope of this paper. 15:1 for wavelet formats and 85% for JPEG is generally considered visually lossless. 44 LERC supports both lossy and lossless modes. Only the latter was tested here. MRF filesize calculation includes the MRF, IDX and PJG auxiliary files 45 ERDAS IMAGINE is unable to write 4-band Lossless MrSID output. 46 IMG output from both ERDAS IMAGINE and Global Mapper produces embedded pyramids by default. File size shown in final table was unchanged for this reason. Global Mapper produced 3-band output. 47 See Appendix A. Reproducible always 48 Global Mapper GeoTIFF Deflate and LZW file size calculated by adding 33% to base image as generating as this was an unintentional test that was missed. 49 ArcGIS reported the file completed after an extraordinary 11 days of processing time, however, see Appendix B where the output is clearly incomplete, despite the file being valid. 50 3-band output which accounts of file size difference, despite being lossless.

Page 22: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 22

Conclusion Due to the challenges encountered to ensure consistent, comparable output, unfortunately many of the numbers are difficult to compare directly. However, if we discount the differences and acknowledge that ECW and JPEG2000 always retained all bands and always included opacity channels, GeoCompressor finished orders of magnitude ahead of all other software, regardless of the format used.

While it would be simple to reduce the test project size to address some of the discrepancies, the size of the project reflects the real-world challenges faced by our Data Provider customers every day.

Understanding these limitations or failure to process images of this size is far more important and shows that a purpose-built application like GeoCompressor allows Hexagon to build a highly optimized compression pipeline far easier than more general-purpose packages. Whether you are a Data Provider frustrated with software deficiencies providing suitable format options, or general users working with large geospatial imagery, GeoCompressor is highly recommended to complement your existing processing workflow.

Appendix A – Data Investigation

Output Variations

As noted above, some formats or software defaulted to output variations including RGB (Red, Green, Blue), RGBA (Red, Green, Blue, Alpha), RGBN (Red, Green, Blue, Near Infrared), RGBNA (Red, Green, Blue,

Near Infrared, Alpha). Although the ECW results above included RGBNA (retaining all input data) the following table was created to show the variations and allow direct comparison.

Target compression for all was 15:1 (default)

ECW output RGB RGBA RGBN RGBNA

Creation time (hh:mm) 00:33 00:33 00:47 00:52

File size (kb) 9,767,042 9,861,600 17,896,538 17,991,104

Page 23: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 23

Lossless JPEG2000 Profiles

Software

Fil

es

ize

(K

b)

Ba

nd

s

No

. T

iles

Til

e S

ize

Pre

cin

ct

Siz

e

Re

so

luti

on

Le

ve

ls

Pro

gre

ss

io

n O

rde

r

Qu

ali

ty

La

ye

rs

Fil

ter

Op

ac

ity

Ba

nd

?

So

p?

Ep

h?

En

co

de

r

GeoCompressor 126,626,142 4 1 x 1 423,460 x

183,720 128 12 RPCL 1 5x3

ECW JPEG 2000 SDK v5.4.0.1431

ERDAS IMAGINE

126,626,887 4 1 x 1 423,460 x

183,720 128 12 RPCL 1 5x3

ECW JPEG 2000 SDK v5.4.0.1431

Competitor A 128,524,942 4 104 x 45 4096 x 4096

256 16 RPCL 30 5x3 Kakadu-v7.5

GDAL OPENJPEG

126,557,717 4 414 x 180 1024 x 1024

512 3 LRCP 1 5x3 OpenJPEG 2.1

GDAL ECWJP2 126,601,306 4 1 x 1 423,460 x 183,720

128 12 RPCL 1 5x3

ECW JPEG 2000 SDK v3.3.0.161

ArcGIS 99,206,112 4 22 x 1 20,000 x 183,720

512 5 PCRL 12 9x7 Kakadu-v7.1

FME 126,601,313 4 1 x 1 423,460 x 183,720

128 12 RPCL 1 5x3

ECW JPEG 2000 SDK v3.3.0.161

ER Mapper 92,326,330 3 1 x 1 423,460 x 183,720

128 12 RPCL 1 5x3

ECW JPEG 2000 SDK v5.4.0.1431

Global Mapper Did not finish ECW JPEG 2000 SDK v3.3.0.161

Page 24: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 24

• Using the default JPEG2000 options from each software yields interesting internal comparisons. For anyone who has used especially large JPEG2000 files from a decoder perspective would often find that different software packages may perform better reading the identical file. This table expresses some of the variations that can lead to these problems. For more information, see the blog JPEG2000 is Slow, or is It?

• Esri output was incomplete, however the file is technically still valid, just not lossless and was included only to highlight profile differences.

• Global Mapper output crashed reliably within the ECW JP2 SDK itself. It was kept in this table to denote the encoder used, which is the not only the v3.3 release from 2006 but the same version used in FME and GDAL that did complete. This is a good example of implementation details regardless of the encoder used.

• Outside of the Hexagon software products, no other Software created an Opacity channel by default, an important feature to ensure usability when integrated into GIS packages. For more information on what happens when opacity bands are not created for any wavelet-based format, see the blog Help – My ECW has Speckled Edges.

• Only the ECWJP2 SDK defaulted to using End of Packet Header (EPH) markers.

• All output examples had GeoGMLJP2 header box created, which was great to see for interoperability and to hope that at some future time, the world and .prj file will go away.

• GDAL OpenJPEG and ArcGIS shared two characteristics:

• A relatively low number of resolution levels given the size of the outputs

• Defaulted to non-RPCL progression order

Overall, JPEG2000 is a complicated format. It is not as simple to draw conclusions over whether it is better to have a large single tile, or thousands of smaller tiles as it is heavily dependent on the decoder library used. The quality layer concept sounds fantastic, but if the decoder ignores them, generating more than one is redundant. The main takeaway from this table should be to acknowledge the differences and be aware of them next time end users may report compatibility issues with a particular software.

Appendix B – Software Workflow

Esri ArcGIS Desktop

Create New Mosaic Dataset

Page 25: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 25

Add Rasters to Mosaic Dataset

Although not directly compared, the time to create the “mosaic” in ArcGIS 10.3 took a surprisingly lengthy 13 minutes.

There was also some level of parallelization during this process.

Page 26: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 26

Mosaic Dataset properties, noting the expected:

• dimensions

• band count

• resolution

Mosaic boundary and footprint vectors

Page 27: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 27

Mosaic to New Raster tool

JPEG2000 output in progress

Performance during the entire export was incredibly slow with minimal CPU and Disk IO. After two failed attempts it was determined this was the expected speed and to let it run.

After 11 days, 1 hour, the process was killed on May 15. It had taken 5 days to write 100mb of data to the OVR file. This was the only indication that the .JP2 was considered complete.

Refer to the JPEG2000 Appendix for further discussion.

Page 28: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 28

Global Mapper

Create new Map Catalog

Preview of mosaic footprints with catalog

metadata matching expected values

Page 29: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 29

Example ECW Export options.

For unknown reasons the Multiband output option is disabled, despite the format supporting this.

JPEG2000 equivalent options. For the same catalog input, the Multiband option is now enabled.

Page 30: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 30

GeoTIFF Export compression options. JPEG is not

listed due to the Multi-band filetype.

An example error seen trying to write BigTIFF JPEG

output.

Enabling “Use tile rather than Strip orientation” resolved this and wrote out 128x128 tiled blocks.

Page 31: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 31

The reproducible crash generating JP2 output in the very old

Hexagon SDK encoder version.

FME

Example workspace using Raster Coercer > Raster Mosaicker > output

Page 32: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 32

2018-04-25 02:02:45|12695.6| 0.0|STATRP|Translation was

SUCCESSFUL with 2 warning(s) (1 feature(s) output)

2018-04-25 02:02:45|12695.6| 0.0|INFORM|FME Session

Duration: 2 hours 35 minutes 31.3 seconds. (CPU: 10877.5s

user, 1817.9s system)

2018-04-25 02:02:45|12695.6| 0.0|INFORM|END -

ProcessID: 4400, peak process memory usage: 8661300 kB,

current process memory usage: 165292 kB

Translation was SUCCESSFUL

Example output compressing 15:1 ECW

Due to percentage target, this equates to 93% quality

GEOPACKAGE_RASTER writer: This format does not support writing the RGB/RGBA bands in the raster with other extra bands. Please remove the extra bands and try again

Example error before using RasterSelector Transformer to select 1:3 bands only

Page 33: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 33

Example of the MRF format options.

ERDAS IMAGINE

Example MosaicPro project overview after importing the inputs.

Page 34: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 34

29/04/18 07:36:18 mosaicprocesspro(7796): The mosaicking time

is 6 hours, 47 minutes,and 5 seconds.[06h:47m:05s]

29/04/18 10:20:22 mosaicprocesspro(7796): The rrd generation and stats/histogram calculation time

is 2 hours, 44 minutes, and 4 seconds.[02h:44m:04s]

29/04/18 10:20:23 mosaicprocesspro(7796): MosaicProcessPro completed in 9 hours, 32 minutes,

and 30 seconds.[09h:32m:30s]

Example output report writing IMG HFA lossless. The second phase includes pyramid and statistics generation, the latter adding overhead.

GDAL

To ensure public reproducible builds were used, release-1911-x64-gdal-2-2-3-mapserver-7-0-7.zip was used. Visual Studio 2017, with OpenJPEG 2.1.0 and Libecwjp2-3.3 drivers added.

> gdalbuildvrt.exe e:\results\2017_hxip_wa_2a_z10_gdal.vrt g:\NWGeo\2017_hxip_wa_2a_z10\*.tif

0...10...20...30...40...50...60...70...80...90...100 - done.

All GDAL based outputs are generated from the VRT virtual mosaic.

A sample of the generated output command lines used are below

> gdal_translate -of GPKG e:\results\2017_hxip_wa_2a_z10_gdal.vrt

e:\results\2017_hxip_wa_2a_z10_gdal_geopackage_jpgpng.gpkg ===

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 24246.470 s

> gdal_translate -of MRF -co COMPRESS=LERC -co OPTIONS="LERC_PREC=0.5" e:\results\2017_hxip_wa_2a_z10_gdal.vrt

e:\results\2017_hxip_wa_2a_z10_gdal_mrf_lerc_lossless.mrf

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 15128.820 s

> gdal_translate -of MRF e:\results\2017_hxip_wa_2a_z10_gdal.vrt

e:\results\2017_hxip_wa_2a_z10_gdal_mrf_png_lossless.mrf

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Page 35: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 35

Execution time: 30455.366 s

> gdal_translate -of JP2openjpeg -co REVERSIBLE=YES -co QUALITY=100 e:\results\2017_hxip_w_z10_gdal.vrt

e:\results\2017_hxip_wa_2a_z10_gdal_lossless_openjpeg.jp2

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

ERROR 1: Prevent buffer overflow (x1: 423460, y1: 183720)

ERROR 1: Marker handler function failed to read the marker segment

ERROR 1: opj_read_header() failed

Execution time: 52188.326 s

> gdal_translate -of JP2ECW -co LARGE_OK=YES -co TARGET=0 e:\results\2017_hxip_wa_2a_z10_gdal.vrt

e:\results\2017p_wa_2a_z10_gdal_lossless_ecw33.jp2

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 42937.571 s

> gdal_translate -of ECW-co TARGET=93 e:\results\2017_hxip_wa_2a_z10_gdal.vrt

e:\results\2017_hxip_wa_2a_z10_gdal_15x.ecw

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 9979.593 s

> gdal_translate -of GTiff -co COMPRESS=DEFLATE -co NUM_THREADS=ALL_CPUS -co PREDICTOR=2 -co TILED=yes -co

BIGTIFF=yes e:\results\2017_hxip_wa_2a_z10_gdal.vrt e:\results\2017_hxip_wa_2a_z10_gdal_deflate.tif

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 3949.307 s

> gdal_translate -of GTiff -co COMPRESS=JPEG -co NUM_THREADS=ALL_CPUS -co TILED=yes -co BIGTIFF=yes

e:\results\2017_hxip_wa_2a_z10_gdal.vrt e:\results\2017_hxip_wa_2a_z10_gdal_jpeg.tif

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 8963.646 s

Page 36: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 36

> gdal_translate -of MRF -co COMPRESS=JPEG e:\results\2017_hxip_wa_2a_z10_gdal.vrt

e:\results\2017_hxip_wa_2a_z10_gdal_mrf_jpeg_85quality.mrf

Input file size is 423460, 183720

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 11251.819 s

Note: GDAL Target compression values tries to reuse JPEG style percentages which is confusing for wavelet rates and is prone to user error under-compressing output, since the default 75% target equates to 4:1 output.

Wavelet 15:1 target =~ 93% “JPEG” quality target

Wavelet 50:1 =~ 98% “JPEG” quality target

Pyramid generation is covered above, however some additional commands can be seen below. A basic timer was placed around the executable that reported total execution time.

> gdaladdo -r average --config NUM_THREADS_OVERVIEW ALL_CPUS --config BIGTIFF_OVERVIEW IF_NEEDED --config

COMPRESS_OVERVIEW DEFLATE e:\results\2017_hxip_wa_2a_z10_gdal_deflate.tif 2 4 8 16 32 64 128 256 512 1024

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 160310.072 s

> gdaladdo -r average --config NUM_THREADS_OVERVIEW ALL_CPUS --config BIGTIFF_OVERVIEW IF_NEEDED --config

COMPRESS_OVERVIEW JPEG

e:\results\2017_hxip_wa_2a_z10_gdal_mrf_jpeg_85quality.mrf 2 4 8 16 32 64 128 256 512 1024

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 158368.921 s

> gdaladdo -r average --config NUM_THREADS_OVERVIEW ALL_CPUS

e:\results\2017_hxip_wa_2a_z10_gdal_mrf_png_lossless.mrf 2 4 8 16 32 64 128 256 512 1024

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

Page 37: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 37

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 176984.936 s

> gdaladdo -r average --config NUM_THREADS_OVERVIEW ALL_CPUS --config BIGTIFF_OVERVIEW IF_NEEDED --config

COMPRESS_OVERVIEW JPEGPNG

e:\results\2017_hxip_wa_2a_z10_gdal_geopackage_jpgpng.gpkg 2 4 8 16 32 64 128 256 512 1024

0...10...20...30...40...50...60...70...80...90...100 - done.

Execution time: 18943.338 s

Page 38: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 38

GeoCompressor

GeoCompressor UI builds the XML Mosaic definition using a wizard interface.

<?xml version="1.0" encoding="UTF-8"?>

<imagecompressor version="1.2"> <compresstask

operation="create" method="tile" tempdir="G:\temp\"

logfile="E:\results\2017_hxip_wa_2a_z10_15x_rgb.log"

memcache="0.25" interactive="false"> <inputs> <file

path="G:\NWGeo\2017_hxip_wa_2a_z10\hxip_m_4512121_nw_10_30.tif"

bandlist="0,1,2" zindex="140" /> <file

path="G:\NWGeo\2017_hxip_wa_2a_z10\hxip_m_4512122_ne_10_30.tif"

bandlist="0,1,2" zindex="139" />

…</inputs> <output version="3"

path="e:\results\2017_hxip_wa_2a_z10_15x_rgba.ecw"> <options>

<option name="targetrate" value="15" /> <option name="genstats"

value="true" /> <option name="colorspace" value="rgb" />

<option name="opacity" value="3" /> </options> </output>

</compresstask> </imagecompressor>

Example XML Project that is

generated

Page 39: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 39

C:\Program Files\Hexagon\GeoCompressor 2018 U2\bin>ImageCompressor.exe -xml g:\temp\2017_hxip_wa_2a_z10_15x_rgba.xml Acquired Geocompressor Professional 16.5 license. Licensed to: Hexagon Geospatial. Expiry Date: 14 April 2020 (553 days) Build: v16.5.0.1523 Hardware Detected -------------- Computer name: MEGATRONX Platform: Windows Server 2012 R2 Standard CPU Model: Genuine Intel(R) CPU @ 2.90GHz CPU Spec: 16 Cores, 32 Threads, 2 Processors, NUMA enabled (32 total) Memory: 127.93 GB Temp dir: G:\temp\ Validating input: ... File 141 of 141 (4305 ms) Input Data -------------- File name: g:\temp\2017_hxip_wa_2a_z10_15x_rgba.xml File type: ERMapper Mosaic Algorithm Data reader: ERM Lib Dimensions: 423,460 x 183,720 px (77.798 gigapixel) Structure: 3 Band, RGB UINT8 (141 image files) Opacity band: false Nodata value: not detected Filesize: 16.24 KB (217.37 GB uncompressed) Projection: EPSG:6339 Pixel Size (width, height): 0.300000,0.300000 Units: Meters Setting mem cache to: 31.98 GB Compression -------------- Start Time: Tue Oct 09 04:35:05 2018 Memory cache -------------- System: 16,384.00 GB Read: 29.63 GB Write: 2.35 GB Method: Tile Threads: 32 Precincts: 25346844 Total Blocks: 6336711 [30.88% NB: 0, B: 2059796, P: 8239240, E: 00:22:42]

GeoCompressor has both a user interface and command line execution. The latter was used to generate these results. The output shown here is the process in progress.

Page 40: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 40

Example CPU usage using default values writing the

mosaic to ECW

Duration: 0 hours 32 mins 59 seconds - Read: 28 mins 23 seconds - Write: 3 mins 53 seconds - Reassembly: 38 seconds Target Ratio: 15:1 Actual Ratio: 30.8:1 Throughput: 150.0 MB / sec Output Data -------------- File Name: e:\results\2017_hxip_wa_2a_z10_15x_rgba.ecw File Type: ECW v3 Data Writer: ECW JPEG2000 SDK v5.4 Dimensions: 423,460 x 183,720 px Structure: 4 Band, RGB UINT8 Opacity band: true Projection: EPSG:6339 Pixel Size (width, height): 0.300000,0.300000 Units: Meters File Size: 9.40 GB

Example of finalized output on completion

Source Material Reference Imagery used was provided by the HxGN Content Program at https://hxgncontent.com.

Technical Reference

HxGN Content Program data specifications are available at https://hxgncontent.com/imagery/data-specifications.

Page 41: GeoCompressor - bynder.hexagon.com · Hexagon GeoCompressor 2018 v16.5.0.1417 March 2018 Competitor “A” (Unable to disclose) Hexagon ERDAS IMAGINE 2018 v16.5.0.912 March 2018

12 July 2019 41

Contact us For more information, please contact us at:

[email protected]

+1 877 463 7327

https://go.hexagongeospatial.com/contact-us-today

About Hexagon Hexagon is a global leader in sensor, software and autonomous solutions. We are putting data to work to boost efficiency, productivity, and quality across industrial, manufacturing, infrastructure, safety, and mobility applications. Our technologies are shaping urban and production ecosystems to become increasingly connected and autonomous — ensuring a scalable, sustainable future. Hexagon’s Geospatial division creates solutions that deliver a 5D smart digital reality with insight into what was, what is, what could be, what should be, and ultimately, what will be. Hexagon (Nasdaq Stockholm: HEXA B) has approximately 20,000 employees in 50 countries and net sales of approximately 4.3bn USD. Learn more at hexagon.com and follow us @HexagonAB. © 2019 Hexagon AB and/or its subsidiaries and affiliates. All rights reserved. Hexagon and the Hexagon logo are registered trademarks of Hexagon AB or its subsidiaries. All other trademarks or service marks used herein are property of their respective owners.