ecw / hpc tutorial: data compression with geocompresor · .tif extension or the .ecw extension as...
TRANSCRIPT
ECW / HPC Tutorial: Data Compression with GeoCompressor
ii
iii
ECW / HPC Tutorial: Data Compression with GeoCompressor Copyright © 2015 Hexagon Geospatial, Inc. All Rights Reserved.
Printed in the United States of America. The information contained in this document is the exclusive property of Hexagon Geospatial. This work is protected under United States copyright law and other international copyright treaties and conventions, to include the Berne and Geneva Phonograms Conventions, the WIPO Copyright Treaty, and the World Trade Organization. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or by any information storage or retrieval system, except as expressly permitted in writing by ERDAS. All requests should be sent to the attention of Manager of Customer Education at the following address: Hexagon Geospatial 5051 Peachtree Corners Circle, Suite 100 Norcross, Georgia 30092-2500 Phone: 770 / 776-3400 Fax: 770 / 776-3000 Web: www.hexagongeospatial.com
The information contained in this document is subject to change without notice.
Warning All information in this document, as well as the software to which it pertains, is proprietary material of Hexagon Geospatial, Inc., and is subject to a Hexagon Geospatial license and non-disclosure agreement. Neither the software nor the documentation may be reproduced in any manner, without the prior written permission of Hexagon Geospatial. Specifications are subject to change without notice.
Trademarks ERDAS, ERDAS IMAGINE, GeoMedia, ImageStation, IMAGINE Advantage, IMAGINE Essentials, IMAGINE Professional, and IMAGINE VirtualGIS are registered trademarks; IMAGINE AutoSync, IMAGINE Developers’ Toolkit, IMAGINE Easytrace, IMAGINE IFSAR DEM, IMAGINE NITF 2.1, IMAGINE OrthoRadar, IMAGINE Radar Interpreter, IMAGINE StereoSAR DEM, IMAGINE Subpixel Classifier and IMAGINE Vector are trademarks. Other brand and product names are the properties of their respective owners.
About This Manual
The Work is an instructional document designed to be an integral part of the training course offered by Hexagon Geospatial. Hexagon Geospatial believes the information in the Work is accurate as of its publication date. Any specifications cited in the Work are subject to change without notice. The information and the software discussed in the Work are subject to change without notice. Intergraph is not responsible for any errors that may appear in the Work.
Intergraph, the Intergraph logo, ERDAS, ERDAS IMAGINE, IMAGINE Essentials, IMAGINE Advantage, IMAGINE Professional, IMAGINE VirtualGIS, GeoMedia, ImageStation, Video Analyst, Stereo Analyst, TopoMouse, Hexagon Smart M.App, and Power Portfolio are trademarks of Intergraph Corporation. Hexagon and the Hexagon logo are registered trademarks of Hexagon AB or its subsidiaries. Other brands and product names are trademarks of their respective owners.
Acknowledgments All data used in this course were downloaded from public data sources and falls within the usage guidelines of those web sites and the organizations making the data publicly available.
iv
v
Table of Contents
Table of Contents .................................................................................................................................. v
This Tutorial ............................................................................................................................................ vi
Section 1: Effective Storage ............................................................................. 7
Compress images using GeoCompressor .......................................................................... 9
Compressing Point Cloud Data ........................................................................................ 19
vi
This Tutorial
Tutorial Objective To instruct how to use the standalone GeoCompressor application to reduce the storage footprint of data by compressing large geospatial image datasets into smaller ECW or JP2 formats. Instruction will also cover taking LiDAR (Light Detection and Ranging) point clouds, which can be quite large, and compress them into a Hexagon Point Cloud (HPC) dataset which gives the data a spatial index and converts the points to voxels that can be utilized in different applications.
Tutorial Data Set The tutorial data is ~2Gb (zipped) in size and can be used in other ERDAS APOLLO tutorials. Some of the data provided in the Work were procured from the Clark County Nevada GIS Department (http://www.clarkcountynv.gov/depts/ccgis) and is provided without warranty of any kind, expressed or implied. Other data used in this course were downloaded from public data sources and falls within the usage guidelines of those web sites and the organizations making the data publicly available.
Tutorial Text Conventions There are several conventions used throughout the tutorial:
Menu items are shown as: In the Aaa tab, in the Bbb group, click Ccc.
Dialog box names, field names, and button names are depicted using Bolded Text.
Information to be entered, either by selecting from a list or by typing, is depicted using Italicized Text.
Tutorial Prerequisites I
Section 1: Effective Storage
Section Objective This section is intended to instruct users how to use GeoCompressor to compress imagery into the ECW format.
Tools Used
GeoCompressor
A standalone program to compress images to the ECW or JPEG 2000 format to simplify format conversion to our optimized formats.
ERDAS IMAGINE
Used to view ECW and JP2 images and header information.
8 Effective Storage
9
Compress images using GeoCompressor
Objective: Managing geospatial imagery remains problematic despite cheaper storage and increasing processing power. Imagery is captured more frequently with more spectral bands and at a higher resolution / bit depth. IT departments have not maintained the velocity of Geospatial imagery in increasing storage allocation and as such storage is still a constraint. For effective storage of terabytes of data, the four critical image format characteristics are: encoding speed, file size, image quality and decoding speed. Wavelet compressed images such as ECW and JPEG2000 are specially designed to meet the 4 critical characteristics for geospatial imagery. ECW provides the industry fastest decompression and compression rates with minimal hardware. It can scale to terabytes in size. It is supported across all major desktop, mobile and server GIS packages. The supplementary files that can accompany imagery, and consume storage space, such as pyramids, overviews or tile caches are not required when using ECW.
The main objective of this exercise is to compress and create a geospatial image which is much smaller in file size than the original dataset with no visual loss in image quality but at the same time can be delivered through high performance Web Services and thus building an effective storage strategy for your geospatial images.
Task 1: Compressing Imagery
GeoCompressor is a standalone program used to convert and compress images to the ECW or JPEG 2000 format It is cross-platform 64bit application with both simple wizard based and command line interfaces.
1. Start GeoCompressor from the Start menu. Start GeoCompressor 2015 GeoCompressor.
2. Select the option to create a “New ECW JP2”
10 Effective Storage
3. Of the three (3) choices, select the Compress a single image or a set of images then click the Next button to progress to the next step.
4. There are three options to add data from the D:\GCData\Orthos folder into the GeoCompressor wizard.
a) Select the Add Images option and navigate to the desired directory, then select one or multiple images (Ctl-Click) and click the Open button.
b) Select the Add Folder option, navigate to the Orthos folder, select it and click the Select Folder button
c) Using an Explorer Window, navigate to the directory, select one or multiple images or the directory itself and drag and drop them into the Wizard.
If the Add Folder option has choosen files that should not be compressed, they can be deleted by selecting the image(s) in the list and click the Delete Image button or right click and Delete. This removes the image(s) from the list.
11
5. After adding the data and clicking Next, the Select Compression Options dialog of the Wizard open up.
6. Select an Output Directory to write the ECW files produced. Select an existing directory or create a new directory and select.
7. In the Filename Template the default is $(FILENAME).$(SRCEXT).ecw. This can be changed by clicking the template button locate to the right of the template field.
8. Change the current template name to GCTrain.$(FILENAME)_$(SRCEXT).ecw
This change will place “GCTrain” as the first part of the output name, followed the input image filename, then the extension of the source file (i.e. tif) and finally the .ecw extension. An example of the output would be: GCTrain.nv_c_3611555_nw_11_1_2006mmdd_tif.ecw.
With this release, there is a minor issue where if the .$(SRCEXT) variable is use and is preceded by a ‘dot’ the output filename will contain two extensions. Example: TestImage.tif.ecw
ERDAS APOLLO will fail to crawl this image as it cannot distigush between the .tif extension or the .ecw extension as the correct extension to catalog. Use an underscore “_” instead of the dot when using this variable.
9. Next, select a Temp Directory to use in the compression process.
10. For the Format, select ECW v3, a Color Space of Auto and a Bit Depth of Auto.
11. The Target Rate should be in-between 15 to 20, the Method can either be Tile or Line and the Threads should be set to ‘0’ for automatic.
When using JPEG 2000 compression, the Method must be Line. Even it Tile is selected the program will ignore and use Line.
12. Click the Next button and the Opacity Channel Options dialog will display. Accept all defaults and click the Next button.
13. The Summary dialog will display.
12 Effective Storage
14. Review the operations in the Summary then click the Finish button.
When using JPEG 2000 It is good practice to read input images from a one disk and write the ouput compressed files to another. This helps IO throughput during the encoding process.
15. In the GeoCompressor dialog, all the images selected will visible. Click on the Start icon to begin the compression process.
a) Once stared the Status column will show one of four (4) states: Pending, Running, Succeeded or Fail.
16. For one of the images that has finished processing, double click on the icon in the Log column. This will open up the log in the systems default text editor.
17. Open one of the images produced and review.
After all records have process, click on the Down icon next to the Delete Task icon. A menu will give four options; select Remove all tasks.
13
Task 2: Mosaic and Compress a set of images.
In this task, the images previously were individually compressed and saved as individual files. Now GeoCompressor will be used to produce a single mosaic image.
1. Click on the New ECW JP2 icon in the GeoCompressor.
2. In the Wizard, select the second option: Mosaic a set of images into a single image and click the Next button.
3. As done in the previous task, add the images from C:\GCData\Orthos to the Wizard.
4. Navigate to an Output Directory where the mosaic image will be saved.
5. In the Filename Template, remove the current template and give the new, single file a name. Example: GCTrain_Mosaic.ecw.
6. Leave the Temp Directory with its default.
7. In the Option:
a) Format – ECW v3 b) Color Space – Multiband c) Bit Depth – Auto d) Target Rate – 15 e) Method – Tile f) Threads – 0
8. Click on the Edit Metadata Defaults button; the Meta Data Options dialog opens.
9. Place a check in the Meta Data Options check box and input the following:
a) Classification – None b) Acquisition Date – Check the box and input today’s date. c) Sensor Name – Leica RC30 d) Author – Your Name e) Copyright – Copyrighted 2015 by Hexagon Geospatial
http://www.intergraph.com/terms.aspx f) Company – Hexagon Geospatial g) Email – None h) Address – 5051 Peachtree Corners Circle, Suite 100, Norcross GA 30092 i) Telephone – +1 770-776-3400
14 Effective Storage
10. Once complete, click the OK button, and then in the Wizard click the Next button.
11. Leave the Opacity Channel Options at its defaults and click Next.
12. Review the Summary and click the Finish button.
13. Review the record to process in the GeoCompressor dialog box. What is different than seen in the previous task?
14. Click Start in the dialog box to begin the mosaic / compression process. When creating a mosaic image from a bunch of individual images, the GeoCompressor Wizard takes all the input information about the process defined by the user and creates an .xml file. This XML file is then used by GeoCompressor to read all the information, including the location of the images to be used. This is why only one record is seen in the interface instead of all the images being used for the mosaic.
15. Open up the image and review the final result.
15
16. Navigate to the directory where the mosaicked image was saved. In this director three files were generated:
a) GCTrain_Mosaic.ecw b) GCTrain_Mosaic.txt c) GCTrain_Mosaic.xml
17. Open the XML file in a text editor (i.e. Notepad++). Notice all the information such as location of individual files, metadata and other information saved.
This XML file can be used if the image needs to be recreated. In the GeoCompressor dialog there is the Load XML option. This allows XML files like this one to run and create compressed images.
16 Effective Storage
Task 3: Update a region of a mosaic with newer imagery.
At times, after a large mosaic has been generated, new information comes in that needs to be incorporated into the mosaic. Instead of recreating the original mosaic, GeoCompressor will be used to incorporate new data into the mosaic imagery without having to re-create the entire process.
This process saves time and resources. The original mosaic may have taken hours or days to create. The update region function takes a fraction of the time needed to create the original image.
1. Navigate to the directory where the mosaic image from Task 2 was created.
2. In that directory there should be three files:
a) GCTrain_Mosaic.ecw b) GCTrain_Mosaic.txt c) GCTrain_Mosaic.xml
3. Delete or Rename the XML file for the existing mosaic.
4. Start GeoCompressor and make sure that no jobs already exist. If they do, select and remove them with the Delete Task option.
5. Click the New ECW JP2 icon and select the third option Update a region within an existing ECW v3 file and click the Next button.
6. In the input image dialog, select Add Images, navigate to C:\GCData\Aria and select the Vegas_aria.img file then click the Next button.
7. For the output File, an existing ECW v3 file must be selected. This is the pre-existing mosaic that will be updated.
8. In the Select Compression Options dialog, all of the options are greyed out except for the number of Threads to use and the Metadata Defaults.
An issue was found in this release where if the existing XML file for the mosaic is not removed or renamed, the GeoCompressor will use the existing XML file for the update process. The mosaic will complete with no errors, but the region to update will not be processed and is ignored.
When updating a region of an existing ECW file, the original image will be changed. There is no option to create a new ECW with the updated are using the Region Update option.
If the original image needs to be preserved, make a back up copy. Once the update is done there is no ‘undo’.
17
9. Click the Next button and leave the defaults for the opacity channel.
10. Click Next for the Summary page and then Finish.
11. Click the Start icon in the GeoCompressor to start the job; review the log and the image once completed.
Original Mosaic and area to
New Data for
18 Effective Storage
12. The band combinations from the mosaic image and the updated imagery are different on purpose for this exercise so the contrast makes it easier to see the new imagery in the older mosaic.
19
Compressing Point Cloud Data
Objective: Point cloud data is typically very large and formats such as LAS do not have a spatial index. Without a spatial index data access in multiple programs, to include ERDAS APOLLO and ERDAS IMAGINE, can be very slow. The Hexagon Point Cloud, or HPC, format is a spatially index format for storing point cloud data that can be up to 10 times smaller than the original point cloud data. GeoCompressor supports single LAS or LAZ file inputs, allowing for the conversion of these formats to the HPC format. Mosaicking or merging is not currently supported.
Task 1: Converting LAS data to HPC format
This task will cover how to take existing LiDAR or Point cloud data in LAS or LAZ format and convert them to the Hexagon Point Cloud (HPC) format.
1. Start up GeoCompressor and select the New HPC icon to start up the Point Compression Wizard.
With this release, point cloud files must be added one at a time. The user cannot drag and drop nor add multiple image in a batch type function.
2. Click on the browse icon in the Select an input file field and navigate to C:\GCData\VegasLAS.
3. Select the LAS file NV_LasVegasValley_2010_000391.las then click the Open button in the lower right.
4. In the Output option, click on the browse button and navigate to the location to place the output HPC file. Name the file the same or give the output file a new name.
5. Click the Next button.
6. In the HPC Conversion Options either use the default Temp Directory or navigate and designate a new directory to use.
20 Effective Storage
7. In the SRS Code section, leave blank. The output SRS will be the same as the input and in this case that is EPSG:32611.
8. The data is in meters so in the Units section either select Auto for auto detect of units or select Meters for the unit choice.
9. Leave the Lookup Table with its default.
10. No input in the Point Resolution field.
11. Since the data is going to be used just for streaming purposes, the only checkbox to check will be the third option: Compress output.
12. Click Next for the Summary, then click the Finish button to get back to the GeoCompressor interface.
13. Notice the single record in the interface. Do not click Start, instead click on the New HPC icon and go through the process again for the remaining 3 HPC files.
14. Once all four LAS files have been added click the Start button to begin the HPC Conversion process.
21