dsm training - preparing auxiliary data

Auxiliary maps for Digital Soil MappingFOSS software (R+SAGA+FWTools)

T. HenglInternationaal Bodemreferentie en Informatie Centrum - ISRIC

Wageningen UR

Digital Soil Mapping training, Sept 3, 2010, JRC Ispra

Programme

I 9:00–10:15 Importance of auxiliary maps for DSM;

I 10:30–12:00 Overview of publicly available global data sets;

I 12:00–14:00 lunch breakI 14:00–15:15 Data exploration (exporting maps to KML)

I Software installation and customization.I Opening and processing worldmaps using R+SAGA/FWTools.I Exercises.

I 15:30–17:00 Preparing auxiliary maps for continental-scaleDSM project;

I Optional: preparation of worldmaps and metadata for yourown case study;


Outline

IntroductionLiteratureDSM and env mapsWorldmaps

SoftwareR code editorsWorking with spatial dataR+SAGA

ExercisesR+FWToolsExport to Google Earth


Literature

Bivand, R., Pebesma, E., Rubio, V., 2008. Applied SpatialData Analysis with R. Use R Series, Springer, Heidelberg, 378p.

Conrad, O., 2007. SAGA — Entwurf, Funktionsumfang undAnwendung eines Systems fur AutomatisierteGeowissenschaftliche Analysen. PhD thesis, University ofGottingen.

Hengl, T., 2009. A Practical Guide to Geostatistical Mapping,2nd edition. University of Amsterdam, 291 p. ISBN978-90-9024981-0.

McBratney, A. B., Santos, M. L. M., Minasny, B., 2003. Ondigital soil mapping. Geoderma, 117(1–2): 3–52.


http://asdar-book.org

http://asdar-book.org

http://webdoc.sub.gwdg.de/diss/2007/conrad/conrad.pdf



http://spatial-analyst.net/book/

http://sciencewatch.com/dr/erf/2010/10feberf/10feberfMcBr/

http://sciencewatch.com/dr/erf/2010/10feberf/10feberfMcBr/

The ASDA(R)-book.org


http://www.asdar-book.org

The ASDA(R) team


A Practical Guide to Geostatistical Mapping


http://spatial-analyst.net/book/

Outline





DSM: the main steps (analysis)

1. Import and organize soil lab data (site/horizon).

2. Filter gross errors (outliers), harmonize data coming fromvarious sources.

3. Fit splines to parameterize soil-depth relationships.

4. Prepare auxiliary maps (grid definition, projection system,data format, . . .).

5. Generate spatial predictions (predictive mapping) and assessaccuracy.

6. Prepare and distribute final maps (with metadata) and reports.


Objectives

I Where to obtain auxiliary environmental maps for DSM?

I Where to obtain and how to use the R+OSGeo software?

I How to combine GIS and DSM operations?

I How to visualize maps using Google Earth/Maps?


Definitions

I Digital Soil Mapping: Analytical production of soil propertyand soil type maps using quantitative techniques, groundrecords (soil profiles), auxiliary maps (covariates) and expertknowledge.

I The main input to DSM is a map; the main output is a map;I There are three main types of auxiliary maps:

1. DEM-derivatives (morphometric, hydrological, climatic, . . .)2. Remote sensing images (thermal images, radar images, gamma

radiometrics, vegetation indices, . . .)3. Thematic maps (geological strata, geomorphological units, . . .)


Definitions


I The main input to DSM is a map; the main output is a map;

I There are three main types of auxiliary maps:




Definitions






Definitions



1. DEM-derivatives (morphometric, hydrological, climatic, . . .)

2. Remote sensing images (thermal images, radar images, gammaradiometrics, vegetation indices, . . .)

3. Thematic maps (geological strata, geomorphological units, . . .)


Definitions




radiometrics, vegetation indices, . . .)

3. Thematic maps (geological strata, geomorphological units, . . .)


Definitions






Auxiliary maps

I Auxiliary maps (environmental predictors or covariates) areequally important input to DSM as the point data.

I If we are lucky, we will be able to explain distribution of soilproperties by using a small sample of speciesoccurrence/attributes.

I Environmental data can come in different resolutions (supportsize), accuracy, coverage, formats etc.

I The publicly available global data sets are in generalunder-used.

I . . . for various reasons (unpopular formats, missing metadata,significant processing required to put it to a usable formatetc.).


Universal model of spatial variation

In statistical terms, spatial variability of some soil variable Zconsists of three components:

Z (s) = Z ∗(s) + ε′(s) + ε′′ (1)

where Z ∗(s) is the deterministic component, ε′(s) is the spatiallycorrelated random component and ε′′ is the pure noise — partiallymicro-scale variation, partially the measurement error. This modelis, in the literature, often referred to as the universal model ofvariation


SCORPAN

A well accepted formulation of the soil–environment correlationmodel is the so-called SCORPAN model McBratney at al. (2003),which is based on the general soil–vegetation formulation:

V × S [x , y , t ] = f

(s[x , y , t ] c[x , y , t ] o[x , y , t ]r [x , y , t ] p[x , y , t ] a[x , y , t ]

)(2)

where V stands for vegetation, S for soil, c stands for climate, ofor organisms (including humans), r is relief, p is parent materialor geology, a is age of the system, x , y are the coordinates and t istime dimension.


Outline





Global repository of publicly available maps


http://commons.wikimedia.org/wiki/Category:Publicly_available_global_data_sets

worldmaps

I I’ve been collecting and sorting/processing various publiclyavailable data sets over the years (at 1–5 km resolution, thereis a lot of free data!).

I The result is a repository with cca 100 unique rasters, thatcan be obtained directly fromhttp://spatial-analyst.net/worldmaps/.

I Each gridded map consists of 7200 columns and 3600 rows;the cell size is 0.05 arcdegrees, which corresponds to about5.6 km; all maps fall on the same grid.

I Maps are projected in the Latitude-Longitude WGS84 system+proj=longlat +ellps=WGS84.

I These maps are ideal for various mapping applications atcontinental and national levels.


http://spatial-analyst.net/wiki/index.php?title=Global_datasets

http://spatial-analyst.net/worldmaps/

The happy triangle guy

GIS analysis

Storage andbrowsing of

geo-data

Statisticalcomputing

KML

GDAL

groundoverlays,

time-series


Software we will use

I R v2.11 (Windows OS) including a list of packages;

I Tinn-R v2.3 (code editor);

I Optional: FWTools v2.4.7 — a list of utilities to handlespatial data; SAGA GIS v2.0.4 — a light GIS excellent foreducational purposes.


http://cran.r-project.org/bin/windows/base/

http://www.sciviews.org/Tinn-R/

http://fwtools.maptools.org

http://www.saga-gis.org

Installing the add-on packages

> install.packages("ctv")

> library(ctv)

> install.views("Spatial")

This will install all connected packages listed at R views Spatial.


http://cran.r-project.org/web/views/Spatial.html

Check your installation

> Sys.getenv(c("OS", "COMPUTERNAME", "R_HOME", "R_LIBS_USER",

+ "PROCESSOR_IDENTIFIER"))

OS

"Windows_NT"

COMPUTERNAME

"PC-IBED193"

R_HOME

"C:\\PROGRA~1\\R\\R-210~1.1"

R_LIBS_USER

"n:/R/win-library/2.10"

PROCESSOR_IDENTIFIER

"x86 Family 6 Model 15 Stepping 6, GenuineIntel"


Outline





Tinn-R


Customizing link between R and Tinn-R:

1. Open Tinn-R select “Options” 7→“R” 7→ Path and check thatthe path is correct;

2. Tinn-R requires R to run either Rterm or Rgui in SDI mode:In Rgui, select “Edit” 7→“GUI preferences” and set SDI andclick on Save.

3. To add the CRTL+R shortcut to Tinn-R:I First open “Options” 7→“Shortcuts” and replace the existing

CRTL+R shortcut to e.g.CRTL+M;I Then, open “R” menu and select “Hotkeys”; add a CRTL+R

shortcut to “Send line”;I To send blocks of text you will need to edit your Rprofile.site

file under “R/etc/” directory;

4. To learn more about how to customize Tinn-R, read theuser_guide.html file under the“Tinn-R/doc/English/user guide/” directory.


http://spatial-analyst.net/scripts/Rprofile.site

Outline





Spatial objects

An advantage of R (as compared to e.g.Matlab) is that you cancreate your own formats and structures for data. But if there aretoo many formats you can easily get lots. In addition, we want tohave smooth links to external formats (R is open!).

To reduce this problem, Bivand et al.(2008) developed new-styleclasses to represent spatial data.


Spatial class

The foundation object is the Spatial class, with just two basic slots(new-style S classes have pre-defined components called slots):

I a bounding box — mostly used for setting up plots;

I a CRS class object — defining the coordinate referencesystem, and may be set to CRS(as.character(NA));

Operations on Spatial* objects should update or copy thesevalues to the new Spatial* objects being created. The most basicspatial data object is a point, which may have 2 or 3 dimensions.


Spatial classes

I for point features: SpatialPoints;SpatialPointsDataFrame;

I for line features: SpatialLines, SpatialLinesDataFrame;

I polygons: SpatialPolygons, SpatialPolygonsDataFrame;

I rasters: SpatialPixels, SpatialPixelsDataFrame,SpatialGrid, SpatialGridDataFrame;


SpatialPoints

> library(sp)

> data(meuse)

> coords <- SpatialPoints(meuse[, c("x", "y")])

> summary(coords)

Object of class SpatialPoints

Coordinates:

min max

x 178605 181390

y 329714 333611

Is projected: NA

proj4string : [NA]

Number of points: 155


SpatialPointsDataFrame

We can add the tabular data to make aSpatialPointsDataFrame object:

> meuse1 <- SpatialPointsDataFrame(coords, meuse)

> str(meuse1, max.level = 2)

Formal class 'SpatialPointsDataFrame' [package "sp"] with 5 slots

..@ data :'data.frame': 155 obs. of 14 variables:

..@ coords.nrs : num(0)

..@ coords : num [1:155, 1:2] 181072 181025 181165 ...

.. ..- attr(*, "dimnames")=List of 2

..@ bbox : num [1:2, 1:2] 178605 329714 181390 333611

.. ..- attr(*, "dimnames")=List of 2

..@ proj4string:Formal class 'CRS' [package "sp"] with 1 slots


Basic methods

I spplot — plotting of spatial objects (maps);

I spsample — sample points from a set of polygons, on a setof lines or from a gridded area;

I bbox — get the bounding box;

I proj4string — get or set the projection (coordinatereference system);

I coordinates — set or retrieve coordinates;

I spTransform — transform coordinates from one CRS toanother (PROJ.4);

I overlay — combine two different spatial objects;


Plotting a SpatialPoints object

> plot(as(meuse1, "Spatial"), axes = TRUE)

> plot(meuse1, add = TRUE)

> plot(meuse1[meuse1$ffreq == 1, ], col = "green", add = TRUE)

178000 179000 180000 181000 182000

3300

0033

1000

3320

0033

3000


Combining statistical and GIS operations

I Because the Spatial*DataFrame family objects behave inmost cases like data frames, most of what we are used todoing with standard data frames just works (but no merge,etc., yet).

I These objects are very similar to typical representations of thesame kinds of objects in geographical information systems, sothey do not suit spatial data that is not geographical (likemedical imaging) as such.

I Because now sp classes for GIS data exits, this opens the doorfor fusing GIS and statistical operations (this has not beenpossible in e.g.2002).


Spatial packages

R now offers a range of contributed packages in spatial statisticsand increasing awareness of the importance of spatial data analysisin the broader community. Current contributed packages withspatial applications:

I point patterns: spatstat, VR:spatial, splancs;

I geostatistics: gstat, geoR, geoRglm, fields, spBayes,RandomFields, VR:spatial, sgeostat, vardiag;

I lattice/area data: spdep, DCluster, spgwr, ade4;

I links to GIS: rgdal, spgrass, RPy, RSAGA;


Let’s create spatial objects!

We can create spatial objects from scratch! For example a DEM:

> dem <- expand.grid(x = seq(100, 600, 100), y = seq(100,

+ 600, 100))

> dem$Z <- as.vector(c(23, 24, 34, 38, 45, 51, 24, 20,

+ 20, 28, 18, 49, 22, 20, 19, 14, 38, 45, 19, 15, 13,

+ 21, 23, 25, 14, 11, 18, 11, 18, 19, 10, 16, 23, 16,

+ 9, 6))

> gridded(dem) <- ~x + y

> dem <- as(dem, "SpatialGridDataFrame")

> str(dem)


Outline





Controlling SAGA from R

> library(RSAGA)

> rsaga.env()

$workspace

[1] "."

$cmd

[1] "saga_cmd.exe"

$path

[1] "C:/PROGRA~1/R/R-210~1.1/library/RSAGA/saga_vc"

$modules

[1] "C:/PROGRA~1/R/R-210~1.1/library/RSAGA/saga_vc/modules"


Getting list of modules

> rsaga.get.modules("ta_channels")

$ta_channels

code name interactive

1 0 Channel Network FALSE

2 1 Watershed Basins FALSE

3 2 Watershed Basins (extended) FALSE

4 3 Vertical Distance to Channel Network FALSE

5 4 Overland Flow Distance to Channel Network FALSE

6 5 D8 Flow Analysis FALSE

7 6 Strahler Order FALSE


Run stream extraction

> rsaga.geoprocessor(lib = "ta_channels", module = 5,

+ param = list(DEM = "dem6.sgrd",

+ DIRECTION = "channels.sgrd", CONNECTION = "route.sgrd",

+ NETWORK = "channels.shp"))

SAGA CMD 2.0.4

library path: C:/PROGRA~1/R/R-210~1.1/library/RSAGA/...

library name: ta_channels

module name : D8 Flow Analysis

author : (c) 2003 by O.Conrad

Load grid: dem6.sgrd...

ready

Parameters

Grid system: 100; 6x 6y; 100x 100y

DEM: dem6

Flow Direction: Flow Direction

Flow Connectivity: Flow Connectivity

Flow Network: Flow Network

Minimum Connectivity: 0

...


Read back to R

> dem$route <- readGDAL("route.sdat")$band1

route.sdat has GDAL driver SAGA

and has 6 rows and 6 columns

> channels <- readOGR("channels.shp", "channels")

OGR data source with driver: ESRI Shapefile

Source: "channels.shp", layer: "channels"

with 32 features and 2 fields

Feature type: wkbLineString with 2 dimensions


Plot the final result

> dem.plt <- spplot(dem[1], main = "DEM", col.regions = topo.colors(25))

> channels.plt <- spplot(dem[2], col.regions = rev(gray(0:20/20)),

+ main = "Flow connectivity", sp.layout = list("sp.lines",

+ channels, col = "red"))

> print(dem.plt, split = c(1, 1, 2, 1), more = T)

> print(channels.plt, split = c(2, 1, 2, 1), more = F)

DEM

10

20

30

40

50

Flow connectivity

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0


Outline





Exercise 1

I Create a working directory called NL, then download themakeRDC.R script.

I Open new session in R and run the script from Tinn-R.

I This will download a land cover map of NL and resample it tothe Dutch coordinate system.


http://spatial-analyst.net/scripts/makeRDC.R

FWTools

I Created and maintained by Frank Warmerdam (director andactive contributor to OSGeo).

I includes OpenEV, GDAL, MapServer, PROJ.4 and OGDI.

I “it is intended to give folks a chance to use the latest andgreatest”.

I FWTools contain a number of (highly efficient!!) utilitiesexcellent for processing large datasets.


http://home.gdal.org/warmerda/

http://www.gdal.org/gdal_utilities.html

Preparing FWTools

There is still no package to control FWTools from R, but we cansimply send command lines using the system command.Before we can use FWTools from R, we need to locate it on ourPC:

> gdalwarp <- gsub("/", "\\\\", dir(path="C:/PROGRA~2/FWTOOL~1.7",

+ pattern="gdalwarp.exe", recursive=TRUE, full.names=TRUE))

> gdalwarp

[1] "C:\\PROGRA~2\\FWTOOL~1.7\\bin\\gdalwarp.exe"

> workd <- paste(gsub("/", "\\\\", getwd()), "\\", sep="")


http://trac.osgeo.org/gdal/wiki/FWTools

http://fwtools.maptools.org

MODIS dataNow we can download some GIS data from web:

> MOD12Q1 <- "ftp://anonymous:[email protected]/

+ MOLT/MOD12Q1.004/2004.01.01/"

> download.file(paste(MOD12Q1,

+ "MOD12Q1.A2004001.h18v03.004.2006117173748.hdf", sep=""),

+ destfile=paste(getwd(),

+ "MOD12Q1.A2004001.h18v03.004.2006117173748.hdf", sep="/"),

+ mode='wb', method='wget')

Resolving e4ftl01u.ecs.nasa.gov... 152.61.4.83

Connecting to e4ftl01u.ecs.nasa.gov|152.61.4.83|:21... connected.

Logging in as anonymous ... Logged in!

==> SYST ... done. ==> PWD ... done.

==> TYPE I ... done. ==> CWD /MOLT/MOD12Q1.004/2004.01.01 ... done.

==> SIZE MOD12Q1.A2004001.h18v03.004.2006117173748.hdf ... 23165983

==> PASV ... done. ==> RETR MOD12Q1.A2004... done.

Length: 23165983 (22M)

0K .......... .......... 0% 64.9K 5m48s

...

22550K .......... .......... 99% 501K 0s

22600K .......... 100% 503K=65s


Reprojecting grids

We can reproject/resample the map to our local coordinate systemusing the gdalwarp functionality (this combines several processingsteps in one function):

> NL.prj <- "+proj=sterea +lat_0=52.15616055555555

+ +lon_0=5.38763888888889 +k=0.999908 +x_0=155000

+ +y_0=463000 +ellps=bessel +units=m +no_defs

+ +towgs84=565.237,50.0087,465.658,

+ -0.406857,0.350733,-1.87035,4.0812"

> system(paste(gdalwarp, " HDF4_EOS:EOS_GRID:\"", workd,

+ "\\MOD12Q1.A2004001.h18v03.004.2006117173748.hdf\"

+ :MOD12Q1:Land_Cover_Type_1 -t_srs \"", NL.prj, "\"

+ IGBP2004NL.tif -r near -te 0 300000 280000 625000

+ -tr 500 500", sep=""))

Creating output file that is 560P x 650L.

Processing input file HDF4_EOS:EOS_GRID:\\MOD12Q1.A2004001...

Using internal nodata values (eg. 255) for image HDF4_EOS:EOS_...

0...10...20...30...40...50...60...70...80...90...100 - done.


http://www.gdal.org/gdalwarp.html

Plot the final resultIn this case we have produced a MODIS-based land cover map forthe whole Netherlands in resolution of 500 m (in local coordinatesystem).


https://lpdaac.usgs.gov/lpdaac/products/modis_products_table/land_cover/yearly_l3_global_0_05deg_cmg/mod12c1

Exercise 2 & 3

I Create a working directory called worldmaps, then downloada selection of worldmaps from the repository (or get a copyfrom the USB stick) and unzip the maps that you need tocomplete the exercises.

I Use any GIS that you find suitable to answer the questions.

I I recommend first testing SAGA GIS, then running theanalysis in R. These maps are Large so it could take time untilyou import/open a map.


http://spatial-analyst.net/worldmaps/

Exercise 4

I We focus on extracting land surface parameters (orgeomorphometric parameters) using a DEM.

I First create a new working directory on your computer, thenDownload the map of world countries (countries) and theglobal Digital Elevation Model at 5.6 km resolution(globedem).

I Resample and subset the DEMs to local coordinate systems— for Germany use the European ETRS89 coordinate system(EPSG:3035), and for Bolivia use the South America AlbersEqual Area Conic coordinate system (ESRI:102033).

I Try also to derive these parameters using the RSAGApackage, i.e.by sending the commands to SAGA from R.


http://spatial-analyst.net/worldmaps/countries.zip

http://spatial-analyst.net/worldmaps/globedem.zip

http://spatialreference.org/ref/epsg/3035/

http://spatialreference.org/ref/esri/102033/

Outline





Writing spatial data to KML

I There are two possibilities to export maps to KML: (a) usingexisting packages, and (b) by writing KML files “by-hand”.

I To export point or line features to KML, use the writeOGR

method that is available in R package rgdal.

I More flexible way to writing KML files is by using loops.


http://code.google.com/apis/kml/documentation/kmlreference.html

http://spatial-analyst.net/wiki/index.php?title=Export_maps_to_GE

dsm training - preparing auxiliary data

Education

denitionsdigital soil

ondigital soil mapping

groundrecords soil proles

jrc ispra

auxiliary maps covariates

soildepth relationships

auxiliary environmental

importance of auxiliary