fundamentals of gis materials by austin troy & brian voigt © 2011 lecture 9: more input methods...

Fundamentals of GIS

Materials by Austin Troy & Brian Voigt © 2011

Lecture 9:More Input Methods and Data

Quality and Documentation

By Austin Troy & Brian VoigtUniversity of Vermont

Fundamentals of GIS

Part 1: Data input methods: Geocoding

------Using GIS--


Fundamentals of GIS

What is Geocoding?

• Convert a list / spreadsheet to spatial data / geographical features

• Needs a mechanism to calculate the geographic coordinate– Address matching: uses street address

database, created from a streets layer– XY coordinates


Fundamentals of GIS

Address Matching Geocoding

• Two required inputs: – 1) a table with the address records, and – 2) a geographic reference layer (e.g. streets,

E911)

• Output: a point file, where each point represents an address record


Fundamentals of GIS

How are addresses matched?• Common method: matching address to street ranges.• Urban areas: usually each street segment (arc)

corresponds to a block. • Each segment has attributes for the left from and to and

right from and to addresses. • Computer knows topological left and right for each street

segment• Step 1:Computer looks for segment with correct name

and address range • Step 2: Computer interpolates the position of the address

point on segment


Fundamentals of GIS

Geocoding Example: 1060 Main Street

1060 Main St

• Locate point on even (upper) side of street

• Position of 1060 is interpolated

Main St1000 1100

1001 1101

L-F-ADDR L-T-ADDR

R-F-ADDR R-T-ADDR

Look for Main street, then for the 1000-1100 block

direction


Fundamentals of GIS

• Create address locator in ArcCatalog

• Defines reference layer • Also where you specify

information about your reference layer that ArcGIS might not know, allowing for more efficient geocoding

• Many “styles” to choose from in for address locators

Address Matching in ArcGIS


Fundamentals of GIS

Geocoding Service• Geocoding styles are necessary because

– Reference layers come in many forms and formats. For instance, a reference layer may have the from right address attribute as fr_rt_add or add_rt_frm

– There are other types of geocoding, besides address geocoding, like geocoding points to the center of zip codes, and there are other types of address geocoding besides street address geocoding, like using a property parcel layer as reference.


Fundamentals of GIS

Geocoding in ArcGIS• In geocoding style interface: choose your

reference file and then specify which attributes in the reference layer correspond with the inputs that ArcGIS needs to do geocoding.

• It also asks for some information about what to expect in your geocoding table (what the required attribute headings are called) and how sensitive to be to things like spelling differences


Fundamentals of GIS

Materials by Austin Troy © 2008

Specify reference file

Specify address range attributes

Specify rules for address list

Specify zone

Fundamentals of GIS

Geocoding in ArcGIS• ArcMap >>> Tools >>>

Geocode Addresses – Select the geocoding

service you want to use• This brings up the

geocoding interface where we specify which field holds the address and which holds the zone

• Also specify an output shapefile or geodatabase and geocoding sensitivity


Fundamentals of GIS

Geocoding Results• How many records were

successfully matched • How many were

unmatchable • How many were

potentially matchable– Interactively match the

potential ones


Fundamentals of GIS

Geocoding and Error• Result is only as good as reference data

– If the streets layer is only accurate to 200 meters, the geocoded points are assumed to be within 200 meters

– If the streets data is consistently 100 meters to the north, then the geocoded points will be the same too

• Some roads layers may have better attributes than other too


Fundamentals of GIS

Geocoding and Error• Here’s an example where the same address list was

geocoded with two different street layers.• Note here how the same house is 100 m off between the

two geocoding attempts

100 m


Fundamentals of GIS


Geocoding and Error• Here we see that many points were coded for Napa1 that

were not coded for Napa2 possibly because Napa1’s street reference layer is newer, and has more streets

Fundamentals of GIS

Geocoding and Error• This error is due to an attribute error in

one of the layers which puts that address in the wrong street segment

300 m


Fundamentals of GIS

Geocoding and Error

Rural road segments: long road segments, less precise

Urban road segments: smaller, more precise

Rural street segments are also more subject to more error because street segments longer, so relies more on interpolation


Fundamentals of GIS

Geocoding in ActionMapping hazard zone properties in L.A. to see effects on property values


Fundamentals of GIS

XY GeocodingWe can also create points from a table by their latitude and longitudeDo this by clicking:

CA haz. waste sites

• Then we specify the X,Y coordinate fields and the spatial reference system


Fundamentals of GIS

Part 2: Data input methods: Digitizing

------Using GIS--


Fundamentals of GIS

Digitizing

• This is generally the process of converting data from analog to digital with a device, such as a digitizing tablet or mouse, to create new vector features

• User defines features by pointing and clicking.

• Table digitizing involves use of a digitizing tablet or table

• A digitizing table is a big table with an electronic mesh that can sense the position of a digitizing cursor

• Transmits X,Y coordinates of each mouse / cursor click to the computer and usually joins those with lines


Fundamentals of GIS

Digitizing• Notice how it is attached with tape

Source: http://ndis.nrel.colostate.edu/ndis/riparian/Tablet.jpg

• If it moves, the map will be inaccurate, because it’s recording position relative to the tablet, not the map

Fundamentals of GIS

Digitizing


• Snapping: Arc will also snap closed any unsnapped lines or polygons and will crop dangling lines, based on user-defined tolerances

• Snap tolerance: won’t snap together

Snap tolerance: will snap together

Dangling arc Snapped to other arc

Fundamentals of GIS

Digitizing

• Digitizing on a tablet requires defining “control points” which allow the conversion of the digitized map to real world coordinates.

• Usually, a corner point on the map of known geographic location is digitized first and its coordinates are assigned in some sort of header file

• “Heads up” digitizing involves scanning a paper map to a digital file, or otherwise obtaining a digital raster map/ image and digitizing “on top” of it on computer


Fundamentals of GIS


Part 3: Spatial Data Quality

Fundamentals of GIS

Data Quality

• Accuracy + Precision = Quality

• Error = f(accuracy, precision)

• Cost vs. quality tradeoff


Fundamentals of GIS


Accuracy• “the degree to which information on a map or in a digital

database matches true or accepted values.”• From Kenneth E. Foote and Donald J. Huebner

http://www.colorado.edu/geography/gcraft/notes/error/error_f.html

• Reflection of how close a measurement represent the actual quantity measured and of the number and severity of errors in a dataset or map.

Image source: http://oopslist.com/

Fundamentals of GIS

Precision• Intensity or level of preciseness, or exactitude in

measurements. The more precise a measurement is, the smaller the unit which you intend to measure

• Hence, a measurement down to a fraction of a cm is more precise than a measurement to a cm

• However, data with a high level of precision can still be inaccurate—this is due to errors

• Each application requires a different level of precision


Fundamentals of GIS

Random and Systematic Error• Error can be systematic or random

• Systematic error can be rectified if discovered, because its source is understood

Image source: http://oopslist.com/Materials by Austin Troy & Brian Voigt © 2011

Fundamentals of GIS

Random and Systematic Error• Systematic errors affect accuracy, but are usually

independent of precision; data can use highly precise methods but still be inaccurate due to systematic error

Accurate and precise: no systematic , little random error

inaccurate and precise: little random error but significant systematic error

Accurate and imprecise: no systematic , but considerable random error

inaccurate and imprecise: both types of error


Fundamentals of GIS

Measurement of Accuracy

• Positional accuracy is often stated as a confidence interval: e.g. 104.2 cm +/- .01 means true value lies between 104.21 and 104.19

• One of the key measurements of positional accuracy is root mean squared error (RMSE); equals squared difference between observed and expected value for observation i divided by total number of observations, summed across each observation i

• This is just a standardized measure of error—how close the predicted measure is to observed


Fundamentals of GIS


Positional Accuracy

• Positional accuracy standards specify that acceptable positional error varies with scale

• Data can have high level of precision but still be positionally inaccurate

• Positional error is inversely related to precision and to amount of processing

Fundamentals of GIS


Accuracy is tied to scale

Fundamentals of GIS

Positional Error Standards• Different agencies have different standards for

positional error

• Example: USGS horizontal positional requirements state that 90% of all points must be within 1/30th of an inch for maps at a scale of 1:20,000 or larger, and 1/50th of an inch for maps at scales smaller than 1:20,000


Fundamentals of GIS

Positional Error Standards• USGS Accuracy standards on the ground:

1:4,800 ± 13.33 feet

1:10,000 ± 27.78 feet

1:12,000 ± 33.33 feet

1:24,000 ± 40.00 feet

1:63,360 ± 105.60 feet

1:100,000 ± 166.67 feet

See image from U. Colorado showing accuracy standards visually

Hence, a point on a map represents the center of a spatial probability distribution of its possible locations

Thanks to Kenneth E. Foote and Donald J. Huebner, The Geographer's Craft Project, Department of Geography, The University of Colorado at Boulder for links






Fundamentals of GIS

Attribute Accuracy• Attribute accuracy and precision refer to quality of

non-spatial, attribute data

• Precision for numeric data means lots of digits

• Example: recording income down to cents, rather than just dollars

• Quantitative measurement errors: e.g. truncation

• A common error is to measure a phenomenon in only one phase of a temporal cycle: bird counts, river flows, average weather metrics, soil moisture


Fundamentals of GIS

Categorical Attributes

• Accuracy refers to amount of misclassification of categorical data

• The chance for misclassification grows as number of possible classes increases; accuracy is a function of precision, or number of classes

• If just classifying as “land and water”, that is not very precise, and not likely to result in an error


Fundamentals of GIS

Other Measures of Data Quality

• Logical consistency

• Completeness

• Data currency/timeliness

• Accessibility

• These apply to both attribute and positional data



Fundamentals of GIS

Examples of Currency and Timeliness


Fundamentals of GIS

Shelburne Road: 1937 Shelburne Road: 2003

Fundamentals of GIS

New North End and Colchester: 2003

New North End and Colchester: 2003

Fundamentals of GIS

Common sources of error• Numerical processing (math operations, data type,

rounding, etc)

• Geocoding (e.g. rural address matching and street interpolation)

• Topological errors from digitizing (overshoots, dangling nodes, slivers, etc)

• Automated classification steps, like unsupervised or supervised land cover classification in remote sensing, can result in processing errors


Fundamentals of GIS

Error Propagation and Cascading


• Propagation: one error leads to another

• Cascading: errors are allowed to propagate unchecked from one layer to the next and on to the final set of products or recommendations

• Cascading error can be managed to a certain extent by conducting “sensitivity analysis”


Fundamentals of GIS

Conflation

• When one layer is better in one way and another is better in another and you wish to get the best of both

• Way of reconciling best geometric and attribute features from two layers into a new one

• Very commonly used for case where one layer has better attribute accuracy or completeness and another has better geometric accuracy or resolution

• Also used where newer layer is produced for some theme but it has lower resolution than older one


Fundamentals of GIS

Conflation

• Attribute conflation: transferring attributes from an attribute rich layer to features in an attribute poor layer

• Feature conflation: improvement of features in one layer based on coordinates and shapes in another, often called rubber sheeting. User either transforms all features or specifies certain features to be kept fixed


Fundamentals of GIS

Metadata

• To avoid many of these errors, good documentation of source data is needed

• Metadata is data documentation, or “data about data”

• Ideally, the metadata describes the data according to federally recognized standards of accuracy

• Almost all state, local and federal agencies are required to provide metadata with geodata they make


http://mapping.usgs.gov/standards/

Fundamentals of GIS

Metadata

1. Information retrieval, cataloguing, querying and searching for data electronically.

2. Describing fitness for use and documenting the usability and quality of data.

3. Describing how to transfer, access or process data

4. Documenting all relevant characteristics of data needed to use it


Fundamentals of GIS

Metadata

• The federal geographic data committee (FGDC) is a federal entity that developed a “Content Standard for Digital Geospatial Metadata” in 1998, which is a model for all spatial data users to follow

• Purpose is: “to provide a common set of terminology and definitions for the documentation of digital geospatial data.”

• All federal agencies are required to use these standards


http://www.fgdc.gov/metadata/metadata.html

Fundamentals of GIS

Metadata


• Metadata usually include sections similar to these

Fundamentals of GIS

Metadata

• Critical components usually break down into:

• Dataset identification, overview

• Data quality

• Spatial reference information

• Data definition

• Administrative information

• Meta metadata


Fundamentals of GIS

Metadata

• Data identification, overview and administrative info:

• General info: name and brief ID of dataset and owner organization, geographic domain, general description/ summary of content, data model used to represent spatial features, intent of production, language used , reference to more detailed documents, if applicable

• Constraints on access and use

• This is usually where info on currency is found


Fundamentals of GIS

Metadata

• Data quality should address:

• Positional accuracy

• Attribute accuracy

• Logical consistency

• Completeness

• Lineage

• Processing steps


Fundamentals of GIS

Metadata

• Spatial reference should include:

• horizontal coordinate system (e.g. State Plane)

• Includes projection used, scale factors, longitude of central meridian, latitude of projection origin, distance units

• Geodetic model (e.g. NAD 83), ellipsoid, semi-major axis


Fundamentals of GIS

Metadata

• Data definition, also known as “Entity and Attribute Information,” should include:

• Entity types (e.g. polygon, raster)

• Information about each attribute, including label, definition, domain of values

• Sometimes will include a data dictionary, or description of attribute codes, while sometimes it will reference a documents with those codes if they are too long and complex


Fundamentals of GIS

Metadata

• Data distribution info usually includes:

• Name, address, phone, email of contact person and organization

• Liability information

• Ordering information, including online and ordering by other media; usually includes fees


Fundamentals of GIS

Metadata

• Metadata reference, or meta-metadata

• This is data about the metadata

• Contains information on

• When metadata updated

• Who made it

• What standard was used

• What constraints apply to the metadata


Fundamentals of GIS

Metadata in ArcGIS• ArcGIS allows you to display, import and export

metadata in and to a variety of Metadata formats:

•

• It defaults to FGDC ESRI which looks like:


Fundamentals of GIS

Metadata in ArcGIS• XML is the most flexible form because its tag

structure allows it to be used in programming; tags can be called as variables or can be created through form interfaces; allows for compatibility across platforms and programs


Fundamentals of GIS

Metadata in ArcGIS• In the past, complete metadata was only available as

text; you had to create most embedded metadata tags yourself. Today many state and nationwide datasets come with complete embedded metadata including full attribute codes

• E.g. NEDs, NLCD, all VCGI data


Fundamentals of GIS

Metadata in ArcGIS• Can edit, import, edit and export metadata in multiple

formats allowing helping with proper sharing of data.

• Can also make templates to save time in repeat documentation of big data sets

• See NPS metadata extension for cool utilities
