ArcGIS Data Reviewer: An IntroductionChandan Banerjee

Jay Cary

Workshop Agenda

• Importance of Data Quality• What is ArcGIS Data Reviewer?• Automated review• Semi-automated review• Managing errors and reporting data quality• Summary/Resources

ArcGIS Data Reviewer: An Introduction

Importance of data quality

ArcGIS Data Reviewer: An Introduction

Defining qualityA business perspective

• Executive- Confidently make decisions- Reduce financial risk- Optimize organizational performance

ArcGIS Data Reviewer: An Introduction

Defining qualityA business perspective

• Executive- Confidently make decisions- Reduce financial risk- Optimize organizational performance

• Manager- Effective data stewardship- Drive increased usage- Maximize productivity

ArcGIS Data Reviewer: An Introduction

Defining qualityA business perspective

• Executive- Confidently make decisions- Reduce financial risk- Optimize organizational performance

• Manager- Effective data stewardship- Drive increased usage- Maximize productivity

• Knowledge worker- Increased efficiencies- Confidence in GIS

ArcGIS Data Reviewer: An Introduction

Defining QualityA Technical Perspective

Spatial Accuracy

Thematic Accuracy

Completeness

Logical Consistency

Temporal Quality

Usability

ISO-19157:2013 Geographic information – Data quality

ArcGIS Data Reviewer: An Introduction

Defining QualitySources of data quality requirements

Industry standards /

Specifications

ArcGIS Data Reviewer: An Introduction

Defining QualitySources of data quality requirements

Industry standards /

Specifications

Subject matter experts

ArcGIS Data Reviewer: An Introduction

Defining QualitySources of data quality requirements

Industry standards /

Specifications

Subject matter experts

Training and experience

ArcGIS Data Reviewer: An Introduction

Defining QualitySources of data quality requirements

Industry standards /

Specifications

Subject matter experts

Training and experience

Quality assurance

plans

ArcGIS Data Reviewer: An Introduction

Data quality managementCapabilities of the ArcGIS Platform

Geodatabase integrity

• Schema constraints• Geoprocessing tools• Data load checks• Versioning

Advanced data types

• Topologies• Geometric networks

ArcGISData Reviewer

• Automated review

• Semi-automated review

• Error management

• Quality reporting

ArcGIS Data Reviewer: An Introduction

What is ArcGIS Data Reviewer?Data Quality Management for ArcGIS

• Provides- Rule-based validation- Interactive tools- Track errors

• For individuals and enterprise- Saves time/money- Less rework

• For multiple domains- Configurable- Extendable

Validate

ReviewSummarize

ArcGIS Data Reviewer: An Introduction

What is ArcGIS Data Reviewer?Data quality management in the ArcGIS platform

• Data Reviewer for Desktop- ArcMap- ArcGIS Pro

• Data Reviewer for Server- Standard or higher

• Data Reviewer API- JavaScript- FLEX

• Web AppBuilder for ArcGIS

Desktop Web Device

Server

Online Content and Services

ArcGIS OnlinePortal

ArcGIS Data Reviewer: An Introduction

Managing Quality ControlQuality Control Processes

ReviewerResults

AutomatedReview

QualityReporting

Semi-automatedReview

ArcGIS Data Reviewer: An Introduction

Automated Review

ArcGIS Data Reviewer: An Introduction

Managing Quality ControlQuality Control Processes

ReviewerResults

AutomatedReview

QualityReporting

Semi-automatedReview

ArcGIS Data Reviewer: An Introduction

Types of quality control

Fast

Consistent and repeatable

Objective

100% coverage

Automatedreview

ArcGIS Data Reviewer: An Introduction

• Implementing quality requirements- 40+ configurable checks

- Feature integrity- Collection rules

- Attribute- Feature and table values

- Spatial- Spatial relationships

Automating Data Validation

http://j.mp/DRCheckPoster104

ArcGIS Data Reviewer: An Introduction

Getting Startedwith Automated Review

ArcGIS Data Reviewer: An Introduction

Demo Scenario

• Department plans to use temporary employees with field data collection devices to map road-related features (signs, street furniture).

• Success CriteriaFeatures should be positionally accurate based on siting

criteriaKey attributes are populated and have the correct valuesAll features should be collected within the survey areaStreamline data collection processesAll errors are corrected and verified

ArcGIS Data Reviewer: An Introduction

AuthoringData Quality Rules

ArcGIS Data Reviewer: An Introduction

Batch ValidationImplementing Cumulative Review

• Reviewer Batch Jobs- Define and encapsulate quality requirements- Captures domain knowledge- Designed once and executed many times

Industry standards /

Specifications

Subject matter experts

Training and experience

Quality assurance

plans

ArcGIS Data Reviewer: An Introduction

Where to startLeveraging templates

• Data Reviewer templates - Electrical utilities- Gas utilities- Local government- Water resources- Water utilities

• Based on Esri industry models• Use as starting point

solutions.arcgis.com

ArcGIS Data Reviewer: An Introduction

Authoring Data Quality Rules

ArcGIS Data Reviewer: An Introduction

Authoring Batch JobsTips and Tricks

Create sample data for testing

Configure and execute from Reviewer Toolbar

Leverage Mini-Browser to assess results

Validate a pilot area with known issues

ArcGIS Data Reviewer: An Introduction

ExecutingAutomated Validation

ArcGIS Data Reviewer: An Introduction

Execute data validation using• ArcMap

Methods for executing data validation

ArcGIS Data Reviewer: An Introduction

Execute data validation using• ArcMap• ArcGIS Pro

Methods for executing data validation

ArcGIS Data Reviewer: An Introduction

Execute data validation using• ArcMap• ArcGIS Pro• Model/Python script

Methods for executing data validation

ArcGIS Data Reviewer: An Introduction

Execute data validation using• ArcMap• ArcGIS Pro• Model/Python script• ArcGIS Workflow Manager

Methods for executing data validation

ArcGIS Data Reviewer: An Introduction

Execute data validation using• ArcMap• ArcGIS Pro• Model/Python script• ArcGIS Workflow Manager• ArcGIS for Server

Methods for executing data validation

http://bit.ly/DR_BVM

ArcGIS Data Reviewer: An Introduction

Executing Automated Validation

ArcGIS Data Reviewer: An Introduction

Demo Scenario

• Department plans to use temporary employees with field data collection devices to map road-related features (signs, street furniture).

• Success CriteriaFeatures should be positionally accurate based on siting

criteriaKey attributes are populated and have the correct valuesAll features should be collected within the survey areaStreamline data collection processesAll errors are corrected and verified

ArcGIS Data Reviewer: An Introduction

Batch validationEvents when validation may be required

Before data loading

During editing

Prior to sharing/publishing maps

Prior to rolling-out new applications

ArcGIS Data Reviewer: An Introduction

Semi-Automated Review

ArcGIS Data Reviewer: An Introduction

Managing Quality ControlQuality Control Processes

ReviewerResults

AutomatedReview

QualityReporting

Semi-automatedReview

ArcGIS Data Reviewer: An Introduction

Types of quality control

Fast

Consistent and repeatable

Objective

100% coverage

Automatedreview

Semi-automatedreview

Guided Workflows

Streamlined processes

Subjective

Sampling

ArcGIS Data Reviewer: An Introduction

Semi-automated review methods

Visual review

• Redlining

• Systematic review

• Sampling

Data inspection

• Feature counts

• Attribute review

• Version differences

Positional accuracy

• Assessment tool

ArcGIS Data Reviewer: An Introduction

Value of Performing Visual Review

Discover Patterns

Compare to trusted sources

Find missing features

ArcGIS Data Reviewer: An Introduction

Semi-automated reviewLeveraging ArcGIS for Desktop

Tools supporting- Selecting/browsing features- Redlining missing features- Flagging existing features in error- Generating random samples- Assessing positional accuracy- Comparing geodatabase versions

ArcGIS Data Reviewer: An Introduction

Semi-automated reviewLeveraging ArcGIS for Server

• Extending quality control workflowsinto other communities

- QC review across ArcGIS platform

- Simple to use tools for error identification

- Manual QC workflow “automation”

ArcGIS Data Reviewer: An Introduction

Demo:Visual Data Review

ArcGIS Data Reviewer: An Introduction

Demo Scenario

• Department plans to use temporary employees with field data collection devices to map road-related features (signs, street furniture).

• GoalsFeatures should be positionally accurate based on siting

criteriaKey attributes are populated and have the correct valuesAll features should be collected within the survey areaStreamline data collection processesAll errors are corrected and verified

ArcGIS Data Reviewer: An Introduction

Reporting Quality

ArcGIS Data Reviewer: An Introduction

Managing Quality ControlQuality Control Processes

ReviewerResults

AutomatedReview

QualityReporting

Semi-automatedReview

ArcGIS Data Reviewer: An Introduction

Data Quality ReportingArcGIS for Desktop

• Adhoc summary tools- Result statistics

• Automated reporting- Automated Check Results- Total Record Count- Sampling

ArcGIS Data Reviewer: An Introduction

Data Quality ReportingArcGIS for Server

Enabling transparency in data quality- Better decision making by communicating

data quality across stakeholders- Open quality reporting- Shared across ArcGIS system- Tools and methods to communicate quality

ArcGIS Data Reviewer: An Introduction

Demo:Reporting Quality

ArcGIS Data Reviewer: An Introduction

Demo Scenario

• Department plans to use temporary employees with field data collection devices to map road-related features (signs, street furniture).

• GoalsFeatures should be positionally accurate based on siting

criteriaKey attributes are populated and have the correct valuesAll features should be collected within the survey areaStreamline data collection processesAll errors are corrected and verified

ArcGIS Data Reviewer: An Introduction

Managing Quality Results

ArcGIS Data Reviewer: An Introduction

Managing Quality ControlQC lifecycle management

REVIEW

Find &Record Errors

CORRECT

Perform Editsor

Note Exceptions

VERIFY

Acceptableor

Unacceptable

ArcGIS Data Reviewer: An Introduction

Managing Quality ResultsDemo:

ArcGIS Data Reviewer: An Introduction

Demo Scenario

• Department plans to use temporary employees with field data collection devices to map road-related features (signs, street furniture).

• GoalsFeatures should be positionally accurate based on siting

criteriaKey attributes are populated and have the correct valuesAll features should be collected within the survey areaStreamline data collection processesAll errors are corrected and verified

ArcGIS Data Reviewer: An Introduction

Workshop Review

• Importance of data quality

• Forms of data quality control- Automated review- Semi-automated review

• ArcGIS Data Reviewer- Automated validation- Semi-automated tools- Error lifecycle management- Reporting

ArcGIS Data Reviewer: An Introduction

Day and Description Type Time Location

Tuesday June 28

ArcGIS Data Reviewer: An Introduction Technical Workshop 8:30 – 9:45am Room 15B

ArcGIS Data Reviewer: Assessing Positional Accuracy Demo Theater 10:30 – 11:15am Demo Theater 4Geodata

ArcGIS Data Reviewer: Integrating ArcGIS Data Reviewer and ArcGIS Workflow Manager to Automate Data Quality Control Workflows

Demo Theater 11:30 – 12:15pm Demo Theater 4Geodata

ArcGIS Data Reviewer: Leveraging Geoprocessing for Data Validation Demo Theater 12:30 – 1:15pm Demo Theater 4

Geodata

ArcGIS Data Reviewer: Quality Assessment for Elevation Raster Datasets Demo Theater 2:30 – 3:15pm Demo Theater 4

Geodata

Wednesday June 29

ArcGIS Data Reviewer: An Introduction Technical Workshop 3:15 – 4:30pm Sapphire Ballroom

A/B

Thursday June 30

ArcGIS Data Reviewer: Advanced Data Validation Concepts Technical Workshop 8:30 – 9:45am Room 32 A/B

ArcGIS Data Reviewer: Implementing Data Quality Reporting in Web Clients Demo Theater 10:30 – 11:15am Demo Theater 6

Server

ArcGIS Data Reviewer: Integrating Data Validation Capabilities into Web Applications Demo Theater 12:30 – 1:15pm Demo Theater 6

Server

Want to learn more this week?

ArcGIS Data Reviewer: An Introduction

Want to learn more?

• Product informationwww.esri.com/datareviewer

ArcGIS Data Reviewer: An Introduction

Want to learn more?

• Documentation- Desktop (desktop.arcgis.com)

ArcGIS Data Reviewer: An Introduction

Want to learn more?

• Documentation- Desktop- Server (server.arcgis.com)

ArcGIS Data Reviewer: An Introduction

Want to learn more?

• Documentation- Desktop- Server

• Training (training.esri.com)- Assessing Data Quality using ArcGIS Data Reviewer (Seminar)- Evaluating Positional Accuracy Using ArcGIS Data Reviewer for Desktop (Seminar)- Data QC with ArcGIS: Automating Validation (Web Course)- Data QC with ArcGIS: Visual Review (Web Course)- Quality Control Using ArcGIS Data Reviewer for Desktop (Instructor-Led)

ArcGIS Data Reviewer: An Introduction

Want to learn more?

• Documentation- Desktop- Server

• Training- Assessing Data Quality using ArcGIS Data Reviewer- Evaluating Positional Accuracy Using ArcGIS Data Reviewer for Desktop- Data QC with ArcGIS: Automating Validation- Data QC with ArcGIS: Visual Review- Quality Control Using ArcGIS Data Reviewer for Desktop

• GeoNet (geonet.esri.com)- Data Reviewer place

ArcGIS Data Reviewer: An Introduction

ArcGIS Data Reviewer: An Introduction