esri uc 2014 | technical workshop | arcgis data reviewer edwin waite & shankar chandrasekaran...

Esri UC 2014 | Technical Workshop |

ArcGIS Data Reviewer

Edwin Waite & Shankar Chandrasekaran

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

A few examples of why data validation matters

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Leaning Tower of Pisa

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

Mars Climate Orbiter (1999)

Esri UC 2014 | Technical Workshop |

Garbage in Garbage Out (GIGO)

Type Presentation Title Here

Buildings

Roads

Water Bodies

Esri UC 2014 | Technical Workshop |

Garbage in Quality Out

Type Presentation Title Here

Buildings

Roads

Water Bodies

Esri UC 2014 | Technical Workshop |

What is ArcGIS Data Reviewer for Server?

• System to support data quality management for ArcGIS

• Automates quality control process

• Monitor and Measure data quality

ReviewValidate Summarize

Esri UC 2014 | Technical Workshop |

Enterprise Quality Management System

Type Presentation Title Here

GIS Data Server

Data Reviewer

Esri UC 2014 | Technical Workshop |

Benefits of Data Reviewer for Server

• Provides accurate data for your applications

• Measure and report data quality

• Eliminate costs associated with poor data quality

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Monitoring your data with validation

Planning and Deploying Data Quality Services

Validate

Esri UC 2014 | Technical Workshop |

Automated ValidationDifferent Type of Issues in GIS Data

Type Presentation Title Here

Esri UC 2014 | Technical Workshop |

Automated Validation

• Checks validate data based on conditions you define.

• Results are written to the Reviewer workspace.

Esri UC 2014 | Technical Workshop |

Automated Validation

• Over 40 rules out of the box

• Ability to extend validation with custom rules

• Rules are authored using ArcGIS Data Reviewer for Desktop

Attribute/Table Feature integrity Spatial Relationships

Item Metadata

Esri UC 2014 | Technical Workshop |

• Group checks into batch jobs

Automated validation

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

ArcGIS Server

Automated Validation Architecture

Type Presentation Title Here

Data Reviewer

Web API

REST

API

Reviewer Service

Scheduler

Validation Service

Data Workspace

Reviewer Workspace

Execution Details

Esri UC 2014 | Technical Workshop |

Automated Review Demo

Planning and Deploying Data Quality Services

Contributor

Organization

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Questions ?

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Analyzing Validation Results

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

• Dashboard- Provides transparency to stakeholders on data quality

- Ability for stakeholders to make informed and accurate decisions

- Interactively summarize results based on different perspectives

- Summaries are derived from results

Summarize data quality

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

Summarize data qualityDashboard - Perspectives

Planning and Deploying Data Quality Services

What are the issues on a geographic extent?

How severe are the issues with my data?

How many types of issues do I have in my data?

How many issues have been reviewed and fixed?

How many issues do I have by feature class?

Esri UC 2014 | Technical Workshop |

Dashboard

Planning and Deploying Data Quality Services

Data Reviewer

Web API

ArcGIS Server

Reviewer Workspace

Dashboard

Reviewer Service

REST

API

Esri UC 2014 | Technical Workshop |

• Results are records that are stored in Reviewer workspace

• Contains information about a feature or a row which failed QC standard

• Query Results- Apply filters

- Sort/Page

Results

Planning and Deploying Data Quality Services

ResultsAutomated Validation

Esri UC 2014 | Technical Workshop |

Results

Planning and Deploying Data Quality Services

• Spatial and non-spatial

• Attributes describe check, problem, and validation information

• Object ID provides link back to feature that failed validation

Esri UC 2014 | Technical Workshop |

ArcGIS Server

Results

Planning and Deploying Data Quality Services

Data Reviewer

Web API

REST

API

Reviewer Service

Results

Reviewer Workspace

Esri UC 2014 | Technical Workshop |

Analyze Results demo

Planning and Deploying Data Quality Services

Contributor

Data Stewards

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Questions ?

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Visual Quality Control

Esri UC 2014 | Technical Workshop |

Visual Review Process

Planning and Deploying Data Quality Services

Reviewer Service

Write Results

Editor

ArcGIS Server

Reviewer

Geodatabase

Esri UC 2014 | Technical Workshop |

• Get feedback from Subject Matter Experts (SME)

• Identify missing or incomplete data

• Track issues reported

Benefits of Visual Review

Type Presentation Title Here

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Demo

Visual Quality Control

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Questions ?

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Deploying Data Quality Services

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

Deploying Data Reviewer ServicesSingle machine deployment

Planning and Deploying Data Quality Services

Server Data WorkspaceReviewer Workspace

Desktop Web Device

Esri UC 2014 | Technical Workshop |

Deploying Data Reviewer ServicesSingle machine Deployment

Planning and Deploying Data Quality Services

ArcGIS Server Single Machine

Inspect and Analyze Map Service

Validate GP Service

Inspect and Analyze Map Service

Inspect and Analyze Map Service

Validate GP Service

Validate GP Service

Esri UC 2014 | Technical Workshop |

Deploying Data Reviewer ServicesMulti-machine deployment

Planning and Deploying Data Quality Services

Data WorkspaceReviewer Workspace

Desktop Web Device

Servers

Esri UC 2014 | Technical Workshop |

Deploying Data Reviewer ServicesMulti-machine deployment

Planning and Deploying Data Quality Services

ArcGIS Servers

Inspect and Analyze Map Service

Validate GP Service

Inspect and Analyze Map Service

Validate GP Service

Inspect and Analyze Map Service

Validate GP Service

Esri UC 2014 | Technical Workshop |

Deploying Data Reviewer ServicesAmazon Elastic Cloud Deployment

Planning and Deploying Data Quality Services

Desktop Web Device

Data Workspace

Reviewer Workspace

Esri UC 2014 | Technical Workshop |

Deploying Data Reviewer ServicesAmazon Elastic Cloud Deployment

Planning and Deploying Data Quality Services

Inspect and Analyze Map Service

Validate GP Service

Inspect and Analyze Map Service

Validate GP Service

AGS - Amazon EC

AGS - Amazon EC

Esri UC 2014 | Technical Workshop |

Data Reviewer ServerOut of the Box

• Server components- Execute Batch Job GP Service

- Reviewer Service

- REST API

• Web components - Client API Reference (JavaScript & Flex)

- Starter Applications (Batch Validation Manager)

- Widgets (Flex & JavaScript)

Planning and Deploying Data Quality Services

Execute BatchJob GP Service

Reviewer Service

Esri UC 2014 | Technical Workshop |

Data Reviewer Platform

Planning and Deploying Data Quality Services

• ArcGIS Data Reviewer for Desktop- Define and author Quality Control Standards- Configure Reviewer Workspace to store results

• ArcGIS Data Reviewer for Server- Schedule and run automated validations- Analyze result stored in Reviewer workspace - Submit SME feedback for Reviewer workspace for visual review

• Client API’s - Develop applications to access data quality services

Esri UC 2014 | Technical Workshop |Esri UC 2014 | Technical Workshop |

Widgets and Starter Apps - Demo

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

ArcGIS Data Reviewer ServerSystem Architecture

Planning and Deploying Data Quality Services

Data WorkspaceBatch Validation

Dashboard

Visual Review

HTTP/REST

Reviewer Workspace

Results

Validation Service

Esri UC 2014 | Technical Workshop |

• Product page- www.esri.com/datareviewer

• Resource Center- http://resources.arcgis.com/en/communities/data-reviewer/

• Training- www.esri.com/training

• Questions & comments- datareviewer@esri.com

Resources

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

Resource Center – Data Reviewer

• Gallery - Apps

- Templates

- Tutorials

• API Documentation

• Live Samples

Day and Description Type Time Location

Tuesday July 15

ArcGIS Data Reviewer: An IntroductionTechnical Workshop

8:30 – 9:45am Room 14A

ArcGIS Data Reviewer: Extending the Validation Framework Demo Theater 9:30 – 10:00amGeodata

Mgmt Theater

ArcGIS Data Reviewer: Leveraging Geoprocessing for Data Validation Demo Theater 10:30 – 11:00amGeodata

Mgmt Theater

ArcGIS Data Reviewer: Assessing Positional Accuracy Demo Theater 1:00 – 1:30pmGeodata

Mgmt Theater

ArcGIS Data Reviewer: Plans for Quality Assessment of Raster Datasets

Demo Theater 5:30 – 6:00pmGeodata

Mgmt Theater

Wednesday July 16

ArcGIS Data Reviewer: An IntroductionTechnical Workshop

8:30 – 9:45am Room 14A

ArcGIS Data Reviewer: Integrating ArcGIS Data Reviewer and ArcGIS Workflow Manager to Automate Data Quality Control Workflows

Demo Theater 11:30 – 12:00amGeodata

Mgmt Theater

ArcGIS Data Reviewer: Special Interest Group Meeting SIG 12:00 – 1:00pm Room 15B

ArcGIS Data Reviewer: Planning and Deploying Data Quality ServicesTechnical Workshop

3:15 – 4:30pm Room 15B

Thursday July 17

ArcGIS Data Reviewer: Integrating Data Validation Capabilities Into Web Applications

Demo Theater 9:30 – 10:00amWeb & Server

Theater

ArcGIS Data Reviewer: Implementing Data Quality Reporting in Web Clients

Demo Theater 10:30 – 11:00amWeb & Server

Theater

Questions ?

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop |

Thank you…

• Please fill out the session survey:

Offering ID: 1313

Online – www.esri.com/ucsessionsurveys

Paper – pick up and put in drop box

Planning and Deploying Data Quality Services

Esri UC 2014 | Technical Workshop | Planning and Deploying Data Quality Services