data quality metrics-considerations for achieving trusted...

©Petroliam Nasional Berhad (PETRONAS) 2014 1

© 2014 PETROLIAM NASIONAL BERHAD (PETRONAS)

All rights reserved. No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise) without the permission of the copyright owner.

Data Quality Metrics –Considerations for Achieving Trusted Data

Open

Philip Lesslar

Technical Assurance, Group Technical Data, PETRONAS

Digital Energy Conference

5th October 2016

Impiana Hotel, Kuala Lumpur


• Trustworthiness and the need for Trusted Data

• Data Quality Metrics (DQM)

• How far can we go with DQM? – The Expert Factor

• So how do we get to trusted data?

• The Quality Data Inventory

• Packaging quality data – The building blocks

• Data delivery and Data Access and Retrieval (DART)

• Quantified benefits and sustainability

• Summary

Contents

Open


Problem Statement

1. Data collection consumes about half the time in business activities.

– 52% in 10 workflows studied.

2. Everyone who acquires data spends time re-checking it.

– 58 man years/year for well header, check shot, deviation & basic logs (find & quality check).

3. Data come from various sources.

Trustworthiness and the Need for Trusted Data

Open


Our 5-step Approach Towards Trusted Data…

1 2 3 4 5

Databank

• Single source of truth for each data type

• Identified mandatory and optional attributes needed for business activities

• Standard and approved process for quality checking (QC) of the data

• Identified Data Custodians accountable for ensuring data capture, availability and longevity

• Business appointed Subject Matter Expert (SME) to advise and QC

• Subject Matter Focal (SMF)

• Audit trail & documentation

Open


It is about measuring data quality…

Data Quality Metrics (DQM) Simplified

E.g. well header must have total depth defined.Business rules

Encoded metrics in application

Data Quality Dashboard

*IQM screenshot (courtesy of Exprodat Inc.)

Features:- Automated rule checking- Consistent- Progressive- Targeted

1

2

3

Total records polled

Total errors found

Quality percent

Trends : Monthly, Weekly, Daily

(1-(1059/141485))*100 = 99.25

Usually written as sequel (SQL) queries or internal proprietary syntax.

User friendly traffic light dashboard allowing for intervention actions.

*IQM = Information Quality Metrics

Open


Business rules

This arrow represents all the quality checking that a wellheader record needs for it to be fit for business purposes.

70%-90%

Checks:• Mandatory attributes• Data quality dimensions – completeness,

duplication, conformity, consistency, integrity, accuracy

Checks:• Relevant control documents• Significance of missing values• Referential integrity• QC & assurance• Fit for purpose• Assignment of quality

indicator

Subject Matter Expert

Fit-for-purpose data, of known

quality

Given any data type (e.g. well header)…

How Far Can We Go with Data Quality Metrics?

Open

© 2016 Petroliam Nasional Berhad (PETRONAS) 7Open

OriginalFormat Data

Reference Data/Metadata

Master Data/Corporate

“Single Source of Truth”

Derived Data Data Collections

Raw SeismicRaw Logs

Units of measure- Linear measures- Pressure

Static (hard) data- Well header- Deviation- Checkshot- Temperature- Pressure

Processed data- Seismic deconvolution- Seismic filtering- Seismic processing- Edited logs- Spliced logs

Composite data- Completion log- Mud log- Paleontological

composites- TRAPIS

Abbreviations- TD, DFE, KB etc

Interpreted (soft) data- Geological markers- Seismic horizons

Interpreted data- Geological markers- Seismic horizons

Data hoards- Projects en masse- Personal stores- Team folders

Valid Lists Data archive- Projects en masse

Range indicators

Comments

Requires:- Official data repository

Requires:- Standards- Implementation

across all impacted tools and databases

Requires:- Clear processes, workflows

and checkpoints- Proper & official repository- Management and security

processes around repository and data access

Requires:- Standard workflows- Standard algorithms- Standard processes- Housekeeping procedures

Requires:- Standard display and

formatting templates- Procedures

Secondary DataPrimary Data

Data Classification – Digital Data

© 2016 Petroliam Nasional Berhad (PETRONAS) 8

DQMData Quality Metrics

QDIQuality Data Inventory+

So How Do We Get To Trusted Data?

Business rules

70%-90%

Subject Matter Expert

Fit-for-purpose data, of known

quality

Open


Checkshots

Well Integrity

3D Models

Daily Operations Report (Facilities)

Pressure Volume Temperature (PVT)

Deviation Data

Pipelines

Well Test

Projects

Data QC exercise

Data submissions

Area of strict & structured checking and management.

Well Schematics

The QDI Dashboard

The QDI Dashboard shows,for the important data typesrequired by the business, thelevel of quality checked, fit-for-purpose data that is in theinventory and available forimmediate use.

etc.

Collective of fit-for-purpose quality checked data, required for business activities, and that can be re-used in the company.

What is the Quality Data Inventory (QDI)?

Open


Filling up the tanks with Trusted Data

Checkshot

Well H

eader

Devia

tion

Well L

ogs (

8)

t1

Checkshot

Well H

eader

Devia

tion

Well L

ogs (

8)

Well I

nte

grity

t2

Pip

elines

Checkshot

Well H

eader

Well L

ogs (

8)

Well I

nte

grity

Pip

elines

tn (future end state)

Schem

atics

Wellte

st

All important data that is required for the business is

properly checked, with signoffs by appropriate

authority.

Over time, the data gets better and better, progress

can be measured and targets can be set.

Amount of quality data increases.

Devia

tion

To provide a structured, consistent and transparent way of showing and demonstrating progress in addressing quality requirements in our data.

What is the Purpose of the Quality Data Inventory?

Open


Quality Data Envelope

Packaging Quality Data - The Building Blocks

Open

Exploration workflow

Facilities workflow

Production Geology workflow

Analytics workflow

Data Type Building Blocks along the Exploration & Production (EP) Value Chain

PRIMARY DATA SECONDARY DATA


Data Delivery Methods

Open

Data in flat files,Loadable into selected

applications

Data pull via plug-ins directly into selected

applications

Near-line robotics

Access from bulk-loaded project workspaces

Official DatabanksThe User

EnvironmentData Delivery Mechanisms

EP D

ata

Types

The amount of Trusted Data will increase over time


Data Delivery – The DART project

Open


Quantified Benefits And Sustainability

Open

• Quantify problems:− 58 manyrs/yr

NPT.− 52% data

downtime.

• Identify causal factors:

− No framework.− Fragmented

data.− Unclear

inventory.− Unknown quality.− Access

difficulties.− Long delivery

time.

What problems are we trying to solve?

• Prioritize key issues.

• Develop framework and end-to-end working model across key data types.

• Deliver data as early as possible for clarity and transparency.

• Secure additional commitments.

Develop & implement the

solutions

• Quantification methodology.

• Ensure capture of initial states in quantified terms.

• Start calculating deltas as soon as solution in place.

Measure how much of the problem has

been solved

• Track continuous progress via register.

• Ensure broad awareness of progress for continued

momentum.

Implement continuous tracking

to move the dot


• Data Quality Metrics go a long way in improving data quality but not to trusted level.

– We have our Data Quality Metrics (DQM) programme and are running daily ~1000 queries against about 28 database instances.

– We also have Quality Data Inventory (QDI) which combines DQM with SME checks to bring data to trusted levels. This is tracked via an automated dashboard.

• A database filled with quality data is of no value until it is easily accessible.

– Our Data Access and ReTrieval (DART) project allows users to find and retrieve data quickly.

• We need to track progress and value of quality data.

– Level of quality data is automatically tracked in dashboard for hands-free transparency

– We have implemented a methodology to calculate and track quantified benefits.

• Sustainability is a function of good processes, transparency, quality and accessibility.

Summary

Open


Thank you

Open

data quality metrics-considerations for achieving trusted...

Documents