cleardq intro pack v0.5
TRANSCRIPT
Data Quality for Financial Performance
Got Bad Data ? How much is bad data really costing your business ?
Why do DQ teams perpetually struggle to solve this problem ?
ClearDQ
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
Data Issues Aren’t going away.The commercial costs are real, and significant.Global technology industry paradigm problem.20
% The damage bill starts est. 20% of FTE.
ClearDQ
Poor Data Quality The Unmanaged business overhead
Costs organizations in excess of 11-13% FTE (some as high as 24% !)
Exacerbated by the Business vs IT “data disconnect” fuelled by IT vendors selling DQ technology tools…
Most Data Warehouses and Business Intelligence efforts fail to address Data Quality and in fact, in many proven cases make Data Quality worse..
Poor DQ wreaks havoc in Regulatory Reporting and Regulatory Markets ( Banking, Energy, Health, etc… ), fines, customer churn, etc…
Vicious Cycle - Poor DQ leads to data duplication, which in turn exacerbates data quality problems.
11
IT
DW
$$
2x
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
Telco Industry Systemic DQ Issues
Regulator TIO refers major systems issues to to regulators - the Australian Communications and Media Authority and the Australian Competition and Consumer Commission (*) Privacy and protection of customers' personal information (Data Classification,
Data Privacy, Data Access, Data Security) Telemarketing practices Misleading sales information Incorrect billing (Billing Data, Data Movement, Data Aggregation, Manual
Adjustments) unfair contract terms, and compliance with consumer guarantees under Australian Consumer Law.Other data quality problematic areas leading to business function problems are: Poor Access to Data, Poor Reporting Solutions, Multiple sources of duplicate data
(merge, filter damage), Poor Data Meaning (Ambiguity), Duplicated Data Costs (Duplicated platforms, teams, processes), Manual Data Efforts, Large Datasets (difficult to manage), Data Retention and Recovery (Data loss), etc..
(*) source: Telecommunication Industry Ombudsman (TIO) - Annual Report 2015)
ClearDQ
ClearDQ Data Quality for Business
Data Quality issues expressed in commercial impact terms - $ and time
A system that places Data Quality under the control of business leaders
Helps IT by instructing IT when, why and what to do, as dictated by the priorities set by the business DQ sponsor.
Hosted app, gets your business team solving their most urgent DQ issues immediately
Strong business relevance for Energy, Banking, Finance, Health, Airline, FMCG and other industry verticals
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
Foundational Data Capabilities
Technology(Design, Build,Operate)
Data EnabledBusinessFunctions & Capabilities(example)
Business Strategy(example)
ProductInnovation
Better Supplier
Management
Simpler Customer
Offer
Ops & Supply Chain
Transformation
Transform Sales
OnlineInnovation
Your Data Mission: Capture, store, secure, move, supply, share and analyse Your Business Data with ease, at speeds, and in consumable formats, that meet our operational and
strategic needs to support fact-based decision making and optimal business execution, within an understood data cost/risk profile.
SocialPlatfor
mDigital Shops Finance
HR
Assets Merchandise Sales Supply
Chain
Services Capabilities
Processes Delivery People
Agile Data Delivery,
Innovation & Data Value Realisation
Corporate Memory - Reusable Foundations, Simplified Technology, Training Uplift, Speed Delivery, Maturity Uplift, Funding Optimisation, Data Innovation
Accountability
Meaning
Single Source
Assurance
Data Quality
Data Moveme
ntAccess
Platforms & Tools
Data Value Maturity & Innovation
Ownership,Governance, Stewardship,
Produce Data,
Consume Data
Business vs IT
Glossary,
MetadataData ModelsRules
Master,Referen
ce & Txn
Data, PK’s FK’s
ProfilingMonitorin
gCleansingEnrichme
ntTriage
Migration, Integratio
n, Extraction
, ETL, ESB, DV, Process Mapping
Apps, Reporting, Analytics, DW / BI
Big Data, Cloud,
Platforms, DB
Engines, Formats
Reporting
Business Case,
Funding, Frameworks, Training,Innovation
MarketingLoyalty
ProjectDelivery
Solution Arch
Artefacts,Proactive Solution
Governance
Funding
Network PlanningProperty Workforc
eCorporat
eSystems
FinancialProducts
Security
IS&O
Customer Value Investment
Effective Teams
Improve Product Margins
Better Partner Network
Loyalty Expansion
StoreUplift
Risk, Controls,
Maint.
Classify,Security, Regulator
y,Archiving, Retention, Recovery
Search and Docs
Documents
ContractsPDF,
WORDEXCELSearch,
Taxonomy
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
Any function can be seriously impacted by bad day. However, Finance, Marketing and Sales functions are (typically) more negatively impacted by bad
data that other functions.
Globally, Data Capability Matrurity is generally LOW or NON-EXISTENT, or UNFUNDED.
ClearDQ
Data Economics & MonetizationLinking Data Capabilities to Data Risks, Costs and Data Value Unlock
Corporate Memory - Reusable Foundations, Simplified Technology, Training Uplift, Speed Delivery, Maturity Uplift, Funding Optimisation, Data Innovation
Accountability
Meaning
Single Source
Assurance
Data Quality
Data Moveme
ntAccess
Platforms & Tools
Data Value Maturity & Innovation
Ownership,Governance, Stewardship,
Produce Data,
Consume Data
Business vs IT
Glossary,
MetadataData ModelsRules
Master,Referen
ce & Txn
Data, PK’s FK’s
ProfilingMonitorin
gCleansingEnrichme
ntTriage
Migration, Integratio
n, Extraction
, ETL, ESB, DV, Process Mapping
Apps, Reporting, Analytics, DW / BI
Big Data, Cloud,
Platforms, DB
Engines, Formats
Reporting
Business Case,
Funding, Frameworks, Training,Innovation
ProjectDelivery
Solution Arch
Artefacts,Proactive Solution
Governance
Risk, Controls,
Maint.
Classify,Security, Regulator
y,Archiving, Retention, Recovery
Search and Docs
Search, Taxonomy
Waste (bad data)
Buy Data Sets
Revenue Leakage
Staff Attrition(due t bad data)-$
+$Sell Data Sets
(Customer Analytics) Cross Sell
(Business Analytics) Automate Data Activity
(Marketing Analytics) Campaign Optimisation
Etc.
Etc.
Business is usually unaware of key Data Capabilities for value to be unlocked
Data Damage exists due to poor (or missing) Data Capabilities
Data Quality is a function of the existence and performance of key Data Capabilities
Data Governance is a fruitless activity if it is not linked to the inspection, evaluation and
performance of Data CapabilitiesData Initiatives built on poor foundations, bad data
and poor governance COST MORE to maintain.
Data must be trusted, accessible and fit-for-purpose, to deliver value.
Data is an “Engineered Product” that acts precisely as it is instructed.
The state and rigor of a company’s Data Capabilities is an accurate reflection of it’s
maturity and values.
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
Start Here, Then Build.
IT DQ Teams struggle and repeatedly fail here (global trend for 15+ years). Not
holistic.
ClearDQ
Data Quality DamageThe base-cost of poor quality data in non-IT (*)
(*) 2012 and 2013, 2015 Research Results – Detailed Customer Assessment Metrics.
GlobalBank
23%FTE
EnergyRetail
13%FTE
HealthRetail
24%FTE
Yours ??
Shadow IT Re Reconcile
Manual Correction
Re Extract De Duplicate Report Preparation
$24m
$3.4m
$60m
$39m
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
Annual DQ Damage Estimator (Telco Examples) Running ClearDQ discovers the accurate detail and $ precision in 2-8 weeks depending on org size.
FTE20% FTE
WasteAvg Salary
$40,000 $45,000 $50,000 $55,000 $60,000 $65,000 $70,000 $75,000 $80,000Due to Bad
DataDamage in
$m $m $m $m $m $m $m $m $m200 40 $2 $2 $2 $2 $2 $3 $3 $3 $3 300 60 $2 $3 $3 $3 $4 $4 $4 $5 $5 500 100 $4 $5 $5 $6 $6 $7 $7 $8 $8 800 160 $6 $7 $8 $9 $10 $10 $11 $12 $13
1000 200 $8 $9 $10 $11 $12 $13 $14 $15 $16 2000 400 $16 $18 $20 $22 $24 $26 $28 $30 $32 5000 1000 $40 $45 $50 $55 $60 $65 $70 $75 $80
10000 2000 $80 $90 $100 $110 $120 $130 $140 $150 $160 15000 3000 $120 $135 $150 $165 $180 $195 $210 $225 $240 20000 4000 $160 $180 $200 $220 $240 $260 $280 $300 $320 25000 5000 $200 $225 $250 $275 $300 $325 $350 $375 $400 30000 6000 $240 $270 $300 $330 $360 $390 $420 $450 $480 35000 7000 $280 $315 $350 $385 $420 $455 $490 $525 $560 40000 8000 $320 $360 $400 $440 $480 $520 $560 $600 $640 50000 10000 $400 $450 $500 $550 $600 $650 $700 $750 $800 55000 11000 $440 $495 $550 $605 $660 $715 $770 $825 $880 60000 12000 $480 $540 $600 $660 $720 $780 $840 $900 $960 80000 16000 $640 $720 $800 $880 $960 $1,040 $1,120 $1,200 $1,280
100000 20000 $800 $900 $1,000 $1,100 $1,200 $1,300 $1,400 $1,500 $1,600 200000 40000 $1,600 $1,800 $2,000 $2,200 $2,400 $2,600 $2,800 $3,000 $3,200 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
Telstra
Optus
AT&T
Verizon
ClearDQ
Annual DQ Damage Estimator (Banking Examples) Running ClearDQ actually discovers the accurate detail and $ precision in 2-8 weeks depending on org size.
FTE20% FTE
WasteAvg Salary
$40,000 $45,000 $50,000 $55,000 $60,000 $65,000 $70,000 $75,000 $80,000Due to Bad
DataDamage in
$m $m $m $m $m $m $m $m $m200 40 $2 $2 $2 $2 $2 $3 $3 $3 $3 300 60 $2 $3 $3 $3 $4 $4 $4 $5 $5 500 100 $4 $5 $5 $6 $6 $7 $7 $8 $8 800 160 $6 $7 $8 $9 $10 $10 $11 $12 $13
1000 200 $8 $9 $10 $11 $12 $13 $14 $15 $16 2000 400 $16 $18 $20 $22 $24 $26 $28 $30 $32 5000 1000 $40 $45 $50 $55 $60 $65 $70 $75 $80
10000 2000 $80 $90 $100 $110 $120 $130 $140 $150 $160 15000 3000 $120 $135 $150 $165 $180 $195 $210 $225 $240 20000 4000 $160 $180 $200 $220 $240 $260 $280 $300 $320 25000 5000 $200 $225 $250 $275 $300 $325 $350 $375 $400 30000 6000 $240 $270 $300 $330 $360 $390 $420 $450 $480 35000 7000 $280 $315 $350 $385 $420 $455 $490 $525 $560 40000 8000 $320 $360 $400 $440 $480 $520 $560 $600 $640 50000 10000 $400 $450 $500 $550 $600 $650 $700 $750 $800 55000 11000 $440 $495 $550 $605 $660 $715 $770 $825 $880 60000 12000 $480 $540 $600 $660 $720 $780 $840 $900 $960 80000 16000 $640 $720 $800 $880 $960 $1,040 $1,120 $1,200 $1,280
100000 20000 $800 $900 $1,000 $1,100 $1,200 $1,300 $1,400 $1,500 $1,600 200000 40000 $1,600 $1,800 $2,000 $2,200 $2,400 $2,600 $2,800 $3,000 $3,200 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ANZ Bank
NAB
Wells FargoBoA, Barclays
ClearDQ
Annual DQ Damage Estimator Running ClearDQ actually discovers the accurate detail and $ precision in 2-8 weeks depending on org size.
FTE20% FTE
WasteAvg Salary
$40,000 $45,000 $50,000 $55,000 $60,000 $65,000 $70,000 $75,000 $80,000Due to Bad
DataDamage in
$m $m $m $m $m $m $m $m $m200 40 $2 $2 $2 $2 $2 $3 $3 $3 $3 300 60 $2 $3 $3 $3 $4 $4 $4 $5 $5 500 100 $4 $5 $5 $6 $6 $7 $7 $8 $8 800 160 $6 $7 $8 $9 $10 $10 $11 $12 $13
1000 200 $8 $9 $10 $11 $12 $13 $14 $15 $16 2000 400 $16 $18 $20 $22 $24 $26 $28 $30 $32 5000 1000 $40 $45 $50 $55 $60 $65 $70 $75 $80
10000 2000 $80 $90 $100 $110 $120 $130 $140 $150 $160 15000 3000 $120 $135 $150 $165 $180 $195 $210 $225 $240 20000 4000 $160 $180 $200 $220 $240 $260 $280 $300 $320 25000 5000 $200 $225 $250 $275 $300 $325 $350 $375 $400 30000 6000 $240 $270 $300 $330 $360 $390 $420 $450 $480 35000 7000 $280 $315 $350 $385 $420 $455 $490 $525 $560 40000 8000 $320 $360 $400 $440 $480 $520 $560 $600 $640 50000 10000 $400 $450 $500 $550 $600 $650 $700 $750 $800 55000 11000 $440 $495 $550 $605 $660 $715 $770 $825 $880 60000 12000 $480 $540 $600 $660 $720 $780 $840 $900 $960 80000 16000 $640 $720 $800 $880 $960 $1,040 $1,120 $1,200 $1,280
100000 20000 $800 $900 $1,000 $1,100 $1,200 $1,300 $1,400 $1,500 $1,600 200000 40000 $1,600 $1,800 $2,000 $2,200 $2,400 $2,600 $2,800 $3,000 $3,200 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800
(*) In November 2013, the average full-time wage is $74,724 before tax. Source Australian Bureau of Statistics© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
2. What is your FTE Average Salary ?
1. How Many FTE
do you have ?
ClearDQ
??
Challenge: Convert the Est. into Precision Numbers (1-8 weeks depending on the workforce size)
(*) In November 2013, the average full-time wage is $74,724 before tax. Source Australian Bureau of Statistics
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
Corporate Memory - Reusable Foundations, Simplified Technology, Training Uplift, Speed Delivery, Maturity Uplift, Funding Optimisation, Data Innovation
Accountability
Meaning
Single Source
Assurance
Data Quality
Data Moveme
ntAccess
Platforms & Tools
Data Value Maturity & Innovation
Ownership,Governance, Stewardship,
Produce Data,
Consume Data
Business vs IT
Glossary,
MetadataData ModelsRules
Master,Referen
ce & Txn
Data, PK’s FK’s
ProfilingMonitorin
gCleansingEnrichme
ntTriage
Migration, Integratio
n, Extraction
, ETL, ESB, DV, Process Mapping
Apps, Reporting, Analytics, DW / BI
Big Data, Cloud,
Platforms, DB
Engines, Formats
Reporting
Business Case,
Funding, Frameworks, Training,Innovation
ProjectDelivery
Solution Arch
Artefacts,Proactive Solution
Governance
Risk, Controls,
Maint.
Classify,Security, Regulator
y,Archiving, Retention, Recovery
Search and Docs
Search, Taxonomy
Accurate $$ Business DamageGranular Business SentimentMapped to Entire IT DATA LandscapeFramework, Method and App
?? ??
Globally, no tools exist to autamate this linkage, at scale, in real-time until
now.
ClearDQ
ClearDQ is made up of 4 key parts…
Data Framework
A proven framework which can be applied as comprehensively or as lightly as needed – to manage your Information challenges.
Data Quality Method
A consistent, repeatable method to track, manage, cost and control DQ issues that impact your business.
Business Web App
A web-based tracking, measurement and collaboration tool to productively manage DQ issues with minimal effort.
Benefits Dashboard
Keeps business managers, GM’s team leaders and your data champions up to date with DQ success metrics.
Data Leadership…
Proven Process…
Smart App… Financial Results…
ProvenFramework
RepeatableMethod
CollaborativeApp
Exec & Team Dashboard
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
DQ User Sentiment Screen Shot (Web App)Granular Data Capability Failures in the Business Population
ClearDQ
DQ Issues Screen Shot (Web App)Granular Data Capability Metrics
ClearDQ
Screen Shot (Web Dashboard)Annual DQ Damage Bill (Minimum Damage **)
Annua l DQ Damage to
PEOPLE.
(**) Does not include Fraud, Revenue Leakage, Lost Sales, Regulatory Fines, etc…)
ClearDQ
Screen Shot (Architectural Damage)Total $ Damage Breakdown, by WORST THEME,
and DEPT.
ClearDQ
Screen Shot (User DQ Sentiment)1.0
Catastrophic Data breakage
ClearDQ
ClearDQ – Benefits
Tracks impact of poor DQ, measured across the entire organization, by team
All DQ Issues are compared, priced and prioritized the SAME way
DQ issues are commercialized into $ and dates
Put the business in control of Data Quality processes, priority and spend
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
Precision $ commercial metrics to satisfy the CFO.
Mapped to 4 Different Data Capability Maturity Models, to satisfy any Enterprise Data Architect or IT leader.
At scale to support any size organization or workforce.
A Business “friendly” enterprise-wide approach, that drives the detailed technology data improvement investments.
Linked to APRA (CPG 235) and any other industry regulator demands.
ClearDQ
Case Study: Major Energy Utility
Pain Points
Suffered major data quality issues and disruption for 3 years after a major system migration. IT made many attempts to solve the DQ issues without success.
Benefits
80+ Data Quality issues were identified, to a total value of $24m in immediate cash OPEX savings. The recovery of the $24m would cost $2m in systems fixes and repairs. Staff waste due to data re-work and data repairs including data duplication, re-extraction, re-reconciling, etc… totaled 13% across a workforce of 500 FTE (not IT !) Data re-work workforce to (65FTE) not focused on their primary business function. Regulatory fines detected and prevented in excess of $5m. Customer Care, Customer churn and and customer satisfaction metrics all positively influenced.
80 24 2 13 65
3 IT
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
Engagement ProcessDepending on the size of the organization
1-2 hours 2-10 days 10-30 days
Train & Implement
DQ Forensic Discovery
DQ Impact Assessment
Consultation & Demo
2-5 days
Is Data Quality a challenge for your organization ? Can our proven approach help you ?
What are the major commercial impacts, risks and team impacts due to poor data quality ? What DQ improvements have been made ? What hasn’t worked ? What needs improving ?
Where are your data issues captured and managed ? Where are your business data rules ? Who manages, applies and measures (and automates) your business data rules ?Who are your data champions ? Where are the biggest DQ improvements yet to be made ?
Who needs to be trained ? What DQ metrics need to be captured & presented ?
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
Get the Business DQ help you need…
Martin SprattEnterprise Data Architect
Author & Trainer
Mob: +61-466-839944
Data Quality
Trusted Data Across
the Enterprise
ISBN: 978-1-906355-21-0
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
About - Clear Strategic IT Partners
Outcomes & Solutions
Training &Advisory
DomainArchitecture
Enterprise Architecture
Other Specialized Architected Solutions, Delivery, Training, Advisory (& Rescue):
Clear DQ Data Quality for Business (v2.0, V3.0)ClearCDO Chief Data Office – Data Monetization PlatformClear RDM Reference Data ManagementClear META Active-Active Metadata ManagementClear BACK Enterprise scale archiving & recoveryClear 360 360 degree view of customer, supplier, part, employee, etc…Clear MIG Data Migrations & TransformationsClear CLOUD Cloud Data Transformation & IntegrationClear PM Project Management Training & Certification (Prince2)Clear MOBI Business reports on the goClear ARC IT Architecture, Training & Strategic Services
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902
ClearDQ
Our growing partner networkTrusted, Credible, CapableIT Project & Porfolio Management
MetaPM
Data Centre
UberGlobal
IT Asset Management
The Mastermind Group
IBM Software
Certus Solutions
Data Architecture
CMMI DMM
EDMCouncil.org
Cloud & Dev Ops
Amazon
Broadcoast
Base2Services
Anatas
Red Hat OpenShift
Big Data & Analytics
Actian
Yellowfin
Hadoop Eco-systems
(Cloudera, MapR, Hortonworks)
Data Virtualization
Cisco
Integration
Anatas
Six Tree
Open Source
Six Tree
Apache.org
Bitnami.com
ClearStrategicIT.comClearDQ.com
Clear.
© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902