lessons in data integrity
DESCRIPTION
Presented at the CDW Technology ConferenceTRANSCRIPT
![Page 1: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/1.jpg)
Data Integrity Making Your Data More Accessible but Less Vulnerable
![Page 2: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/2.jpg)
Introduction: About Me
• Senior Manager, Accenture • Expert in Business Analytics and Data
Management • 20 years of Client Project experience in
Systems Integration, Operations • Industry experience in Financial Services,
Government, Healthcare, High Technology, Manufacturing, Retail, Telecommunications
![Page 3: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/3.jpg)
About Accenture
• URL: http://www.accenture.com • Net Revenues: US$13.67 billion for fiscal 2004 (12 mos. ended
Aug. 31, 2004) • Exchange/Ticker: NYSE/ACN • Employees: Over 110,000 • Global Reach: Over 110 offices in 48 countries • Clients: Approximately 2,500, including 84 of the Fortune
Global 100, two-thirds of the Fortune Global 500, and government agencies in 26 countries.
• Ranked #1 in Customer Relationship Management and at the top of the Supply Chain Management and Business Consulting rankings
![Page 4: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/4.jpg)
What To Expect
• An overview of Data Integrity challenges • Successful approaches to Data Integrity • Observations and “Lesson’s Learned”
![Page 5: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/5.jpg)
Google “define:data integrity”
• A condition in which data has not been altered or destroyed in an unauthorized manner. (http://www.nrc.gov/site-help/eie/terms_id.html)
• The state that exists when computerized information is predictably related to its source and has been subjected to only those processes which have been authorized by the appropriate personnel. www.utmb.edu/is/security/glossary.htm
• (IEEE) The degree to which a collection of data is complete, consistent, and accurate. Syn: data quality.
![Page 6: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/6.jpg)
Data Integrity In The News
“Poor record-keeping and the mixing of one day’s leftover hamburger into the next day’s production” created E.coli bacteria contamination that caused several illness and the food services company to lose their largest client. Source: New York Times
“Database giant ChoicePoint said late Wednesday that 145,000 consumers nationwide were placed at risk by a recent data theft at the company. Previously, the company had suggested the theft only affected California residents.” Source: MSNBC, Feb. 18, 2005
![Page 7: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/7.jpg)
Rapid Explosion Of Digital Data
• Structured Data – To & From Databases
• Unstructured Data – Text – Spreadsheets – Audio – Video – Images
![Page 8: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/8.jpg)
Emerging Data Challenges
• Digital Rights Management (DRM) • Biometrics • Compression techniques • Encryption techniques (e.g., steganography) • Competing Metadata standards • Proliferation of unique, nonstandard data
definitions and structures for XML tags
![Page 9: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/9.jpg)
Counting The Costs
What is the estimated cost of customer data integrity problems on American businesses?
a) $200 billion b) $400 billion c) $600 billion d) $800 billion
![Page 10: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/10.jpg)
Customer Data Problems
“Poor quality customer data costs U.S. businesses an estimated $611 billion per year in postage, printing, and staff overhead….”
— “Taking Out the (Data) Garbage,” The Data Warehousing Institute, 4/2/2003
![Page 11: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/11.jpg)
Many Sources Of Customer Data • Pervasive collection of electronic data
– Credit Cards, ATMs, Debit – Phone Records, Check Scans
• Exchange of electronic data – Insurance, medical diagnosis, employment
verification, tax collections • Archived electronic records
– Credit History, Criminal Records, Property Transactions, Licenses & Permits
![Page 12: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/12.jpg)
Example
A major bank had 251 different customer files that it had to analyze just to answer the question, “Who is our best customer?”
![Page 13: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/13.jpg)
Customer Data Variations
Life Event Changes Data Entry Errors Identity Variations
One Life . . . . . . Multiple Identities
![Page 14: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/14.jpg)
Customer Data Integrity Tools
Capabilities Description Analysis Automated Data Assessment "
CRM
Cleansing Data Extract, Reengineering, Transformation and/or Cleansing " " " "
Cleansing Name & Address
Cleanses and enriches name and address data " " " " "
Metadata Quality
Provides quality assessment or metadata management "
Data Defect Prevention
Prevents data errors in source applications that create and update data " "
Rule Discovery
Analyzes and discovers rule in business data " " " "
Service Provider
Provides the quality services or has third party suppliers who use their products. " " "
![Page 15: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/15.jpg)
Other Technologies
• Network and Virus Protection • Authentication and Encryption • Highly-Available Storage • Reliable Network Infrastructure • Secure Remote Access • Monitoring and Audit • Data Profiling and Cleansing
![Page 16: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/16.jpg)
But, wait…
• Are tools the complete solution? • Are there deeper problems? • What risks are hidden below the surface?
1994 Statistic: only 16% of IT projects are successful
2004 Statistic: only 34% of IT projects are successful
![Page 17: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/17.jpg)
Lesson Learned: Management
• Data Management results in Data Integrity • “Management” means
– Accountable Decision-Makers – Defining what the data means – Measuring results – Changing to improve
![Page 18: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/18.jpg)
Key Success Factors
1. Defining the business need 2. Integrating business and
technology professionals 3. Securing sponsorship 4. Taking an iterative
approach 5. Using proven
communication principles
![Page 19: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/19.jpg)
Defining The Business Need
![Page 20: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/20.jpg)
Integrating Business & Technology
• Business users own the data – Requirements for data quality – Content – Definitions
• IT group designs and builds – Ensures technology meets requirements – Provides technical expertise
![Page 21: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/21.jpg)
Sponsorship is crucial
• Lack of executive sponsorship is #1 reason projects fail
• Sponsors provide: – Budget – Business content & context – Influence – Decision authority
![Page 22: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/22.jpg)
Iterative Approach
• “Big Bang” projects often fail • Break projects into short-duration releases
– Set clear objectives – Define “success” in business value
• Incorporate “lessons learned” in next releases
![Page 23: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/23.jpg)
Communication
• Managing expectations of key stakeholders – Use status meetings and
reports – Project and budget
tracking • Marketing the project’s
benefit • Ensuring that key users will
provide their input
![Page 24: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/24.jpg)
Parting Advice
• Plan for constant change – Technology – Executive Leadership
• There are no “silver bullets” – Not one cause, not one solution
• Process is easier to duplicate than talent – Incremental improvement is the norm
• Plans, Actions/Decisions, and Metrics – You have to be moving to steer
![Page 25: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/25.jpg)
Questions?
?
![Page 26: Lessons In Data Integrity](https://reader034.vdocuments.us/reader034/viewer/2022042613/5474fa14b4af9f9d0a8b58c2/html5/thumbnails/26.jpg)
References
• Robb, Drew. Taking Out the (Data) Garbage, 4/2/2003. http://www.tdwi.org/Publications/display.aspx?id=6626&t=y
• Agosta, Lou. The Essential Guide to Data Warehousing
• English, Larry P. Improving Data Warehouse and Business Information Quality
• Mitnick, Kevin D. The Art of Deception : Controlling the Human Element of Security
• Stoll, Clifford. The Cuckoo's Egg: Tracking a Spy Through the Maze of Computer Espionage