with the emergence of big data, where do relational ... · relational databases vs. big data with...
Post on 26-Sep-2020
2 Views
Preview:
TRANSCRIPT
With the Emergence of Big Data, Where do Relational Technologies Fit?
VP Product Marketing CA Technologies
DAMA Chicago June 2013
Donna Burbank
Copyright © 2013 CA. All rights reserved.
Who am I?
More than more than 15 years of experience in the areas of
data management, metadata management, and enterprise
architecture. – Currently is VP of Product Marketing for CA’s data modeling
solutions.
– Brand Strategy and Product Management roles at Computer
Associates and Embarcadero Technologies
– Senior consultant for PLATINUM technology’s information
management consulting division in both the U.S. and Europe.
– Worked with dozens of Fortune 500 companies worldwide in the
U.S., Europe, Asia, and Africa and speaks regularly at industry
conferences.
– Co-author of several books including:
Data Modeling for the Business
Data Modeling Made Simple with CA ERwin Data Modeler r8
– Board member of DAMA Rocky Mountain chapter
Copyright © 2013 CA. All rights reserved.
“Big Data” Survey
How familiar are you with "Big Data" technologies?
A. Very familiar. We are using Big Data technologies now.
B. Somewhat. I know what it is, but am not using it.
C. Not at all. This is new to me.
Copyright © 2013 CA. All rights reserved.
DAMA Down Under
I presented a similar topic to DAMA Sydney several weeks ago.
Who knew that DAMA had its own wallaby?
Big Data a Growing Priority for Data Modelers
5 Copyright 2012, DATAVERSITYTM
Copyright © 2013 CA. All rights reserved.
Technological and Culture Shift in Society
1960 1970 1980 1990 2000 2010
Copyright © 2013 CA. All rights reserved.
Technological and Culture Shift in Data Management
1960 1970 1980 1990 2000 2010
• Mainframes • Flat Files • In-House Development • Waterfall Methodology
• Individual PC’s • Relational Databases • Democratization of Computing
• Client/Server Computing • Relational Databases • Data Warehousing • Packaged Applications • RAD Development
• Dot COM Revolution • Changing Business Models
• Dot COM Bubble Bursts • Data Integration • “Do More with Less”
• Cloud Computing • “Big Data”, NoSQL • Data Vaults • Agile Development
Copyright © 2013 CA. All rights reserved.
Traditional Relational Technologies and “Big Data” a Paradigm Shift
•
•
•
•
•
•
•
•
•
•
•
•
Design Implement Discover Document
Unstructured
Data
Copyright © 2013 CA. All rights reserved.
Emergence
I love my new Levis jeans.
Is Levi coming to the party?
Sale #LEVIS 20% at Macys.
LOL. TTYL. Leving soon.
Copyright © 2013 CA. All rights reserved.
Relational Databases vs. Big Data
With the rise of massive volume, real-time data streams, traditional
relational database technologies cannot scale.
– Google, Yahoo, Facebook, Twitter, MySpace
– Corporate Data: email logs, click stream data, medical diagnostics research, etc.
– Huge volumes: With Google, for example, you’re basically “ingesting” the entire internet.
Issues that arise with Big Data
– Can’t store in a single, relational database Too Big
– Can’t easily index Too Big
– Can’t write traditional SQL Unstructured
– Can’t design data structures “top-down” Data is constantly in flux and in creation.
Copyright © 2013 CA. All rights reserved.
Relational Databases vs. “Big Data”
Big Data Solutions address these issues by: – Distributed data storage across servers
– Stores data in files
– Post-creation analysis, real-time
– Data is structured as it is accessed
– Natural-language, application-level processing
– Distributed hash tables – pull record by name, similar to application
programming
Copyright © 2013 CA. All rights reserved.
The Business Case Remains the Same Single, Consistent View of Information – from all Sources
Tell me what
customers are
saying about
our product.
Sybase
SAP
DB2 Oracle
SQL
Server
MS SQL
Azure
Informix Teradata
DBA
Which customer
database do you
want me to pull this
from? We have 25.
Data
Architect
And, by the way, the
databases all store
customer information in a
different format. “CUST_NM”
on DB2, “cust_last_nm” on
Oracle, etc. It’s a mess.
I love my new Levis jeans.
Is Levi coming to the party?
Sale #LEVIS 20% at Macys.
LOL. TTYL. Leving soon.
Big Data
Traditional/Relational Data
Copyright © 2013 CA. All rights reserved.
The Four “V’s” of Big Data
You’ve most likely heard of the “3 V’s of Big Data”
olume
elocity
ariety
Copyright © 2013 CA. All rights reserved.
The Four “V’s” of Big Data
The 4th “V” is now more commonly-mentioned
Remember the business need
– Why are we analyzing this?
– What is the value to the business?
– What aspects of the “volume” are worth keeping?
alue
I love my new Levis jeans.
Is Levi coming to the party?
Sale #LEVIS 20% at Macys.
LOL. TTYL. Leving soon.
Copyright © 2013 CA. All rights reserved.
Are Data Models Still Relevant?
From Data Modeling for the Business by Hoberman, Burbank, Bradley, Technics Publications, 2009
Copyright © 2013 CA. All rights reserved.
Do Relational Databases Go Away?
What Relational Databases are good for: Structured data, that can be designed “up front” with predictable queries
Ease of query and retrieval
Reducing redundancy
Data consistency & integrity - Increase data quality
The same reasons you always have, actually
What “Big Data” solutions are good for: Analyzing large volumes of unstructured data that is updated in real-time
Finding correlations between data from multiple sources.
Analyzing raw, real-time data BEFORE it goes into a Data Warehouse or Transactional Database
Copyright © 2013 CA. All rights reserved.
The Two Solutions Can Work Together
Big Data
I love my new Levis jeans.
Is Levi coming to the party?
Sale #LEVIS 20% at Macys.
LOL. TTYL. Leving soon.
Customer Sentiment Database
Traditional Data Warehouse
Case Study: ARTS Analysis w/ Big Data feeds Traditional DW
18 Copyright © 2013 CA. All rights reserved.
ARTS developed the ARTS Operational Data Model and ARTS Data Warehouse
Model to provide retailers and their suppliers with a mature data architecture
for developing retail IT solutions.
– “ERwin enables us to build, update and manage models efficiently so we can easily incorporate new features, such as big data analytics, to meet evolving retailer needs.”
– Using social analytics data to enhance Customer information -> then feeding that data
into their traditional data warehouse.
Big Data is analyzed before it gets to a structured BI/DW environment with only
the discovered information deemed valuable to the business. The process steps
include:
– Observe the data to determine patterns that makes sense to the business
– Orient or organize these patterns into usable analytical assessments
– Decide on which patterns make sense to pursue
– Act or implement these results into usable BI/DW implementations
Copyright © 2013 CA. All rights reserved.
Case Study: Disney Internet Moved from Traditional DW to Hadoop
Outgrew their Traditional Data Warehouse—Moving to a Big Data Solution to
support customer sentiment analysis
Presentation from Hadoop World 2011
Before (2009)
After (2011)
Copyright © 2013 CA. All rights reserved.
Case Study: Facebook Moved from Big Data to Traditional DW (!!)
Started with Hadoop, then saw the need for a traditional data warehouse
Presented keynote at TDWI Chicago in May—video replay available.
Needed a single source of reference for core business data
– All data in one place- “managed chaos”
– Define the core elements of the business, leave the rest alone
– 100,000s of tables in Hadoop down to a few dozen in Data Warehouse
What data warehouse was good at:
– Operational analysis (e.g. how many users logged in by region?)
– Faster query time (1 minute on DW, over 1 hour on Hadoop)
What Big Data was good at:
– Exploratory analysis (e.g. Where are users posting from—how can we infer a location if one is not
listed?)
“The genius of AND and the tyranny of OR” – Jim Collins, author of “Good to Great”
– Both solutions have their place
Copyright © 2013 CA. All rights reserved.
Case Study: Facebook Moved from Big Data to Traditional DW (!!)
Challenge in both Big Data and Data Warehouse solutions—business definitions
and business value of data.
e.g. How many users logged in yesterday?
– What do you mean by user?
– Does user include mobile devices?
– If a user posted from Spotify, is that a user?
Copyright © 2013 CA. All rights reserved.
Case Study: Facebook Moved from Big Data to Traditional DW (!!)
Challenge in both Big Data and Data Warehouse solutions—business definitions
and business value of data.
e.g. How many users logged in yesterday?
– What do you mean by user?
– Does user include mobile devices?
– If a user posted from Spotify, is that a user?
Sound familiar?
Copyright © 2013 CA. All rights reserved.
Case Study: Facebook Moved from Big Data to Traditional DW (!!)
Challenge in both Big Data and Data Warehouse solutions—business definitions
and business value of data.
e.g. How many users logged in yesterday?
– What do you mean by user?
– Does user include mobile devices?
– If a user posted from Spotify, is that a user?
Sound familiar?
alue
olume
elocity
ariety
A Data Model Can Provide a Central, Business-Centric View of the Data Landscape
Conceptual Data Model – Business View of Data (concepts)
Logical Data Model
Relational (entities) Logical Data Model
Hadoop (files)
SQL
Server
Physical Data Model
DBMS (e.g. SQL Server)
Logical Data Model
Cassandra (KV pairs)
Unified Logical Model
(Mixed Notation)
XML
DB2 Oracle Teradata
Physical Data Model
File (e.g. HDFS) Physical Data Model
Copyright © 2013 CA. All rights reserved.
Copyright © 2013 CA. All rights reserved.
Summary
We are in a period of “disruptive” technology
– Rapid rate of change, Massive volumes of data
– Social changes: more participatory, engaged
– Distributed, web-based implementations
– Real-time analysis required
Create a fit-for-purpose solution
– Relational databases are still great for operational systems
– Big Data offers new opportunities for analysis across distributed systems of diverse data
As with any Age of Change, the basics still apply
– The “hard stuff” still needs to be done: analysis, data quality, etc.
– Data models are valuable to document business requirements and technical implementation
– Data models can create a unified view of the technical infrastructure as well, but tools and
research are in early stages.
Have fun! This is an exciting time to be in Data Management
top related