dba to data scientist -satyendra

© Copyright 2013. Apps Associates LLC. 1

DBA to Data Scientist with Oracle Big Data

August 23rd , 2014


Satyendra Kumar Pasalapudi

Associate Practice Director – IMS @ Apps Associates

Co Founder & Vice President of AIOUG

@pasalapudi

http://www.aioug.org/index.php


Agenda

• What is Big Data

• Big Data Growth

• 4 Phases of Big Data

• NoSQL Databases

• Hadoop Basics

• Big Data Appliance

• Skills Required for DBA Scientist


Big Data Growth

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=Pq2UnPH7Gg1AxM&tbnid=U9-GZOoVX1Nh_M:&ved=0CAUQjRw&url=http://www.forbes.com/sites/davefeinleib/2012/10/15/how-forresters-former-cmo-is-using-big-data-to-build-next-generation-crm/&ei=HvTHUbHTPIarrAfemYCAAQ&bvm=bv.48293060,d.aGc&psig=AFQjCNGl0v61aVrSz5sxynp5NdBlA-6sjg&ust=1372144939486562

http://www.mycustomer.com/files/siftmedia-mycustomer/images/Domo infographic data never sleeps.jpg


Cost effectively manage

and analyze

all available data in its

native form

unstructured,

structured, streaming

ERP CRM

RFID

Website

Network Switches

Social Media

Billing

Big data Challenge

90%

Of the world’s data has been created in the last two years

Source: IBM


3 Macro Trends Driving Disruption


Gen X Stats


Big Data – High Data Varity & Velocity


How Did Big Data Evolve?

• More people interacting with data

• Smartphones

• Internet

• Greater volumes of data being generated (machine-to-machine generation)

• Sensors

• General Packet Radio Services (GPRS)


What Is Big Data?

Big data is defined as voluminous unstructured data from many different sources, such as:

• Social networks

• Banking and financial services

• E-commerce services

• Web-centric services

• Internet search indexes

• Scientific searches

• Document searches

• Medical records

• Weblogs


Big Data

• Extremely large datasets that are hard to deal with using Relational Databases

– Storage/Cost

– Search/Performance

– Analytics and Visualization

• Need for parallel processing on hundreds of machines

– ETL cannot complete within a reasonable time

– Beyond 24hrs – never catch up


Characteristics of Big Data

Volume Variety Velocity Value

Social Networks

RSS Feeds

Micro Blogs


Characteristics of Big Data


Big Data Eco System

Financial services Discover fraud patterns based on multi-years worth of credit card transactions and in a time scale that does not allow new patterns to accumulate significant losses. Measure transaction processing latency across many business processes by processing and correlating system log data.

Internet retailer Discover fraud patterns in Internet retailing by mining Web click logs. Assess risk by product type and session/Internet Protocol (IP) address activity.

Retailers Perform sentiment analysis by analyzing social media data.

Drug discovery Perform large-scale text analytics on publicly available information sources.

Healthcare Analyze medical insurance claims data for financial analysis, fraud detection, and preferred patient treatment plans. Analyze patient electronic health records for evaluation of patient care regimes and drug safety.

Mobile telecom Discover mobile phone churn patterns based on analysis of CDRs and correlation with activity in subscribers’ networks of callers.

IT technical support Perform large-scale text analytics on help desk support data and publicly available support forums to correlate system failures with known problems.

Scientific research Analyze scientific data to extract features (e.g., identify celestial objects from telescope imagery).

Internet travel Improve product ranking (e.g., of hotels) by analysis of multi-years worth of Web click logs.

Big Data /Hadoop Use Cases


The Four Phases of Data Conversion

Acquire Organize Analyze Decide

1 4 3 2


Database Market Disruption

$30B Database Market Being Disrupted


Operational vs. Analytical Databases


Growth is the New Reality

Instagram gained nearly 1 million users overnight when they expanded to Android


Draw Something Viral Growth


How Do You Take This Growth?


Scaling Out RDBMS


RDBMS are Not Enough?


NoSQL Technology Scales Out


A New Technology


Use Cases


Relational vs. Documental Data Model

JSON or JavaScript Object Notation, is a text-based open standard designed for human-readable data interchange. It is derived from the JavaScript scripting language for representing simple data structures and associative arrays, called objects. Despite its relationship to JavaScript, it is language-independent, with parsers available for many languages

http://en.wikipedia.org/wiki/Open_standard

http://en.wikipedia.org/wiki/Human-readable



http://en.wikipedia.org/wiki/JavaScript

http://en.wikipedia.org/wiki/Data_structure

http://en.wikipedia.org/wiki/Data_structure

http://en.wikipedia.org/wiki/Associative_array

http://en.wikipedia.org/wiki/Language-independent_specification




Brewer's CAP Theorem


NoSQL Technology Spectrum


Operational vs. Analytical Databases


Hadoop Design Principles

• System shall manage and heal itself

– Automatically and transparently route around failure

– Speculatively execute redundant tasks if certain nodes are detected to be slow

• Performance shall scale linearly

– Proportional change in capacity with resource change

• Compute should move to data

– Lower latency, lower bandwidth

• Simple core, modular and extensible


Hadoop Intro

• At Google MapReduce operation are run on a special file system called Google File System (GFS) that is highly optimized for this purpose.

• GFS is not open source.

• Doug Cutting and others at Yahoo! reverse engineered the GFS and called it Hadoop Distributed File System (HDFS).

• The software framework that supports HDFS, MapReduce and other related entities is called the project Hadoop or simply Hadoop.

• Projects Nutch and Lucene were started with “search” as the application in mind;


Hadoop Intro

• Hadoop Distributed file system and mapreduce were found to have applications beyond search.

• HDFS and MapReduce were moved out of Nutch as a sub-project of Lucene and later promoted into a apache project Hadoop


Hadoop History

• Dec 2004 – Google GFS paper published

• July 2005 – Nutch uses MapReduce

• Feb 2006 – Starts as a Lucene subproject

• Apr 2007 – Yahoo! on 1000-node cluster

• Jan 2008 – An Apache Top Level Project

• Jul 2008 – A 4000 node test cluster

• May 2009 – Hadoop sorts Petabyte in 17 hours


What & Where is Hadoop Used For?

Search

• Yahoo, Amazon, Zvents

Log Processing

• Facebook, Yahoo, ContextWeb. Joost, Last.fm

Recommendation Systems

• Facebook

Data Warehouse

• Facebook, AOL

Video and Image Analysis

• New York Times, Eyealike


What & Where is Hadoop Used For?

Amazon.com, Ancestry.com, Akamai, American Airlines, AOL, Apple, AVG , eBay, Electronic Arts, Hortonworks, Federal Reserve Board of Governors, Foursquare, Fox Interactive Media, Google, HewlettPackard, IBM, ImageShack, ISI, InMobi, Intuit, Joost, Last.fm, LinkedIn, Microsoft, NetApp, Netflix, Ooyala, Riot Games, Spotify, Qualtrics, The New York Times, SAP AG, SAS Institute, StumbleUpon, Twitter, Yodlee


Hadoop Ecosystem

HDFS (Hadoop Distributed File System)

HBase (key-value store)

MapReduce (Job Scheduling/Execution System)

Data Access

Sqoop Flume

Client Access

Hue Hive(Sql)

Pig(Pl/Sql)

Zoo

Kee

pe

r (C

oo

rdin

atio

n)

(Streaming/Pipes APIs)

Ch

ukw

a (M

on

ito

rin

g)

Data Mining

Mahout

OS – Redhat, Suse, Ubuntu,Windows

Commodity Hardware

Java Virtual Machine

Networking

Orchestration

Oozie


PIG

• Compiled into a series of MapReduce jobs

– Easier to program

– Optimization opportunities

• grunt> A = LOAD 'student' USING PigStorage() AS (name:chararray, age:int, gpa:float);

• grunt> B = FOREACH A GENERATE name;


Hive

Managing and querying structured data • MapReduce for execution • SQL like syntax • Extensible with types,

functions, scripts • Metadata stored in a RDBMS

(MySQL) • Joins, Group By, Nesting • Optimizer for number of

MapReduce required

hive> SELECT a.foo FROM invites a WHERE a.ds='<DATE>‘;


Sqoop

• It supports incremental loads of a single table or a free form SQL query as well as saved jobs which can be run multiple times to import updates made to a database since the last import

• Imports can also be used to populate tables in Hive or HBase

• Exports can be used to put data from Hadoop into a relational database


HDFS Architecture


Architecture Overview


HDFS Distributions


Hadoop 2.0

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=A2ql2vprIy1-9M&tbnid=KXM_k3zvnu_14M:&ved=0CAUQjRw&url=http://hortonworks.com/hadoop/yarn/&ei=zyZ9UqzXJ8mWiAfR_IGgCQ&bvm=bv.56146854,d.aGc&psig=AFQjCNHvWs4AyHht_VRSMJ42zphHBU-kRw&ust=1384020037430301


Oracle Big Data Appliance: Introduction

Oracle Big Data Appliance: Introduction

• Oracle Big Data Appliance is an engineered system containing both hardware and software components. Oracle Big Data Appliance delivers:

‒ A complete and optimized solution for big data

‒ Single-vendor support for both hardware and software

‒ An easy-to-deploy solution

‒ Tight integration with Oracle Database


Oracle Big Data Appliance: Where It Stands?

Big Data Appliance

Data Variety

Unstructured

Schema-less

Schema

Information Acquire Organize Analyze


Oracle Big Data: Software Components

Oracle Big Data Appliance

Oracle NoSQL Database

Oracle Big Data Connectors

Open Source R Distribution

Cloudera Manager & Cloudera’s Distribution Including Apache Hadoop

Oracle Linux 5.6 and Java Hotspot VM


Oracle Big Data with Oracle Exadata


Oracle Big Data Solution

Oracle BI Foundation Suite

Oracle Real-Time Decisions

Endeca Information Discovery

Decide

Oracle Advanced Analytics

Oracle Database

Oracle Spatial & Graph

Acquire – Organize – Analyze


Oracle Data Integrator

Stream

Oracle Event Processing

Apache Flume

Oracle GoldenGate


Cloudera Hadoop

Oracle R Distribution


Mapping the Phases with Software

Acquire Phase

– Hadoop Distributed File System

– Oracle NoSQL Database

Organize Phase

– Hadoop Software Framework

– Oracle Data Integrator

Analyze Phase

– R Statistical Programming Environment

– Oracle Data Warehouse


Analyze Phase

Analyze Database

+ Oracle R Enterprise

Statistical Functions

Data Mining Algorithms

Query Capabilities


Oracle Big Data: Software Components

Oracle Big Data Appliance



Open Source R Distribution

Cloudera Manager & Cloudera’s Distribution Including Apache Hadoop

Oracle Linux 5.6 and Java Hotspot VM


Oracle Big Data Sql

• New Technology Bridges Oracle, Hadoop, and NoSQL Data Stores

Using Oracle Big Data SQL, organizations can:

• Combine data from Oracle Database, Hadoop and NoSQL in a single SQL query

• Query and analyze data in Hadoop and NoSQL

• Integrate big data analysis into existing applications and architectures

• Extend security and access policies from Oracle Database to data in Hadoop and NoSQL

• Maximize query performance on all data using Smart Scan

• One Fast SQL Query for All Your Data


Data Science

Source :http://wikibon.org/blog/role-of-the-data-scientist/


Data Scientist

Source :http://wikibon.org/blog/role-of-the-data-scientist/

http://wikibon.org/blog/wp-content/uploads/2011/03/data-science-full.jpg


DBA to Data Scientist

Hadoop HDFS Map Reduce NoSQL Database Hive Pig OR All the above with Big Data Appliance


Big Data Eco System


On Premise Hadoop as RDBMS “active archive”

SALES 2013

Oracle Database

Structured Data Analytics from Apps

SALES 2012

SALES 2011

SALES 2010

SALES 2011

SALES 2010

“Hive” provides an SQL-like query layer over Hadoop and MapReduce

Unstructured + Structured Data Analytics from Apps

Hadoop for Structured Archive and Unstructured data


AWS EMR as RDBMS “active archive”

SALES 2013

Oracle Database

Structured Data Analytics from Apps

SALES 2012

SALES 2011

SALES 2010

SALES 2011

SALES 2010

“Hive” provides an SQL-like query layer over Amazon EMR

Unstructured + Structured Data Analytics from Apps

AWS EMR for Structured Archive and Unstructured data

Amazon Elastic MapReduce (Amazon EMR)

Thank You! @pasalapudi

dba to data scientist -satyendra

Technology

apps associates llc

scale text analytics

big data

query layer

york times

years worth

oracle database

structured archive