UNDERSTANDING BIG DATAS THINK BIG

MEHMET BURAK AKGÜN

First of All What is Big Data?

• BIG-DATA is collected as non-structural from different sources Such as Social media sharing, network logs, blogs, photos, videos, log files, etc..

• BIG-DATA is Can not be analyzed before Enormous size and / or diversity information over the data

1) Why Big Data ? How did we come this point? 2) What is the Big Data’s Components? 3) How will the integration with existing systems? 4) Is there any BIG Data Platform / Applications?

• Almost all data scientists believed that too important advantages of companies using with the ability to analyze information collected from all sources .

• For example, retailers; according to scientists analyze they can through data can improve their operations by 60 percent profit margin. Similar rates are also valid for the public sector.

• According to research, The US healthcare using to data scientists can save 300 million dollars a year.

• According to estimates in 2020, from household appliances to cars and phones and about 50 billion devices will produce data and will be silently communicate with each other.

• For companies which wish to take decisions in the forward-looking , predictions should be critically correct from these «Big Data».

● Energy companies, using smart grids and meters, consisting of the use of data relating to individual subscribers, store, handle the event.

● Banks has become 7/24 branch as according to the information they store regarding collecting customers , recognizing the user, the Internet branch knows the day nor to enter and accordingly the main page menu that makes the most efficient, customers who reminders, offers customizable interfaces, rich, fast and convenient branch .

● Hospitals saving datas on their databases for effective to provide medical services

● This information will be stored as "Big Data"

DATA = PRODUCT

Google without using conventional methods, created their technologies of requirenments,improving itself was a success. Google has billions of web pages on the Google File System keeps uses Big Table in the database, using MapReduce for processing Big Data. See http://www.google.com/about/datacenters Google and Amazon publishes academic articles are related to their work. Some developers who inspired by the articles such as Doug Cutting created similar technology as secret. These are usually the most beautiful examples of the Apache Lucene, Left, Hadoop, HBase such projects. Each of these projects can successfully use the Big Datas . The second generation of the companies such as Facebook, Twitter, Linkedin, are going a step forward by publishing them store it as open-source projects developed for Big data. Cassandra, Hive, Pig, Voldemort, Storm, indextank projects are examples.

Peki Büyük Veri nasıl depolanacak? Nasıl işlenecek? Nasıl analiz edilecek?

Volume

of Tweets create daily.

12+ terabytes

Variety

of different types of data.

100’s Veracity

decision makers trust their information.

Only 1 in 3

4 «components» for being Big

trade events per second.

5+ million

Velocity

• It is expecting to reach 3 billion users.In August 2015, 12TB / day «log data» is producing. The end of 2014 expected reach is 500+ million users. 160 million users are online.

•

•

• 100 million active users. 12+ TB of data tweets / day! ..

Social Networks and Social Work

Google process 24 Petabytes data every day

4.6 Billion mobile phone exist

2 Billion Internet users annual traffic in 2014 equals 667 Exabytes

Social Networks and Social Work

Users are not just people! ..

Each engine 10TB / 30 Mins data produces.

“Data generated by machines and sensors will exceed that generated by social media by at least a factor of 10.” *

Leon Katsnelson Program Director, Big Data & Cloud Computing IBM

By the way, We don't have enough space to store all this data!

We actually appear as a game company.In fact we are data analysis company.. Ken Rudin, Zynga VP of Analytics

• Offers completely free game facilities. • Gaining revenue by selling virtual goods. • The monthly average has 232m active users. • 95% of players never visited shop! • With Using Big Data analysis they disturbed to the game world.

Four Entry Points of Big Data Unlock Big

Data

Simplify Your Warehouse

Preprocess Raw Data

Analyse Streaming

Data

IBM Big Data Platform Systems

Management Application

Development Visualization & Discovery

Accelerators

Information Integration & Governance

Hadoop System

Stream Computing

Data Warehouse

BI / Reporting Exploration / Visualization

Functional App

Industry App

Predictive Analytics

Content Analytics

Analytic Applications

Applications for Big Data Analytics

Homeland Security

Finance Smarter Healthcare Multi-channel sales

Telecom

Manufacturing

Traffic Control

Trading Analytics Fraud and Risk

Log Analysis

Search Quality

Retail: Churn, NBO

"Companies which give importance to Social media is gaining"

In Turkey Organized by the Teradata "Big Data: Great Opportunity" themed event brought together senior executives of the company in Istanbul. President of Teradata EMEA Hermann Wimmer, ‘companies that want to increase their profitability by preventing competitors from social media by analyzing the data obtained in a short time, said he helped to make quick decisions.’

Technological solution is preferring when it provides advantages

Big Data and Open Source Nested

• Open Source Community contribution made over the years

- Apache Hadoop ve Jaql, Apache Derby, Apache Geronimo, Apache Jakarta

- Eclipse: was founded by IBM. - Lucene ;IBM Lucene Extension Library (ILEL) - DRDA, XQuery, SQL, XML4J, XERCES, HTTP,

Java, Linux... • Open source IBM Softwares

– WebSphere: Apache – Rational: Eclipse and Apache – InfoSphere: Eclipse and Apache

• IBM’s BigInsights (Hadoop) is %100 open source software.

Hadoop • Open Source

• Distributed Computing

• Very Simply MapReduce

• Connected

computers

• Ebay – 532 node 532x8 Core – 4256 Core • Facebook – 1100 node 8800 core 12 PB

Storage • LinkedIn – 1200 + 580 + 120 node • Quantcast – 3000 node 3.5 PB 1PB+ daily

data

Example Hadoop Installations

Hadoop • Components:

– Data Storage - HDFS - Distributed Disk File System – Data Processing - MapReduce

HDFS • Google File System • Distributed Disk File System • File Clustering(Usually Files sizes are over then GBs • Fault tolerant • Replication • HDFS is knowledge about the location of physical.

HDFS • Accessible by with Hadoop Shell , Java API or Web UI • Four application node:

– NameNode – manages the metadata of the file – SystemJob Tracker – MapReduce – Task Tracker – MapReduce – Data Node – Informations hiding with NameNode

BIG DATA is not just HADOOP

Manage & store huge volume of any data

Hadoop File System MapReduce

Manage streaming data Stream Computing

Analyze unstructured data Text Analytics Engine

Data Warehousing Structure and control data

Integrate and govern all data sources

Integration, Data Quality, Security, Lifecycle Management, MDM

Understand and navigate federated big data sources Federated Discovery and Navigation

MapReduce • Big Data Processing -- Google • Distributed Calculation Model • Key - the value binary data processing • Easy programming framework – for use:you should improve map() ve reduce() functions

Map: First Step • Make ready the next processing elements

while matching with a key – Data Cleaning – Simply Calculating – Split Strings

Reduce: Last Step • Takes the list of values for Same key with

Iterators – Filtering – Combining – Samping

So reduced.. The result is written to HDFS or HBase

Architecture:MapReduce HDFS CLUSTER

K1,V1 K1,V1

…..

K1,V1 K1,V1

…

K1,V1 K1,V1

….

K1,V1 K1,V1

…

M K2,V2 K2,V2

M K2,V2 K2,V2

M K2,V2 K2,V2 K2,V2 M …

M M M M

K2,V2 K2,V2 K2,V2 K2,V2 K2,V2 K2,V2 K2,V2 K2,V2 K2,V2 …

GROUPING RANKING

K2, Iterable<V2>

K2, Iterable<V2>

K2, Iterable <V2>

R K3,V3 K3,V3 …

Apache Pig • Yahoo!

• Designed to easy analysis of large data sets

• Easy than Map Reduce function in Java – Pig Lan coding – Can be improved

• Similar with each languages

• 10 lines of Pig code may be equivalent to hundreds of lines Java

Run To Pig ! • Grunt – Shell • Java interface • Eclipse and IntelliJ IDEA plugins

%tweets = load ‘/today/tweets’ as (user, mention, tweet) %twitters = group tweets by mention

Apache Hive • Data Warehouse Project for Hadoop • Summarizing Data • Instant queries • SQL-like language –HiveQL

– Allows to define custom mapper and reducer Special Note * : Hive compiler translates the MapReduce operations to SQL queries

HBase Every time we do not need to relational databases We need Scalability Table size can be very large so We want very fast access

Distributed Key-- Sorted Persistent Map

HBase Google – Big Table clon Works on HDFS *fault tolerance *scalability *MapReduces input-output Hbase = HDFS + Random read/ write

HBase Where to Use: Social Media Recommended Systems Search Engines Intelligence and Monitoring Services Financial Systems - fraud

Apache Mahout • Let us go beyond the simple analysis

• Classification - Email Spam - Call Center • Clustering - finding new news • Recommended Systems

Apache Mahout

Ready Algorithms:: • Classification:

• Logisac Regression • Bayesian • Random Forests

– Application • Call forwarding • Face recognition

Apache Mahout

Ready Algorithms: Recommendation:

• Distributed Item based • Matrix Factorizaaon • Non distributed item/user based

– Application: • eCommerce - Amazon • Movie recommendation - netflix • Music recommendation - LastFM • Venue Recommendation - foursquare

In summary:Some Big Data Applications Log Analytics (IT for IT) Smart Grid / Smarter Utilities RFID Tracking & Analytics Fraud / Risk Management & Modeling 360° View of the Customer Warehouse Extension Email / Call Center Transcript

Analysis Call Detail Record Analysis IBM Watson

The IBM Big Data Platform

IBM Big Data Platform IBM Big Data & Netezza Product Group

InfoSphere BigInsights Hadoop-based, low latency, diverse and high-volume data

analysis

Hadoop

IBM Netezza High Capacity Appliance Archived questionable

structural data

IBM Netezza 1000 BI+Ad Hoc Structured Data Analysis

IBM Smart Analytics System

Structural analysis of operational data

IBM Informix Timeseries Time-structured analytics

IBM InfoSphere Warehouse

High volume, structural veri analizi

Stream Computing InfoSphere Streams Fluid analysis for low latency data

MPP Data Warehouse

Information Integration InfoSphere Information

Server High volume data integration and transformation

Big Data Exploration: Value & Diagram

File Systems

Relational Data

Content Management

Email

CRM

Supply Chain

ERP

RSS Feeds

Cloud

Custom Sources

Data Explorer

Application/ Users

Find, Visualize & Understand all big data to improve business knowledge

• Greater efficiencies in business processes

• New insights from combining and analyzing data types in new ways

• Develop new business models with resulting increased market presence and revenue

Data Analysis in Different Diversity Making the analysis on the data in mixed feature.

Dinamic Data Analysis High volume flow data, ad-hoc analysis

Explore and experiment Data on Ad-hoc analysis, discovery and inspection data Manage and Plans Data Rules,Data integrity checking and application

Very High Volume Data Analysis The data in the PB scale appropriate price / performance criteria for analysis

What does IBM platform?