big data hub workshop

Intro to Big Data Workshop

14 March

Big Data Hub https://www.facebook.com/BigDataHub

Introduction

2

Make it Easy

BidData Community : Intro to BigData Workshop

Introduction

3 BidData Community : Intro to BigData Workshop

Who, we are…..

Introduction


Community Vision Building a Big Data Knowledge hub in Egypt.

Introduction


Mission Spreading BD science and engineering

awareness

Introduction


Objectives ● Training 120 4th year student from

Engineering and computer science faculties.

● Running 5 awareness sessions targeting five universities in the first phase.

Introduction


Monofia University

Introduction


Call for volunteers Business Development

Administration

Development

Administrative & Logistics

Introduction


Today RoadMap 1. Big Data Introduction 2. Hadoop installation

3. Small Java application

Introduction


Ready ?

Eng. Youssef Eassa

BigData and Ecosystem

BigData & Ecosystem


Data Vs. Information Vs. Knowledge

BigData & Ecosystem


Data Scientist VS Data Engineer

BigData & Ecosystem


What is the maximum file size you have? Movies/Files/Streaming video that you have used?

What is the maximum download speed you get? How much time to just transfer?

BigData & Ecosystem


How do you process

Distributed System

Massive data?

Distributed Computing System

BigData & Ecosystem


Now, What is difference between

Big Data Massive Data and

BigData & Ecosystem


From Where Data are Generated?

Social media and networks (all of us are generating data)

Scientific instruments (collecting all sorts of data)

Mobile devices (tracking all objects all the time)

Sensor technology and networks (measuring all kinds of data)

BigData & Ecosystem


Big Data in one Minute

BigData & Ecosystem


What makes Big Data that big?

BigData & Ecosystem


Data Evolution

BigData & Ecosystem


The Problem

BigData & Ecosystem


Hadoop != Database

Hadoop

Hadoop is a Distributed Storage and Computation Framework

BigData & Ecosystem


BIGDATA is not just HADOOP

BigData & Ecosystem


BigData Platforms

BigData & Ecosystem


Core Components of Hadoop

Storage (HDFS) Processing (MapReduce)

BigData & Ecosystem


BigData & Ecosystem


HDFS to Store Big Data

BigData & Ecosystem


BigData & Ecosystem


MapReduce :

BigData & Ecosystem


Hadoop 2.0: Next-gen platform

BigData & Ecosystem


Store all data in one place Interact with data in multiple ways

BigData & Ecosystem


Hadoop 2.0 Projects

• YARN • HDFS Federation Aka HDFS 2.0

BigData & Ecosystem


YARN: Architecture

BigData & Ecosystem


HDFS Federation: Architecture

BigData & Ecosystem


Big Data Ecosystem

BigData & Ecosystem


BigData & Ecosystem


It’s the time for Implementaion

Eng. Ahmed Shouman

BigData Practical Activity

Hadoop Installation

BigData practice


What’s Hadoop? Is a Java-based programming framework that supports the processing of large data sets in a distributed computing environment.


BigData practice

Hadoop installation modes

○ Stand alone mode. ○Pseudo distributed mode. ○Fully distributed mode.


BigData practice

What’s Hadoop?


BigData practice

Setting up our Lab:


BigData practice

Prerequisites:


BigData practice

Install JAVA & Configure hosts file

○ sudo apt-get update ○ sudo apt-get install sun-java6-jdk ○ Java –version

○Hosts File ○ #gedit /etc/hosts


BigData practice

Install & Configure SSH ○ sudo apt-get install openssh-server ○ ssh-keygen -t rsa -P "“


BigData practice

Download & Install


BigData practice

Download & Install

○ In the Linux Terminal, Write: “ wget http://supergsego.com/apache/hadoop/common/hadoop-1.2.1/hadoop-

1.2.1-bin.tar.gz ” & hit ENTER


BigData practice

Editing .bashrc file ○ #gedite ~/.bashrc ○ Add the following lines at the end of the file


BigData practice

Main Installation ○ #tar –zxvf hadoop-1.2.1-bin.tar.gz


BigData practice

Editing hadoop-env.sh

○ #gedite /opt/hadoop/conf/hadoop-env.sh


BigData practice

Editing conf/*-site.xml files ○ 1- “Core-site.xml” File: ○ #gedit /opt/hadoop/conf/core-site.xml


BigData practice

Editing conf/*-site.xml files ○ 2-”Mapred-site.xml” File ○ #gedit /opt/hadoop/conf/mapred-site.xml


BigData practice

Editing conf/*-site.xml files ○ 3-”hdfs-site.xml” File. ○ #gedit /opt/hadoop/conf/hdfs-site.xml


BigData practice

Formatting Namenode F.S. ○ #hadoop namenode –format


BigData practice

Firing Hadoop Deamons ○ #start-all.sh


BigData practice

Testing Installation ○ Localhost:50070


BigData practice


BigData practice

Thank You Special thanks to


BigData practice

Questions? Like us on FB:

https://www.facebook.com/BigDataHub


BigData Workshop

big data hub workshop

Data & Analytics