Download - The Hadoop Ecosystem for Developers
The Hadoop Ecosystem
Zohar Elkayam & Ronen Fidel
Brillix
Agenda
• Big Data – The Challenge
• Introduction to Hadoop
– Deep dive into HDFS
– MapReduce and YARN
• Improving Hadoop: tools and extensions
• NoSQL and RDBMS
2
About Brillix
• Brillix is a leading company that specialized in Data Management
• We provide professional services and consulting for Databases, Security and Big Data solutions
3
Who am I?
• Zohar Elkayam, CTO at Brillix
• DBA, team leader, instructor and a senior consultant for over 17 years
• Oracle ACE Associate
• Involved with Big Data projects since 2011
• Blogger – www.realdbamagic.com
4
Big Data
"Big Data"??
Different definitions
“Big data exceeds the reach of commonly used hardware environments
and software tools to capture, manage, and process it with in a tolerable
elapsed time for its user population.” - Teradata Magazine article, 2011
“Big data refers to data sets whose size is beyond the ability of typical
database software tools to capture, store, manage and analyze.”
- The McKinsey Global Institute, 2012
“Big data is a collection of data sets so large and complex that it
becomes difficult to process using on-hand database management
tools.” - Wikipedia, 2014
6
A Success Story
8
More success stories
9
MORE stories..
• Crime Prevention in Los Angeles
• Diagnosis and treatment of genetic diseases
• Investments in the financial sector
• Generation of personalized advertising
• Astronomical discoveries
10
Examples of Big Data Use Cases Today
MEDIA/ENTERTAINMENT
Viewers / advertising effectiveness
COMMUNICATIONS
Location-based advertising
EDUCATION &RESEARCH
Experiment sensor analysis
CONSUMER PACKAGED GOODS
Sentiment analysis of what’s hot, problems
HEALTH CARE
Patient sensors, monitoring, EHRs
Quality of care
LIFE SCIENCES
Clinical trials
Genomics
HIGH TECHNOLOGY / INDUSTRIAL MFG.
Mfg quality
Warranty analysis
OIL & GAS
Drilling exploration sensor analysis
FINANCIALSERVICES
Risk & portfolio analysis
New products
AUTOMOTIVE
Auto sensors reporting location, problems
RETAIL
Consumer sentiment
Optimized marketing
LAW ENFORCEMENT & DEFENSE
Threat analysis -social media monitoring, photo analysis
TRAVEL &TRANSPORTATION
Sensor analysis for optimal traffic flows
Customer sentiment
UTILITIES
Smart Meter analysis for network capacity,
ON-LINE SERVICES / SOCIAL MEDIA
People & career matching
Web-site
optimization
11
Most Requested Uses of Big Data
• Log Analytics & Storage
• Smart Grid / Smarter Utilities
• RFID Tracking & Analytics
• Fraud / Risk Management & Modeling
• 360° View of the Customer
• Warehouse Extension
• Email / Call Center Transcript Analysis
• Call Detail Record Analysis
12
The Challenge
Big Data Big Problems
• Unstructured• Unprocessed• Un-aggregated• Un-filtered• Repetitive• Low quality• And generally messy
Oh, and there is a lot of it
14
The Big Data Challenge
15
Big Data: Challenge to Value
Business
Value
High Variety
High Volume
High Velocity
Today
Deep Analytics
High Agility
Massive Scalability
Real TimeTomorrow
Challenges
16
Volume
• Big data come in one size: Big.
• Size is measured in Terabyte(1012), Petabyte(1015), Exabyte(1018), Zettabyte (1021)
• The storing and handling of the data becomes an issue
• Producing value out of the data in a reasonable time is an issue
17
Some Numbers
• How much data in the world?– 800 Terabytes, 2000– 160 Exabytes, 2006 (1EB = 1018B)– 4.5 Zettabytes, 2012 (1ZB = 1021B)– 44 Zettabytes by 2020
• How much is a zettabyte?– 1,000,000,000,000,000,000,000 bytes– A stack of 1TB hard disks that is 25,400 km high
18
Data grows fast!
19
Growth Rate
How much data generated in a day?
– 7 TB, Twitter
– 10 TB, Facebook
20
Variety
• Big Data extends beyond structured data: including semi-structured and unstructured information: logs, text, audio and videos
• Wide variety of rapidly evolving data types requires highly flexible stores and handling
21
Structured & Un-Structured
Un-Structured Structured
Objects Tables
Flexible Columns and Rows
Structure Unknown Predefined Structure
Textual and Binary Mostly Textual
22
Big Data is ANY data:
Unstructured, Semi-Structure and Structured
• Some has fixed structure
• Some is “bring own structure”
• We want to find value in all of it
23
Data Types by Industry
24
Velocity
• The speed in which the data is being generated and collected
• Streaming data and large volume data movement
• High velocity of data capture – requires rapid ingestion
• Might cause the backlog problem
25
Global Internet Device Forecast
26
Internet of Things
27
Veracity
• Quality of the data can vary greatly
• Data sources might be messy or corrupted
28
So, What Defines Big Data?
• When we think that we can produce value from that data and want to handle it
• When the data is too big or moves too fast to handle in a sensible amount of time
• When the data doesn’t fit conventional database structure
• When the solution becomes part of the problem
29
Handling Big Data
Big Data in Practice
• Big data is big: technological infrastructure solutions needed
• Big data is messy: data sources must be cleaned before use
• Big data is complicated: need developers and system admins to manage intake of data
32
Big Data in Practice (cont.)
• Data must be broken out of silos in order to be mined, analyzed and transformed into value
• The organization must learn how to communicate and interpret the results of analysis
33
Infrastructure Challenges
• Infrastructure that is built for:
– Large-scale
– Distributed
– Data-intensive jobs that spread the problem across clusters of server nodes
34
Infrastructure Challenges (cont.)
• Storage:– Efficient and cost-effective enough to capture and
store terabytes, if not petabytes, of data
– With intelligent capabilities to reduce your data footprint such as:
• Data compression
• Automatic data tiering
• Data deduplication
35
Infrastructure Challenges (cont.)
• Network infrastructure that can quickly import large data sets and then replicate it to various nodes for processing
• Security capabilities that protect highly-distributed infrastructure and data
36
Introduction To Hadoop
Apache Hadoop
• Open source project run by Apache (2006)
• Hadoop brings the ability to cheaply process large amounts of data, regardless of its structure
• It Is has been the driving force behind the growth of the big data Industry
• Get the public release from:http://hadoop.apache.org/core/
38
Hadoop Creation History
39
Key points
• An open-source framework that uses a simple programming model to enable distributed processing of large data sets on clusters of computers.
• The complete technology stack includes
– common utilities
– a distributed file system
– analytics and data storage platforms
– an application layer that manages distributed processing, parallel computation, workflow, and configuration management
• Cost-effective for handling large unstructured data sets than conventional approaches, and it offers massive scalability and speed
40
Why use Hadoop?
Cost Flexibility
Near linear
performance up
to 1000s of nodes
Leverages
commodity HW &
open source SW
Versatility with
data, analytics &
operation
Scalability
41
No, really, why use Hadoop?
• Need to process Multi Petabyte Datasets• Expensive to build reliability in each application• Nodes fail every day
– Failure is expected, rather than exceptional– The number of nodes in a cluster is not constant
• Need common infrastructure– Efficient, reliable, Open Source Apache License
• The above goals are same as Condor, but– Workloads are IO bound and not CPU bound
42
Hadoop Benefits
• Reliable solution based on unreliable hardware• Designed for large files• Load data first, structure later• Designed to maximize throughput of large scans• Designed to leverage parallelism• Designed to scale• Flexible development platform• Solution Ecosystem
43
Hadoop Limitations
• Hadoop is scalable but it’s not fast
• Some assembly required
• Batteries not included
• Instrumentation not included either
• DIY mindset
44
Hadoop Components
Hadoop Main Components
• HDFS: Hadoop Distributed File System –distributed file system that runs in a clustered environment.
• MapReduce – programming paradigm for running processes over a clustered environments.
47
HDFS is...
• A distributed file system
• Redundant storage
• Designed to reliably store data using commodity hardware
• Designed to expect hardware failures
• Intended for large files
• Designed for batch inserts
• The Hadoop Distributed File System
48
HDFS Node Types
HDFS has three types of Nodes
• Namenode (MasterNode)– Distribute files in the cluster– Responsible for the replication between
the datanodes and for file blocks location
• Datanodes– Responsible for actual file store– Serving data from files(data) to client
• BackupNode (version 0.23 and up)• It’s a backup of the NameNode
49
Typical implementation
• Nodes are commodity PCs
• 30-40 nodes per rack
• Uplink from racks is 3-4 gigabit
• Rack-internal is 1 gigabit
50
MapReduce is...
• A programming model for expressing distributed computations at a massive scale
• An execution framework for organizing and performing such computations
• An open-source implementation called Hadoop
51
MapReduce paradigm
• Implement two functions:• MAP - Takes a large problem and divides into sub problems
and performs the same function on all subsystemsMap(k1, v1) -> list(k2, v2)
• REDUCE - Combine the output from all sub-problemsReduce(k2, list(v2)) -> list(v3)
• Framework handles everything else (almost)
• Value with same key must go to the same reducer
52
Typical large-data problem
• Iterate over a large number of records
• Extract something of interest from each
• Shuffle and sort intermediate results
• Aggregate intermediate results
• Generate final output
Map
Reduce
53
Divide and Conquer
54
MapReduce - word count example
function map(String name, String document):
for each word w in document:
emit(w, 1)
function reduce(String word, Iterator
partialCounts):
totalCount = 0
for each count in partialCounts:
totalCount += count
emit(word, totalCount)
55
MapReduce Word Count Process
56
MapReduce Advantages
Example: $HADOOP_HOME/bin/hadoop jar @HADOOP_HOME/hadoop-
streaming.jar \
- input myInputDirs \
- output myOutputDir \
- mapper /bin/cat \
- reducer /bin/wc
• Runs programs (jobs) across many computers
• Protects against single server failure by re-run failed steps
• MR jobs can be written in Java, C, Phyton, Ruby and
others
• Users only write Map and Reduce functions
57
MapReduce is good for...
• Embarrassingly parallel algorithms
• Summing, grouping, filtering, joining
• Off-line batch jobs on massive data sets
• Analyzing an entire large dataset
58
MapReduce is OK for...
• Iterative jobs (i.e., graph algorithms)
• Each iteration must read/write data to disk
• IO and latency cost of an iteration is high
59
MapReduce is NOT good for...
• Jobs that need shared state/coordination
• Tasks are shared-nothing
• Shared-state requires scalable state store
• Low-latency jobs
• Jobs on small datasets
• Finding individual records
60
Deep Dive into HDFS
HDFS
• Appears as a single disk• Runs on top of a native filesystem
– Ext3,Ext4,XFS
• Based on Google's Filesystem GFS• Fault Tolerant
– Can handle disk crashes, machine crashes, etc...
• Based on Google's Filesystem (GFS or GoogleFS)– gfs-sosp2003.pdf
• http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/gfs-sosp2003.pdf
– http://en.wikipedia.org/wiki/Google_File_System
62
HDFS is Good for...
• Storing large files– Terabytes, Petabytes, etc...– Millions rather than billions of files– 100MB or more per file
• Streaming data– Write once and read-many times patterns– Optimized for streaming reads rather than random reads– Append operation added to Hadoop 0.21
• “Cheap” Commodity Hardware– No need for super-computers, use less reliable commodity hardware
63
HDFS is not so good for...
• Low-latency reads– High-throughput rather than low latency for small chunks of
data– HBase addresses this issue
• Large amount of small files– Better for millions of large files instead of billions of small files
• For example each file can be 100MB or more
• Multiple Writers– Single writer per file– Writes only at the end of file, no-support for arbitrary offset
64
HDFS: Hadoop Distributed File System
• A given file is broken down into blocks (default=64MB), then blocks are replicated across cluster (default=3)
• Optimized for:– Throughput– Put/Get/Delete– Appends
• Block Replication for:– Durability– Availability– Throughput
• Block Replicas are distributed across servers and racks
65
HDFS Architecture
• Name Node : Maps a file to a file-id and list of Map Nodes
• Data Node : Maps a block-id to a physical location on disk
• Secondary Name Node: Periodic merge of Transaction log
66
HDFS Daemons
• Filesystem cluster is manager by three types of processes– Namenode
• manages the File System's namespace/meta-data/file blocks• Runs on 1 machine to several machines
– Datanode• Stores and retrieves data blocks• Reports to Namenode• Runs on many machines
– Secondary Namenode• Performs house keeping work so Namenode doesn’t have to• Requires similar hardware as Namenode machine• Not used for high-availability – not a backup for Namenode
67
Files and Blocks
• Files are split into blocks (single unit of storage)
– Managed by Namenode, stored by Datanode
– Transparent to user
• Replicated across machines at load time
– Same block is stored on multiple machines
– Good for fault-tolerance and access
– Default replication is 3
68
HDFS Blocks
• Blocks are traditionally either 64MB or 128MB– Default is 128MB
• The motivation is to minimize the cost of seeks as compared to transfer rate– 'Time to transfer' > 'Time to seek'
• For example, lets say– seek time = 10ms– Transfer rate = 100 MB/s
• To achieve seek time of 1% transfer rate– Block size will need to be = 100MB
69
Block Replication
• Namenode determines replica placement • Replica placements are rack aware
– Balance between reliability and performance• Attempts to reduce bandwidth• Attempts to improve reliability by putting replicas on multiple racks
– Default replication is 3• 1st replica on the local rack• 2nd replica on the local rack but different machine• 3rd replica on the different rack
– This policy may change/improve in the future
70
Data Correctness
• Use Checksums to validate data– Use CRC32
• File Creation– Client computes checksum per 512 byte
– Data Node stores the checksum
• File access– Client retrieves the data and checksum from Data Node
– If Validation fails, Client tries other replicas
71
Data Pipelining
• Client retrieves a list of Data Nodes on which to place replicas of a block
• Client writes block to the first Data Node
• The first Data Node forwards the data to the next Data Node in the Pipeline
• When all replicas are written, the Client moves on to write the next block in file
72
Client, Namenode, and Datanodes
• Namenode does NOT directly write or read data
– One of the reasons for HDFS’s Scalability
• Client interacts with Namenode to update Namenode’s HDFS namespace and retrieve block locations for writing and reading
• Client interacts directly with Datanode to read/write data
73
Name Node Metadata
• Meta-data in Memory– The entire metadata is in main memory– No demand paging of meta-data
• Types of Metadata– List of files– List of Blocks for each file– List of Data Nodes for each block– File attributes, e.g. creation time, replication factor
• A Transaction Log– Records file creations, file deletions. etc.
74
Namenode Memory Concerns
• For fast access Namenode keeps all block metadata in-memory– The bigger the cluster - the more RAM required
• Best for millions of large files (100mb or more) rather than billions• Will work well for clusters of 100s machines
• Hadoop 2+– Namenode Federations
• Each namenode will host part of the blocks• Horizontally scale the Namenode
– Support for 1000+ machine clusters
75
Using HDFS
Reading Data from HDFS
1. Create FileSystem
2. Open InputStream to a Path
3. Copy bytes using IOUtils
4. Close Stream
77
1: Create FileSystem
• FileSystem fs = FileSystem.get(new Configuration());
– If you run with yarn command, DistributedFileSystem (HDFS) will be created
• Utilizes fs.default.name property from configuration
• Recall that Hadoop framework loads core-site.xml which sets property to hdfs (hdfs://localhost:8020)
78
2: Open Input Stream to a Path
...InputStream input = null;try {input = fs.open(fileToRead);...• fs.open returns org.apache.hadoop.fs.FSDataInputStream
– Another FileSystem implementation will return their own custom implementation of InputStream
• Opens stream with a default buffer of 4k• If you want to provide your own buffer size use
– fs.open(Path f, int bufferSize)
79
3: Copy bytes using IOUtils
IOUtils.copyBytes(inputStream, outputStream, buffer);
• Copy bytes from InputStream to OutputStream
• Hadoop’s IOUtils makes the task simple
– buffer parameter specifies number of bytes to buffer at a time
80
4: Close Stream
...
} finally {
IOUtils.closeStream(input);
...
• Utilize IOUtils to avoid boiler plate code that catches IOException
81
ReadFile.java Example
public class ReadFile {
public static void main(String[] args) throws IOException {
Path fileToRead = new Path("/user/sample/sonnets.txt");
FileSystem fs = FileSystem.get(new Configuration()); // 1: Open FileSystem
InputStream input = null;
try {
input = fs.open(fileToRead); // 2: Open InputStream
IOUtils.copyBytes(input, System.out, 4096); // 3: Copy from Input to Output
} finally {
IOUtils.closeStream(input); // 4: Close stream
}
}
}
$ yarn jar my-hadoop-examples.jar hdfs.ReadFile
82
Reading Data - Seek
• FileSystem.open returns FSDataInputStream
– Extension of java.io.DataInputStream
– Supports random access and reading via interfaces:
• PositionedReadable : read chunks of the stream
• Seekable : seek to a particular position in the stream
83
Seeking to a Position
• FSDataInputStream implements Seekableinterface– void seek(long pos) throws IOException
• Seek to a particular position in the file
• Next read will begin at that position• If you attempt to seek past the file boundary IOException is emitted
• Somewhat expensive operation – strive for streaming and not seeking
– long getPos() throws IOException• Returns the current position/offset from the beginning of the
stream/file
84
SeekReadFile.java Examplepublic class SeekReadFile {
public static void main(String[] args) throws IOException {
Path fileToRead = new Path("/user/sample/readMe.txt");
FileSystem fs = FileSystem.get(new Configuration());
FSDataInputStream input = null;
try {
input = fs.open(fileToRead);
System.out.print("start postion=" + input.getPos() + ": ");
IOUtils.copyBytes(input, System.out, 4096, false);
input.seek(11);
System.out.print("start postion=" + input.getPos() + ": ");
IOUtils.copyBytes(input, System.out, 4096, false);
input.seek(0);
System.out.print("start postion=" + input.getPos() + ": ");
IOUtils.copyBytes(input, System.out, 4096, false);
} finally {
IOUtils.closeStream(input);
}
}
}
85
Run SeekReadFile Example
$ yarn jar my-hadoop-examples.jar hdfs.SeekReadFile
start position=0: Hello from readme.txt
start position=11: readme.txt
start position=0: Hello from readme.txt
86
Write Data
1. Create FileSystem instance
2. Open OutputStream
– FSDataOutputStream in this case
– Open a stream directly to a Path from FileSystem
– Creates all needed directories on the provided path
3. Copy data using IOUtils
87
WriteToFile.java Example
public class WriteToFile {public static void main(String[] args) throws IOException {
String textToWrite = "Hello HDFS! Elephants are awesome!\n";InputStream in = new BufferedInputStream(
new ByteArrayInputStream(textToWrite.getBytes()));Path toHdfs = new Path("/user/sample/writeMe.txt");Configuration conf = new Configuration();FileSystem fs = FileSystem.get(conf); // 1: Create FileSystem instanceFSDataOutputStream out = fs.create(toHdfs); // 2: Open OutputStreamIOUtils.copyBytes(in, out, conf); // 3: Copy Data
}}
88
Run WriteToFile
$ yarn jar my-hadoop-examples.jar hdfs.WriteToFile
$ hdfs dfs -cat /user/sample/writeMe.txt
Hello HDFS! Elephants are awesome!
90
MapReduce and YARN
Hadoop MapReduce
• Model for processing large amounts of data in parallel– On commodity hardware– Lots of nodes
• Derived from functional programming– Map and reduce functions
• Can be implemented in multiple languages– Java, C++, Ruby, Python (etc...)
92
The MapReduce Model
• Imposes key-value input/output• Defines map and reduce functions
map: (K1,V1) → list (K2,V2)reduce: (K2,list(V2)) → list (K3,V3)1. Map function is applied to every input key-value pair2. Map function generates intermediate key-value pairs3. Intermediate key-values are sorted and grouped by key4. Reduce is applied to sorted and grouped intermediate key-values5. Reduce emits result key-values
93
MapReduce Programming Model
94
MapReduce in Hadoop (1)
95
MapReduce in Hadoop (2)
96
MapReduce Framework
• Takes care of distributed processing and coordination
• Scheduling– Jobs are broken down into smaller chunks called tasks.– These tasks are scheduled
• Task Localization with Data– Framework strives to place tasks on the nodes that host
the segment of data to be processed by that specific task– Code is moved to where the data is
97
MapReduce Framework
• Error Handling
– Failures are an expected behavior so tasks are automatically re-tried on other machines
• Data Synchronization
– Shuffle and Sort barrier re-arranges and moves data between machines
– Input and output are coordinated by the framework
98
Map Reduce 2.0 on YARN
• Yet Another Resource Negotiator (YARN)• Various applications can run on YARN
– MapReduce is just one choice (the main choice at this point)– http://wiki.apache.org/hadoop/PoweredByYarn
• YARN was designed to address issues with MapReduce1– Scalability issues (max ~4,000 machines)– Inflexible Resource Management
• MapReduce1 had slot based model
99
MapReduce1 vs. YARN
• MapReduce1 runs on top of JobTracker and TaskTracker daemons– JobTracker schedules tasks, matches task with TaskTrackers– JobTracker manages MapReduce Jobs, monitors progress– JobTracker recovers from errors, restarts failed and slow tasks
• MapReduce1 has inflexible slot-based memory management model– Each TaskTracker is configured at start-up to have N slots– A task is executed in a single slot– Slots are configured with maximum memory on cluster start-up– The model is likely to cause over and under utilization issues
100
MapReduce1 vs. YARN (cont.)
• YARN addresses shortcomings of MapReduce1– JobTracker is split into 2 daemons
• ResourceManager - administers resources on the cluster• ApplicationMaster - manages applications such as MapReduce
– Fine-Grained memory management model• ApplicationMaster requests resources by asking for
“containers” with a certain memory limit (ex 2G)• YARN administers these containers and enforces memory usage• Each Application/Job has control of how much memory to
request
101
Daemons
• YARN Daemons– Node Manger
• Manages resources of a single node• There is one instance per node in the cluster
– Resource Manager• Manages Resources for a Cluster• Instructs Node Manager to allocate resources• Application negotiates for resources with Resource Manager• There is only one instance of Resource Manager
• MapReduce Specific Daemon– MapReduce History Server
• Archives Jobs’ metrics and meta-data
102
Old vs. New Java API
• There are two flavors of MapReduce API which became known as Old and New
• Old API classes reside under– org.apache.hadoop.mapred
• New API classes can be found under– org.apache.hadoop.mapreduce– org.apache.hadoop.mapreduce.lib
• We will use new API exclusively• New API was re-designed for easier evolution• Early Hadoop versions deprecated old API but deprecation was removed• Do not mix new and old API
103
Developing First MapReduce Job
MapReduce
• Divided in two phases– Map phase
– Reduce phase
• Both phases use key-value pairs as input and output
• The implementer provides map and reduce functions
• MapReduce framework orchestrates splitting, and distributing of Map and Reduce phases– Most of the pieces can be easily overridden
105
MapReduce
• Job – execution of map and reduce functions to accomplish a task
– Equal to Java’s main
• Task – single Mapper or Reducer
– Performs work on a fragment of data
106
Map Reduce Flow of Data
107
First Map Reduce Job
• StartsWithCount Job
– Input is a body of text from HDFS
• In this case hamlet.txt
– Split text into tokens
– For each first letter sum up all occurrences
– Output to HDFS
108
Word Count Job
109
Starts With Count Job
1. Configure the Job– Specify Input, Output, Mapper, Reducer and Combiner
2. Implement Mapper– Input is text – a line from hamlet.txt– Tokenize the text and emit first character with a count of
1 - <token, 1>
3. Implement Reducer– Sum up counts for each letter– Write out the result to HDFS
4. Run the job
110
1: Configure Job
• Job class– Encapsulates information about a job– Controls execution of the job
Job job = Job.getInstance(getConf(), "StartsWithCount");
• A job is packaged within a jar file– Hadoop Framework distributes the jar on your behalf– Needs to know which jar file to distribute– The easiest way to specify the jar that your job resides in is by calling
job.setJarByClassjob.setJarByClass(getClass());
– Hadoop will locate the jar file that contains the provided class
111
1: Configure Job - Specify Input
TextInputFormat.addInputPath(job, new Path(args[0]));
job.setInputFormatClass(TextInputFormat.class);
• Can be a file, directory or a file pattern– Directory is converted to a list of files as an input
• Input is specified by implementation of InputFormat - in this case TextInputFormat– Responsible for creating splits and a record reader– Controls input types of key-value pairs, in this case LongWritable
and Text– File is broken into lines, mapper will receive 1 line at a time
112
Side Note – Hadoop IO Classes
• Hadoop uses it’s own serialization mechanism for writing data in and out of network, database or files– Optimized for network serialization– A set of basic types is provided– Easy to implement your own
• org.apache.hadoop.io package– LongWritable for Long– IntWritable for Integer– Text for String– Etc...
113
1: Configure Job – Specify Output
TextOutputFormat.setOutputPath(job, new Path(args[1]));job.setOutputFormatClass(TextOutputFormat.class);
• OutputFormat defines specification for outputting data from Map/Reduce job
• Count job utilizes an implemenation ofOutputFormat - TextOutputFormat
– Define output path where reducer should place its output• If path already exists then the job will fail
– Each reducer task writes to its own file• By default a job is configured to run with a single reducer
– Writes key-value pair as plain text
114
1: Configure Job – Specify Output
job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);
• Specify the output key and value types for both mapper and reducer functions– Many times the same type– If types differ then use
• setMapOutputKeyClass()• setMapOutputValueClass()
115
1: Configure Job
• Specify Mapper, Reducer and Combiner– At a minimum will need to implement these classes
– Mappers and Reducer usually have same output key
job.setMapperClass(StartsWithCountMapper.class);
job.setReducerClass(StartsWithCountReducer.class);
job.setCombinerClass(StartsWithCountReducer.class);
116
1: Configure Job
• job.waitForCompletion(true)
– Submits and waits for completion
– The boolean parameter flag specifies whether output should be written to console
– If the job completes successfully ‘true’ is returned, otherwise ‘false’ is returned
117
Our Count Job is configured to
• Chop up text files into lines• Send records to mappers as key-value pairs
– Line number and the actual value
• Mapper class is StartsWithCountMapper– Receives key-value of <IntWritable,Text>– Outputs key-value of <Text, IntWritable>
• Reducer class is StartsWithCountReducer– Receives key-value of <Text, IntWritable>– Outputs key-values of <Text, IntWritable> as text
• Combiner class is StartsWithCountReducer
118
1: Configure Count Job
public class StartsWithCountJob extends Configured implements Tool{@Overridepublic int run(String[] args) throws Exception {
Job job = Job.getInstance(getConf(), "StartsWithCount");job.setJarByClass(getClass());
// configure output and input sourceTextInputFormat.addInputPath(job, new Path(args[0]));job.setInputFormatClass(TextInputFormat.class);
// configure mapper and reducerjob.setMapperClass(StartsWithCountMapper.class);job.setCombinerClass(StartsWithCountReducer.class);job.setReducerClass(StartsWithCountReducer.class);
119
StartsWithCountJob.java (cont.)
// configure outputTextOutputFormat.setOutputPath(job, new Path(args[1]));job.setOutputFormatClass(TextOutputFormat.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);
return job.waitForCompletion(true) ? 0 : 1;}public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new StartsWithCountJob(), args);
System.exit(exitCode);}
}
120
2: Implement Mapper class
• Class has 4 Java Generics parameters– (1) input key (2) input value (3) output key (4) output value– Input and output utilizes hadoop’s IO framework
• org.apache.hadoop.io
• Your job is to implement map() method– Input key and value– Output key and value– Logic is up to you
• map() method injects Context object, use to:– Write output– Create your own counters
121
2: Implement Mapper
public class StartsWithCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable countOne = new IntWritable(1);private final Text reusableText = new Text();
@Overrideprotected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {StringTokenizer tokenizer = new StringTokenizer(value.toString());while (tokenizer.hasMoreTokens()) {
reusableText.set(tokenizer.nextToken().substring(0, 1));context.write(reusableText, countOne);
}}
}
122
3: Implement Reducer
• Analogous to Mapper – generic class with four types– (1) input key (2) input value (3) output key (4) output value– The output types of map functions must match the input types of reduce
function• In this case Text and IntWritable
– Map/Reduce framework groups key-value pairs produced by mapper by key
• For each key there is a set of one or more values• Input into a reducer is sorted by key• Known as Shuffle and Sort
– Reduce function accepts key->setOfValues and outputs key-value pairs• Also utilizes Context object (similar to Mapper)
123
3: Implement Reducer
public class StartsWithCountReducer extendsReducer<Text, IntWritable, Text, IntWritable> {
@Overrideprotected void reduce(Text token,
Iterable<IntWritable> counts,Context context) throws IOException, InterruptedException {int sum = 0;
for (IntWritable count : counts) {sum+= count.get();
}context.write(token, new IntWritable(sum));
}}
124
3: Reducer as a Combiner
• Combine data per Mapper task to reduce amount of data transferred to reduce phase
• Reducer can very often serve as a combiner– Only works if reducer’s output key-value pair types are the
same as mapper’s output types
• Combiners are not guaranteed to run– Optimization only– Not for critical logic
• More about combiners later
125
4: Run Count Job
$ yarn jar my-hadoop-examples.jar \mr.wordcount.StartsWithCountJob \/user/sample/readme.txt \/user/sample/wordcount
126
Output of Count Job
• Output is written to the configured output directory
– /user/sample/wordCount/
• One output file per Reducer
– part-r-xxxxx format
• Output is driven by TextOutputFormat class
127
$yarn command
• yarn script with a class argument command launches a JVM and executes the provided Job
$ yarn jar HadoopSamples.jar \mr.wordcount.StartsWithCountJob \/user/sample/hamlet.txt \/user/sample/wordcount/
• You could use straight java but yarn script is more convenient– Adds hadoop’s libraries to CLASSPATH– Adds hadoop’s configurations to Configuration object
• Ex: core-site.xml, mapred-site.xml, *.xml
– You can also utilize $HADOOP_CLASSPATH environment variable
128
Input and Output
MapReduce Theory
• Map and Reduce functions produce input and output– Input and output can range from Text to Complex data
structures– Specified via Job’s configuration– Relatively easy to implement your own
• Generally we can treat the flow asmap: (K1,V1) → list (K2,V2)reduce: (K2,list(V2)) → list (K3,V3)– Reduce input types are the same as map output types
130
Map Reduce Flow of Data
map: (K1,V1) → list (K2,V2)
reduce: (K2,list(V2)) → list (K3,V3)
131
Key and Value Types
• Utilizes Hadoop’s serialization mechanism for writing data in and out of network, database or files– Optimized for network serialization– A set of basic types is provided– Easy to implement your own
• Extends Writable interface– Framework’s serialization mechanisms– Defines how to read and write fields– org.apache.hadoop.io package
132
Key and Value Types
• Keys must implement WritableComparableinterface– Extends Writable and java.lang.Comparable<T>– Required because keys are sorted prior reduce phase
• Hadoop is shipped with many default implementations of WritableComparable<T>– Wrappers for primitives (String, Integer, etc...)– Or you can implement your own
133
WritableComparable<T>
Implementations
Hadoop’s Class Explanation
BooleanWritable Boolean implementation
BytesWritable Bytes implementation
DoubleWritable Double implementation
FloatWritable Float implementation
IntWritable Int implementation
LongWritable Long implementation
NullWritable Writable with no data
134
Implement Custom
WritableComparable<T>
• Implement 3 methods– write(DataOutput)
• Serialize your attributes
– readFields(DataInput)• De-Serialize your attributes
– compareTo(T)• Identify how to order your objects• If your custom object is used as the key it will be sorted
prior to reduce phase
135
BlogWritable – Implemenation
of WritableComparable<T>
public class BlogWritable implements
WritableComparable<BlogWritable> {
private String author;
private String content;
public BlogWritable(){}
public BlogWritable(String author, String content) {
this.author = author;
this.content = content;
}
public String getAuthor() {
return author;
public String getContent() {
return content;
...
...
136
BlogWritable – Implemenation
of WritableComparable<T>
...
@Override
public void readFields(DataInput input) throws IOException {
author = input.readUTF();
content = input.readUTF();
}
@Override
public void write(DataOutput output) throws IOException {
output.writeUTF(author);
output.writeUTF(content);
}
@Override
public int compareTo(BlogWritable other) {
return author.compareTo(other.author);
}
}
137
Mapper
• Extend Mapper class– Mapper<KeyIn, ValueIn, KeyOut, ValueOut>
• Simple life-cycle1. The framework first calls setup(Context)2. for each key/value pair in the split:
• map(Key, Value, Context)3. Finally cleanup(Context) is called
138
InputSplit
• Splits are a set of logically arranged records– A set of lines in a file– A set of rows in a database table
• Each instance of mapper will process a single split– Map instance processes one record at a time
• map(k,v) is called for each record
• Splits are implemented by extending InputSplitclass
139
InputSplit
• Framework provides many options for InputSplit implementations
– Hadoop’s FileSplit
– HBase’s TableSplit
• Don’t usually need to deal with splits directly
– InputFormat’s responsibility
140
Combiner
• Runs on output of map function• Produces outpu
map: (K1,V1) → list (K2,V2)combine: (K2,list(V2)) → list (K2,V2)reduce: (K2,list(V2)) → list (K3,V3)
• Optimization to reduce bandwidth– NO guarantees on being called– Maybe only applied to a sub-set of map outputs
• Often is the same class as Reducer• Each combine processes output from a single split
141
Combiner Data Flow
142
Sample StartsWithCountJob
Run without Combiner
143
Sample StartsWithCountJob
Run with Combiner
144
Specify Combiner Function
• To implement Combiner extend Reducer class
• Set combiner on Job class
–
job.setCombinerClass(StartsWithCountReducer.class);
145
Reducer
• Extend Reducer class– Reducer<KeyIn, ValueIn, KeyOut, ValueOut>– KeyIn and ValueIn types must match output types of mapper
• Receives input from mappers’ output– Sorted on key– Grouped on key of key-values produced by mappers– Input is directed by Partitioner implementation
• Simple life-cycle – similar to Mapper– The framework first calls setup(Context)– for each key → list(value) calls
• reduce(Key, Values, Context)
– Finally cleanup(Context) is called
146
Reducer
• Can configure more than 1 reducer– job.setNumReduceTasks(10);– mapreduce.job.reduces property
• job.getConfiguration().setInt("mapreduce.job.reduces", 10)
• Partitioner implementation directs key-value pairs to the proper reducer task– A partition is processed by a reduce task
• # of partitions = # or reduce tasks
– Default strategy is to hash key to determine partition
implemented by HashPartitioner<K, V>
147
Partitioner Data Flow
148
HashPartitioner
public class HashPartitioner<K, V> extends Partitioner<K, V> {public int getPartition(K key, V value, int numReduceTasks) {
return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;}
}
• Calculate Index of Partition:– Convert key’s hash into non-negative number
• Logical AND with maximum integer value
– Modulo by number of reduce tasks
• In case of more than 1 reducer– Records distributed evenly across available reduce tasks
• Assuming a good hashCode() function
– Records with same key will make it into the same reduce task– Code is independent from the # of partitions/reducers specified
149
Custom Partitioner
public class CustomPartitionerextends Partitioner<Text, BlogWritable>{
@Overridepublic int getPartition(Text key, BlogWritable blog,
int numReduceTasks) {int positiveHash =blog.getAuthor().hashCode()& Integer.MAX_VALUE;
//Use author’s hash only, AND with//max integer to get a positive value
return positiveHash % numReduceTasks;}
}
• All blogs with the same author will end up in the same reduce task
150
Component Overview
151
Improving Hadoop
Improving Hadoop
• Core Hadoop is complicated so some tools were added to make things easier
• Hadoop Distributions collect these tools and release them as a whole package
153
Noticeable Distributions
• Cloudera
• MapR
• HortonWorks
• Amazon EMR
154
HADOOP Technology Eco System
155
Improving Programmability
• Pig: Programming language that simplifies Hadoop actions: loading, transforming and sorting data
• Hive: enables Hadoop to operate as data warehouse using SQL-like syntax.
156
Pig
• “is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. “
• Top Level Apache Project– http://pig.apache.org
• Pig is an abstraction on top of Hadoop– Provides high level programming language designed for data processing– Converted into MapReduce and executed on Hadoop Clusters
• Pig is widely accepted and used– Yahoo!, Twitter, Netflix, etc...
157
Pig and MapReduce
• MapReduce requires programmers– Must think in terms of map and reduce functions– More than likely will require Java programmers
• Pig provides high-level language that can be used by– Analysts– Data Scientists– Statisticians– Etc...
• Originally implemented at Yahoo! to allow analysts to access data
158
Pig’s Features
• Join Datasets
• Sort Datasets
• Filter
• Data Types
• Group By
• User Defined Functions
159
Pig’s Use Cases
• Extract Transform Load (ETL)– Ex: Processing large amounts of log data
• clean bad entries, join with other data-sets
• Research of “raw” information– Ex. User Audit Logs
– Schema maybe unknown or inconsistent
– Data Scientists and Analysts may like Pig’s data transformation paradigm
160
Pig Components
• Pig Latin– Command based language– Designed specifically for data transformation and flow expression
• Execution Environment– The environment in which Pig Latin commands are executed– Currently there is support for Local and Hadoop modes
• Pig compiler converts Pig Latin to MapReduce– Compiler strives to optimize execution– You automatically get optimization improvements with Pig updates
161
Pig Code Example
162
Hive
• Data Warehousing Solution built on top of Hadoop
• Provides SQL-like query language named HiveQL– Minimal learning curve for people with SQL expertise
– Data analysts are target audience
• Early Hive development work started at Facebook in 2007
• Today Hive is an Apache project under Hadoop– http://hive.apache.org
163
Hive Provides
• Ability to bring structure to various data formats
• Simple interface for ad hoc querying, analyzing and summarizing large amounts of data
• Access to files on various data stores such as HDFS and HBase
164
When not to use Hive
• Hive does NOT provide low latency or real time queries
• Even querying small amounts of data may take minutes
• Designed for scalability and ease-of-use rather than low latency responses
165
Hive
• Translates HiveQL statements into a set of MapReduce Jobs which are then executed on a Hadoop Cluster
166
Hive Metastore
• To support features like schema(s) and data partitioning Hive keeps its metadata in a Relational Database
– Packaged with Derby, a lightweight embedded SQL DB• Default Derby based is good for evaluation an testing
• Schema is not shared between users as each user has their own instance of embedded Derby
• Stored in metastore_db directory which resides in the directory that hive was started from
– Can easily switch another SQL installation such as MySQL
167
Hive Architecture
168
1: Create a Table
• Let’s create a table to store data from $PLAY_AREA/data/user-posts.txt
169
1: Create a Table
170
2: Load Data Into a Table
171
3: Query Data
172
3: Query Data
173
Databases and DB Connectivity
• HBase: column oriented database that runs on HDFS.
• Sqoop: a tool designed to import data from relational databases (HDFS or Hive).
174
HBase
• Distributed column-oriented database built on top of HDFS, providing Big Table-like capabilities for Hadoop
175
When do we use HBase?
• Huge volumes of randomly accessed data.
• HBase is at its best when it’s accessed in a distributed fashion by many clients.
• Consider HBase when you’re loading data by key, searching data by key (or range), serving data by key, querying data by key or when storing data by row that doesn’t conform well to a schema.
176
When not to use Hbase
• HBase doesn’t use SQL, don’t have an optimizer, doesn’t support in transactions or joins.
• If you need those things, you probably can’t use Hbase
177
HBase Example
Example: create ‘blogposts’, ‘post’, ‘image’ ---create table
put ‘blogposts’, ‘id1′, ‘post:title’, ‘Hello World’ ---insert value
put ‘blogposts’, ‘id1′, ‘post:body’, ‘This is a blog post’ ---insert value
put ‘blogposts’, ‘id1′, ‘image:header’, ‘image1.jpg’ ---insert value
get ‘blogposts’, ‘id1′ ---select records
178
Sqoop
• Sqoop is a command line tool for moving data from RDBMS to Hadoop
• Uses MapReduce program or Hive to load the data
• Can also export data from HBase to RDBMS
• Comes with connectors to MySQL, PostgreSQL, Oracle, SQL Server and DB2.
Example:
$bin/sqoop import --connect 'jdbc:sqlserver://10.80.181.127;username=dbuser;password=dbpasswd;database=tpch' \
--table lineitem --hive-import
$bin/sqoop export --connect 'jdbc:sqlserver://10.80.181.127;username=dbuser;password=dbpasswd;database=tpch' --table lineitem --export-dir /data/lineitemData
179
Improving Hadoop – More useful tools
• For improving coordination: Zookeeper
• For Improving log collection: Flume
• For improving scheduling/orchestration: Oozie
• For Monitoring: Chukwa
• For Improving UI: Hue
180
ZooKeeper
• ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services
• It allows distributed processes to coordinate with each other through a shared hierarchal namespace which is organized similarly to a standard file system
• ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions
181
Flume
• Flume is a distributed system for collecting log data from many sources, aggregating it, and writing it to HDFS
• Flume maintains a central list of ongoing data flows, stored redundantly in Zookeeper.
182
Oozie
• Oozie is a workflow scheduler system to manage Hadoop jobs
• Oozie workflow is a collection of actions arranged in a control dependency DAG specifying a sequence of actions execution
• The Oozie Coordinator system allows the user to define workflow execution bases on intervals or on-demand
183
Spark
Fast and general MapReduce-like engine for large-scale data processing
• Fast
– In memory data storage for very fast interactive queries Up to 100 times faster then Hadoop
• General
– Unified platform that can combine: SQL, Machine Learning , Streaming , Graph & Complex analytics
• Ease of use
– Can be developed in Java, Scala or Python
• Integrated with Hadoop
– Can read from HDFS, HBase, Cassandra, and any Hadoop data source.
184
Spark is the Most Active Open Source
Project in Big Data
185
The Spark Community
186
Key Concepts
Resilient Distributed Datasets
• Collections of objects spread across a cluster, stored in RAM or on Disk
• Built through parallel transformations
• Automatically rebuilt on failure
Operations
• Transformations(e.g. map, filter, groupBy)
• Actions(e.g. count, collect, save)
Write programs in terms of transformations on
distributed datasets
187
Unified Platform
• Continued innovation bringing new functionality, e.g.:• Java 8 (Closures, LambaExpressions)• Spark SQL (SQL on Spark, not just Hive)• BlinkDB(Approximate Queries)• SparkR(R wrapper for Spark)
188
Language Support
189
Data Sources
• Local Files– file:///opt/httpd/logs/access_log
• S3• Hadoop Distributed Filesystem
– Regular files, sequence files, any other Hadoop InputFormat
• Hbase• Can also read from any other Hadoop data source.
190
Resilient Distributed Datasets (RDD)
• Spark revolves around RDDs
• Fault-tolerant collection of elements that can be operated on in parallel
– Parallelized Collection: Scala collection which is run in parallel
– Hadoop Dataset: records of files supported by Hadoop
191
Hadoop Tools
192
Hadoop cluster
Cluster of machine running Hadoop at Yahoo! (credit: Yahoo!)
193
Big Data and NoSQL
The Challenge
• We want scalable, durable, high volume, high velocity, distributed data storage that can handle non-structured data and that will fit our specific need
• RDBMS is too generic and doesn’t cut it any more –it can do the job but it is not cost effective to our usages
195
The Solution: NoSQL
• Let’s take some parts of the standard RDBMS out to and design the solution to our specific uses
• NoSQL databases have been around for ages under different names/solutions
196
Example Comparison: RDBMS vs. Hadoop
Typical Traditional RDBMS Hadoop
Data Size Gigabytes Petabytes
Access Interactive and Batch Batch – NOT Interactive
Updates Read / Write many times Write once, Read many times
Structure Static Schema Dynamic Schema
Scaling Nonlinear Linear
Query Response
Time
Can be near immediate Has latency (due to batch processing)
197
Best Used For:
Structured or Not (Flexibility)
Scalability of Storage/Compute
Complex Data Processing
Cheaper compared to RDBMS
Relational Database
Best Used For:
Interactive OLAP Analytics
(<1sec)
Multistep Transactions
100% SQL Compliance
Best when used together
Hadoop And Relational Database
198
The NOSQL Movement
• NOSQL is not a technology – it’s a concept.
• We need high performance, scale out abilities or an agile structure.
• We are now willing to sacrifice our sacred cows: consistency, transactions.
• Over 200 different brands and solutions (http://nosql-database.org/).
199
NoSQL, NOSQL or NewSQL
• NoSQL is not No to SQL
• NoSQL is not Never SQL
• NOSQL = Not Only SQL
200
Why NoSQL?
• Some applications need very few database features, but need high scale.
• Desire to avoid data/schema pre-design altogether for simple applications.
• Need for a low-latency, low-overhead API to access data.
• Simplicity -- do not need fancy indexing – just fast lookup by primary key.
201
Why NoSQL? (cont.)
• Developer friendly, DBAs not needed (?).
• Schema-less.
• Agile: non-structured (or semi-structured).
• In Memory.
• No (or loose) Transactions.
• No joins.
202
203
Is NoSQL a RDMS Replacement?
NOWell... Sometimes it does…
204
RDBMS vs. NoSQL
Rationale for choosing a persistent store:Relational Architecture NoSQL Architecture
High value, high density, complexData
Low value, low density, simple data
Complex data relationships Very simple relationships
Schema-centric Schema-free, unstructured or semistructured Data
Designed to scale up & out Distributed storage and processing
Lots of general purposefeatures/functionality
Stripped down, special purposedata store
High overhead ($ per operation) Low overhead ($ per operation)
205
Scalability and Consistency
Scalability
• NoSQL is sometimes very easy to scale out
• Most have dynamic data partitioning and easy data distribution
• But distributed system always come with a price: The CAP Theorem and impact on ACID transactions
207
ACID Transactions
Most DBMS are built with ACID transactions in mind:• Atomicity: All or nothing, performs write operations as a single
transaction• Consistency: Any transaction will take the DB from one
consistent state to another with no broken constraints, ensures replicas are identical on different nodes
• Isolation: Other operations cannot access data that has been modified during a transaction that has not been completed yet
• Durability: Ability to recover the committed transaction updates against any kind of system failure (transaction log)
208
ACID Transactions (cont.)
• ACID is usually implemented by a locking mechanism/manager
• Distributed systems central locking can be a bottleneck in that system
• Most NoSQL does not use/limit the ACID transactions and replaces it with something else…
209
• The CAP theorem states that in a distributed/partitioned application, you can only pick two of the following three characteristics:
– Consistency.
– Availability.
– Partition Tolerance.
CAP Theorem
210
CAP in Practice
211
NoSQL BASE
• NoSQL usually provide BASE characteristics instead of ACID.
BASE stands for:– Basically Available– Soft State– Eventual Consistency
• It means that when an update is made in one place, the other partitions will see it over time - there might be an inconsistency window
• read and write operations complete more quickly, lowering latency
212
Eventual Consistency
213
Types of NoSQL
NoSQL Taxonomy
Type Examples
Key-Value Store
Document Store
Column Store
Graph Store
215
SQL comfort zone
siz
e
Complex
Typical
RDBMS
Key
ValueColumn
Store
Graph
DATABASE
Document
Database Performance
NoSQL Map
216
Key Value Store
• Distributed hash tables.• Very fast to get a single value.• Examples:
– Amazon DynamoDB– Berkeley DB– Redis– Riak– Cassandra
217
Document Store
• Similar to Key/Value, but value is a document
• JSON or something similar, flexible schema
• Agile technology
• Examples:– MongoDB
– CouchDB
– CouchBase
218
Column Store
• One key, multiple attributes
• Hybrid row/column
• Examples:– Google BigTable
– Hbase
– Amazon’s SimpleDB
– Cassandra
219
How Records are Organized?
• This is a logical table in RDBMS systems
• Its physical organization is just like the logical one: column by column, row by row
Row 1
Row 2
Row 3
Row 4
Col 1 Col 2 Col 3 Col 4
220
Query Data
• When we query data, records are read at the order they are organized in the physical structure
• Even when we query a single column, we still need to read the entire table and extract the column
Row 1
Row 2
Row 3
Row 4
Col 1 Col 2 Col 3 Col 4
Select Col2 From MyTable
Select *From MyTable
221
How Does Column Store Keep Data
Organization in row store Organization in column store
Select Col2 From MyTable
222
Graph Store
• Inspired by Graph Theory• Data model: Nodes, relationships, properties
on both• Relational Database have very hard time to
represent a graph in the Database• Example:
– Neo4j– InfiniteGraph– RDF
223
• An abstract representation of a set of objects where some pairs are connected by links.
• Object (Vertex, Node) – can have attributes like name and value
• Link (Edge, Arc, Relationship) – can have attributes like type and name or date
What is Graph
NODEEdge
224
Graph Types
Undirected Graph
Directed Graph
Pseudo Graph
Multi Graph
NODEEdge
NODE
NODEEdge
NODE
NODE
NODE NODE
225
More Graph Types
Weighted Graph
Labeled Graph
Property Graph
NODE10
NODE
NODELike
NODE
NODE NODEfriend, date 2015
Name:yosi,
Age:40
Name:ami,
Age:30
226
Relationships
ID:1TYPE:F
NAME:alice
ID:2TYPE:M
NAME:bob
ID:1TYPE:G
NAME:NoSQL
ID:1TYPE:F
NAME:dafna
TYPE: member
Since:2012
227
228
Q&A
Conclusion
• The Challenge of Big Data
• Hadoop Basics: HDFS, MapReduce and YARN
• Improving Hadoop and Tools
• NoSQL and RDBMS
230