large-scale file systems and map-reduce single-node architecture memory disk cpu google example: 20+...
Post on 16-Jan-2016
214 Views
Preview:
TRANSCRIPT
Large-scale file systems and Map-Reduce
Single-node architecture
Memory
Disk
CPU
Google example:
• 20+ billion web pages x 20KB = 400+ Terabyte• 1 computer reads 30-35 MB/sec from disk• ~4 months to read the web
• ~1,000 hard drives to store the web• Takes even more to do something useful with the data• New standard architecture is emerging:• Cluster of commodity Linux nodes• Gigabit ethernet interconnect
Slide based on www.mmds.com
Distributed File Systems
• Files are very large, read/append.• They are divided into chunks.– Typically 64MB to a chunk.
• Chunks are replicated at several compute-nodes.• A master (possibly replicated) keeps track of all
locations of all chunks.
Slide based on www.mmds.com
Commodity clusters: compute nodes• Organized into racks.• Intra-rack connection typically gigabit speed.• Inter-rack connection faster by a small factor.• Recall that chunks are replicated
Some implementations:• GFS (Google File System –
proprietary). In Aug 2006 Google had ~450,000 machines
• HDFS (Hadoop Distributed File System – open source).
• CloudStore (Kosmix File System, open source).
Slide based on www.mmds.com
• Problems with Large-scale computing on commodity hardware
• Challenges:– How do you distribute computation?– How can we make it easy to write distributed
programs?– Machines fail:• One server may stay up 3 years (1,000 days)• If you have 1,000 servers, expect to loose 1/day• People estimated Google had ~1M machines in 2011
– 1,000 machines fail every day!
Slide based on www.mmds.com
Slide based on www.mmds.com
• Issue: Copying data over a network takes time• Idea:– Bring computation close to the data– Store files multiple times for reliability
• Map-reduce addresses these problems– Google’s computational/data manipulation model– Elegant way to work with big data– Storage Infrastructure – File system• Google: GFS. Hadoop: HDFS
– Programming model• Map-Reduce
Slide based on www.mmds.com
• Problem:– If nodes fail, how to store data persistently?
• Answer:– Distributed File System:• Provides global file namespace• Google GFS; Hadoop HDFS;
• Typical usage pattern– Huge files (100s of GB to TB)– Data is rarely updated in place– Reads and appends are common
Racks of Compute Nodes
File
Chunks
Replication
Slide based on www.mmds.com
3-way replication offiles, with copies ondifferent racks.
Replication
Slide based on www.mmds.com
Map-Reduce• You write two functions, Map and Reduce.– They each have a special form to be explained.
• System (e.g., Hadoop) creates a large number of tasks for each function.– Work is divided among tasks in a precise way.
Slide based on www.mmds.com
Map-Reduce Algorithms• Map tasks convert inputs to key-value pairs.– “keys” are not necessarily unique.
• Outputs of Map tasks are sorted by key, and each key is assigned to one Reduce task.
• Reduce tasks combine values associated with a key.
Slide based on www.mmds.com
Simple map-reduce example: Word Count• We have a large file of words, one word to a line• Count the number of times each distinct word
appears in the file• Sample application: analyze web server logs to find
popular URLs• Different scenarios:– Case 1: Entire file fits in main memory– Case 2: File too large for main mem, but all
<word, count> pairs fit in main mem– Case 3: File on disk, too many distinct words to
fit in memorySlide based on www.mmds.com
Word Count• Map task: For each word, e.g. CAT output (CAT,1)
• Total output: (w1,1), (w1,1), …., (w1,1) (w2,1), (w2,1), …., (w2,1) …… Hash each (w,1) to bucket h(w) in [0,r-1] in local intermediate file. r is the number of reducers
• Master: Group by key: (w1,[1,1,…,1]), (w2,[1,1,…,1]), Push group (w,[1,1,..,1]) to reducer h(w)
• Reduce task: Reducer h(w)
Read : (w,[1,1,…,1]) Aggregate: each (w,[1,1,…,1]) into (w,sum) Output: (w,sum) into common output file• Since addition is commutative and associative the map task could have sent : (w1,sum1), (w2,sum2), …
• Reduce task would receive: (wi,sumi,1), (wi,sumi,2), … (wj,sumj,1), (wj,sumj,2), … and output (wi,sumi), (wj,sumj), ….
Slide based on www.mmds.com
Partition Function
• Inputs to map tasks are created by contiguous splits of input file
• For reduce, we need to ensure that records with the same intermediate key end up at the same worker
• System uses a default partition function e.g., hash(key) mod R
• Sometimes useful to override – E.g., hash(hostname(URL)) mod R ensures URLs
from a host end up in the same output file
Slide based on www.mmds.com
Coordination
• Master data structures– Task status: (idle, in-progress, completed)– Idle tasks get scheduled as workers become
available– When a map task completes, it sends the master
the location and sizes of its R intermediate files, one for each reducer
– Master pushes this info to reducers• Master pings workers periodically to detect
failuresSlide based on www.mmds.com
Data flow• Input, final output are stored on a distributed file system
– Scheduler tries to schedule map tasks “close” to physical storage location of input data
• Intermediate results are stored on local FS of map and reduce workers
• Output is often input to another map-reduce task
• Master data structures– Task status: (idle, in-progress, completed)– Idle tasks get scheduled as workers become available– When a map task completes, it sends the master the location and sizes
of its R intermediate files, one for each reducer– Master pushes this info to reducers
• Master pings workers periodically to detect failures
Slide based on www.mmds.com
Failures
• Map worker failure– Map tasks completed or in-progress at worker
are reset to idle (result sits locally at worker)– Reduce workers are notified when task is
rescheduled on another worker• Reduce worker failure– Only in-progress tasks are reset to idle
• Master failure– Map-reduce task is aborted and client is notified
Slide based on www.mmds.com
How many Map and Reduce jobs?
• M map tasks, R reduce tasks• Rule of thumb:– Make M and R much larger than the number of
nodes in cluster– One DFS chunk per map is common– Improves dynamic load balancing and speeds
recovery from worker failure• Usually R is smaller than M, because output
is spread across R files
Slide based on www.mmds.com
Relational operators with map-reduce
Selection
Map task: If C(t) is true output pair (t,t)
Reduce task: With input (t,t) output t
Selection is not really suitable for map-reduce,everything could have been done in the map task
)(RC
Relational operators with map-reduce
Projection
Map task: Let t’ be the projection of t. Output pair (t’,t’)
Reduce task: With input (t’,[t’,t’,…,t’] ) output t’
Here the duplicate elimination is done by the reduce task
)(RL
Relational operators with map-reduce
• Union RSMap task: for each tuple t of the chunk of R or S output (t, t)Reduce task: input is (t,[t]) or (t,[t, t]). Output t
• Intersection R SMap task: for each tuple t of the chunk output (t,t)Reduce task: if input is (t,[t,t]), output t if input is (t,[t]) , output nothing
• Difference R – SMap task: for each tuple t of R output (t,R) for each tuple t of S output (t,S)Reduce task: if input is (t,[R]), output t if input is (t,[R,S]) , output nothing
Joining by Map-Reduce• Suppose we want to compute • R(A,B) JOIN S(B,C), using k Reduce tasks.– I.e., find tuples with matching B-values.
• R and S are each stored in a chunked file.
• Use a hash function h from B-values to k buckets.– Bucket = Reduce task.
• The Map tasks take chunks from R and S, and send:– Tuple R(a,b) to Reduce task h(b).
• Key = b value = R(a,b).– Tuple S(b,c) to Reduce task h(b).
• Key = b; value = S(b,c).
Slide based on www.mmds.com
Reducetask i
Map tasks sendR(a,b) if h(b) = i
Map tasks sendS(b,c) if h(b) = i
All (a,b,c) such thath(b) = i, and (a,b)is in R, and (b,c) isin S.
• Key point: If R(a,b) joins with S(b,c), then both tuples are sent to Reduce task h(b).
• Thus, their join (a,b,c) will be produced there and shipped to the output file.
Slide based on www.mmds.com
Mapping tuples in joins
Mapper for R(1,2)
R(1,2) (2, (R,1))
Mapper for R(4,2)R(4,2)
Mapper for S(2,3)
S(2,3)
Mapper for S(5,6)
S(5,6)
(2, (R,4))
(2, (S,3))
(5, (S,6))
Reducerfor B = 2
Reducerfor B = 5
(2, [(R,1), (R,4), (S,3)])
(5, [(S,6)])
Slide based on www.mmds.com
Output of the Reducers
Reducerfor B = 2
Reducerfor B = 5
(2, [(R,1), (R,4), (S,3)])
(5, [(S,6)])
(1,2,3), (4,2,3)
Slide based on www.mmds.com
Relational operators with map-reduce
Grouping and aggregation: A,agg(B)(R(A,B,C))
Map task: for each tuple (a,b,c) output (a,[b]) Reduce task: if input is (a,[b1, b2, …, bn]), output (a,agg(b1, b2, …, bn))
for example (a, b1+b2+ …+bn)
Matrix-vector multiplication using map-reduce
j=1
If vector doesn’t fit in main memory
Divide matrix and vector into stripes:
Each map task gets a chunk of stripe i of the matrixand the entire stripe i of the vector and producespairs
Reduce task i gets all pairs and producespairs
Slide based on www.mmds.com
Example:
MAPPERS:
REDUCERS:
Examples:
• Hamming distance 1 between bit-strings
• Matrix multiplication in one MR-round
• Matrix multiplication in two MR-rounds
• Three-way joins in two rounds and in one round
Relational operators with map-reduceThree-Way Join
• We shall consider a simple join of three relations, the natural join
R(A,B) ⋈ S(B,C) ⋈ T(C,D).
• One way: cascade of two 2-way joins, each implemented by map-reduce.
• Fine, unless the 2-way joins produce large intermediate relations.
Slide based on www.mmds.com
Another 3-Way Join
• Reduce processes use hash values of entire S(B,C) tuples as key.
• Choose a hash function h that maps B- and C-values to k buckets.
• There are k2 Reduce processes, one for each (B-bucket, C-bucket) pair.
Slide based on www.mmds.com
Job of the Reducers
• Each reducer gets, for certain B-values b and C-values c :
1. All tuples from R with B = b,2. All tuples from T with C = c, and3. The tuple S(b,c) if it exists.
• Thus it can create every tuple of the form (a, b, c, d) in the join.
Slide based on www.mmds.com
Mapping for 3-Way Join
We map each tuple S(b,c) to ((h(b), h(c)), (S, b, c)).
We map each R(a,b) tuple to ((h(b), y), (R, a, b)) for all y = 1, 2,…,k.
We map each T(c,d) tuple to ((x, h(c)), (T, c, d)) for all x = 1, 2,…,k.
Keys Values
Aside: even normalmap-reduce allowsinputs to map toseveral key-valuepairs.
Slide based on www.mmds.com
Assigning Tuples to Reducers
h(b) = 0
1
2
3
h(c) = 0 1 2 3
S(b,c) whereh(b)=1; h(c)=2
R(a,b), whereh(b)=2
T(c,d), whereh(c)=3
Slide based on www.mmds.com
DB = R(1,1)R(1,2)R(2,1)R(2,2)
S(1,1)S(1,2)S(2,1)S(2,2)
T(1,1)T(1,2)T(2,1)T(2,2)
..etc
R(1,1)R(2,1)
S(1,1) T(1,1)T(1,2)
R(1,1)R(2,1)
S(1,2) T(2,1)T(2,2)
R(1,2)R(2,2)
S(2,1) T(1,1)T(1,2)
R(1,2)R(2,2)
S(2,2) T(2,1)T(2,2)
MapperR(1,1) (1,1,(R,1))(1,2,(R,1))
R(1,1) S(1,1) T(1,1) R(2,1) S(1,1) T(1,1) R(2,1) S(1,1) T(1,2) R(1,1) S(1,1) T(1,2)
MapperS(1,2) (1,2,(S,1,2))
top related