parallel programming and map-reduce · • map-reduce is a programming model + distributed system...

Parallel Programming and Map-ReduceEverything Data

CompSci 216 Spring 2018

Announcements (Wed. Mar 28)•  Homework #10 will be posted on Mon, Apr

2.– Due in 2 weeks (Mon, Apr 16) – Last homework!

•  Homework #9:–  Fri Mar 30, 11:59 AM: Submit visualizations–  Sun Apr 1, 11:59 PM: Vote on visualizations

•  Project deadlines: – Mon 4/2, 11:59 PM: Mid-Term Report is due.

2

3

1.2 trillion searches per year

4

131,000,000 pages mentioning Einstein

Size of the entire corpus??

5

http://www.worldwidewebsize.com/

Size of the entire corpus??

Trend 1: Data centers6

http://designtaxi.com/news/353991/Where-The-Internet-Lives-A-Glimpse-Inside-Google-s-Private-Data-Centers/

Trend 2: Multicore7

Copyright © National Academy of Sciences. All rights reserved.

The Future of Computing Performance: Game Over or Next Level?

POWER IS NOW LIMITING GROWTH IN COMPUTING PERFORMANCE 91

expected performance. One might think that it should therefore be pos-sible to continue to scale performance by doubling the number of proces-sor cores. And, in fact, since the middle 1990s, some researchers have argued that chip multiprocessors (CMPs) can exploit capabilities of CMOS technology more effectively than single-processor chips.21 However, dur-ing the 1990s, the performance of single processors continued to scale at the rate of more than 50 percent per year, and power dissipation was still not a limiting factor, so those efforts did not receive wide attention. As single-processor performance scaling slowed down and the air-cooling power-dissipation limit became a major design constraint, researchers and industry shifted toward CMPs or multicore microprocessors.22

21 Kunle Olukotun, Basem A. Nayfeh, Lance Hammond, Ken Wilson, and Kunyung Chang, 1996, The case for a single-chip multiprocessor, Proceedings of 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Cambridge, Mass., October 1-5, 1996, pp. 2-11.

22 Listed here are some of the references that document, describe, and analyze this shift: Michael Bedford Taylor, Walter Lee, Jason Miller, David Wentzlaff, Ian Bratt, Ben Greenwald, Henry Hoffmann, Paul Johnson, Jason Kim, James Psota, Arvind Saraf, Nathan Shnidman, Volker Strumpen, Matt Frank, Saman Amarasinghe, and Anant Agarwal, 2004, Evaluation of the raw microprocessor: An exposed-wire-delay architecture for ILP and streams, Pro-ceedings of the 31st Annual International Symposium on Computer Architecture, Munich,

10

100

1,000

10,000

1985 1990 1995 2000 2005 2010Year of Introduction

FIGURE 3.3 Microprocessor-clock frequency (MHz) over time (1985-2010).http://www.nap.edu/catalog.php?record_id=12980

Copy

right

© N

atio

nal A

cade

my

of S

cienc

es. A

ll rig

hts

rese

rved

.

The

Futu

re o

f Com

putin

g Pe

rform

ance

: G

ame

Ove

r or N

ext L

evel

?

POW

ER I

S N

OW

LIM

ITIN

G G

RO

WTH

IN

CO

MPU

TIN

G P

ERFO

RM

AN

CE

91

expe

cted

per

form

ance

. One

mig

ht t

hink

tha

t it

shou

ld t

here

fore

be

pos-

sibl

e to

con

tinue

to s

cale

per

form

ance

by

doub

ling

the

num

ber

of p

roce

s-so

r co

res.

And

, in

fac

t, si

nce

the

mid

dle

1990

s, s

ome

rese

arch

ers

have

ar

gued

that

chi

p m

ultip

roce

ssor

s (C

MPs

) can

exp

loit

capa

bilit

ies

of C

MO

S te

chno

logy

mor

e ef

fect

ivel

y th

an s

ingl

e-pr

oces

sor

chip

s.21

How

ever

, dur

-in

g th

e 19

90s,

the

perf

orm

ance

of s

ingl

e pr

oces

sors

con

tinue

d to

sca

le a

t th

e ra

te o

f mor

e th

an 5

0 pe

rcen

t per

yea

r, an

d po

wer

dis

sipa

tion

was

stil

l no

t a

limiti

ng f

acto

r, so

tho

se e

ffor

ts d

id n

ot r

ecei

ve w

ide

atte

ntio

n. A

s si

ngle

-pro

cess

or p

erfo

rman

ce s

calin

g sl

owed

dow

n an

d th

e ai

r-co

olin

g po

wer

-dis

sipa

tion

limit

beca

me

a m

ajor

des

ign

cons

trai

nt, r

esea

rche

rs a

nd

indu

stry

shi

fted

tow

ard

CM

Ps o

r m

ultic

ore

mic

ropr

oces

sors

.22

21 K

unle

Olu

kotu

n, B

asem

A. N

ayfe

h, L

ance

Ham

mon

d, K

en W

ilson

, and

Kun

yung

Cha

ng,

1996

, The

cas

e fo

r a

sing

le-c

hip

mul

tipro

cess

or, P

roce

edin

gs o

f 7th

Inte

rnat

iona

l Con

fere

nce

on A

rchi

tect

ural

Sup

port

for

Prog

ram

min

g La

ngua

ges

and

Ope

ratin

g Sy

stem

s, C

ambr

idge

, M

ass.

, Oct

ober

1-5

, 199

6, p

p. 2

-11.

22

Lis

ted

here

are

som

e of

the

ref

eren

ces

that

doc

umen

t, de

scri

be, a

nd a

naly

ze t

his

shift

: M

icha

el B

edfo

rd T

aylo

r, W

alte

r Lee

, Jas

on M

iller

, Dav

id W

entz

laff

, Ian

Bra

tt, B

en G

reen

wal

d,

Hen

ry H

offm

ann,

Pau

l Joh

nson

, Jas

on K

im, J

ames

Pso

ta, A

rvin

d Sa

raf,

Nat

han

Shni

dman

, Vo

lker

Str

umpe

n, M

att

Fran

k, S

aman

Am

aras

ingh

e, a

nd A

nant

Aga

rwal

, 200

4, E

valu

atio

n of

the

raw

mic

ropr

oces

sor:

An

expo

sed-

wir

e-de

lay

arch

itect

ure

for

ILP

and

stre

ams,

Pro

-ce

edin

gs o

f the

31s

t Ann

ual I

nter

natio

nal S

ympo

sium

on

Com

pute

r A

rchi

tect

ure,

Mun

ich,

10100

1,00

0

10,0

00

1985

1990

1995

2000

2005

2010

Year

of I

ntro

duct

ion

FIG

UR

E 3.

3 M

icro

proc

esso

r-cl

ock

freq

uenc

y (M

Hz)

ove

r tim

e (1

985-

2010

).

Moore’s Law: # transistors on integrated circuits doubles every 2 years

Need to think “parallel”

•  Data resides on different machines•  Split computation onto different

machines/cores

8

But … parallel programming is hard!

Low level code needs to deal with a lot of issues … •  Failures

– Loss of computation– Loss of data

•  Concurrency•  …

9

Map-Reduce10

Programming Model Distributed System

•  Simple model

•  Programmer only describes the logic

•  Works on commodity hardware

•  Scales to thousands of machines

•  Ship code to the data, rather than ship data to code

•  Hides all the hard systems problems from the programmer•  Machine failures•  Data placement•  …

+

Map-Reduce Programming Model11

map

reduce

f f f f f f f f

rows of dataset

g

Map-Reduce Programming Model12

map!

!!, !! ! list! !!, !! !!reduce

!!!, list(!!) ! list! !!, !! !

!

map

reduce

f f f f f f f

g g

Example 1: Word Count

•  Input: A set of documents, each containing a list of words– Each document is a row– E.g., search queries, tweets, reviews, etc.

•  Output: A list of pairs <w, c>–  c is the number of times w appears across all

documents.

13

Word Count: Map

<docid, {list of words}> à {list of <word, 1>}

•  The mapper takes a document d and creates n key value pairs, one for each word in the document.

•  The output key is the word•  The output value is 1

–  (count of each appearance of a word)

14

Word Count: Reduce<word, {list of counts}> à <word, sum(counts)>

•  The reducer aggregates the counts (in this case

1) associated with a specific word.

15

Map-Reduce Implementation16

Map Phase (per record computation)

Reduce Phase (global computation)

Shuffle

map!

!!, !! ! list! !!, !! !!reduce

!!!, list(!!) ! list! !!, !! !

!

Split




Shuffle

Split

Split phase partitions the data across different mappers (… think different machines)




Shuffle

Split

Each mapper executes user defined map code on the partitions in parallel




Shuffle

Split

Data is shuffled such that there is one reducer per output key (… again think different machines)




Shuffle

Split

Each reducer executes the user defined reduce code in parallel.

Map Reduce Implementation

•  After every map and reduce phase, data is wrijen onto disk–  If machines fail during the reduce phase, then

no need to rerun the mappers.

21

Writing out the output can be slow. Should minimize number of map-reduce phases.

Mappers, Reducers and Workers

•  Physical machines are called workers•  Multiple mappers and reducers can run on

the same worker.

•  More workers implies …… more parallelism (faster computation) … … but more (communication) overhead …

22

Map Reduce Implementation

•  All reducers start only after all the mappers complete.

•  Straggler: A mapper or reducer that takes a long time

23

Back to Word Count

•  Map: <docid, {list of words}> à {list of <word, 1>}

•  Reduce: <word, {list of counts}> à <word, sum(counts)>

•  Number of records output by the map phase equals the number of words across all documents.

24

Map-Combine-Reduce

•  Combiner is a mini-reducer within each mapper. – Helps when the reduce function is

commutative and associative.

•  Aggregation within each mapper reduces the communication cost.

25

Word Count … with combiner•  Map:

<docid, {list of words}> à {list of <word, 1>}

•  Combine: <word, {list of counts}> à <word, sum(counts)>

•  Reduce: <word, {list of counts}> à <word, sum(counts)>

26

Mapper

Reducer

Example 2: K most frequent words

•  Need multiple Map-Reduce steps

•  Map: <docid, {list of words}> à {list of <word, 1>}

•  Reduce: <word, {list of counts}> à <_ , (word, sum(counts))>

•  Reduce: <_ , {list of (word, count) pairs}> à

<_ , {list of words with k most frequent counts}>

27

Example 3: Distributed Grep

•  Input: String•  Output: {list of lines that match the string}

•  Map:<lineid, line> à <lineid, line> // if line matches string

•  Reduce: // do nothing

28

Example 4: Matrix Multiplication29

M N P

36 CHAPTER 2. MAP-REDUCE AND THE NEW SOFTWARE STACK

the Reduce function applies each of them to the list of values associated witha given key and produces a tuple consisting of the key, including componentsfor all grouping attributes if there is more than one, followed by the results ofeach of the aggregations.

2.3.9 Matrix Multiplication

If M is a matrix with element mij in row i and column j, and N is a matrixwith element njk in row j and column k, then the product P = MN is thematrix P with element pik in row i and column k, where

pik =!

j

mijnjk

It is required that the number of columns of M equals the number of rows ofN , so the sum over j makes sense.

We can think of a matrix as a relation with three attributes: the row number,the column number, and the value in that row and column. Thus, we could viewmatrix M as a relation M(I, J, V ), with tuples (i, j, mij), and we could viewmatrix N as a relation N(J, K, W ), with tuples (j, k, njk). As large matrices areoften sparse (mostly 0’s), and since we can omit the tuples for matrix elementsthat are 0, this relational representation is often a very good one for a largematrix. However, it is possible that i, j, and k are implicit in the position of amatrix element in the file that represents it, rather than written explicitly withthe element itself. In that case, the Map function will have to be designed toconstruct the I, J , and K components of tuples from the position of the data.

The product MN is almost a natural join followed by grouping and ag-gregation. That is, the natural join of M(I, J, V ) and N(J, K, W ), havingonly attribute J in common, would produce tuples (i, j, k, v, w) from each tuple(i, j, v) in M and tuple (j, k, w) in N . This five-component tuple represents thepair of matrix elements (mij , njk). What we want instead is the product ofthese elements, that is, the four-component tuple (i, j, k, v × w), because thatrepresents the product mijnjk. Once we have this relation as the result of onemap-reduce operation, we can perform grouping and aggregation, with I andK as the grouping attributes and the sum of V × W as the aggregation. Thatis, we can implement matrix multiplication as the cascade of two map-reduceoperations, as follows. First:

The Map Function: For each matrix element mij , produce the key value pair"

j, (M, i, mij)#

. Likewise, for each matrix element njk, produce the key valuepair

"

j, (N, k, njk)#

. Note that M and N in the values are not the matricesthemselves. Rather they are names of the matrices or (as we mentioned for thesimilar Map function used for natural join) better, a bit indicating whether theelement comes from M or N .

The Reduce Function: For each key j, examine its list of associated values.For each value that comes from M , say (M, i, mij)

#

, and each value that comes

x =

Matrix Multiplication•  Assume the input format is <matrix id, row, col, entry>

•  E.g.: <M, i, j, mij>

•  Map:<_ , (M, i, j, mij)> à <(i,k), (M, i, mij)> … for all k

<_ , (N, j, k, njk)> à <(i,k), (N, k, njk)> … for all i•  Reduce:

<(i,k), {(M, i, j, mij), (N, j, k, njk) …}> à { <(i,k), Σj mijnjk>}

30

Example 5: Join two tables

•  Input: file1 and file2, with schema <key, value>•  Output: keys appearing in both files.

•  Map: <_ , (file1, key, value)> à (key, file1)<_ , (file2, key, value)> à (key, file2)

•  Reduce: <key , {list of fileids}> à <_, key>// if list contains both file1 and file2.

31

Map-side Join

•  Suppose file2 is very small …

•  Map: <_ , (file1, key, value, {keys in file2})> à (_ , key) // If key is also in file2

•  Reduce: // do nothing <_ , {list of keys}> à <_ , {list of keys}>

32

Send contents of file2 to all the mappers

Example 5: Join 3 tables

•  Input: 3 Tables– User (id:int, age:int)– Page (url:varchar, category:varchar)– Log (userid: int, url:varchar)

•  Output: Ages of users and types of urls they clicked.

33

Multiway Join•  Join( Page, Join (User , Log))•  Map:

<_ , (User, id, age)> à (id, (User, key, value))<_ , (Log, userid, url)> à (userid, (Log, userid, url))

•  Reduce: <id, {list of records}> à <_, {records from User} x {records from Log}>// if list contains records both from User and Log.

•  Map: <_ , (User, Log, id, age, userid, url)> à (url, (User, Log, id, age, userid, url))<_ , (Page, url, category)> à (url, (Page, url, category))

•  Reduce: <url, {list of records}> à <_, {records from User x Log} x {records from Page}>// if list contains records both from User x Log and Page.

34

Summary

•  Map-reduce is a programming model + distributed system implementation that make parallel programming easy.– Programmer does not need to worry about

systems issues.

•  Computation is a series of Map and Reduce jobs. – Parallelism is achieved within each Map and

Reduce phase.

35

parallel programming and map-reduce · • map-reduce is a programming model + distributed system...

Documents