Data Structures and Algorithms in Parallel Computing

Lecture 4

Parallel computer

• A parallel computer consists of a set of processors that work together on solving a problem

• Moore’s law has broken down in 2007 because of increasing power consumption– O(f3) where f is the frequency– Tianhe-2 has 3.12 million processor cores and consumes 17.8

MW– It is as cheap to put 2 cores on the same chip as putting just one– 2 cores running at f/2 consume only ¼ of the power of a single

core running at f– However it is hard to achieve the same total computational

speed in practice

Bulk Synchronous Parallelism

• BSP model 1st proposed in 1989 • Alternative to the PRAM model• Used for distributed memory computers• Fast local memory access• Algorithm developers need not worry about

network details, only about global performance• Efficient algorithms which can be run on many

different parallel computers

BSP

• Processors + network + synchronization• Superstep– Concurrent parallel computation– Message exchanges between processors– Barrier synchronization

• All processors reaching this point wait for the rest

Supersteps

• A BSP algorithm is a sequence of supersteps– Computation superstep

• Many small steps– Example: floating point operations (addition, subtraction, etc.)

– Communication superstep• Communication operations each transmitting a data word

– Example: transfer a real number between 2 processors

• In theory we distinguish between the 2 types of supersteps

• In practice we assume a single superstep

Communication superstep

• h-relation– Superstep in which every processor sends and receives

at most h data words– h=max{hs, hr}

• hs is the maximum number of data words sent by a processor

• hd is the maximum number of data words received by a processor

• Cost– T(h)=hg+l

• Where g is the time per data word and l is the global synchronization time

Time of an h-relation on a 4-core Apple iMac desktop

• Taken from www.staff.science.uu.nl/~bisse101/Book/PSC/psc1_2.pdf

Computation superstep

• T=w+l– Where w is the maximum number of flops of a

processor in a superstep, and l is the global synchronization time

• Processors with less than w flops wait idle

Total cost of a BSP algorithm

• Add cost of all supersteps• a+bg+cl– g and l are a function of the number of processors– a, b, and c depend on p and the problem size n

BSP implementations

• Google Pregel• MapReduce• Apache Giraph– Open source implementation of Pregel

• Apache Hama– Inspired from Pregel

• BSPLib• …

Pregel framework

• Computations consist of a sequence of iterations called supersteps

• During a superstep, the framework invokes a user defined function for each vertex which specifies the behavior at a single vertex V and a single superstep S The function can: – Read messages sent to V in superstep S-1 – Send messages to other vertices that will be received in superstep S+1 – Modify the state of V and of the outgoing edges – Make topology changes (add/remove/update edges/vertices)

http://people.apache.org/~edwardyoon/documents/pregel.pdf

Model of computation: Progress

• In superstep 0 all vertices are active• Only active vertices participate in a superstep– They can go inactive by voting for halt– They can be reactivated by an external message

from another vertex• The algorithm terminates when all vertices

have voted for halt and there are no messages in transit

Model of computation: Vertex

Pregel execution

Pregel executionMaster-worker paradigm

1. Partition graphI. Random hash basedII. Custom (mincut, …)

2. Assign vertices to workers (processors)3. Mark all vertices as active4. Each active vertex executes a Compute() function and delivers messages

which were sent in the previous superstep1. Get/Modify current vertex value using GetValue()/MutableValue()

5. Respond to master with active vertices for the next superstep

All workers execute the same code! Master is used for coordination

Fault tolerance

• At the start of each superstep the master instructs workers to save their state

• Master pings workers to see who is running• If failure is detected, master reasigns

partitions to available workers

Example: find largest value in a graph

What’s next?

• BSP model– SSSP, connected components, pagerank

• Vertex centric vs. subgraph centric• Load balancing– Importance of partitioning and graph type

• ...