parallel algorithms on networks of processors

Parallel Algorithms on Networks of Processors

Roy (Hutch) Pargas, PhD Computer Science (UNC Chapel Hill)School of Computing, Clemson University

[email protected]

2

OutlineWhat are parallel algorithms? Why use them?

Challenges for parallel algorithm designers

Choosing a network

Partitioning the data

Designing the algorithm

Example

Recurrences (binary tree)

Analysis (Speedup and Efficiency)

Summary and Conclusions

Roy Pargas, Clemson University [email protected]

July 30, 2011

50 Golden Years Ateneo Mathematics Program Quezon City, Philippines

3

Why Parallel Computation?


July 30, 2011


4



July 30, 2011


5



July 30, 2011


6



July 30, 2011


7



July 30, 2011


8



July 30, 2011


9

New ProcessorsFaster and Cheaper


July 30, 2011


January2011

10

Partition the Data


July 30, 2011


11

Organize the Processors


July 30, 2011


12

Build a Network


July 30, 2011


13

Build a Network


July 30, 2011


14

Choosing a Network


July 30, 2011


15

Choosing a Network


July 30, 2011


16

Choosing a Network


July 30, 2011


17

Choosing a Network


July 30, 2011


18

Choosing a Network


July 30, 2011


19

Are There Really Any Multiprocessing

Systems in Use Today?


July 30, 2011


20

Are There Really Any Multiprocessing

Systems in Use Today?


July 30, 2011


HamburgJune 2011Top 500

21

SupercomputersNEC/HP Tsubame

(Japan)


July 30, 2011


1.192 petaflops ≈ 1.28 quadrillion floating point

operations per sec

73,278 Xeon cores

Infiniband grid network

22


July 30, 2011


SupercomputersDawning Nebulae

(China)1.27 petaflops ≈ 1.36

quadrillion floating point operations per sec

9280 Intel 6-core Xeon processors = 55,680 cores

Infiniband grid network

23


July 30, 2011


SupercomputersCray Jaguar (USA)

1.75 petaflops ≈ 1.876 quadrillion floating point

operations per sec

224,256 AMD cores

3D torus network

24


July 30, 2011


SupercomputersNUDT Tianhe-1A

(China)2.566 petaflops ≈ 2.75

quadrillion floating point operations per sec

14336 CPUs

Undisclosed proprietary network

25


July 30, 2011


SupercomputersFujitsu “K” (Japan)

K = “kei” = Japanese for 10 quadrillion

8.162 petaflops ≈ 9 quadrillion floating point operations per

sec

68,544 8-core SPARC64 processors = 548,352

cores

3-dimensional torus network called Tofu

26


July 30, 2011


TOP500

Top 500 Computers in the World

http://en.wikipedia.org/wiki/TOP500


27

Where Does that Leave Us?


July 30, 2011


28



July 30, 2011


In a wonderful playground of mathematical algorithmic design where imagination and creativity are key!

29




July 30, 2011


30




July 30, 2011


31




July 30, 2011


32

Challenges for Parallel Algorithm

DesignersChoosing a network

Partitioning the problem

Designing the parallel algorithm


July 30, 2011


33

So Let’s Try It:

Choosing a network




July 30, 2011


34

So Let’s Try It:Elliptic Partial Diff

EqnsChoosing a network




July 30, 2011


35

Elliptic PDEs


July 30, 2011


Problems involving second-order elliptic partial differential equations are equilibrium problems. Given a region R bounded by a curve C and that the unknown function z satisfies Laplace’s or Poisson’s equation in R, the objective is to approximate the value of z at any point in R. The method of finite differences is an often used numerical method for solving this problem. The basic strategy is to approximate the differential equation by a difference equation and to solve the difference equation.

36

Designing the Algorithm

Why Solve Linear Recurrences?


July 30, 2011


37



The problem: Solving PDEs using the Method of Finite Differences leads to


July 30, 2011


38




Solving Block Tridiagonal Systems which leads to


July 30, 2011


39





Solving Tridiagonal Systems which leads to


July 30, 2011


40






Solving Linear Recurrences


July 30, 2011


41






Solving Linear Recurrences many many many times


July 30, 2011


42






Solving Linear Recurrences many many many times

Why Use a Binary Tree? Roy Pargas, Clemson University [email protected]

July 30, 2011


43

Linear Recurrences


July 30, 2011


Key Idea: Successfully solving the original pde problem depends upon solving recurrences quickly and efficiently.

44

Linear Recurrences


July 30, 2011


Consider the following set of n equations:x0 = a0

x1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

45

Linear Recurrences


July 30, 2011


Consider the following set of n equations:x0 = a0

x1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

Can we solve for xi in parallel?

46

Linear Recurrences


July 30, 2011


For uniformity:x0 = a0 + b0 x-1 b0=0, x-1=dummy variablex1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

47

Linear Recurrences


July 30, 2011


For uniformity:x0 = a0 + b0 x-1

x1 = a1 + b1 x0

x2 = a2 + b2 x1

...

xn-1 = an-1 + bn-1 xn-2

Observe, ifxi = a + b xj

xj = a’ + b’ xk

Thenxi = (a + ba’) +bb’ xk

= a” + b” xk

48

Linear Recurrences


July 30, 2011


Notation changex0 = a0 + b0 x-1 C0,-1 = (a0,b0)x1 = a1 + b1 x0 C1,0 = (a1,b1)x2 = a2 + b2 x1 C2,1 = (a2,b2)...

xn-1 = an-1 + bn-1 xn-2 Cn-1,n-2 = (an-1,bn-1)

Observe, ifxi = a + b xj

xj = a’ + b’ xk

Thenxi = (a + ba’) +bb’ xk

= a” + b” xk

49

Linear Recurrences


July 30, 2011


Notation changex0 = a0 + b0 x-1 C0,-1 = (a0,b0)x1 = a1 + b1 x0 C1,0 = (a1,b1)x2 = a2 + b2 x1 C2,1 = (a2,b2)...

xn-1 = an-1 + bn-1 xn-2 Cn-1,n-2 = (an-1,bn-1)

Observe, ifxi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)

Thenxi = (a + ba’) +bb’ xk Ci,j Cj,k =

= a” + b” xk Ci,k = (a+ba’,bb’)

50

Linear Recurrences


July 30, 2011


To summarize:xi = a + b xj Ci,j = (a,b)xj = a’ + b’ xk Cj,k = (a’,b’)


= a” + b” xk Ci,k = (a+ba’,bb’)

51

Linear Recurrences


July 30, 2011




= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:

52

Linear Recurrences


July 30, 2011




= a” + b” xk Ci,k = (a+ba’,bb’)

One last observation:If any variable is expressed in terms of

the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved.

53

Linear Recurrences


July 30, 2011




= a” + b” xk Ci,k = (a+ba’,bb’)


the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved. So Ci,-1 = (a,b)

54

Linear Recurrences


July 30, 2011




= a” + b” xk Ci,k = (a+ba’,bb’)


the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved. So Ci,-1 = (a,b) means that b=0

55

Linear Recurrences


July 30, 2011




= a” + b” xk Ci,k = (a+ba’,bb’)


the dummy variable x-1 (e.g., x0 = a0 + b0 x-1) then that variable is solved. So Ci,-1 = (a,b) means that b=0 and that xi = a + b x-1 = a + 0 x-1 = a

56

C0,-

1

C1,

0

C2,

1

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6 Roy Pargas, Clemson University [email protected]

July 30, 2011


Linear Recurrences

57

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

58

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

59

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

60

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

61

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

62

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

63

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

64

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

Solved variable

65

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

Solved variables

66

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C6,5C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1C7,-1

How do we solve for the other variables?

67

C6,5

C7,-1

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

C7,-

1

In the downsweep!

68

C6,5

C7,-1

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

(x7,x-

1)

69

C6,5

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

C7,-

1

(x7,x-

1)

70

C6,5

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

C7,3 C3,-1

x3

(x7,x-

1)

71

C6,5

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C7,5 C5,3C3,1 C1,-1

C7,

3

C3,-

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

72

C6,5

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

C3,1 C1,-1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5C7,5 C5,3

x1

73

C6,5

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

C7,

5

C5,

3

C3,

1

C1,-

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x7,x5) (x5,x3) (x3,x1) (x1,x-1)

74

(x7,x5)

C6,5

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-1C1,0C2,1C3,2C4,3

C5,4C7,6

C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

75

(x7,x5)

(x7,x-1)

Linear Recurrences


July 30, 2011


C0,-

1

C1,

0

C3,

2

C4,

3

C5,

4

C6,

5

C7,

6

C2,

1

x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1

(x7,x6) (x6,x5) (x5,x4)(x0,x-1)(x1,x0)(x4,x3) (x3,x2) (x2,x1)

76

(x7,x5)

(x7,x-1)

Linear Recurrences


July 30, 2011


x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1


(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

77

(x7,x5)

(x7,x-1)

Linear Recurrences


July 30, 2011


x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1


(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

Leaves contain solutions!

78

(x7,x5)

(x7,x-1)

Linear Recurrences


July 30, 2011


x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1


(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

But we can do better!

79

(x7,x5)

(x7,x-1)

Linear Recurrences


July 30, 2011


x3

(x7,x-

1)

(x7,x3) (x3,x-1)

x5 x1

(x5,x3) (x3,x1) (x1,x-1)

x6 x4 x3 x1


(x7,x6) (x0,x-

1)(x6,x5) (x5,x4) (x4,x3) (x3,x2) (x2,x1) (x1,x0)

Pipelining the solution

80


July 30, 2011


Linear RecurrencesPipelining the solution

81


July 30, 2011



82


July 30, 2011



83


July 30, 2011



84


July 30, 2011



85


July 30, 2011



86


July 30, 2011



87


July 30, 2011



88


July 30, 2011



89


July 30, 2011



90


July 30, 2011



91

AnalysisT1 = Time on one processor

Tn = Time on n processors

S = Speedup = T1/Tn (ideal: S = n)

E = Efficiency = S/n (ideal: E = 1)


July 30, 2011


92

Speedup and Efficiency

Single PassAssume 1 floating point operation requires 1 time unit

T1 = (n−1) (1 mult + 1 add) = 2n−2 time units

n leaves 2n processors T2n = (log2n)( 2 mults + 1 add) // upsweep

+ (log2n)(1 mult + 1 add) // downsweep= 5 log2n time units

S = Speedup = T1/T2n = (2n−2)/(5 log2n)

E = Efficiency = S/2n = (2n−2)/[ (5 log2n) (2n) ]


July 30, 2011


93

Speedup


July 30, 2011


0 200 400 600 800 1000 12000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Series1


Single Pass

94

Speedup Efficiency


July 30, 2011


0 200 400 600 800 1000 12000

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Series1

0 50 100 150 200 250 300 3500

0.01

0.02

0.03

0.04

0.05

0.06

Series1


Single Pass

95

With Pipelining. Assume kn equations for large k

T1 = (kn−1) (1 mult + 1 add) = 2kn−2 time units

n leaves 2n processors T2n = (log2n)( 2 mults + 1 add) // pipefill up

+ (k – 2 log 2n) (5) // pipeline on k−2log 2n waves

+ (log2n)( 1 mult + 1 add) // pipedrain

down= 5(k – log2n) time units

S = Speedup = T1/T2n = (2kn−2)/[5(k – log2n )]

E = Efficiency = S/2n = (2kn−2)/[5(k – log2n ) (2n)]


July 30, 2011



With Pipelining

96


July 30, 2011



With PipeliningSpeedup

0 200 400 600 800 1000 12000

50

100

150

200

250

300

350

400

450

Series1

97


July 30, 2011



With PipeliningSpeedup Efficiency

0 200 400 600 800 1000 12000

50

100

150

200

250

300

350

400

450

Series1

0 200 400 600 800 1000 12000.1999

0.2000

0.2001

0.2002

0.2003

Series1

98

Technique Can Work For


July 30, 2011


Second-order linear recurrencesx0 = a0

x1 = a1 + b1 x0

x2 = a2 + b2 x1 + c2 x0

...

xn-1 = an-1 + bn-1 xn-2 + cn-1 xn-3

Higher order linear recurrences

99

Technique Can Work For


July 30, 2011


Quotients of linear recurrencesx0 = a0

xi = (ai + bi xi-1)/(ci + di xi-1) i=1,2,…, n-1

Other recurrences

100



July 30, 2011


101


Chip technology is going to get even better/faster/cheaper for the foreseeable future.


July 30, 2011


102



July 30, 2011


Chip technology is going to get even better/faster/cheaper for the foreseeable future. (Ignore the naysaying pundits!)

103



More massively parallel processing systems are going to be built and will become even better/faster/cheaper.


July 30, 2011


104



July 30, 2011




The challenging world of parallel algorithmic design beckons and awaits creative minds.

105




The challenging world of parallel algorithmic design beckons and awaits creative minds. Yes, this means you!


July 30, 2011


106

Where in the World is Clemson

University?


We are here!


July 30, 2011

107

Thank you for your kind attention!


July 30, 2011


Questions?

108

Extra Slides


July 30, 2011


109


July 30, 2011


Links

1. Fujitsu K Computer (K = “kei” = Japanese word for 10 quadrillion)

http://www.fujitsu.com/global/about/tech/k/http://en.wikipedia.org/wiki/K_computer

2. NUDT “Tianhe-1A” Computerhttp://blog.zorinaq.com/?e=36http://en.wikipedia.org/wiki/Tianhe-I

3. Cray Jaguarhttp://en.wikipedia.org/wiki/Jaguar_(computer)http://www.nccs.gov/jaguar/

4. Dawning Nebulaehttp://en.wikipedia.org/wiki/Dawning_Information_Industryhttp://www.theregister.co.uk/2010/05/31/top_500_supers_jun2010/

5. NEC/HP Tsubame 2.0 http://en.wikipedia.org/wiki/TOP500

http://www.fujitsu.com/global/about/tech/k/

http://en.wikipedia.org/wiki/K_computer

http://en.wikipedia.org/wiki/K_computer

http://blog.zorinaq.com/?e=36

http://blog.zorinaq.com/?e=36

http://en.wikipedia.org/wiki/Tianhe-I

http://en.wikipedia.org/wiki/Tianhe-I

http://en.wikipedia.org/wiki/Jaguar_(computer

http://www.nccs.gov/jaguar/

http://www.nccs.gov/jaguar/

http://en.wikipedia.org/wiki/Dawning_Information_Industry

http://en.wikipedia.org/wiki/Dawning_Information_Industry

http://www.theregister.co.uk/2010/05/31/top_500_supers_jun2010/





110

Hypercube


July 30, 2011


0-Degree

111

Hypercube


July 30, 2011


0-Degree

112

Hypercube


July 30, 2011


0-Degree

113

Hypercube


July 30, 2011


0-Degree

114

Hypercube


July 30, 2011


0-Degree

1-Degree

0 1

115

Hypercube


July 30, 2011


0-Degree

1-Degree

0 1

0 1

116

Hypercube


July 30, 2011


0-Degree

1-Degree

0 1

0 1

0 1

117

Hypercube


July 30, 2011


0-Degree

1-Degree

0 1

0 1

0 1

118

Hypercube


July 30, 2011


0-Degree

1-Degree

0 1

00

01

10

11

2-Degree

119

Hypercube


July 30, 2011


3-Degree

0-Degree

1-Degree

0 1

00

01

10

11

2-Degree

120

Hypercube


July 30, 2011


3-Degree

4-Degree

0-Degree

1-Degree

0 1

00

01

10

11

2-Degree

121

Hypercube


July 30, 2011


3-Degree

4-Degree

5-Degree

122

All-to-All Communication

(Hypercube)


July 30, 2011


Problem

Motivation

parallel algorithms on networks of processors

Documents

clemson university

networkroy pargas

processorsroy pargas

cheaperroy pargas

dataroy pargas

conclusions roy pargas

efficient parallel algorithms

multiprocessing systems