cs4402 – parallel computing lecture 1: classification of parallel computers classification of...
TRANSCRIPT
![Page 1: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/1.jpg)
CS4402 – Parallel Computing
Lecture 1:Classification of Parallel Computers
Classification of Parallel Computation
Important Laws of Parallel Compuation
![Page 2: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/2.jpg)
How I used to make breakfast……….
![Page 3: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/3.jpg)
How to set family to work...
![Page 4: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/4.jpg)
How finally got to the office in time….
![Page 5: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/5.jpg)
What is Parallel Computing?
In the simplest sense, parallel computing is the simultaneous use of multiple computing resources to solve a problem.
Parallel computing is the solution for "Grand Challenge Problems“: weather and climate biological, human genome chemical and nuclear reactions
Parallel Computing is a necessity for some commercial applications: parallel databases, data mining computer-aided diagnosis in medicine
Ultimately, parallel computing is an attempt to minimize time.
![Page 6: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/6.jpg)
![Page 7: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/7.jpg)
![Page 8: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/8.jpg)
Grand Challenges Problems
![Page 10: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/10.jpg)
Reason 1: Speedup
![Page 11: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/11.jpg)
Reason 2: Economy
Resources already available. Taking advantage of non-local resources Cost savings - using multiple "cheap" computing resources instead of
paying for time on a supercomputer.
A parallel system is cheaper than a better processor. Transmission speeds. Limits to miniaturization. Economic limitations.
![Page 12: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/12.jpg)
Reason 3: Scalability
![Page 13: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/13.jpg)
13
![Page 14: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/14.jpg)
14
Types of || Computers
Parallel Computers
Hardware Software
Shared memory
Distributed memory
Hybrid memory
SIMD MIMD
![Page 15: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/15.jpg)
![Page 16: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/16.jpg)
![Page 17: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/17.jpg)
17
![Page 18: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/18.jpg)
![Page 19: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/19.jpg)
![Page 20: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/20.jpg)
![Page 21: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/21.jpg)
![Page 22: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/22.jpg)
![Page 23: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/23.jpg)
The Banking Analogy
Tellers: Parallel Processors
Customers: tasks Transactions: operations Accounts: data
![Page 24: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/24.jpg)
Vector/Array
Each teller/processor gets a very fine-grained task
Use pipeline parallelism
Good for handling batches when operations can be broken down into fine-grained stages
![Page 25: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/25.jpg)
SIMD (Single-Instruction-Multiple-Data)
All processors do the same things or idle
Phase 1: data partitioning and distributed
Phase 2: data-parallel processing
Efficient for big, regular data-sets
![Page 26: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/26.jpg)
Systolic Array
Combination of SIMD and Pipeline parallelism
2-d array of processors with memory at the boundary
Tighter coordination between processors
Achieve very high speeds by circulating data among processors before returning to memory
![Page 27: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/27.jpg)
MIMD(Multi-Instruction-Multiple-Data)
Each processor (teller) operates independently
Need synchronization mechanism by message passing or mutual exclusion (locks)
Best suited for large-grained problems
Less than data-flow parallelism
![Page 28: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/28.jpg)
28
Important Laws of || Computing.
![Page 29: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/29.jpg)
29
![Page 30: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/30.jpg)
30
![Page 31: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/31.jpg)
31
![Page 32: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/32.jpg)
32
![Page 33: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/33.jpg)
33
![Page 34: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/34.jpg)
34
![Page 35: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/35.jpg)
35
![Page 36: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/36.jpg)
36
Important Consequences
f=0 when no serial part S(n)=n perfect speedup.
f=1 when everything is serial S(n)=1 no parallel code.
fn
nnS
)1(1)(
![Page 37: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/37.jpg)
37
Important Consequences
S(n) is increasing when n is increasing
S(n) is decreasing when f is increasing.
fn
nnS
)1(1)(
![Page 38: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/38.jpg)
38
Important Consequences
no matter how many processors are being used the speedup cannot increase above
Examples: f = 5% S(n) < 20 f = 10% S(n) < 10 f = 20% S(n) < 5.
ffn
nnS
1
)1(1)(
![Page 39: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/39.jpg)
39
![Page 40: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/40.jpg)
40
![Page 41: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/41.jpg)
41
![Page 42: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/42.jpg)
42
Gustafson’s Law - More
![Page 43: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/43.jpg)
43
Gustafson’s Speed-up
pnsT
TpnTs
TimeParallel
TimeSequentialnS
)(
snnsnsnS )1()1()( When s+p=1
Important Consequences:
1) S(n) is increasing when n is increasing
2) S(n) is decreasing when n is increasing
3) There is no upper bound for the speedup.
![Page 44: CS4402 – Parallel Computing Lecture 1: Classification of Parallel Computers Classification of Parallel Computation Important Laws of Parallel Compuation](https://reader035.vdocuments.us/reader035/viewer/2022062408/56649f045503460f94c186e5/html5/thumbnails/44.jpg)
44
To read:
1. John L. Gustafson, Re-evaluating Amdahl's Law,
http://www.scl.ameslab.gov/Publications/Gus/AmdahlsLaw/Amdahls.html
2. Yuan Shi, Re-evaluating Amdahl's and Gustafson’s Laws,
http://www.cis.temple.edu/~shi/docs/amdahl/amdahl.html
3. Wilkinson’s book,
1. sections of the laws of parallel computing
2. sections about types of parallel machines and compuation