![Page 1: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/1.jpg)
CS 470/570:Introduction to Parallel and Distributed
Computing
![Page 2: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/2.jpg)
Lecture 1 Outline
• Course Details• General Issues in Parallel Software Development• Parallel Architectures• Parallel Programming Models
![Page 3: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/3.jpg)
General Information
• Instructor: Thomas Gendreau• Office: 211 Wing• Office Phone: 785-6813• Office Hours: – Monday, Wednesday, Friday 2:15-3:15– Thursday 1:00-3:00 – Feel free to see me at times other than my office hours
• email: [email protected]• web: cs.uwlax.edu/~gendreau• Textbook: Introduction to Parallel Programming by Peter S.
Pacheco
![Page 4: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/4.jpg)
Grading
• 240 points: 12 quizzes (best 12 out of 14)• 100 points: Assignments • 100 points Cumulative final exam – 4:45-6:45 Wednesday December 17
• 440 total points• A quiz will be given in class every Friday and on Wednesday
November 21st. No makeup quizzes will be given. Unusual situations such as multi-week illness will be handled on an individual basis.
![Page 5: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/5.jpg)
General Issues/Terminology
• Identify possible parallelism• Match amount of parallelism to the architecture• Scalability• Synchronization
– Mutual exclusion
• Choose a parallel programming model• Performance
– Speedup– Amdahl’s law
• Data distribution
![Page 6: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/6.jpg)
General Issues/Terminology
• Why study parallel programming now?– Evolution of computer architecture– Computer networks as potential parallel computers– Manage large data sets– Solve computationally difficult problems
![Page 7: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/7.jpg)
General Issues/Terminology
• Parallelism in computer systems– Multiprogrammed OS– Instruction level parallelism (pipelines)– Vector processing– Multiprocessor machines• Multicores
– Networks of processors
![Page 8: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/8.jpg)
Parallel Architectures
• Classics Taxonomy (Flynn)– Single Instruction Single Data Stream: SISD– Single Instruction Multiple Data Streams: SIMD– Multiple Instructions Multiple Data Streams: MIMD
![Page 9: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/9.jpg)
SIMD
• Single control Unit • Multiple ALUs• All ALUs execute the same instruction on local
data
![Page 10: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/10.jpg)
MIMD
• Multiple control units and ALUs • Separate stream of control on each processor• Possible organization– Shared Physical Address Space• Uniform or Non-uniform Memory Access• Cache Coherance issues
– Distributed Physical Address Space• Network of Computers
![Page 11: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/11.jpg)
Parallel Programming Models
• How is parallelism expressed?– Process– Task– Thread– Implicitly
• How is information accurately shared among parallel actions?
• How are parallel actions synchronized?
![Page 12: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/12.jpg)
Parallel Programming Models
• Shared Address Space Programming– Shared variables
• Message Based Programming– Send/receive values among cooperating processes
• Programming models are independent of the underlying architecture
• SPMD– Single Program Multiple Data
![Page 13: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/13.jpg)
Shared Address Space Programming
• Processes• Threads • Directive Model
![Page 14: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/14.jpg)
Unix Processes
• Heavyweight processes– Requires the creation of a copy of the parent’s
data space– Process creation has high overhead
• Example functions– fork– waitpid
![Page 15: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/15.jpg)
Threads
• Lightweight process– Does not require the creation of a copy of the
parent’s data space
• Example pthread functions– pthread_create– pthread_join
![Page 16: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/16.jpg)
Directive Based Parallel Programming
• Directives in source code tell compiler where parallelism should be used
• Threads are implicitly created• OpenMP– Compiler directives– #pragma omp contruct clause– #pragma omp parallel for
![Page 17: CS 470/570:Introduction to Parallel and Distributed Computing](https://reader035.vdocuments.us/reader035/viewer/2022072010/56649dbc5503460f94aaf0a7/html5/thumbnails/17.jpg)
Message Based Parallel Programming
• Send/Receive messages to share values• Synchronous communication• Asynchronous communication• Blocking• Non-blocking