shared-memory model and threads intel software college introduction to parallel programming – part...
TRANSCRIPT
![Page 1: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/1.jpg)
Shared-Memory Model and Threads
Intel Software College
Introduction to Parallel Programming – Part 2
![Page 2: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/2.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
2Shared-Memory Model and Threads
Intel® Software College
Objectives
At the end of this module you should be able to:
Describe the shared-memory model of parallel programming
Describe the differences between the fork/join model and the general threads model
Demonstrate how to implement domain and functional decompositions using threads
Decide whether a variable in a multithreaded program should be shared or private
![Page 3: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/3.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
3Shared-Memory Model and Threads
Intel® Software College
Cooperating Parallel Processes
Parallel computing multiple processes working together to speed the solution of a task
Working together process cooperation
Kinds of cooperation
Sharing information (communication)
Keeping out of each other’s way (synchronization)
![Page 4: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/4.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
4Shared-Memory Model and Threads
Intel® Software College
The Shared-Memory Model
Shared Memory
CPU
PrivateMemory
CPU
PrivateMemory
CPU
PrivateMemory
![Page 5: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/5.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
5Shared-Memory Model and Threads
Intel® Software College
Evaluating Parallel Models
How do processes share information?
How do processes synchronize?
In shared-memory model, both accomplished through shared variables
Communication: buffer
Synchronization: semaphore
![Page 6: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/6.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
6Shared-Memory Model and Threads
Intel® Software College
Methodology
Study problem, sequential program, or code segment
Look for opportunities for parallelism
Use threads to express parallelism
![Page 7: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/7.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
7Shared-Memory Model and Threads
Intel® Software College
What Is a Process?
A program in some state of execution
Code
Data
Logical address space
Information about a process includes
Process state
Program counter
Values of CPU registers
Memory management information
![Page 8: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/8.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
8Shared-Memory Model and Threads
Intel® Software College
What Is a Thread?
“A unit of control within a process” — Carver and Tai
Main thread executes program’s “main” function
Main thread may create other threads to execute other functions
Threads have own program counter, copy of CPU registers, and stack of activation records
Threads share process’s data, code, address space, and other resources
Threads have lower overhead than processes
![Page 9: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/9.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
9Shared-Memory Model and Threads
Intel® Software College
Another Way of Looking at It
So why does this view matter?
Thread 0(Program Counter, Registers, etc.)
AddressSpace
File Descriptors
Other Data
Process
![Page 10: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/10.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
10Shared-Memory Model and Threads
Intel® Software College
Why This View Matters
Thread 0(Program Counter, Registers, etc.)
AddressSpace
File Descriptors
Other Data
Process
Thread 1(Program Counter, Registers, etc.)
![Page 11: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/11.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
11Shared-Memory Model and Threads
Intel® Software College
What Are Threads Good For?
Making programs easier to understand
Overlapping computation and I/O
Improving responsiveness of GUIs
Improving performance through parallel execution
![Page 12: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/12.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
12Shared-Memory Model and Threads
Intel® Software College
Fork/Join Programming Model
When program begins execution, only master thread active
Master thread executes sequential portions of program
For parallel portions of program, master thread forks (creates or awakens) additional threads
At join (end of parallel section of code), extra threads are suspended or die
![Page 13: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/13.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
13Shared-Memory Model and Threads
Intel® Software College
MasterThread
Fork JoinJoin Fork
Oth
er
Th
read
s
![Page 14: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/14.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
14Shared-Memory Model and Threads
Intel® Software College
Relating Fork/Join to Code
for {
}
for {
}
Sequential code
Parallel code
Sequential code
Parallel code
Sequential code
![Page 15: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/15.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
15Shared-Memory Model and Threads
Intel® Software College
More General Threads Model
When program begins execution, only one user thread, called the main thread, is active
The main thread can create other threads, which execute other functions
Created threads can also create additional threads
How this is done varies according to programming language or API
![Page 16: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/16.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
16Shared-Memory Model and Threads
Intel® Software College
Fork/Join versus General Model
You can implement fork/join in the general model
Hence fork/join a special case of general model
More structured
More easily optimized
General model
More flexible
Better support for functional decompositions
![Page 17: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/17.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
17Shared-Memory Model and Threads
Intel® Software College
Incremental Parallelization
Sequential program a special case of threaded program
Programmers can add parallelism incrementally
Profile program execution
Repeat
Choose best opportunity for parallelization
Transform sequential code into parallel code
Until further improvements not worth the effort
![Page 18: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/18.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
18Shared-Memory Model and Threads
Intel® Software College
Utility of Threads
Threads are flexible enough to implement
Domain decomposition
Functional decomposition
Pipelining
![Page 19: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/19.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
19Shared-Memory Model and Threads
Intel® Software College
Domain Decomposition Using Threads
SharedMemory
Thread 0 Thread 2
Thread 1
f ( )
f ( )
f ( )
![Page 20: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/20.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
20Shared-Memory Model and Threads
Intel® Software College
SharedMemory
Functional Decomposition Using Threads
Thread 0 Thread 1
e ( )
g ( )h ( )
f ( )
![Page 21: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/21.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
21Shared-Memory Model and Threads
Intel® Software College
Pipelining Using Threads
Thread 0 Thread 2Thread 1
Shared Memory
Input Output
e ( ) f ( ) g ( )
Dataset 2
Data sets5, 6, ...
Dataset 4 Data
set 3 Dataset 1
![Page 22: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/22.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
22Shared-Memory Model and Threads
Intel® Software College
Shared versus Private Variables
SharedVariables
PrivateVariables
PrivateVariables
Thread
Thread
![Page 23: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/23.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
23Shared-Memory Model and Threads
Intel® Software College
Domain Decomposition
Sequential Code:
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
![Page 24: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/24.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
24Shared-Memory Model and Threads
Intel® Software College
Domain Decomposition
Sequential Code:
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Thread 0:
for (i = 0; i < 500; i++) a[i] = foo(i);
Thread 1:
for (i = 500; i < 1000; i++) a[i] = foo(i);
![Page 25: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/25.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
25Shared-Memory Model and Threads
Intel® Software College
Domain Decomposition
Sequential Code:
int a[1000], i;
for (i = 0; i < 1000; i++) a[i] = foo(i);
Thread 0:
for (i = 0; i < 500; i++) a[i] = foo(i);
Thread 1:
for (i = 500; i < 1000; i++) a[i] = foo(i);
SharedPrivate
![Page 26: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/26.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
26Shared-Memory Model and Threads
Intel® Software College
Functional Decompositionint e;
main () { int x[10], j, k, m; j = f(x, k); m = g(x, k); ...}
int f(int *x, int k){ int a; a = e * x[k] * x[k]; return a;}
int g(int *x, int k){ int a; k = k-1; a = e / x[k]; return a;}
![Page 27: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/27.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
27Shared-Memory Model and Threads
Intel® Software College
Functional Decompositionint e;
main () { int x[10], j, k, m; j = f(x, k); m = g(x, k);}
int f(int *x, int k){ int a; a = e * x[k] * x[k]; return a;}
int g(int *x, int k){ int a; k = k-1; a = e / x[k]; return a;}
Thread 0
Thread 1
![Page 28: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/28.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
28Shared-Memory Model and Threads
Intel® Software College
Functional Decompositionvolatile int e;
main () { int x[10], j, k, m; j = f(x, k); m = g(x, k);}
int f(int *x, int k){ int a; a = e * x[k] * x[k]; return a;}
int g(int *x, int k){ int a; k = k-1; a = e / x[k]; return a;}
Thread 0
Thread 1
Static variable: Shared
![Page 29: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/29.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
29Shared-Memory Model and Threads
Intel® Software College
Functional Decompositionvolatile int e;
main () { int x[10], j, k, m; j = f(x, k); m = g(x, k);}
int f(int *x, int k){ int a; a = e * x[k] * x[k]; return a;}
int g(int *x, int k){ int a; k = k-1; a = e / x[k]; return a;}
Thread 0
Thread 1
Heap variable: Shared
![Page 30: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/30.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
30Shared-Memory Model and Threads
Intel® Software College
Functional Decompositionvolatile int e;
main () { int x[10], j, k, m; j = f(x, k); m = g(x. k);}
int f(int *x, int k){ int a; a = e * x[k] * x[k]; return a;}
int g(int *x, int k){ int a; k = k-1; a = e / x[k]; return a;}
Thread 0
Thread 1
Function’s local variables: Private
![Page 31: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/31.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
31Shared-Memory Model and Threads
Intel® Software College
Shared and Private Variables
Shared variables
Static variables
Heap variables
Contents of run-time stack at time of call
Private variables
Loop index variables
Run-time stack of functions invoked by thread
![Page 32: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/32.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
32Shared-Memory Model and Threads
Intel® Software College
The Shared-Memory Model (Reprise)
Shared Memory
CPU
PrivateMemory
CPU
PrivateMemory
CPU
PrivateMemory
![Page 33: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/33.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
33Shared-Memory Model and Threads
Intel® Software College
The Threads Model
Shared Variables
Thread
PrivateVariables
Thread
PrivateVariables
Thread
PrivateVariables
![Page 34: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/34.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
34Shared-Memory Model and Threads
Intel® Software College
References
Jim Beveridge and Robert Wiener, Multithreading Applications in Win32®, Addison-Wesley (1997).
David R. Butenhof, Programming with POSIX® Threads, Addison-Wesley, (1997).
Richard H. Carver and Kuo-Chung Tai, Modern Multithreading: Implementing, Testing, and Debugging Java and C++/Pthreads/ Win32 Programs, Wiley-Interscience (2006).
Doug Lea, “A Java Fork/Join Framework,” Proceedings of the ACM 2000 Conference on Java Grande, pp. 36-43.
Michael J. Quinn, Parallel Programming in C with MPI and OpenMP, McGraw-Hill (2004).
![Page 35: Shared-Memory Model and Threads Intel Software College Introduction to Parallel Programming – Part 2](https://reader034.vdocuments.us/reader034/viewer/2022052618/5513f25e5503463a298b609f/html5/thumbnails/35.jpg)
Copyright © 2006, Intel Corporation. All rights reserved.
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
35Shared-Memory Model and Threads
Intel® Software College