lappeenranta university of technology / jp ct30a7001 concurrent and parallel computing introduction...
Post on 28-Jan-2016
214 Views
Preview:
TRANSCRIPT
Lappeenranta University of Technology / JP
CT30A7001 Concurrent and Parallel
ComputingIntroduction to concurrent
and parallel computing
Lappeenranta University of Technology / JP
Definitions• Concurrent computing
– Concurrent computing is the concurrent (simultaneous) execution of multiple interacting computational tasks. Concurrent computing is related to parallel computing, but focuses more on the interactions between tasks.
• Parallel computing– Parallel computing is a form of computation in which many
instructions are carried out simultaneously.• Distributed computing
– In distributed computing a program is split up into parts that run simultaneously on multiple computers communicating over a network. Distributed computing is a form of parallel computing, but parallel computing is most commonly used to describe program parts running simultaneously on multiple processors in the same computer.
• Grid computing– Grid computing is a form of distributed computing whereby a "super
and virtual computer" is composed of a cluster of networked, loosely-coupled computers, acting in concert to perform very large tasks.
Lappeenranta University of Technology / JP
Motivation
• The ever increasing need of computing power has traditionally been fulfilled by producing new more powerful processors=> physical limits restrict the growth
(Clock rates have increased from 40 MHz (1988) to over 2 GHz (2002); Multiple instructions/cycle, multiple processors)
Lappeenranta University of Technology / JP
Performance – Cost ratio
Cost
Perf
orm
an
ce
1960s
1990s
1980s
1970s
Lappeenranta University of Technology / JP
Can’t you go any faster ?
• More power can be achieved by using multiple computers (parallel computing)=> allows us to break the limits set for the
single processor
Lappeenranta University of Technology / JP
Trends ?
• Faster processors will evolve !!– Moore’s law (statement)– Development of parallel applications take
time ?
• However– Memory access times lag behind (10%
growth)– The data transfer speed lags behind the
processing capacity– Software lags behind the hardware
Lappeenranta University of Technology / JP
Moore’s law
Lappeenranta University of Technology / JP
Parallel computing ?
• Non-”von Neumann” type of computing• Some problem is solved at the same time
in two or more locations (processors …)– Example: library workers– Problem need to have some inherent
parallelism– Problem is solved in some parallel
architecture– Problem is solved by some algorithm
Lappeenranta University of Technology / JP
Library workers example
• 1 or more workers return books into shelves
• Aisles are divided according to the alphabet– A-E, F-J, K-O, P-T, U-Z
• How to divide work among workers ?
Lappeenranta University of Technology / JP
1 Worker
A-E
F-JK-O
P-T
U-Z
AlgorithmWork
Lappeenranta University of Technology / JP
2 Workers
A-E
F-JK-O
P-T
U-Z
Lappeenranta University of Technology / JP
2 Workers
A-E
F-JK-O
P-T
U-Z
Lappeenranta University of Technology / JP
2 Workers
A-E
F-JK-O
P-T
U-Z
Lappeenranta University of Technology / JP
Things to consider
• The amount of work– When do we need several workers ?
• The number of workers– How many workers is enough ?
• Internal algorithm– How do we divide the work ?
• External algorithm (communication)– How is the work is synchronized
Lappeenranta University of Technology / JP
Utilization
Lappeenranta University of Technology / JP
Inherent parallelism
• Parallel computing can only be applied to those problems that have some inherent parallelism
• Parallelism is an inherent property that cannot be increased !– Some problems are naturally sequential– One can affect how well the parallelism
is used
Lappeenranta University of Technology / JP
Benefits of parallel computing
• Many problems are naturally parallel although they are modeled sequentially– Interrupts, polling => loss of information
• There are problems that cannot be solved sequentially
• Performance may increase linearly (some times even more)
• In future natural parallelism is commonly utilized – There are no sequential computers (inherent
parallelism)
Lappeenranta University of Technology / JP
Parallel computing goals• Different types of goals:
– Reduce the execution timeExample: weather forecasts
– Solve bigger problems (more accurate models)Example: Mobile network simulations
– Run multiple applications at the same time Multithreading
– Natural approach
Lappeenranta University of Technology / JP
Example: Weather forecast
Compute weather forecast for 3000*3000 miles area up to 11 miles high
Divide the model into parts of size 0.1*0.1*0.1 miles => 1011 parts Compute weather forecast for two days (48 hours)
Computing one part takes ~100 operations and parts are computed ~100 times within these 48 hours => 1015 operations to compute
If we have a single 1000 MIPS workstation computations take ~280 hours !!!
If we use 1000 workstations capable of 100 MIPS the computation takes only ~3 hours
Lappeenranta University of Technology / JP
Applicability ?
• Parallel computing can be used for solving all those problems that contain some parallel properties
• Time consuming task !– Performance– Portability– Scalability
Lappeenranta University of Technology / JP
Application areas
• Technical applications– floating point operations, computing
grids
• Business applications– decision support systems,
simulations, ...
• Network applications– multimedia, video-on-demand, ...
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
Parallel architectures
• Problems are computed in parallel in parallel architectures
• There exits several different approaches:– Massively parallel processors (MPP)– Symmetrical multiprocessor (SMP)– Vector -processors– Cluster machines
Lappeenranta University of Technology / JP
Examples Cray-1
Avalon Cluster - 140
Lappeenranta University of Technology / JP
Rank Manufacturer Computer/Procs Rmax
Rpeak Installation Site
Country/Year
1 NECEarth-Simulator/ 5120
35860.00
40960.00
Japan/2002
2 Hewlett-PackardASCI Q - AlphaServer SC ES45/1.25 GHz/ 4096
7727.0010240.00
USA/2002
3 Hewlett-PackardASCI Q - AlphaServer SC ES45/1.25 GHz/ 4096
7727.0010240.00
USA/2002
4 IBMASCI White, SP Power3 375 MHz/ 8192
7226.0012288.00
USA/2000
5 Linux NetworXMCR Linux Cluster Xeon 2.4 GHz - Quadrics/ 2304
5694.0011060.00
USA/2002
6 Hewlett-PackardAlphaServer SC ES45/1 GHz/ 3016
4463.006032.00
USA/2001
7 Hewlett-PackardAlphaServer SC ES45/1 GHz/ 2560
3980.005120.00
France/2001
8 HPTiAspen Systems, Dual Xeon 2.2 GHz - Myrinet2000/
1536
3337.006758.00
USA/2002
9 IBMpSeries 690 Turbo 1.3GHz/ 1280
3241.006656.00
UK/2002
10 IBMpSeries 690 Turbo 1.3GHz/ 1216
3164.006323.00
USA/2002
TOP500 - 2003
Lappeenranta University of Technology / JP
Top 500 - 2006
TOP 500 - 2010
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
No 54
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
Hyperthreadingwithin processor
Use of threadswithin workstation
Message passing inside cluster
GRID
Nu
mb
er
of
poss
ible
use
rs in
crease
s
Use
r inte
ractio
n in
crease
s (req
uire
s kn
ow
led
ge)
Geog
rap
hic size
incre
ase
s
Lappeenranta University of Technology / JP
Algorithms
• Selection of the algorithm is important as different algorithms have different suitability
• The fastest sequential algorithm may not be the best algorithm to be parallelized !
• The selection of algorithm may depend on the selected parallel architecture
Lappeenranta University of Technology / JP
Example: Sorting a sequence
• Traditional approach: O(nlogn) time
• Simple Merge-Sort algorithm:O(logn loglogn) time
• Pipelined sorting algorithm:O(logn) time
Lappeenranta University of Technology / JP
Lappeenranta University of Technology / JP
Note !
• Parallel computing does NOT help if the bottleneck of the system is in memory, in disk system or in network !
• Parallel computing may produce surprises– resources are competed=> winner is not known beforehand=> non-deterministic behavior of the program=> deadlock ? (mutual exclusion)
Lappeenranta University of Technology / JP
Aspects slowing the future• Not all problems can be solved in parallel
– Example: Digging a well vs. a dike
• Parallel programming requires a new way of thinking (compare OO-programming)
• Sequential algorithm may not be suitable for parallel execution
• Partitioning and load balancing are not an easy task
• Tools are to be developed at the moment
Lappeenranta University of Technology / JP
Tools for parallel computing
• Threads– http://www.sun.com/software/Products/Developer-products
/threads/
• PVM (Parallel Virtual Machine)– http://www.epm.ornl.gov/pvm/
• MPI (Message Passing Interface)– http://www.epm.ornl.gov/~walker/mpi/– http://www.netlib.org/utk/papers/mpi-book/mpi-book.html
• OpenMP
top related