grid computing, b. wilkinson, 20047a.1 computational grids

Grid Computing, B. Wilkinson, 2004 7a.1

Computational Grids


Computational Problems

• Problems that have lots of computations and usually lots of data.


Demand for Computational Speed

• Continual demand for greater computational speed from a computer system than is currently possible

• Areas requiring great computational speed include numerical modeling and simulation of scientific and engineering problems.

• Computations must be completed within a “reasonable” time period.


Grand Challenge Problems

One that cannot be solved in a reasonable amount of time with today’s computers. Obviously, an execution time of 10 years is always unreasonable.

Examples• Modeling large DNA structures• Global weather forecasting• Modeling motion of astronomical bodies.


Weather Forecasting

• Atmosphere modeled by dividing it into 3-dimensional cells.

• Calculations of each cell repeated many times to model passage of time.


Global Weather Forecasting Example

• Suppose global atmosphere divided into cells of size 1 mile 1 mile 1 mile to a height of 10 miles - about 5 108 cells.

• Suppose each calculation requires 200 floating point operations. In one time step, 1011 floating point operations necessary.

• To forecast weather over 7 day period using 1-minute intervals, a computer operating at 1Gflops (109 floating point operations/s) takes 106 seconds or over 10 days.


Modeling Motion of Astronomical Bodies

• Each body attracted to each other body by gravitational forces. Movement of each body predicted by calculating total force on each body.

• With N bodies, N - 1 forces to calculate for each

body, or ≈ N2 calculations.

(N log2 N for an efficient approximate algorithm.)

• After determining new positions of bodies, calculations repeated.


• A galaxy might have, say, 1011 stars.

• Even if each calculation done in 1 ms (extremely optimistic figure), it takes almost a year for one iteration using the N log2 N algorithm.

• 100 years for 100 iterations. Typically require millions of iterations.


Astrophysical N-body simulation by Scott Linssen (undergraduate UNC-Charlotte student).


High Performance Computing (HPC)

• Traditionally, achieved by using the multiple computers together - parallel computing.

• Simple idea! -- Using multiple computers (or processors) simultaneously should be able can solve the problem faster than a single computer.


Using multiple computers or processors

• Key concept - dividing problem into parts that can be computed simultaneously.

• Parallel programming - programming a computing platform consisting of more than one processor or computer.

• Concept very old (50 years).


High Performance Computing

• Long History:– Multiprocessor system of various

types (1950’s onwards)

– Supercomputers (1960s-80’s)

– Cluster computing (1990’s)

– Grid computing (2000’s) ??Maybe, but let’s first look at how to achieve HPC.


Speedup Factor

ts is execution time on a single processor

tp is execution time on a multiprocessor.

S(p) gives increase in speed by using multiprocessor.

Best sequential algorithm for single processor. Parallel algorithm usually different.

S(p) = Execution time using one processor (best sequential algorithm)Execution time using a multiprocessor with p processors

=ts

tp


Maximum Speedup

Maximum speedup is usually p with p processors (linear speedup).

Possible to get superlinear speedup (greater than p) but usually a specific reason such as:

– Extra memory in multiprocessor system– Non-deterministic algorithm


Maximum Speedup Amdahl’s law

Serial section Parallelizable sections

(a) One processor

(b) Multipleprocessors

fts (1 - f)ts

ts

(1 - f)ts /ptp

p processors


Speedup factor is given by:

This equation is known as Amdahl’s law

S(p) = ts p=fts + (1 − f)ts/p 1 + (p − 1)f


Speedup against number of processors

Even with infinite number of processors, max. speedup limited to 1/f .

Example: With only 5% of computation being serial, max. speedup 20, irrespective of number of

processors.

4

8

12

16

20

4 8 12 16 20

f = 20%

f = 10%

f = 5%

f = 0%

Number of processors, p


Superlinear Speedup Example Searching

(a) Searching each sub-space sequentially

tsts/p

Start Time

Δt

Solution foundxts/p

-Sub spacesearch

x indeterminate


(b) Searching each sub-space in parallel

Solution found

Δt


Question

What is the speed-up now?


Worst case for sequential search when solution found in last sub-space search. Then parallel version offers greatest benefit, i.e.

Sp()

p 1–p

------------⎝⎠⎛⎞ ts tΔ+×

tΔ---------------------------------------- ∞→=

asΔ t tends to zero


Least advantage for parallel version when solution found in first sub-space search of the sequential search, i.e.

Actual speed-up depends upon which subspace holds solution but could be extremely large.

Sp() tΔtΔ

-----1==


Types of Parallel Computers

Two principal types:

1. Single computer containing multiple processors - main memory is shared, hence called “Shared memory multiprocessor”

2. Multiple computer system


Conventional ComputerConsists of a processor executing a program stored in a (main) memory:

Each main memory location located by its address within a single memory space.

Main memory

Processor

Instructions (to processor)Data (to or from processor)


Shared Memory Multiprocessor• Extend single processor model - multiple processors

connected to multiple memory modules• Each processor can access any memory module:

Processors

Interconnectionnetwork

Memory modulesOneaddressspace


Examples

• Dual Pentiums

• Quad Pentiums


Programming Shared Memory Multiprocessors

• Threads - programmer decomposes program into parallel sequences (threads), each being able to access variables declared outside threads. Example: Pthreads

• Use sequential programming language with preprocessor compiler directives, constructs, or syntax to declare shared variables and specify parallelism. Examples: OpenMP (an industry standard), UPC (Unified Parallel C) -- needs compilers.


• Parallel programming language with syntax to express parallelism. Compiler creates executable code -- not now common.

• Use parallelizing compiler to convert regular sequential language programs into parallel executable code - also not now common.


Multiple ComputersMessage-passing multicomputer

Complete computers connected through and interconnection network:

Processor

Interconnectionnetwork

Local

Computers

Messages

memory


Networked Computers as a Computing Platform

• Became a very attractive alternative to expensive supercomputers and parallel computer systems for high-performance computing in 1990’s.

• Several early projects. Notable:– Berkeley NOW (network of workstations)

project. – NASA Beowulf project.


Key Hardware Advantages:

• Very high performance workstations and PCs readily available at low cost.

• Latest processors can easily be incorporated into the system as they become available.


Programming Clusters

• Usually based upon explicit message-passing.

• Common approach -- a set of user-level libraries for message passing. Example:– Parallel Virtual Machine (PVM) - late 1980’s.

Became very popular in mid 1990’s. – Message-Passing Interface (MPI) - standard

defined in 1990’s and now dominant.


Beowulf Clusters

• Name given to a group of interconnected “commodity” computers designed to achieve high performance with low cost.

• Typically using commodity interconnects (high-speed Ethernet).

• Typically Linux OS.

Beowulf comes from name given by NASA Goddard Space Flight Center cluster project.


Cluster Interconnects

• Originally fast Ethernet on low cost clusters• Gigabit Ethernet - easy upgrade path

More Specialized/Higher Performance• Myrinet - 2.4 Gbits/sec - disadvantage: single vendor• Infiniband - may be important as Infiniband

interfaces may be integrated on next generation PCs


Dedicated cluster with a master node

Dedicated Cluster User

Switch

Master node

Compute nodes

Up link

2nd Ethernetinterface

External network


WCU Department of Mathematics and CS

leo I cluster(now dismandled)

Front-end

16 ×16 100 /Mb s

8 Compute nodes

Internet

F or expansion

LEO I CLUSTER

SOFTWAREROCKSREDHA T LINUXMPICHGLOB 2.4US

P entium III500 , 384 Mhz Mb yte

main memory

P entium IIIs500 , 128 MhzMb yte mainmemory

Ether net switch

Being replaced with Pentium IVs and Gigabit Ethernet.


Message-Passing Programming using User-level Message Passing Libraries

Two primary mechanisms needed:

1. A method of creating separate processes for execution on different computers

2. A method of sending and receiving messages


Multiple program, multiple data model(MPMD)

Sourcefile

Executables

Processor 0 Processor p - 1

Compile to suitprocessor

Sourcefile


Single Program Multiple Data Model(SPMD)

• Different processes merged into one program.

• Control statements select different parts for

each processor to execute.

• All executables started together - static process creation


Single Program Multiple Data Model(SPMD)

Sourcefile

Executables

Processor 0 Processor p − 1

Compile to suitprocessor

Basic MPI way


Multiple Program Multiple Data Model(MPMD)

• Separate programs for each processor.

• One processor executes master process.

• Other processes started from within master process - dynamic process creation.


Multiple Program Multiple Data Model(MPMD)

Process 1

Process 2spawn();

Time

Start executionof process 2


“Point-to-point” send and receive routines

Process 1 Process 2

send(&x, 2);

recv(&y, 1);

x y

Movementof data

Generic syntax (actual formats later)

Passing a message between processes using send() and recv() library calls:


Synchronous Message PassingRoutines that return when message transfer completed.

Synchronous send routine• Waits until complete message can be accepted by

the receiving process before sending the message.

Synchronous receive routine• Waits until the message it is expecting arrives.


Synchronous send() and recv() using 3-way protocol

Process 1 Process 2

send();

recv();Suspend

Time

processAcknowledgment

MessageBoth processescontinue

(a) When send() occurs before recv()

Process 1 Process 2

recv();

send();Suspend

Time

process

Acknowledgment

MessageBoth processescontinue

(b) When recv() occurs before send()

Request to send

Request to send


• Synchronous routines intrinsically perform two actions:– They transfer data and – They synchronize processes.


Asynchronous Message Passing• Do not wait for actions to complete before

returning.

• More than one version depending upon semantics for returning.

• Usually require local storage for messages.

• They do not synchronize processes and allow processes to move forward sooner. Must be used with care.


MPI Definitions of Blocking and Non-Blocking

• Blocking - return after their local actions complete, though message transfer may not have been completed.

• Non-blocking - return immediately.

Assumes data storage not modified by subsequent statements prior to being used for transfer, and it is left to the programmer to ensure this.


How message-passing routines return before transfer completed

Process 1 Process 2

send();

recv();

Message buffer

Readmessage buffer

Continueprocess

Time

Message buffer needed between source and destination to hold message:


Asynchronous (blocking) routines changing to synchronous routines

• Buffers only of finite length and a point could be reached when send routine held up because all available buffer space exhausted.

• Then, send routine will wait until storage becomes re-available - i.e then routine behaves as a synchronous routine.


Message Tag

• Used to differentiate between different types of messages being sent.

• Message tag is carried within message.


Message Tag Example

Process 1 Process 2

send(&x,2,5);

recv(&y,1,5);

x y

Movementof data

W aits for a message from process 1 with a tag of 5

To send a data, x, with message tag 5 from process, 1, to destination process, 2, and assign to y:


Wild Card

• If message tag matching not required, wild card message tag used.

• Then, recv() will match with any send().


Collective Message Passing Routines

Have routines that send message(s) to a group of processes or receive message(s) from a group of processes

Higher efficiency than separate point-to-point routines although not absolutely necessary.


BroadcastSending same message to all processes concerned with problem.

bcast();

buf

bcast();

data

bcast();

datadata

Process 0 Process p − 1 1Process

Action

Code

MPI form


Scatter

scatter();

buf

scatter();

data

scatter();

datadata


Action

Code

MPI form

Sending each element of an array in root process to a separate process. Contents of ith location of array sent to ith process.


Gather

gather();

buf

gather();

data

gather();

datadata


Action

Code

MPI form

Having one process collect individual values from set of processes.


Reduce

reduce();

buf

reduce();

data

reduce();

datadata


+

Action

Code

MPI form

Gather operation combined with arithmetic/logical operation.Example: Values gathered and added together:


Grid Computing

• A grid is a form of multiple computer system.

• For solving computational problems, it could be viewed as the next step after cluster computing, and the same programming techniques used.

Why is this not necessarily true?


• VERY expensive, sending data across network costs millions of cycles

• Bandwidth shared with other users

• Links unreliable


Computational Strategies

• As a computing platform, a grid favors situations with absolute minimum communication between computers.

• Next class will look at these strategies and details of MPI programming.

grid computing, b. wilkinson, 20047a.1 computational grids

Documents

n log2 n algorithm

n bodies

execution time

passage of time

time step

parallel computing

reasonable time period

computer system