A Presentation on Parallel Computing

-Ameya Waghmare(Rno 41,BE CSE)Guided by-Dr.R.P.Adgaonkar(HOD),CSE Dept.

• Parallel computing is a form of computation in Parallel computing is a form of computation in which many instructions are carried out which many instructions are carried out simultaneously operating on the principle that operating on the principle that large problems can often be divided into large problems can often be divided into smaller ones, which are then solved smaller ones, which are then solved concurrently (in parallel).concurrently (in parallel).

Why is it required?Why is it required?

With the increased use of computers in every With the increased use of computers in every sphere of human activity,computer scientists sphere of human activity,computer scientists are faced with two crucial issues today.are faced with two crucial issues today.

Processing has to be done faster like never Processing has to be done faster like never

beforebefore Larger or complex computation problems Larger or complex computation problems

need to be solvedneed to be solved

Increasing the number of transistors as per Increasing the number of transistors as per

Moore’s Law isn’t a solution,as it also Moore’s Law isn’t a solution,as it also increases the frequency scaling and power increases the frequency scaling and power consumption.consumption.

Power consumption has been a major issue Power consumption has been a major issue recently,as it causes a problem of processor recently,as it causes a problem of processor heating.heating.

The perfect solution is The perfect solution is PARALLELISMPARALLELISM In hardware as well as software.In hardware as well as software.

Difference With Distributed Difference With Distributed ComputingComputing

When different processors/computers work on When different processors/computers work on a single common goal,it is a single common goal,it is parallel computing.parallel computing.

Eg.Ten men pulling a rope to lift up one Eg.Ten men pulling a rope to lift up one rock,supercomputers implement parallel rock,supercomputers implement parallel computing.computing.

Distributed computing Distributed computing is where several different is where several different computers work separately on a multi-faceted computers work separately on a multi-faceted computing workload.computing workload.

Eg.Ten men pulling ten ropes to lift ten different Eg.Ten men pulling ten ropes to lift ten different rocks,employees working in an office doing their rocks,employees working in an office doing their own work. own work.

Difference With Cluster ComputingDifference With Cluster Computing

A A computer clustercomputer cluster is a group of linked computers, is a group of linked computers, working together closely so that in many respects they working together closely so that in many respects they form a single computer.form a single computer.

Eg.,In an office of 50 employees,group of 15 doing Eg.,In an office of 50 employees,group of 15 doing some work,25 some other,and remaining 10 some work,25 some other,and remaining 10 something else.something else.

Similarly,in a network of 20 computers,16 working on a Similarly,in a network of 20 computers,16 working on a common goal,whereas 4 on some other common goal.common goal,whereas 4 on some other common goal.

Cluster Computing Cluster Computing is a specific case of parallel is a specific case of parallel computing.computing.

Difference With Grid ComputingDifference With Grid Computing

Grid ComputingGrid Computing makes use of computers communicating over the Internet to work on a given problem.

Eg.When 3 persons,one of them from USA,another from Japan and a third from Norway are working together online on a common project.

Websites like Wikipedia,Yahoo!Answers,YouTube,FlickR or open source OS like Linux are examples of grid computing.

Again,it serves a san example of parallel computing.

The Concept Of PipeliningThe Concept Of Pipelining

In computing, a In computing, a pipelinepipeline is a set of data is a set of data processing elements connected in series, so processing elements connected in series, so that the output of one element is the input of that the output of one element is the input of the next one. The elements of a pipeline are the next one. The elements of a pipeline are often executed in parallel or in time-sliced often executed in parallel or in time-sliced fashion; in that case, some amount of buffer fashion; in that case, some amount of buffer storage is often inserted between elements.storage is often inserted between elements.

Approaches To Parallel ComputingApproaches To Parallel Computing

Flynn’s TaxonomyFlynn’s Taxonomy SISD(Single Instruction Single Data)SISD(Single Instruction Single Data) SIMD(Single Instruction Multiple Data)SIMD(Single Instruction Multiple Data) MISD(Multiple Instruction Single Data)MISD(Multiple Instruction Single Data) MIMD(Multiple Instruction Multiple Data)MIMD(Multiple Instruction Multiple Data)

Approaches Based On ComputationApproaches Based On Computation

Massively ParallelMassively Parallel Embarrassingly Parallel Embarrassingly Parallel Grand Challenge ProblemsGrand Challenge Problems

Massively Parallel SystemsMassively Parallel Systems

It signifies the presence of many It signifies the presence of many independent units or entire microprocessors, independent units or entire microprocessors, that run in parallel. that run in parallel.

The term massive connotes hundreds if not The term massive connotes hundreds if not

thousands of such units.thousands of such units.

Example:the Earth Example:the Earth Simulator(Supercomputer from 2002-Simulator(Supercomputer from 2002-

2004)2004)

Embarrassingly Parallel SystemsEmbarrassingly Parallel Systems

An embarrassingly parallel system is one for An embarrassingly parallel system is one for which no particular effort is needed to segment which no particular effort is needed to segment the problem into a very large number of parallel the problem into a very large number of parallel tasks.tasks.

Examples include surfing two websites Examples include surfing two websites simultaneously , or running two applications on a simultaneously , or running two applications on a home computer.home computer.

They lie to an end of spectrum of parallelisation They lie to an end of spectrum of parallelisation where tasks can be readily parallelised.where tasks can be readily parallelised.

Grand Challenge ProblemsGrand Challenge Problems

A grand challenge is a fundamental problem in A grand challenge is a fundamental problem in science or engineering, with broad applications, science or engineering, with broad applications, whose solution would be enabled by the whose solution would be enabled by the application of high performance computing application of high performance computing resources that could become available in the resources that could become available in the near future.near future.

Grand ChallengesGrand Challenges were USA policy terms set as were USA policy terms set as goals in the late 1980s for funding high-goals in the late 1980s for funding high-performance computing and communications performance computing and communications research in part in response to the Japanese 5th research in part in response to the Japanese 5th Generation (or Next Generation) 10-year project.Generation (or Next Generation) 10-year project.

Types Of ParallelismTypes Of Parallelism

• Bit-LevelBit-Level• InstructionalInstructional• DataData• TaskTask

Bit-Level ParallelismBit-Level Parallelism

When an 8-bit processor needs to add two 16-When an 8-bit processor needs to add two 16-bit integers,it’s to be done in two steps. bit integers,it’s to be done in two steps.

The processor must first add the 8 lower-The processor must first add the 8 lower-order bits from each integer using the order bits from each integer using the standard addition instruction, standard addition instruction,

Then add the 8 higher-order bits using an add-Then add the 8 higher-order bits using an add-with-carry instruction and the carry bit from with-carry instruction and the carry bit from the lower order additionthe lower order addition

Instruction Level ParallelismInstruction Level Parallelism

The instructions given to a computer for The instructions given to a computer for processing can be divided into groups, or re-processing can be divided into groups, or re-ordered and then processed without changing ordered and then processed without changing the final result.the final result.

This is known as instruction-level parallelism.This is known as instruction-level parallelism. i.e.,ILP.i.e.,ILP.

An ExampleAn Example

1. e = a + b1. e = a + b 2. f = c + d2. f = c + d 3. g = e * f3. g = e * f Here, instruction 3 is dependent on Here, instruction 3 is dependent on

instruction 1 and 2 .instruction 1 and 2 . However,instruction 1 and 2 can be However,instruction 1 and 2 can be

independently processed.independently processed.

Data ParallelismData Parallelism

Data parallelism focuses on distributing the Data parallelism focuses on distributing the

data across different parallel computing data across different parallel computing nodes.nodes.

It is also called as loop-level parallelism.It is also called as loop-level parallelism.

An IllustrationAn Illustration

In a data parallel implementation, CPU A In a data parallel implementation, CPU A could add all elements from the top half of the could add all elements from the top half of the matrices, while CPU B could add all elements matrices, while CPU B could add all elements from the bottom half of the matrices. from the bottom half of the matrices.

Since the two processors work in parallel, the Since the two processors work in parallel, the job of performing matrix addition would take job of performing matrix addition would take one half the time of performing the same one half the time of performing the same operation in serial using one CPU alone.operation in serial using one CPU alone.

Task ParallelismTask Parallelism

Task Parallelism focuses on distribution of Task Parallelism focuses on distribution of tasks across different processors.tasks across different processors.

It is also known as functional parallelism or It is also known as functional parallelism or control parallelismcontrol parallelism

An ExampleAn Example

As a simple example, if we are running code As a simple example, if we are running code on a 2-processor system (CPUs "a" & "b") in a on a 2-processor system (CPUs "a" & "b") in a parallel environment and we wish to do tasks parallel environment and we wish to do tasks "A" and "B" , it is possible to tell CPU "a" to do "A" and "B" , it is possible to tell CPU "a" to do task "A" and CPU "b" to do task 'B" task "A" and CPU "b" to do task 'B" simultaneously, thereby reducing the runtime simultaneously, thereby reducing the runtime of the execution.of the execution.

Key Difference Between Data And Task Key Difference Between Data And Task ParallelismParallelism

Data ParallelismData Parallelism It is the division of

threads(processes) or instructions or tasks internally into sub-parts for execution.

A task ‘A’ is divided into sub-parts and then processed.

Task ParallelismTask Parallelism It is the divisions among It is the divisions among

threads(processes) or threads(processes) or instructions or tasks instructions or tasks themselves for execution.themselves for execution.

A task ‘A’ and task ‘B’ A task ‘A’ and task ‘B’ are processed separately are processed separately by different processors.by different processors.

Implementation Of Parallel ComputImplementation Of Parallel Computing In Software

When implemented in software(or rather When implemented in software(or rather algorithms), the terminology calls it ‘parallel algorithms), the terminology calls it ‘parallel programming’.programming’.

An algorithm is split into pieces and then An algorithm is split into pieces and then

executed, as seen earlier.executed, as seen earlier.

Important Points In Parallel Important Points In Parallel ProgrammingProgramming

Dependencies-A typical scenario when line 6 Dependencies-A typical scenario when line 6 of an algorithm is dependent on lines 2,3,4 of an algorithm is dependent on lines 2,3,4 and 5and 5

Application Checkpoints-Just like saving the Application Checkpoints-Just like saving the algorithm, or like creating a backup point.algorithm, or like creating a backup point.

Automatic Parallelisation-Identifying Automatic Parallelisation-Identifying dependencies and parallelising algorithms dependencies and parallelising algorithms automatically.This has achieved limited automatically.This has achieved limited success. success.

Implementation Of Parallel Computing Implementation Of Parallel Computing In HardwareIn Hardware

When implemented in hardware, it is called as When implemented in hardware, it is called as ‘parallel processing’.‘parallel processing’.

Typically,when a chunk of load for execution Typically,when a chunk of load for execution

is divided for processing by units like is divided for processing by units like cores,processors,CPUs,etc. cores,processors,CPUs,etc.

An Example:Intel Xeon Series An Example:Intel Xeon Series ProcessorsProcessors

ReferencesReferenceshttp://portal.acm.org/citation.cfm?http://portal.acm.org/citation.cfm?

id=290768&coll=portal&dl=ACMid=290768&coll=portal&dl=ACMhttp://www-users.cs.umn.edu/~karypis/parbook/http://www-users.cs.umn.edu/~karypis/parbook/www.cs.berkeley.edu/~yelick/cs267-sp04/lectures/01/www.cs.berkeley.edu/~yelick/cs267-sp04/lectures/01/

lect01-introlect01-introwww.cs.berkeley.edu/~demmel/cs267_Spr99/Lectures/www.cs.berkeley.edu/~demmel/cs267_Spr99/Lectures/

Lect_01_1999bLect_01_1999bhttp://www.intel.com/technology/computing/dual-http://www.intel.com/technology/computing/dual-

core/demo/popup/dualcore.swfcore/demo/popup/dualcore.swfwww.parallel.ru/ftp/computers/intel/xeon/24896607.pdfwww.intel.comwww.intel.com

ANY QUERIES? Thank You!