cs 159 two lecture introduction parallel processing: a ......two lecture introduction parallel...

CS 159Two Lecture Introduction

Parallel Processing:

A Hardware Solution&

A Software Challenge

We’re on the Road to Parallel Processing

Outline

Hardware Solution (Day 1)

Software Challenge (Day 2)

Opportunities

Outline

Hardware Solution

Software Challenge

Opportunities

Technical

Technical

Technical

Outline

Hardware Solution

Software Challenge

Opportunities

Business

Business

Business

Technical

Technical

Technical

Outline

Hardware Solution

Software Challenge

Opportunities

Business Technical

Hardware Solution - Technical

Evolution of Computer Architectures

Computer HardwareEvolution Highway

Micro Macro

Hardware&

Software

Car&

Driver



Micro-Scopic View



Micro-Scopic View

29,000 Transistors 291,000,000

5 MHz Clock Frequency 2.9 GHz

1978 Year 2006

Intel 8086 Intel Core 2 Duo



Micro-Scopic View

In 28 Years from 1978 to 2006:

• Number of Transistors Increased 10,034X

• Clock Frequency Increased 586X

Primary Driver / Facilitator was (and is) Moore’s Law:

• Number of Transistors Doubles every 18-24 Months

• Stated by Gordon Moore, Intel Co-Founder in 1965

• Prediction has been proven valid over a long term

• “Prediction” has been the “Law” for over 40 years



Micro-Scopic View

Historically, Huge Performance Gains came from

Huge Clock Frequency Increases

Unfortunately ……….

Source: Patterson, Computer Organization and Design



Micro-Scopic View

Clock Rate Limits Have Been Reached

Blocked by Heat and Power “Wall”



Micro-Scopic View

• Power (and Heat) Grows as Frequency3

Power Voltage2 x Frequency

Voltage Frequency

Power Frequency3

Single Core Multi-Core



Micro-Scopic View

• How can HW Performance Continue to Increase?

Single Core vs. Dual Core

Single Core clocked at 2f Dual Core clocked at f

2f Throughput f + f

8f 3 Heat f 3 + f 3

Sequential Processing Parallel Processing



Micro-Scopic View

Instruction-Level Parallelism (ILP) was also Heavily Used

Implemented On-Chip via Hardware

Transparent to Software (No impact on Programmers)

We will Study Two Types:

Pipelining (Intra-Instruction Parallelism)

Multi-Function Units (Inter-Instruction Parallelism)

ILP has provided reasonable speedups in the past,

Unfortunately…….



Micro-Scopic View Instruction-Level Parallelism Limits have been Reached too

100

10

1

Single-IssuePipelined

Superscalar(LimitedLook-Ahead)

Superscalar(Aggressive Look-Ahead)

AggressivenessOf ILP

Diminishing returns w.r.t.larger instruction window,higher issue-width

Power growsexponentially

Watts,IPC

Effort

Performance

ScalarIn-Order

ModeratePipelining,Look-Ahead

Very Deep Pipe,AggressiveLook-Ahead

Made sense to goSuperscalar andMulti-Function:Good ROI

Very little gain forsubstantial effort


Evolution of Computer Architectures Gain - to - Effort Ratio of ILP beyond “Knee” of Curve Diminishing Returns due to

Increased Cost and Complexity of Extracting ILP



Micro-Scopic View Summary

Clock Frequency Scaling Limits have been Reached

Era of Single Core Performance Increases has Ended

All Future Micro-Processor Designs will be Multi-Core

Evident in Chip Manufacturer’s RoadMaps

No More “Free Lunch” for Software Programmers

Multiple Cores Will Directly Expose Parallelism to SW

Instruction Level Parallelism Limits have been Reached




1

Moore’s Law Continues to 2x Transistors / 24 mos, but

It will be used to Increase Number of Cores Instead

2 4 ……

Single Core Multi-Core




Computer HardwareEvolution Highway

Micro Macro



Macro-Scopic View

Personal Computer

Nodes: 1

Location: Desktop



Macro-Scopic View

Cluster Computer

Nodes: 10’s – 100’s

Location: Local



Macro-Scopic View

ExampleCluster

Computer



Macro-Scopic View

Grid Computer

Nodes: 1,000’s

Location: Distributed



Macro-Scopic View

ExampleGrid

Computer



Macro-Scopic View

Cloud Computer

Nodes: 10,000’s

Location: Highly

Distributed



Macro-Scopic View

ExampleCloud

ComputerEC2

AzureApp Engine



Macro-Scopic View Summary

Single Node Many Nodes


Hardware Solution - Business



Micro

Macro


Computer Processing Power:

• Has a Highly Elastic Supply and Demand Curve

• Increased Supply Generates Increased Demand

Software is Like a Gas

• It Expands to Fill any size Hardware Container

Computer Power

New Applications


Hardware

Software


Goal of Computer Hardware and Software Designers

• Continually Increase Performance and Lower Cost

• Operate at Optimum Point on Technology Curve


Even Visionaries Sometime Forget

No Matter How Much Computer Power People Have,

They Always Want More

“640K should be enough for anybody”

“There is a world market for maybe five computers”

Bill Gates, 1981

Thomas Watson, 1943


Technology Curve

Cost / Performancevs.

Performance

Performance

Cost

Performance

Under-Utilization

Optimum UtilizationOver-

Utilization


Technology Curve

Sequential Processing


Performance

Performance

Cost

Performance

2XX

$

$$$

$$$$$$


Technology Curve


Performance

Performance

Cost

Performance

X

$

Parallel Processing$$

Sequential Processing$$$$$$

Parallel Processing is really an Evolution in

Micro- and Macro-Architecture Hardware

That provides a Solution to:

• The Heat and Power Wall

• The Limitations of ILP

• Cost-Effective Higher Performance

Parallel Processing is also a Software Challenge

Key Points

Hardware Solution

cs 159 two lecture introduction parallel processing: a ......two lecture introduction parallel...

Documents