![Page 1: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/1.jpg)
CS 159Two Lecture Introduction
Parallel Processing:
A Hardware Solution&
A Software Challenge
![Page 2: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/2.jpg)
We’re on the Road to Parallel Processing
![Page 3: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/3.jpg)
Outline
Hardware Solution (Day 1)
Software Challenge (Day 2)
Opportunities
![Page 4: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/4.jpg)
Outline
Hardware Solution
Software Challenge
Opportunities
Technical
Technical
Technical
![Page 5: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/5.jpg)
Outline
Hardware Solution
Software Challenge
Opportunities
Business
Business
Business
Technical
Technical
Technical
![Page 6: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/6.jpg)
Outline
Hardware Solution
Software Challenge
Opportunities
Business Technical
![Page 7: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/7.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Computer HardwareEvolution Highway
Micro Macro
Hardware&
Software
Car&
Driver
![Page 8: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/8.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
![Page 9: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/9.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
29,000 Transistors 291,000,000
5 MHz Clock Frequency 2.9 GHz
1978 Year 2006
Intel 8086 Intel Core 2 Duo
![Page 10: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/10.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
In 28 Years from 1978 to 2006:
• Number of Transistors Increased 10,034X
• Clock Frequency Increased 586X
Primary Driver / Facilitator was (and is) Moore’s Law:
• Number of Transistors Doubles every 18-24 Months
• Stated by Gordon Moore, Intel Co-Founder in 1965
• Prediction has been proven valid over a long term
• “Prediction” has been the “Law” for over 40 years
![Page 11: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/11.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
![Page 12: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/12.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
![Page 13: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/13.jpg)
Historically, Huge Performance Gains came from
Huge Clock Frequency Increases
Unfortunately ……….
![Page 14: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/14.jpg)
Source: Patterson, Computer Organization and Design
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
Clock Rate Limits Have Been Reached
![Page 15: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/15.jpg)
Blocked by Heat and Power “Wall”
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
![Page 16: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/16.jpg)
• Power (and Heat) Grows as Frequency3
Power Voltage2 x Frequency
Voltage Frequency
Power Frequency3
Single Core Multi-Core
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
• How can HW Performance Continue to Increase?
![Page 17: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/17.jpg)
Single Core vs. Dual Core
Single Core clocked at 2f Dual Core clocked at f
2f Throughput f + f
8f 3 Heat f 3 + f 3
Sequential Processing Parallel Processing
![Page 18: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/18.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View
Instruction-Level Parallelism (ILP) was also Heavily Used
Implemented On-Chip via Hardware
Transparent to Software (No impact on Programmers)
We will Study Two Types:
Pipelining (Intra-Instruction Parallelism)
Multi-Function Units (Inter-Instruction Parallelism)
ILP has provided reasonable speedups in the past,
Unfortunately…….
![Page 19: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/19.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View Instruction-Level Parallelism Limits have been Reached too
100
10
1
Single-IssuePipelined
Superscalar(LimitedLook-Ahead)
Superscalar(Aggressive Look-Ahead)
AggressivenessOf ILP
Diminishing returns w.r.t.larger instruction window,higher issue-width
Power growsexponentially
Watts,IPC
![Page 20: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/20.jpg)
Effort
Performance
ScalarIn-Order
ModeratePipelining,Look-Ahead
Very Deep Pipe,AggressiveLook-Ahead
Made sense to goSuperscalar andMulti-Function:Good ROI
Very little gain forsubstantial effort
Hardware Solution - Technical
Evolution of Computer Architectures Gain - to - Effort Ratio of ILP beyond “Knee” of Curve Diminishing Returns due to
Increased Cost and Complexity of Extracting ILP
![Page 21: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/21.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View Summary
Clock Frequency Scaling Limits have been Reached
Era of Single Core Performance Increases has Ended
All Future Micro-Processor Designs will be Multi-Core
Evident in Chip Manufacturer’s RoadMaps
No More “Free Lunch” for Software Programmers
Multiple Cores Will Directly Expose Parallelism to SW
Instruction Level Parallelism Limits have been Reached
![Page 22: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/22.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View Summary
![Page 23: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/23.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Micro-Scopic View Summary
1
Moore’s Law Continues to 2x Transistors / 24 mos, but
It will be used to Increase Number of Cores Instead
2 4 ……
![Page 24: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/24.jpg)
Single Core Multi-Core
Sequential Processing Parallel Processing
![Page 25: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/25.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Computer HardwareEvolution Highway
Micro Macro
![Page 26: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/26.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
Personal Computer
Nodes: 1
Location: Desktop
![Page 27: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/27.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
Cluster Computer
Nodes: 10’s – 100’s
Location: Local
![Page 28: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/28.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
ExampleCluster
Computer
![Page 29: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/29.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
Grid Computer
Nodes: 1,000’s
Location: Distributed
![Page 30: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/30.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
ExampleGrid
Computer
![Page 31: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/31.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
Cloud Computer
Nodes: 10,000’s
Location: Highly
Distributed
![Page 32: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/32.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View
ExampleCloud
ComputerEC2
AzureApp Engine
![Page 33: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/33.jpg)
Hardware Solution - Technical
Evolution of Computer Architectures
Macro-Scopic View Summary
Single Node Many Nodes
Sequential Processing Parallel Processing
![Page 34: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/34.jpg)
Hardware Solution - Business
Evolution of Computer Architectures
Sequential Processing Parallel Processing
Micro
Macro
![Page 35: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/35.jpg)
Hardware Solution - Business
Computer Processing Power:
• Has a Highly Elastic Supply and Demand Curve
• Increased Supply Generates Increased Demand
Software is Like a Gas
• It Expands to Fill any size Hardware Container
Computer Power
New Applications
Sequential Processing Parallel Processing
Hardware
Software
![Page 36: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/36.jpg)
Hardware Solution - Business
Goal of Computer Hardware and Software Designers
• Continually Increase Performance and Lower Cost
• Operate at Optimum Point on Technology Curve
Sequential Processing Parallel Processing
Even Visionaries Sometime Forget
No Matter How Much Computer Power People Have,
They Always Want More
“640K should be enough for anybody”
“There is a world market for maybe five computers”
Bill Gates, 1981
Thomas Watson, 1943
![Page 37: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/37.jpg)
Hardware Solution - Business
Technology Curve
Cost / Performancevs.
Performance
Performance
Cost
Performance
Under-Utilization
Optimum UtilizationOver-
Utilization
![Page 38: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/38.jpg)
Hardware Solution - Business
Technology Curve
Cost / Performancevs.
Performance
Performance
Cost
Performance
Under-Utilization
Optimum UtilizationOver-
Utilization
![Page 39: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/39.jpg)
Hardware Solution - Business
Technology Curve
Sequential Processing
Cost / Performancevs.
Performance
Performance
Cost
Performance
2XX
$
$$$
$$$$$$
![Page 40: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/40.jpg)
Hardware Solution - Business
Technology Curve
Sequential Processing
Cost / Performancevs.
Performance
Performance
Cost
Performance
2XX
$
$$$
$$$$$$
![Page 41: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/41.jpg)
Hardware Solution - Business
Technology Curve
Cost / Performancevs.
Performance
Performance
Cost
Performance
X
$
Parallel Processing$$
Sequential Processing$$$$$$
![Page 42: CS 159 Two Lecture Introduction Parallel Processing: A ......Two Lecture Introduction Parallel Processing: A Hardware Solution & A Software Challenge We’re on the Road to Parallel](https://reader030.vdocuments.us/reader030/viewer/2022012003/60ac8f51f85d5e58a24dec74/html5/thumbnails/42.jpg)
Parallel Processing is really an Evolution in
Micro- and Macro-Architecture Hardware
That provides a Solution to:
• The Heat and Power Wall
• The Limitations of ILP
• Cost-Effective Higher Performance
Parallel Processing is also a Software Challenge
Key Points
Hardware Solution