cache memory by ed martinez. the fastest and most expensive memory on a computer system that is...

Cache MemoryCache Memory

By Ed MartinezBy Ed Martinez

The fastest and most expensive memory on a computer The fastest and most expensive memory on a computer system that is used to store collections of data.system that is used to store collections of data.

Uses very short time to access recently and frequently Uses very short time to access recently and frequently used data. This helps improve computing performance used data. This helps improve computing performance by decreasing data access time.by decreasing data access time.

With the speeds of modern CPUs ever increasing, With the speeds of modern CPUs ever increasing, accessing main memory has become a major accessing main memory has become a major performance bottleneck. Effective use of cache can help performance bottleneck. Effective use of cache can help to minimize this problem. to minimize this problem.

What is Cache Memory?What is Cache Memory?

Cache LevelsCache Levels Cache are referred to in levels, such as L1 or L2.Cache are referred to in levels, such as L1 or L2.

The level describes the connection or physical proximity of the The level describes the connection or physical proximity of the cache to the CPU.cache to the CPU.

Traditionally, L1 cache, usually the smaller of the two, is located on Traditionally, L1 cache, usually the smaller of the two, is located on the processor and runs at the same speed as the processor. L2 the processor and runs at the same speed as the processor. L2 cache is outside, but near the processor and runs at the speed of cache is outside, but near the processor and runs at the speed of the motherboard or FSB.the motherboard or FSB.

Recent designs have begun to integrate L2 cache onto the Recent designs have begun to integrate L2 cache onto the processor card or into the CPU chip itself, like L1 cache. This processor card or into the CPU chip itself, like L1 cache. This speeds access to the larger L2 cache, which improves the speeds access to the larger L2 cache, which improves the computer’s performance.computer’s performance.

CPU RoadmapCPU Roadmap

This photo shows a road map of the inside of the CPU, notice the This photo shows a road map of the inside of the CPU, notice the

different areas, and their functions. Can you find the L1 cache?different areas, and their functions. Can you find the L1 cache?

Cache MemoryCache Memory

This photo shows level 2 cache memory on the Processor board, This photo shows level 2 cache memory on the Processor board, beside the CPU beside the CPU

Cache MappingCache Mapping

AssociativeAssociative

DirectDirect

Set AssociativeSet Associative

Associative MappingAssociative Mapping Sometimes known as fully associative mapping, any block of Sometimes known as fully associative mapping, any block of

memory can be mapped to any line in the cache.memory can be mapped to any line in the cache.

The block number is appended to the line as a tag.The block number is appended to the line as a tag.

There is a bit to indicate whether the line is valid or not.There is a bit to indicate whether the line is valid or not.

Difficult and expensive to implement.Difficult and expensive to implement.

Direct MappingDirect Mapping Each block in main memory can be loaded to only one line in cache.Each block in main memory can be loaded to only one line in cache.

Each cache line contains a tag to identify the block of main memory Each cache line contains a tag to identify the block of main memory in the cache.in the cache.

This is easy to implement, but inflexible.This is easy to implement, but inflexible.

Set-Associative MappingSet-Associative Mapping Blocks can be mapped to a subset of the lines in cache.Blocks can be mapped to a subset of the lines in cache.

A block can be mapped to either 2 or 4 lines in cache, usually A block can be mapped to either 2 or 4 lines in cache, usually referred to as “Two-way” or “Four-way”. referred to as “Two-way” or “Four-way”.

LRU is the most popular replacement policy used with this mapping LRU is the most popular replacement policy used with this mapping process. process.

More flexible than direct mapping, but easier to implement than More flexible than direct mapping, but easier to implement than associative mapping.associative mapping.

Why use a cache replacement Why use a cache replacement policy?policy?

Cache is small compared to available system memoryCache is small compared to available system memory

Cache maintains copies of recently referenced data items to speed Cache maintains copies of recently referenced data items to speed access timeaccess time

Once full it must use a replacement policy to decide how to handle Once full it must use a replacement policy to decide how to handle additional data itemsadditional data items

The replacement policy can either ignore the new data or decide The replacement policy can either ignore the new data or decide which of the old data items to remove to make roomwhich of the old data items to remove to make room

Cache Replacement Policy (cont)Cache Replacement Policy (cont)

Two main issues in determining which policy to use:Two main issues in determining which policy to use:

Increase the hit ratio by keeping the items that are referenced Increase the hit ratio by keeping the items that are referenced most frequentlymost frequently

Policy should be inexpensive to implementPolicy should be inexpensive to implement

Cache Replacement PoliciesCache Replacement Policies

FIFOFIFO

LRULRU

RandomRandom

FIFO ReplacementFIFO Replacement FIFO (First in, first out) – the oldest data item in the cache frame is FIFO (First in, first out) – the oldest data item in the cache frame is

replaced.replaced.

Easy to implement and has fast operation.Easy to implement and has fast operation.

Since it maintains no reference statistics, depending on the Since it maintains no reference statistics, depending on the sequence of the data you may be constantly removing and replacing sequence of the data you may be constantly removing and replacing the same block of memory.the same block of memory.

LRU ReplacementLRU Replacement Least Recently Used – The item that was accessed least recently is Least Recently Used – The item that was accessed least recently is

replaced with the new data item. replaced with the new data item.

Easy to implement, follows the idea that items which have been Easy to implement, follows the idea that items which have been recently used are likely to be used again.recently used are likely to be used again.

Keeps a list of data items that are currently in the cache. Keeps a list of data items that are currently in the cache. Referenced data items are moved to the front of the listReferenced data items are moved to the front of the list

Data items at the back of the list are removed as space is neededData items at the back of the list are removed as space is needed

Primary drawback is that it requires time to analyze the reference Primary drawback is that it requires time to analyze the reference statistics to determine which item to remove. Also requires statistics to determine which item to remove. Also requires additional cache space to maintain the statisticsadditional cache space to maintain the statistics

Random ReplacementRandom Replacement Random – a randomly generated address determines which data Random – a randomly generated address determines which data

item in the cache frame to replace.item in the cache frame to replace.

Has no need to store statistics, so it doesn’t waste time analyzing Has no need to store statistics, so it doesn’t waste time analyzing anything.anything.

Performance is close to that of the other policies, and the Performance is close to that of the other policies, and the implementation is fairly simple.implementation is fairly simple.

Cache Replacement PoliciesCache Replacement Policies Lets assume a computer system has an associative, a direct-Lets assume a computer system has an associative, a direct-

mapped, or a two-way set-associative cache of 8 bytes.mapped, or a two-way set-associative cache of 8 bytes.

The CPU accesses the following locations in order (tThe CPU accesses the following locations in order (the subscript is the low-order 3 bits of its address in physical memory ): ):

A0 B0 C2 A0 D1 B0 E4 F5 A0 C2 D1 B0 G3 C2 H7 I6 A0 B0

Associative CacheAssociative Cache(FIFO Replacement Policy)(FIFO Replacement Policy)

Data A B C A D B E F A C D B G C H I A B

CACHE

A A A A A A A A A A A A A A A I I I

B B B B B B B B B B B B B B B A A

C C C C C C C C C C C C C C C B

D D D D D D D D D D D D D D

E E E E E E E E E E E E

F F F F F F F F F F F

G G G G G G

H H H H

Hit? * * * * * * *

Hit ratio = 7/18Hit ratio = 7/18


Direct mapped CacheDirect mapped Cache



CACHE

0 A B B A A B B B A A A B B B B B A B

1 D D D D D D D D D D D D D D

2 C C C C C C C C C C C C C C C C

3 G G G G G G

4 E E E E E E E E E E E E

5 F F F F F F F F F F F

6 I I I

7 H H H H

Hit? * * *


Two-way set associative cacheTwo-way set associative cache(LRU Replacement Policy)(LRU Replacement Policy)




C

A

C

H

E

0 A-0 A-1 A-1 A-0 A-0 A-1 E-0 E-0 E-1 E-1 E-1 B-0 B-0 B-0 B-0 B-0 B-1 B-0

0 B-0 B-0 B-1 B-1 B-0 B-1 B-1 A-0 A-0 A-0 A-1 A-1 A-1 A-1 A-1 A-0 A-1

1 D-0 D-0 D-0 D-1 D-1 D-1 D-0 D-0 D-0 D-0 D-0 D-0 D-0 D-0

1 F-0 F-0 F-0 F-1 F-1 F-1 F-1 F-1 F-1 F-1 F-1

2 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-1 C-1 C-1

2 I-0 I-0 I-0

3 G-0 G-0 G-1 G-1 G-1 G-1

3 H-0 H-0 H-0 H-0

Hit? * * * * * * *

Replacement with 2 byte line sizeReplacement with 2 byte line size

Now lets consider data lines of two bytes. The data pairs that make up Now lets consider data lines of two bytes. The data pairs that make up the lines are:the lines are:

A and J; B and D; C and G; E and F; and I and H.A and J; B and D; C and G; E and F; and I and H.

Associative Cache with 2 byte line Associative Cache with 2 byte line size (FIFO Replacement Policy)size (FIFO Replacement Policy)



A and J; B and D; C and G; E and F; and I and HA and J; B and D; C and G; E and F; and I and H


C

A

C

H

E

A A A A A A A A A A A A A A I I I I

J J J J J J J J J J J J J J H H H H

B B B B B B B B B B B B B B B A A

D D D D D D D D D D D D D D D J J

C C C C C C C C C C C C C C C B

G G G G G G G G G G G G G G G D

E E E E E E E E E E E E

F F F F F F F F F F F F

Hit? * * * * * * * * * * *

Direct-mapped CacheDirect-mapped Cachewith line size of 2 byteswith line size of 2 bytes

Hit ratio 7/18Hit ratio 7/18


CACHE

0 A B B A B B B B A A B B B B B B A B

1 J D D J D D D D J J D D D D D D J D

2 C C C C C C C C C C C C C C C C

3 G G G G G G G G G G G G G G G G

4 E E E E E E E E E E E E

5 F F F F F F F F F F F F

6 I I I I

7 H H H H

Hit? * * * * * * *



Two-way set Associative CacheTwo-way set Associative Cachewith line size of 2 byteswith line size of 2 bytes



C

A

C

H

E

0 A-0 A-1 A-1 A-0 A-1 A-1 E-0 E-0 E-1 B-0 B-0 B-0 B-0 B-0 B-0 B-0 B-1 B-0

1 J-0 J-1 J-1 J-0 J-1 J-1 F-0 F-0 F-1 D-0 D-0 D-0 D-0 D-0 D-0 D-0 D-1 D-0

0 B-0 B-0 B-1 B-0 B-0 B-1 B-1 A-0 A-0 A-1 A-1 A-1 A-1 A-1 A-1 A-0 A-1

1 D-0 D-0 D-1 D-0 D-0 D-1 D-1 J-0 J-0 J-1 J-1 J-1 J-1 J-1 J-1 J-0 J-1

2 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-0 C-1 C-1 C-1 C-1

3 G-0 G-0 G-0 G-0 G-0 G-0 G-0 G-0 G-0 G-0 G-0 G-0 G-1 G-1 G-1 G-1

2 I-0 I-0 I-0 I-0

3 H-0 H-0 H-0 H-0

Hit? * * * * * * * * * * * *



Cache PerformanceCache Performance Primary reason for including cache memory in a computer is to Primary reason for including cache memory in a computer is to

improve system performance by reducing the time needed to improve system performance by reducing the time needed to access data in memory.access data in memory.

The cache hit ratio – hits/(hits + misses).The cache hit ratio – hits/(hits + misses).

A well-designed cache will have a hit ratio close to 1. A well-designed cache will have a hit ratio close to 1. Cache hits outnumber the misses by far. When more data is Cache hits outnumber the misses by far. When more data is

retrieved from the faster cache memory rather than from main retrieved from the faster cache memory rather than from main memory system performance is improved. memory system performance is improved.

Cache Performance (cont)Cache Performance (cont)

h TM

0.0 60 ns

0.1 55 ns

0.2 50 ns

0.3 45 ns

0.4 40 ns

0.5 35 ns

0.6 30 ns

0.7 25 ns

0.8 20 ns

0.9 15 ns

1.0 10 ns

Hit ratios and average memory access times

As the number of hits increase the average memory access time decreases

h = hit ratio TM = weighted average cache access

time.

ReferencesReferences Carpielli, John D, Computer Systems Organization & ArchitectureCarpielli, John D, Computer Systems Organization & Architecture

Comer, Douglas E, Essentials of Computer ArchitectureComer, Douglas E, Essentials of Computer Architecture

http://www.kids-online.net/learn/click/details/http://www.kids-online.net/learn/click/details/micropromicropro.html.html

http://www.oit.edu/faculty/conek_images/documents/Ch6_ECOA.ppt http://www.oit.edu/faculty/conek_images/documents/Ch6_ECOA.ppt

http://www.kids-online.net/learn/click/details/micropro.html



cache memory by ed martinez. the fastest and most expensive memory on a computer system that is...

Documents