csc1401 lecture05 - cache memory

Madam Raini Hassan

Office: C5 - 23, Level 5, KICT Building

Department: Computer Science

Emails: [email protected], [email protected] 1 Semester II 2014/2015

Du’a for Study

Semester II 2014/2015 2

LECTURE 05

Cache Memory (Chapter 4)

Outline

• Computer Memory System

Overview

Characteristics of Memory

Systems

The Memory Hierarchy

• Cache Memory Principles

• Elements of Cache Design

Cache Addresses

Cache Size

Mapping Function

Replacement Algorithms

Write Policy

Line Size

Number of Caches

• Pentium 4 Cache

Organization


Introduction

• The complex subject of computer memory is made

more manageable if we classify memory systems

according to their characteristics.


Key Characteristics of Computer

Memory Systems

Table 4.1 Key Characteristics of Computer Memory Systems Semester II 2014/2015 6

7

1. Location

• Refers to whether memory is internal and external to the computer

• Internal memory is often equated with main memory

• Processor requires its own local memory, in the form of registers

• Cache is another form of internal memory

• External memory consists of peripheral storage devices that are accessible to the processor via I/O controllers

Semester II 2014/2015

8

2. Capacity

• Memory is typically expressed in terms of bytes.

• In other words, the capacity of a memory is the total number of bits that can be stored.

• Word

– A group of bits in memory that represents instructions or data.

– The natural unit of organization (1 byte = 8 bits)

– Common word lengths – 8, 16, 32 bits

• Byte

– The smallest unit of binary data is the bit and a collection of 8 bit is known as byte (1 byte = 8 bits)


9

3. Unit of Transfer

• For internal memory (main memory), this is the

number of bits read out of or written into memory at a

time.

• The unit of transfer need not equal a word on an

addressable unit.

• For external memory, data are often transferred in units

larger than words; referred to as block.


Sequential access

Memory is organized into units of data called records

Access must be made in a specific linear sequence

Access time is variable

Direct access

Involves a shared read-write mechanism

Individual blocks or records have a unique address based on physical

location

Access time is variable

4. Access Methods (1)


Random access

Each addressable location in memory has a unique, physically wired-in

addressing mechanism

The time to access a given location is independent of the sequence of prior

accesses and is constant

Any location can be selected at random and directly addressed and

accessed

Main memory and some cache systems are random access

Associative

A word is retrieved based on a portion of its contents rather than its address

Each location has its own addressing mechanism and retrieval time is

constant independent of location or prior access patterns

Cache memories may employ associative access

4. Access Methods (2)


The two most important characteristics of memory

Three performance parameters are used:

Access time (latency)

• For random-access memory it is the time it takes to perform a read or write operation

• For non-random-access memory it is the time it takes to position the read-write mechanism at the desired location

Memory cycle time

• Access time plus any additional time required before second access can commence.

• Additional time may be required for transients to die out on signal lines or to regenerate data if they are read destructively.

• Concerned with the system bus, not the processor.

Transfer rate

• The rate at which data can be transferred into or out of a memory unit

• For random-access memory it is equal to 1/(cycle time)

5. Capacity and Performance


13

6. Physical Types

• Semiconductor

– RAM

• Magnetic surface memory

– Disk & Tape

• Optical

– CD & DVD

• Magneto-optical

– Note book computer


14

7. Physical Characteristics

• Decay • Volatility

– Information decays naturally or is lost when electrical power is switched off

– Example; semiconductor memory

• Non-volatile – Once recorded, information remains without deterioration until

deliberately changed – No electrical power is needed to retain information – Example: Magnetic-surface memories and semiconductor memory

• Erasable • Non-erasable

– Cannot be altered, except by destroying the storage unit – Semiconductor memory of this type is known as read-only memory

(ROM)

• Power consumption


15

8. Organization

• Physical arrangement of bits into words.


Principle of Locality

• Principle of locality (or locality of reference):

• Program accesses a relatively small portion of the address space at any instant of time.

• Temporal locality and spatial locality.

Address Space 0 2

n - 1

Probability of reference


Principle of Locality (2)

Temporal locality (locality in time):

Keep most recently accessed data items closer to the processor.

Spatial locality (locality in space):

Move blocks consisting of contiguous words to ‘upper’ levels.

Block is a unit of transfer.


Principle of Locality (3)

What programming constructs lead to principle of

locality?

Spatial Temporal

Data arrays loop counters

Code no branch/jump loop


The Memory Hierarchy

• Design constraints on a computer’s memory can be summed up by three questions:

– How much, how fast, how expensive

• There is a trade-off among capacity, access time, and cost

– Faster access time, greater cost per bit

– Greater capacity, smaller cost per bit

– Greater capacity, slower access time

• The way out of the memory dilemma is not to rely on a single memory component or technology, but to employ a memory hierarchy


Memory Hierarchy – Basic

Theoretical Diagram

• Registers

• L1 Cache

• L2 Cache

• Main memory

• Disk cache

• Disk

• Optical

• Tape


Memory Hierarchy – Simplified “Real

Life” Diagram


Memory Hierarchy – Practical Diagram


Memory Hierarchy – Current

Control

Data

path

Processor

Reg

s

Second-

ary

Memory L2

Cache

L1 $

Main

Memory

Speed(ns): 0.5ns 2ns 6ns 100ns 10,000,000ns

Size (MB): 0.0005 0.05 1-4 100-1000 100,000

Cost ($/MB): -- $100 $30 $1 $0.05

Technology: Regs SRAM SRAM DRAM Disk


24

The 3 Parameters

• Capacity, speed and cost increase moving from top to down.

• First, access time get bigger; e.g.: CPU can be accessed in a few nanoseconds.

• Second, the storage capacity increases as we go downwards.

• CPU registers – around 128 bytes, caches – a few megabytes, main memory – thousands of megabytes, and magnetic disks – a few gigabytes.

• Third, the number of bits per RM spent increases down the hierarchy.


The Bottom Line

• How much?

– Capacity

• How fast?

– Time is money

• How expensive?


So you want fast?

• It is possible to build a computer which uses only static

RAM (see later)

• This would be very fast

• This would need no cache

– How can you cache?

• This would cost a very large amount


27

Cache Memory Principles

• Small amount of fast memory

• Sits between normal main memory and CPU

• May be located on CPU chip or module


Cache and Main Memory


29

Other definitions of Cache

1. Cache memory is random access memory (RAM) that a computer microprocessor can access more quickly than it can access regular RAM. As the microprocessor processes data, it looks first in the cache memory and if it finds the data there (from a previous reading of data), it does not have to do the more time-consuming reading of data from larger memory.

2. A cache is a small capacity fast memory, too expensive to be used for the whole of RAM. It acts as an intermediate store between the CPU and main memory and is used to improve the overall speed of the computer since it is used primarily to keep the most active portions of the program being used.

3. A special, high-speed memory unit.

4. It is a temporary, very fast, digital storage for a central processor.


30

Cache operation - overview

• CPU requests contents of memory location

• Check cache for this data

• If present, get from cache (fast)

• If not present, read required block from main memory to cache

• Then deliver from cache to CPU

• Cache includes tags to identify which block of main memory is in each cache slot


Cache/Main Memory Structure


Cache Read

Operation


Typical Cache Organization


Elements of Cache Design

Table 4.2 Elements of Cache Design Semester II 2014/2015 34

+ 1. Cache Addresses

• Virtual memory

– Facility that allows programs to address memory from a logical point of view, without regard to the amount of main memory physically available

– When used, the address fields of machine instructions contain virtual addresses

– For reads to and writes from main memory, a hardware memory management unit (MMU) translates each virtual address into a physical address in main memory


Logical

and

Physical

Caches


37

2. Cache Size

• Cost

– More cache is expensive

• Speed

– More cache is faster

– Checking cache for data takes time


Table 4.3

Cache Sizes

of Some

Processors

a Two values

separated by a

slash refer to

instruction and

data caches.

b Both caches

are instruction

only; no data

caches. Semester II 2014/2015 38

3. Mapping Function

• Because there are fewer cache lines than main memory

blocks, an algorithm is needed for mapping main memory

blocks into cache lines

• Three techniques can be used:

1. Direct

2. Associative

3. Set Associative


40

3. Mapping Function (Example)

• Cache of 64kByte

• Cache block of 4 bytes

– i.e. cache is 16k (214) lines of 4 bytes

• 16MBytes main memory

• 24 bit address

– (224=16M)


41

3. Mapping Function: Direct

Mapping

Direct Mapping

• The simplest technique.

• Each block of main memory maps to only one possible

cache line

– i.e. if a block is in cache, it must be in one specific place

• Address is in two parts


42


Mapping

Direct Mapping

• Least Significant w bits identify unique word

• Most Significant s bits specify one memory block

• The MSBs (Most Significant Bits) are split into a cache

line field r and a tag of s-r (most significant)


3. Mapping

Function:

Direct

Mapping


3. Mapping Function: Direct Mapping

Cache Organization


3. Mapping

Function:

Direct

Mapping

Example



Mapping Summary

• Address length = (s + w) bits

• Number of addressable units = 2s+w words or bytes

• Block size = line size = 2w words or bytes

• Number of blocks in main memory = 2s+ w/2w = 2s

• Number of lines in cache = m = 2r

• Size of tag = (s – r) bits


47


Mapping Pros & Cons

• Simple

• Inexpensive

• Cons: Fixed location for given block

– If a program accesses 2 blocks that map to the same line

repeatedly, then the blocks will be continually swapped in the

cache, caused the cache misses to be very high


48

3. Mapping Function: Fully

Associative Cache

• To reduce the conflict misses of direct mapped caches without affecting its fast access time is to remember what was discarded in case it is needed again.

• Since the discarded data has already been fetched, it can be used again at a small cost.

• Such recycling is possible by using a victim cache, also known as fully associative cache.

• Typical size is 4 to 16 cache lines

• Residing between direct mapped L1 cache and the next level of memory


3. Mapping Function: Fully Associative

Cache Organization


50

3. Mapping Function: Associative

Associative Mapping

• Permits each main memory block to be loaded into any line of the cache.

• The cache control logic interprets a memory address simply as a Tag and a Word field.

• Tag uniquely identifies block of memory.

• To determine whether a block is in the cache, the cache control logic must simultaneously examine every line’s Tag for a match .

• Cache searching gets expensive.


3. Mapping

Function:

Associative

Mapping


3. Mapping

Function:

Associative

Mapping

Example


3. Mapping Function: Associative

Mapping Summary

• Address length = (s + w) bits



• Number of blocks in main memory = 2s+ w/2w = 2s

• Number of lines in cache = undetermined

• Size of tag = s bits


54

3. Mapping Function: Set-Associative

Set-Associative Mapping

• Compromise that exhibits the strengths of both the direct and associative approaches while reducing their disadvantages

• Cache consists of a number of sets

• Each set contains a number of lines

• A given block maps to any line in a given set – Block B can be in any line of set i

• e.g. 2 lines per set – 2 way associative mapping

– A given block can be in one of 2 lines in only one set


Mapping From Main

Memory

to Cache:

k-Way

Set Associative


k-Way

Set Associative

Cache

Organization


3. Mapping Function: Set

Associative Mapping Summary • Address length = (s + w) bits



• Number of blocks in main memory = 2s+w/2w=2s

• Number of lines in set = k

• Number of sets = v = 2d

• Number of lines in cache = m = kv = k * 2d

• Size of cache = k * 2d+w words or bytes

• Size of tag = (s – d) bits


3. Mapping Function: Varying

Associativity Over Cache Size


4. Replacement Algorithms

• Once the cache has been filled, when a new block

is brought into the cache, one of the existing

blocks must be replaced.

• For direct mapping there is only one possible line

for any particular block and no choice is possible.

• For the associative and set-associative techniques a

replacement algorithm is needed.

• To achieve high speed, an algorithm must be

implemented in hardware.


61

4. Replacement Algorithms:

Direct mapping • No choice

• Each block only maps to one line

• Replace that line


62


Associative & Set Associative

• Hardware implemented algorithm (speed)

1. Least Recently used (LRU)

2. First in first out (FIFO)

3. Least frequently used

4. Random



LRU

Least recently used (LRU)

– Most effective

– Replace that block in the set that has been in the cache

longest with no reference to it

– Because of its simplicity of implementation, LRU is the

most popular replacement algorithm



FIFO

First-in-first-out (FIFO)

– Replace that block in the set that has been in the cache

longest.

– Easily implemented as a round-robin or circular buffer

technique.



LFU

Least Frequently Used (LFU)

– Replace that block in the set that has experienced the

fewest references

– Could be implemented by associating a counter with each

line


When a block that is resident in the cache is to be replaced there

are two cases to consider:

If the old block in the cache has not been altered then it may be overwritten with a new block without first writing

out the old block

If at least one write operation has been performed on a word in that line of the

cache then main memory must be updated by writing the line of cache out to the block of memory before bringing

in the new block

There are two problems to contend with:

More than one device may have access to main memory

A more complex problem occurs when multiple processors are attached to the

same bus and each processor has its own local cache - if a word is altered in

one cache it could conceivably invalidate a word in other caches

5. Write Policy


5. Write Policy

In other words…

• Must not overwrite a cache block unless main

memory is up to date

• Multiple CPUs may have individual caches

• I/O may address main memory directly

2 methods of write policy are…


5. Write Policy: Write Through

Write Through

– Simplest technique

– All write operations are made to main memory as

well as to the cache

– The main disadvantage of this technique is that it

generates substantial memory traffic and may

create a bottleneck


5. Write Policy: Write Back

Write Back

– Minimizes memory writes

– Updates initially are made only in the cache

– Update bit for cache slot is set when update occurs

– If block is to be replaced, write to main memory only if update bit is set

– Other caches get out of sync

– Resulting portions of main memory are invalid and hence accesses by I/O modules can be allowed only through the cache

– This makes for complex circuitry and a potential bottleneck


6. Line Size

• When a block of data is retrieved and placed in the

cache not only the desired word but also some number

of adjacent words are retrieved.

• As the block size increases the hit ratio will at first

increase because of the principle of locality.

• As the block size increases more useful data are brought

into the cache.

• The hit ratio will begin to decrease as the block becomes

bigger and the probability of using the newly fetched

information becomes less than the probability of

reusing the information that has to be replaced.


6. Line Size

• Two specific effects come into play:

– Larger blocks reduce the number of blocks that fit into a

cache

– As a block becomes larger each additional word is farther

from the requested word


7. Number of Caches

• When caches were originally introduced, the typical

system had a single cache.

• More recently, the use of multiple caches has become

the norm.


7. Number of Caches: Multilevel Caches

• As logic density has increased it has become possible to

have a cache on the same chip as the processor

• The on-chip cache reduces the processor’s external bus

activity and speeds up execution time and increases overall

system performance

– When the requested instruction or data is found in the on-chip

cache, the bus access is eliminated

– On-chip cache accesses will complete appreciably faster than

would even zero-wait state bus cycles

– During this period the bus is free to support other transfers


7. Number of Caches: Multilevel Caches

• Two-level cache:

– Internal cache designated as level 1 (L1)

– External cache designated as level 2 (L2)

• Potential savings due to the use of an L2 cache depends

on the hit rates in both the L1 and L2 caches

• The use of multilevel caches complicates all of the design

issues related to caches, including size, replacement

algorithm, and write policy


7. Number of Caches: Unified Versus

Split Caches

• Has become common to split cache:

– One dedicated to instructions

– One dedicated to data

– Both exist at the same level, typically as two L1 caches

• Advantages of unified cache:

– Higher hit rate

• Balances load of instruction and data fetches automatically

• Only one cache needs to be designed and implemented


7. Number of Caches: Unified Versus

Split Caches

• Trend is toward split caches at the L1 and unified caches

for higher levels

• Advantages of split cache:

– Eliminates cache contention between instruction fetch/decode

unit and execution unit

• Important in pipelining


Pentium 4 Cache Organization

• The evolution of cache organization is seen clearly in

the evolution of Intel microprocessors (Table 4.4)


Table 4.4

Intel

Cache

Evolution

78

csc1401 lecture05 - cache memory

Education

internal memory main

capacity memory

local memory

sequential access memory

cache memory chapter

computer internal memory

nonrandomaccess memory

main memory processor