Ch7 MEMORY HIERARCHY

OVERVIEW Accessing memory has always been one of the primary bottlenecks in computing. Fast memory is expensive, and only very high-performance, expensive computers are built with very large, fast memories. The rest of us live with slow (50 ns access, these days) DRAM memory. Consider: Todays machines may have a clock cycle of 1 ns, but memory access requires 50 nsec! Obviously, a disconnect in performance. If we had to live with a simple memory of this type, wed have a performance differential (assuming that of the instructions are loads and stores and all instructions have to do a fetch): Perf: 50ns memory/1.0 nsec memory=0.75*50+0.25*140/0.75*1.0+0.25*2.0 = 50 Obviously, this is a problem. Thus, the necessity of better memory performance. A number of techniques are used to greatly improve memory performance. Most important is, of course, cache memory. What is cache memory? A special fast memory that stores RECENTLY ACCESSED locations, and locations adjacent to those. 1. Level 1 cache memory is integrated with the CPU, runs full speed 2. NOT random access, instead, its structured to store recent program and data locations. 3. As long as required instructions and data are in cache, a hit occurs, and memory access does not slow down CPU (possibly 2-cycle Load/Store) 4. If info is NOT in cache, a miss occurs, and the CPU has to stop to await the slow process of memory being uploaded. 5. Because the amount of space available on the CPU die for cache memory is small (32 Kbytes-128 kbytes), often a LEVEL 2 cache is integrated into the computers motherboard. This is a cache memory that is larger than level 1 cache (256 kbyte-4 Mbyte), but necessarily slower because it is off-chip. Usually 10-25 nsec access. Why does cache memory work? If locations were accessed totally randomly, one would not expect many hits. However, the execution of a program is anything but random:
1

Principle of locality: Temporal locality: if an address is referenced, it is likely to be accessed again soon. Programs tend to consist of loops, so the first time through a loop may be slow because all the instructions need to be uploaded from memory. Data is often accessed within the loop too, so L/S addresses tend to be the same time after time. Spatial locality: If an address is referenced, it is likely that close-by locations will also be referenced. Loops consist of adjacent instructions. Data: related variables tend to be spaced closely together (arrays, structs, variable definitions). These principles suggest that it is indeed worthwhile to use a cache, and that rather than uploading one instruction or data location at a time, it may be useful to upload several. WHY? Memory (DRAM) requires substantial overhead to get the first location out. Subsequent locations can be accessed much faster. ADDR RAS CAS DATA IN This is true because two cycles are needed to latch the address (1/2 the address at a time), but todays DRAMS are pipelined so that only the first address needs to be given; all subsequent locations just require a little control handshaking. Consequently: cache memory operates on BLOCKS of adjacent bytes. A whole block is loaded from DRAM (or Level 2 cache) at a time. Block sizes tend to be 16-64 bytes (4-16 words). DRAM DATA OUT (1,2 or 4 bits)

Thus, the primary purpose of cache memory is to improve the average access time of instruction and data fetches. Today, faster memory is also available (SDRAM), at a premium price. Access times are down to 8 ns, allowing full-speed 133 MHz memory bus operation. Virtual memory and paging But, there are other aspects to the memory hierarchy system. Consider a user of, say, a UNIX system on which a number of other users are operating. That user wants to be certain that: No other user interferes with his/her program execution Multiple programs can run concurrently on the machine, even if they all start at Address 0 The machine will not run out of memory, even if a program requires 10 times as many locations as are available in DRAM! The memory hierarchy of todays computers ensures that these are all true. Memory protection hardware compares accessed locations to the range of addresses allowed to the currently-running (in CPU) program, and causes an exception if the access is not legal (memory access violation). Active programs (called processes) share CPU resources: periodically an external timer generates an interrupt, and the O.S. decides whether to give a different process CPU time. This happens about every 10 ms. Virtual memory allows the second need to be met. Addresses accessed by the CPU are translated so that the target address may be in one of many different locations in DRAM. This allows all programs to start at location 0, as long as the mapping is different for all the programs. Locations in memory are organized into blocks of adjacent locations called PAGES; each page can be arbitrarily mapped. Demand paging allows writing memory blocks that have not been recently used to the hard disk drive, temporarily caching them at a lower level! This frees up DRAM memory for other uses. When a program needs a location that has been paged out to disk, the hardware detects the problem and

causes an exception, and the operating system takes over and retrieves the page.

CACHE MEMORY (7.2) Consider a RAM memory read operation. The system responds to an applied address by outputting the bits stored at that location. A cache memory must work differently. It stores recently used locations somewhere in the cache, and must retrieve the information by address. A cache is organized into LINES. Each line is comprised of: 1) One block of data (or instructions) 2) Address information that allows determining whether the CPU address requested is located in the line 3) A few housekeeping control bits. Most systems today have a UNIFIED CACHE that contains both instructions and data, but a few have separate caches for data and instructions to allow simultaneous IF and Load/Store operations. Questions that must be addressed here in considering caches are: 1) Where can a block be placed in the cache? 2) How is a block found if it is in the cache? 3) Which block should be replaced on a cache miss? 4) What happens on a write? Well consider three different cache architectures. Lets assume that each line contains 32 bytes. ADDR: A31-A5=block number, A4-A2=word in block, A1-A0=byte # Architecture 1: FULLY-ASSOCIATIVE CACHE. This is the hardest system to implement, but the most general, and so it is pretty much a theoretical construct. A block can be placed ANYWHERE in the FA cache. This means that the full address of the block must be stored in the cache. Hardest is that a LRU (least-recently used) algorithm must be implemented to determine which block should be replaced if there is a miss. This takes a lot of slow circuitry! Block diagram:
5

Tag ADDR

Data DATA STATUS

=? Addr 5-31,

hit 2-4

MUX Data out

What happens: each line has an independent address comparator that tries to match the CPU address with the line address. If one matches, we have a hit UNLESS the line isnt in use, and we have a spurious match. One of the status bits must therefore be line valid. On a hit, the appropriate word in the line is passed via mux to the CPU (on a read). Well talk about writes later. On a miss (read), the LRU hardware decides which line to replace, and the CPU is halted until the requested block is uploaded from DRAM or L2 cache. The blocks address is copied along with the data, and the line valid flag is set. Architecture 2: DIRECT-MAPPED CACHE This is the opposite case from the above: each block can only go to one line in the cache, resulting in the possibility that lines can be overwritten frequently. But it is very simple structurally. To avoid the possibility that nearby data or instructions can overwrite each others line in the cache, the lower CPU address bits are used to index into the cache. The location of the data or instructions in cache is found by taking the modulo of the of the address in memory and the number of locations in cache. If a power of 2, you just take the low number bits for the address in cache and the tag is the upper number of bits. Note that in a unified cache, if data and instructions have the same bit patterns, they will interfere with each other. Likewise, a long loop that has twice as many instructions as cache capacity will also render the cache
6

useless because instructions will constantly overwrite each other in the cache. ADDR: (A16-A5) 000 tag 001 A31-A17 002 FFF Index =? CPU A31-A17 A4-A2 DATA OUT data (32 bytes) status

In this case, there is only ONE comparator, and no LRU hardware. So simple! Yet, the failings of this architecture have led to an intermediate approach. Architecture 3: SET-ASSOCIATIVE CACHE This is an intermediate approach. The cache is divided into 2-8 sets that are each direct-mapped. However, data may reside in any one of the slots in the sets, at the direct-mapped address. A read results in one comparison in each set. If a hit, data is taken from that set. If a miss, a crude LRU algorithm is used to determine which sets element to replace. A 4 way set associative cache means that there are 4 slots of data in each set. There could be a number of sets in the total cache. SET 1 ADDR: (A16-A5) 000 001 002 A31-A17 FFF =?
7

DATA (32 bytes)

STATUS

CPU A31-A17 A4-A2 DATA OUT SET 2,3,4 same as set 1 (some additional control circuitry needed, plus an RLU implementation- which is sometimes just random replacement). Lets reconsider our questions: 1) Where can a block be placed in the cache? FA-anywhere, DM-one place, SA-one location in each set. 2) How is a block found if it is in the cache? FA-one of N comparators, DM-one comparator, SA-one of a few 3) Which block should be replaced on a cache miss? FA-LRU algorithm, DM-constrained, SA-LRU or random. A common LRU algorithm involves a status bit for each line that is periodically reset. When a hit occurs, the bit is set again. After a while, only the non-accessed lines have bits reset, and can be replaced 4) What happens on a write? In other words, if we update data in the program that we want to store in memory? This is a bit tricky. In some cases (esp. with a DM cache), we may want to avoid writing data to the cache, in case it overwrites instructions or data. A write does not guarantee a subsequent read! But, DRAM writes DO have a time penalty. Thus, there are a number of policies: On a HIT: Write-through (or store-through) Information written BOTH to block in cache AND the lower-level memory. Write-back (copy back) Information written ONLY to block in cache. A dirty bit in status is set to indicate that a write to memory needs to be done. On a MISS: Write allocate (fetch on write) Block is loaded on a write miss, and one of the above actions is taken.

No-write allocate (write around) the block is modified at the lower level only; the cache is not updated.

How can we quantify cache performance? 1) CPUtime = (CPUclockCycles + MemoryStallCycles) * ClockCycleTime 2) MemoryStallCycles = ReadStallCycles + WriteStallCycles = Instructs/program * misses/instruct * MissPenalty 3) ReadStallCycles = Reads/Program x ReadMissRate x ReadMissPenalty Write cycles stalls are more complex because they depend on write protocols. If write-allocate, then a miss involves a read miss penalty. Otherwise, penalty may be minimal (or nonexistent) to transfer data into write buffer. If write buffer write-to-memory doesnt complete before next read miss, however, there will be additional stalls on the read 4)WriteStallCycles = (Writes/program * writeMissRate * WriteMissPenalty) + WriteBufferStalls IMPROVING PERFORMANCE To improve performance, what can we do? Look at the components of Expression 3: A) Reduce Reads/Program these can be reduced by making the compiler smarter. The compiler must choose whether a variable remains in memory or is manipulated in a register location, within a window of the program. Good strategies lead to fewer loads and stores. Reduce ReadMissRate by: Larger cache overall
9

B)

C)

D)

Larger associativity in the cache (4-way vs. 2 way Set Assoc.) Optimizing block size (larger is better to a certain point) Compiler optimizations (e.g. ordering of variables in blocks) Reduce ReadMissPenalty by: Faster main memory (Fig below) Wider memory bus (more bits uploaded at once) (Fig below) Additional cache level(s) example follows PREFETCH instructions and or data- special instruction indicates to memory system that certain address will be read soon, giving memory system time to load it (while CPU is active with preceeding code!). Write optimizations can also be used to improve performance: Defer main memory writes in a WRITE BUFFER. A memory write operation is delayed until there is no read. Note that read addresses must be compared with addresses in write buffer! Use write-back strategies.

The miss penalty can be improved either by increasing the bandwidth or data width of the path between DRAM memory and cache: (example on page 561) CPU 32 L1 cache 128 L2 cache 128 DRAM Normal bandwidth Memory access interleaved (4x faster) High bandwidth (4x faster) 32 32 32

10

SAMPLE PROBLEM USING LEVEL 2 CACHE A system has a CPU with avg CPI of 1.5 and a clock speed 1.0 GHz. Main memory access block transfer time is 150 nsec. The miss rate for a (dataintensive) program is 10%. How much faster will the machine be if a secondary cache is used that transfers a block in 20 nsec, with a 2% miss rate? Consider the system without Level 2 cache: MainMemoryMissPenalty = 150 ns / 1ns clock cycle = 150 clock cycles EffectiveCPI = BaseCPI + memoryStalls/instruction = 1.5 + 0.1 * 150 = 16.5 **** According to Amdahls law, improving CPU wont help much! **** With second-level cache: Level2MissPenalty = 20ns/1ns clock = 20 clocks TotalCPI = base + Lev2 stalls +mainMemStalls = 1.5 + 0.1*20 + 0.02*150 = 1.5 + 2 + 3 = 6.5. Still somewhat far from optimal! Performance = 16.5 / 6.5 = 2.54 Definitely want Level 2 cache in there. But also want a reduced miss rate!

Problems: 1) (7.1) Describe the general characteristics of a program that would exhibit very little temporal and spatial locality with regard to data accesses. Solution: This is a problem where you have no arrays or structures. Memory accesses jump around and are only accessed once. 2) (7.7) Here is a series of address references given as word addresses: 1, 4, 8, 5, 20, 17, 19, 56, 9, 11, 4, 43, 5, 6, 9, 17. Assuming a direct mapped

11

cache with 16 one-word blocks that are initially empty, label each reference in the list as a hit or a miss and show the final contents of the cache. Solution: 1 Miss 4 Miss 8 Miss 5 Miss 20 Miss 17 Miss 19 Miss 56 Miss 9 Miss 11 Miss 4 Miss 43 Miss 5 Hit 6 Miss 9 Hit 17 Hit

Value Slot 0

17 1 2

19 4 3 4

5 5

6 6

56 9 8 9

43 10 11 12 13 14 15

3. (7.8) Using the same references as problem 2, show the hits and misses and final cache contents for a direct mapped cache with four word blocks and a total size of 16 words. Solution. What this means is we still have the same total cache space, but since we have 4 word blocks and only 16 total blocks, we can only have 4 slots. The hits and misses: 1 Miss 4 Miss 8 Miss 5 Hit
12

20 Miss 17 Miss 19 Hit 56 Miss 9 Miss 11 Hit 4 Miss 43 Miss 5 Hit 6 Hit 9 Miss 17 Hit Final contents: Start of word block

16 17 18 19

4 5 6 7 1

8 9 10 11 2 3

Block Number

4) (7.11) Consider a memory hierarchy using one of the three organizations for main memory shown in Figure 7.13 (a: one-word wide memory, b: fourword-wide memory, c: four word interleaved memory). Assume that the cache block size is 16 words. If the main memory latency for a new access is 10 cycles and the transfer time is 1 cycle, what are the miss penalties for each of these organizations? Solution: For a: one-word wide memory: This requires 16 main memory accesses to retrieve a cache block and words are transferred 1 at a time so: 1 + 16*10 + 16*1 = 177 clock cycles.

13

For b: four-word-wide: This requires 4 main memory accesses to retrieve a cache block and they are transferred 4 at a time: 1 + 4*10 + 4*1 = 45 clock cycles For c: four word interleaved: This requires 4 main memory accesses to retrieve a cache block and they are transferred 1 at a time: 1 + 4*10 + 16*1 = 57 5) (7.12) Suppose a processor with a 16 word block size has an effective miss rate per instruction of 0.5%. Assume that the CPI without cache misses is 1.2. Using the memories from problem 4, how much faster is this processor when using the wide memory than narrow or interleaved? Solution: First we have to calculate the effective CPI for each memory: a: Effective CPI = 1.2 + .005*177 = 2.085 b: Effective CPI = 1.2 + .005*45 = 1.425 c: Effective CPI = 1.2 + .005*57 = 1.485 Since it is the same software and CPU, we can compare CPI directly. Wide over narrow = 2.085-1.425 = 1.46 Wide over interleaved = 1.485/1.425 = 1.04 6) (7.20) Using the following series of references, 1, 4, 8, 5, 20, 17, 19, 56, 9, 11, 4, 43, 5, 6, 9, 17, show the hits and misses and final cache contents for a two-way set associative cache with one word blocks and a total size of 16 words. Solution: Ok, two way set assoc. means we have two slots in each group. Each slot is only one word so we have 8 slots. 1 Miss 4 Miss 8 Miss 5 Miss 20 Miss 17 Miss 19 Miss 56 Miss 9 Miss
14

11 Miss 4 Hit 43 Miss 5 Hit 6 Miss 9 Hit 17 Hit Final state: Block # 0 1 2 3 4 5 6 7 Element #1 56 17 43 4 5 6 Element #2 8 9 11 20

7) (7.38) If all misses are classified into one of three categories: compulsory (blocks not there since we are just starting), capacity (blocks not there since they couldnt all fit, some had to be replaced), or conflict (misses in direct mapped or set assoc. that wouldnt occur in a full assoc.), which misses are likely to be reduced when a program is rewritten so as to require less memory? How about if the clock rate of the machine that the program is running on is increased? How about if the associativity of the existing cache is increased? Solution: Less memory: will likely reduce compulsory and capacity misses. Maybe conflicttough to tell Increase clock rate : nothing. Clock rate doesnt effect hit rate. Increase assoc.: should reduce conflict.

Homework:
15

1) (7.5) Describe the general characteristics of a program that would exhibit very high amounts of temporal locality, but very little spatial locality with regard to instruction fetches. 2) (7.9) Compute the total number of bits required to implement the cache in Figure 7.10. This number is different from the size of the cache, which usually refers to the number of bytes of data stored in the cache. The number of bits needed to implement the cache represents the total amount of memory needed for storing all of the data, tags and valid bits. 3) (7.13) Cache C1 is direct-mapped with 16 one-word blocks. Cache C2 is direct-mapped with 4 four-word blocks. Assume that the miss penalty for C1 is 8 clock cycles and C2 is 11 clock cycles. Assuming that the caches are initially empty, find a reference string for which C2 has a lower miss rate but spends more cycles on cache misses than C1. Use word addresses. 4) (7.21) Using the series of references: 1, 4, 8, 5, 20, 17, 19, 56, 9, 11, 4, 43, 5, 6, 9, 17, show the hits and misses and final cache contents for a fully associative cache with one-word blocks and a total size of 16 words. Assume least recently used replacement.

16