CH 6 - PIPELINING

INTRODUCTION - basics of pipelining and the problems that are generated by this technique. Basic Principle is simple: Divide up the instruction execution into discrete parts, much like the multicycle processor: IF - Instruction fetch ID - Register read/ Instruction decode EXE - Execute operation MEM - Memory Access WB - Write back Note that all instructions will write back on the 5th cycle (normally); data will remain on hold during mem access stage if there is no mem access. Key: can execute several instructions in parallel- up to five- because we have five 'functional units' in our pipeline. If instruction n is being executed, n+1 is doing register read, n+2 is being fetched, n-1 is being memory accessed, and n-2 being written back. How does this improve performance? Before, each instruction would have taken 5 cycles. Now, a new instruction might be fetched every cycle- giving an upper-limit performance increase of: MaxSpeedup = CyclesBefore/EffectiveCyclesAfter = pipelineStages It looks as if each instruct takes one cycle! However, execution for a given instruction is the same, or more (due to delayed writeback), than for multicycle. Let's look at an example:
ClockCycle: add $2,$3,$4 lw $4,0($2) beq $4,$0,foo sw $2,0($6) 1 2 if id if 3 4 5 6 7 8 exe mem wb id exe mem wb if id exe mem wb if id exe mem wb 9 10

Much better, right? Well, yes and no. There are a few problems. HAZARDS are situations where the result of an instruction must be determined in order to be able to execute later instructions.

There are three basic types of hazards: 1) Structural hazards occur when the hardware is incapable of supporting the operations needed in the same clock cycle. MIPS is so simple that it does not have structural hazards. If, however, program and data memory were unified, then mem read could not be done at the same time as IF. Or, if there were a two-cycle divide unit that was not pipelined, couldn't have adjacent divide instructions. Solution: STALL. Continue the earlier instruction's execution, but hold back all later instructions one cycle so the conflict can be resolved. This inserts a BUBBLE into the pipeline where useful work is not being done. 2) Control hazards occur because until a conditional branch is evaluated, or a jump address determined, the address of the next instruction is unclear. Question: in the example above, when is the address of the instruction after the branch available? (After bra exe, could do sw if). Solutions: a) Stall until decision known (2 clocks IF branch calculated early) b) Assume "branch-not-taken" and inhibit WB if branch c) Assume "branch-taken", which causes at least one stall d) Branch prediction: keep track of each branch address and whether the last three times branch was taken or not more often. *** Prediction is used in most systems. NO STALLS unless wrong. e) Delayed branch. The instruction after the bra is assumed to always be executed. "Early decision" branch detection could then be used to decide branch at end of id. Allows only bzero, bnz. The instruction after the branch is in the "branch delay slot". Often theres no instruction that works, so nop used. 3) Data hazards are most common. This happens when a register value calculated in a previous instruction has not yet been written back by the time a later instruction needs it in the id stage. Question: Can you find examples in our code? ($2:add->lw, $4: lw->beq; $2:add->sw is NOT a hazard- we can assume that a wb and read in same

cycle can be supported by the hardware. Write completes in beginning of clock, read at end). Solution: Non-load data hazards can be fixed using data forwarding, Load data hazards can't be fixed and must invoke stalls. Note that for non-load instructions, the results are available after the exe stage, and the next instruction doesn't need the result until IT'S exe stage. Thus, the result from the first instruction can be forwarded from the exe output to the exe input, bypassing the register altogether (this technique is also called data bypassing). This is done with multiplexers. Let's reexamine our program snippet. If we used just stalls, we'd have to:
ClockCycle: add $2,$3,$4 lw $4,0($2) beq $4,$0,foo sw $2,0($6) 1 2 3 4 5 6 7 8 if id exe mem wb if STA STA id exe mem wb if STA STA STA STA id 9 10 11 12 13 14

exe mem if id exe mem wb

And the sw WB would occur on clock cycle 14! Note that this is not unusually "unfortunate" code; data hazards are the rule more than the exception. But it is folly to just wait for the branch decision to be made. If we use the schemes we have discussed, and we make the correct branch decision, then we would have:
ClockCycle: add $2,$3,$4 lw $4,0($2) beq $4,$0,foo sw $2,0($6) 1 2 if id if 3 4 5 6 7 8 9 exe mem wb id exe mem wb if STA id exe mem wb if STA id exe mem wb 10 (data forward) (data forward)
(branch predict)

Only one stall, due to the unavoidable load data hazard! A performance improvement of 14/9 = 56%!

That's the background on pipelining, next we'll consider how to extend our simple MIPS to do pipelining.

MIPS pipeline implementation (Sec. 6.2) - fig. 6.10/6.12 To consider how to implement the pipelined MIPS, we'll have to go back to assuming separate program and data memories, to avoid memory contention problems for L/S instructions. Let's start back at the single cycle block diagram, reorganized as 6.10. This shows the datapath elements associated with the five pipeline stages: IF: fetch instruction, update PC=PC+4 ID: register access EXE: execute, calculate branch addr MEM: Load/Store only WB: dest. MUX only Just like in the multicycle implementation, the key factor is that we need to use registers to save the results of each stage so that they can be available as inputs to the next stage in the next clock cycle. The four pipeline registers are shown in the next Figure. The IF/ID register contains the IR and updated PC value. A few of the details are glossed over in this figure. Most egregious is that the write register addressing scheme does not extend through the other three pipeline registers. Control signals and data have to travel together through the registers. Note that registers are updated all at once at the clock edge. It is assumed that the clock is slow enough that all signals settle in time before the edge. Let's run our little program snippet on this architecture. Note that there is no forwarding available, and ... when is the branch decision made? (during exe, PC not updated with branch address until mem).
ClockCycle: add $2,$3,$4 lw $4,0($2) beq $4,$0,foo sw $2,0($6) 1 2 if id if 3 4 5 6 7 8 exe mem wb id exe mem wb if id 9 10 11 12

exe mem if

id

exe

PIPELINE CONTROL (6.3, handout Fig. 6.30) In this lecture, well consider the control signals for a basic pipeline, and then expand to detection and (if possible) circumvention of hazards. The key concept here is that control signals must go through the pipeline registers in step with the data buses, so they can be applied in the correct stage at the correct clock cycle. Look at the handout. We see that once control signals are decoded, they are carried along through the pipeline registers to later stages. Dividing the control signals into the appropriate pipeline stage, we have: IF Nothing to assert (registers always clocked) ID Nothing to write here. RegWrite controlled in last stage. EXE RegDst, ALUop, ALUSrc applied here MEM Branch, MemRead, MemWrite needed here. Branch PC here! WB MemToReg and RegWrite. NOTE that register write happens here! WORK through the example in the text pp. 471-476. You should be able to identify which instruction is doing what in which portion of the pipeline. DATA HAZARDS and FORWARDING (6.4) In order to deal with data hazards, we need to do the following: 1. Detection of data hazards 2. Forward data where possible 3. Stall where forwarding not possible (Loads) Lets consider these steps one at a time. Detection of data hazards. First, consider how far apart instructions can be and still have data hazard problems. Clock Add $1,$2,$3 Add $4,$5,$1 Add $6,$7,$1 Add $8,$9,$1 1 if 2 id if 3 exe id if 4 mem exe id if 5 wb mem exe id 6 7 8 9

wb mem wb exe mem wb

Since we can do reg fetch and writeback in same cycle, The last add is independent of the first. Thus, the second and third are dependent. So we need to examine whether any register values of an instruction are generated by either of the two previous instructions. This is done in the execution stage by pipelining the rs and rt addresses, and comparing them to the rd address in the following two stages: id/exe exe/mem mem/wb

rd rs rt

Forward Detect

If any address matches (rs=rd_mem, rs=rd_wb, rt=rd_mem,rt=rd_wb), data must be forwarded UNLESS the operation is a load. Forwarding is achieved by adding TWO MORE inputs to the MUX at the ALU inputs, which are connected to the datastream outputs from the mem and WB stages. See Handout. What happens with a load? Clock: 1 2 Lw $1, 0($2) if id Add $3,$4,$1 if Add $5,$6,$1 Lw $1, 0($2) Add $3,$4,$10 Add $5,$6,$1 if id if 3 exe id if exe id if 4 mem stall stall 5 wb exe id 6 7 8

mem wb (data haz) exe mem wb (no data haz) (data haz)

mem wb exe mem wb id exe mem wb

Data forwarding can be used after one cycle. Stall required if instruction after load has data hazard with load. Thus, we still need to be able to do a stall. Heres how: Need a HAZARD DETECTION UNIT (in the books pseudocode): If(ID/EX.MemRead and ((ID/EX.rt=IF/ID.Rs) or (ID/EX.rt = IF/ID.rt)) then stall the pipelines if and id stages for one cycle. Note that exe continues (to do memory load), while the following instructions (in id, if) are stalled one cycle. Note that ID/EX.rt is actually the destination address for the Load, since the RegDst MUX is in the EX stage! The stall is accomplished by inhibiting clocking of the PC and IF/ID pipeline. Also, the control signals for the bubble are set to 0, which inhibits writeback, mem write, and branching. Thats all!! BRANCH HAZARDS (6.6) Reduction of branch hazards can be achieved in a number of ways: 1) Assume branch not taken. This will give no stalls IF branch shouldnt be taken, but will give 3 stalls if it IS taken, if new address not calculated until mem unit Clock: 1 2 3 4 5 6 7 8 Beq $1, $2, foo if id exe mem wb +1 if id exe mem (wb) +2 if id exe mem (wb) +3 if id exe mem (wb) foo: instruct if id exe mem *** write/writeback inhibited for +1, +2, +3 *** 2) Reducing branch delay. Can improve above situation by one cycle by pre-calculating branch address. PC+4 can be calculated in the IF stage, and the offset can be calculated in the ID stage. Normally would still have to wait for exe to determine result, so earliest that new IF could occur would be while branch is in mem (2 stalls). We can go one better by using new hardware that determines if rs, rt equal and put it in ID/reg fetch stage (see diagram). Then, decision
7

can be made in ID stage, and branched-to IF can happen during EXE, giving only ONE delay 3) Dynamic branch prediction. Based on actual history of execution! Uses branch prediction buffer or branch history table. Small memory that is indexed by lowest address bits, and has high address bits and recent history: A2-A6 A7-A31 A7-A31 H0-H1 =? Branch decision Match: instruction is branch

Where decision is based on state machine that predicts based on two of three last branch outcomes at that address. When branch resolved, the decision outcome bits are modified. 4) Delayed Branch instruction after branch ALWAYS executed. Used to reduce bubbles even if branch prediction is wrong. Or for jumps. EXCEPTIONS (6.7) As usual, exceptions mess up the elegance of the architectural model. Exceptions can happen in several pipeline stages (id=bad instruction, exe=overflow, mem=memory fault) and in some of these cases we need to RESTART the program from the instruction that faulted. External interrupts need to be dealt with as well. Need logic that COMPLETES (WB) instructions BEFORE the faulting instruction, but disables writeback of the faulting instruction or later ones. Again, control bits of the faulting instruction and later ones can be zeroed out. Also need to save the address (+4) of the faulting instruction in the EPC, and the reason in the CAUSE register. This means passing the address+4 through the pipeline registers (Fig. 6.55) allows saving exe exceptions but not mem.

For interrupts, there is some flexibility as to where the pipeline should complete. Somewhat arbitrary. SUPERSCALAR PROCESSORS (6.8) In the hunt for greater performance, the step after pipelining was superscalar. A superscalar CPU is capable of issuing (fetching) more than one instruction at a time. Usually the ORDER of a superscalar is 2 or 4 (simultaneous instructions). How is this done? 1) The cache memory can send 2 or 4 instructions at a time to the CPU. (The path from DRAM to cache may be narrower). 2) The IF and ID stages run in tandem for these instructions, usually, so that the register set is multiple ported. 3) Once the instructions are decoded, they are scheduled for issue to one of a collection of functional units: Int unit IF ID Int unit Float unit Load/Store Branch unit 4) Instructions may be issued OUT OF ORDER if there is no functional unit available at the right time (structural hazard). This is called dynamic scheduling. 5) Some instructions (div, float ops) take longer, so instructions can complete OUT OF ORDER 6) WRITEBACK is done by collecting results and writing them back in order, with appropriate data forwarding. Control and exceptions are VERY COMPLICATED! A SCOREBOARD is often used to keep track of instructions as they are executing, and data/control dependencies between instructions. Nifty techniques can be implemented, such as executing both the nonbranch code and the branch-to code at the same time, then tossing out the instructions that were not to be executed.
9

Writeback

The DEC Alpha 21264 was the first processor to implement a superscalar processor with deep pipelines (nine stages for integer ops only!), 4th order, max of 6 instructions issued at once, first 1GHz clock CPU. Problems: 1) (6.2) Show the forwarding paths needed to execute the following three instructions: add add add $2, $3, $4 $4, $5, $6 $5, $3, $4

Solution: we need a forwarding path from the second instruction to the third since it depends on the first ( $4) 2) (6.3) How could we modify the following code to make use of a delayed branch slot? Loop: lw $2, 100($3) Addi $3, $3, 4 Beq $3, $4, Loop addi $3, $3, 4 Beq $3, $4, Loop Lw $2, 96($3)

Solution:

Loop:

3) (6.11) Consider executing the following code on the pipelined datapath of Figure 6.46. Add Add Add Add Add $1, $2, $3 $4, $5, $6 $7, $8, $9 $10, $11, $12 $13, $14, $15

10

At the end of the fifth cycle of execution, which registers are being read and which registers being written? Solution: In the fifth cycle, register $1 will be written and $11 and $12 will be read. 4) (6.12) With regard to the last problem, explain what the forwarding unit is doing during the fifth cycle of execution. If any comparisons are being made, mention them. Solution: The forwarding unit is looking at the instructions in the fourth and fifth stages and checking to see whether they intend to write to the register file and whether the register written is being used as an ALU input. Thus, it is comparing 8 = 4? 8 = 1? 9 = 4? 9 = 1? 5) (6.23) Normally we want to maximize performance on our pipelined datapath with forwarding and stalls on use after a load. Rewrite this code to minimize performance while still obtaining the same result. Lw Lw Add Add Add Sw Beq Solution: Lw Add Sw Lw Add Add Beq $3, 0($5) $7, $7, $3 $6, 0($5) $4, 4($5) $8, $8, $4 $10, $7, $8 $10, $11, Loop $3, 0($5) $4, 4($5) $7, $7, $3 $8, $8, $4 $10, $7, $8 $6, 0($5) $10, $11, Loop

11

Homework: 1) (6.4) Identify all of the data dependencies in the following code. Which dependencies are data hazards that will be removed via forwarding? Add Add Sw Add $2, $5, $4 $4, $2, $5 $5, 100($2) $3, $2, $4

2) (6.9) Given Figure 6.71, determine as much as you can about the five instructions in the five pipeline stages. If you cannot fill in a field of an instruction, state why. 3) (6.14) Consider a program consisting of 100 lw instructions and in which each instruction is dependent upon the instruction before it. What would the actual CPI be if the program were run on the pipelined datapath of Figure 6.45? 4) (6.15) Consider executing the following code on the pipelined datapath of Figure 6.46. add $5, $6, $7 lw $6, 100($7) sub $7, $6, $8 How many cycles will it take to execute this code? State what hazards there are and how they are fixed.

12