Weighted Fair Queuing (WFQ)

Prof. T. Szymanski
McMaster University

Overview
! Internet Flows & Fairness
! Fluid-Flow Generalized Processor Sharing (GPS)
! Packet-Flow Weighted Fair Queuing (WFQ)
! Definition
! Computing Finishing Times
! Examples
! Jitter (Inter-Departure Time) Distribution
! Commercial Implementations
! Other Scheduling Methods

4DN4 - WFQ, pg 2

Prof. T Szymanski

Internet Flow/Session
"

"

Definition: A flow is a sequence of packets sent from one IP source

to one IP destination [text, p594]
the source and destination addresses are defined by an IP address
and a port number
Bandwidth

Example of the bandwidth requirement of a traffic flow. [1, Fig 7.52, p551]

Prof. T Szymanski

4DN4 - WFQ, pg 3

Flow of Packets
"

"

"

"

"

[text, p539]

An IP router = a node where IP packets arrive, are segmented into

fixed-size cells, queued at the input queues, scheduled and
transmitted through the switch, reassembled and buffered at the
output queues, queued at the output fiber awaiting transmission, and
eventually transmitted onto the outgoing fiber .
An IP router will contain Input Queues (IQs) and Output Queues
(OQs)
A typical backbone router with 40 Gbps links will buffer typically up to
5 million IP packets per port
Packets from competing traffic flows compete for buffer space and
transmission bandwidth along the outgoing transmission line
How does one define fairness?

4DN4 - WFQ, pg 4

Prof. T Szymanski

Flow of Packets Through Routers

Consider the end-to-end QoS of a packet traversing N switches [1,
Fig 7.40, p539]
Each packet traverses N queueing systems, and experiences delay
and jitter at each queueing system.

4DN4 - WFQ, pg 5

Prof. T Szymanski

Fairness
"

Intuitively
"

"

"

Any excess bandwidth should be equally shared amongst the

flows.

Protection against Starvation

"

"

Competing flows should be guaranteed the bandwidth they

have reserved (assuming they can reserve bandwidth)

Fairness prevents Starvation or Deadlock, where flows are

deprived of access to buffers/bandwidth [text, p540]

Fair scheduling
"

Automatically isolates heavy users, to avoid resource hogging

"

Provides dynamic sharing of unused bandwidth allocations.

4DN4 - WFQ, pg 6

Prof. T Szymanski

Requirements of Fair Scheduling

"

An ideal scheduler:
" is easy to implement
" is totally fair in resource sharing
" provides protection against starvation
" provides guaranteed performance bounds
" offers different flows different QoS; each flow given a
weight, ie a guaranteed fraction of resource usage
Packet
voice
Interactive video
Email

w1

w2
w3

Flows entering the same queue are given weights, w1, w2, and w3, and share the bandwidth r
accordingly.

Prof. T Szymanski

4DN4 - WFQ, pg 7

Problem with Round-Robin Service

"

"

"

(pg 544, text)

Suppose 4 flows arrive at an IP router. Why not service the

packets in round-robin order ?
Suppose packets in flow 1 are large, packets in flows 2 and 3
are small, and packets in flow 4 are very small.
Servicing queues in round-robin order would give flow 1 much
more service than the other flows, since it has much larger
packets. Similarly, flow 4 would receive very little bandwidth.

4DN4 - WFQ, pg 8

Prof. T Szymanski

Generalized Processor Sharing(GPS)

"

GPS is an idealized Fluid Model, developed at MIT, which

provides an upper bound on Fairness [1, p542]
"
"

"

"

Each flow has its own buffer.

Transmission bandwidth is divided among the flows
according to flow weights
Conceptually, packets in buffers like a fluid that is drained
continuously.

How it works (assume all flow weights are equal for now) :
"

"

"

Assume initially that we service individual bits from each

flow, in a round-robin order
Given n flows in the system and transmission capacity = C
bits/sec, then each flow is guaranteed transmission rate =
(C/n) bits/sec
A flow could experience a higher rate when the other flows
are not using their allocation of bandwidth.

4DN4 - WFQ, pg 9

Prof. T Szymanski

Fluid (Bit-by-Bit) Round-Robin Service in GPS

let R(t) denote the current round of service at time t; R(t) is also called
Virtual Time. Virtual time increases quickly when few buffers are active.
Virtual time increases slowly when many buffers are active.
4DN4 - WFQ, pg 10

Prof. T Szymanski

Definition of a GPS Round

"

"

"

"

"

(text, Fig. 7.46, pg. 544)

One round consists of a round-robin service cycle in which all n

queues are offered service (1 bit of service each, for now).
Therefore, the amount of real time elapsed per round = (number
of active queues) * (time serving one bit from each active queue)
nactive(t) queues = number of active (non-empty) queues at time t
In round of service, all nactive(t) queues (i.e. non-empty queues)
are serviced (1 bit for now).
The Round Number R(t) = the number of rounds of service
completed by the GPS round-robin server at the current time t

4DN4 - WFQ, pg 11

Prof. T Szymanski

Packet Finishing Time

"

"

"

"

The Fluid-Based GPS system effectively computes the ideal finishing

time of a packet in a perfectly fair system
The finishing time of a packet is easily expressed in terms of rounds
(Virtual-Time), rather than real time (i.e. a packet with 100 bits will
finish service after 100 rounds have expired)
In fluid-based GPS, packets are said to depart the system when they
reach their finishing times => Packets with smallest finishing times are
transmitted first.
The GPS scheduler maintains several state variables:
1) The current round number R(t) or the current Virtual Time
2) The finishing time (in rounds) for every packet in the system,
3) The last finishing time (in rounds) of the last packet in every queue.

4DN4 - WFQ, pg 12

Prof. T Szymanski

Finishing time: Empty Queue Case

(text, pg 545)

"

GPS Finishing times are expressed in rounds of service.

"

Let F(f,k) denote the finishing time of packet k from flow f in a queue

"

"

Suppose packet k from flow f arrives at an empty queue at real time t

(f,k) and round (ie virtual time) R( t(f,k) ).
The Packet has length P(f,k) bits. Therefore, it will complete its
transmission after P(f,k) rounds of additional service (since each
round services one bit)

"

Assuming all flows request equal service:

"

The finishing time, expressed in rounds, is clearly:

F(f,k) = R( t(f,k) ) + P(f,k)

(1)

Prof. T Szymanski

4DN4 - WFQ, pg 13

Finishing time: NonEmpty Queue Case

"
"

"

"
"

Suppose packet k from flow f arrives at a nonempty queue

It must wait for all the packets ahead of it (and in particular packet
k-1 which is directly ahead of it) to be serviced before it is serviced.
The finishing time of the packet ahead of it (packet k-1) in the queue
is F(f, k-1).
Assuming all flows request equal service:
The current packet needs P(f,k) additional rounds of service,
therefore the finishing time of this packet is clearly:
F(f,k) = F(f,k-1) + P(f,k)

4DN4 - WFQ, pg 14

(2)

Prof. T Szymanski

Finishing time: Empty and NonEmpty Queues

From the last slides we have:
"

F(f,k) = R( t(f,k) )+ P(f,k)

- Empty Queue Case

(1)

"

F(f,k) = F(f,k-1) + P(f,k)

- NonEmpty Queue Case (2)

Combining equations (1) and (2) we get

F(f,k) = max{F(f,k-1), R( t(f,k) )} + P(f,k)

This is all great, but how does it help us?

We can now compute the ideal departure time of packets in a perfectly fair GPS
system, expressed in rounds, when each flow requests equal service.
However, can we really build IP routers which service packets in round-robin order
at the bit-level (ie one bit of service per round) ? No, but we do not need to. We only
need to know the ideal finishing time of each packet in a GPS system.
Also, what if flows request different levels of service?
Prof. T Szymanski

4DN4 - WFQ, pg 15

Packet-oriented GPS == WFQ

(pg 544)

Let a Weighted Fair Queueing (WFQ) Server run in parallel with an ideal
GPS server. The WFQ server only services whole IP packets (it doesnt
split them into bits).
Given variable length IP packets, suppose the WFQ server will service
entire packets in the same order as the perfectly fair service order of a
fluid-flow GPS system, i.e., the next packet to be transmitted over a
transmission line by the WFQ server is the packet with the smallest GPS
Virtual Finishing time amongst all packets waiting for that transmission
line
Call this system Packet-oriented GPS, to distinguish it from the Fluidflow GPS. It is also called Weighted Fair Queueing. This scheme is
implemented in CISCO routers.

4DN4 - WFQ, pg 16

Prof. T Szymanski

Packet-by-Packet WFQ vs Fluid-Flow GPS

"

"

"

"

"

Following slides will illustrate the differences between ideal fluid

flow GPS and Packet-by-Packet Weighted-Fair-Queuing.
Assume buffers 1 and 2 each have one L-bit packet to transmit at
time t = 0, and that no subsequent packets arrive
Assume the transmission capacity is C = L bits/second, then we
can transmit 1 packet/sec .
The idealized fluid-flow GPS system transmits both packets
simultaneously at a rate = ! packet/sec and both packets
complete at time t = 2 seconds.
The packet-by-packet WFQ system transmits (1) the packet from
buffer 1 at rate = 1 packet/sec, then (2) the packet from buffer 2 at
rate = 1 packet/sec, so the packet completion times are 1 and 2
seconds, respectively.

Prof. T Szymanski

4DN4 - WFQ, pg 17

GPS vs Packet-by-Packet WFQ,

(Fig 7.45, pg 543)

Note: Y-axis = fraction of packet in buffer1 and buffer 2. Both buffers contain 1 L -bit packet initially.

4DN4 - WFQ, pg 18

Prof. T Szymanski

Packets with Different Lengths

(Fig 7.46, pg 546)

4DN4 - WFQ, pg 19

Adding Flow Weights to GPS or WFQ

"
"

"
"

"

"

"

Prof. T Szymanski

(pg 546)

In a real IP router, different flows will have different bandwidth requirements

Let each flow j have its own buffer and a weight Wj which determines its
relative share of bandwidth C (bits/sec)
In each service round, the server services Wj bits from flow j
When all flows are active, each flow j receives the fluid service at the rate C*
(Wj) /sum(Wj == guaranteed service rate for each flow
When some flows are idle, the excess bandwidth is shared amongst active flows
according to the weight of the active flows
the next slide illustrates an example of Weighted Fair Queueing - WFQ. Let flow
1 have weight W1=(1/4), and flow 2 have weight W2 = (3/4)
Both flows have packet size = 1. The time to complete each packet is (L bits)/
(Wj * C bits/sec), where Wj is expressed as a fraction. At t=0, based upon the 2
active flows, the time to complete flow #1 = L bits/(1/4 * C bits/sec) = 4 L/C
seconds. Time to complete flow #2 = L bits/(3/4 * C bits/sec) = (4/3) L/C
seconds. In this case (L/C) = 1. Once flow 2 is finished it service, since there are
no other active flows, flow 1 starts receiving service at the rate L bits/sec.
(When making such graphs, one difficulty is keeping track of the discontinuities,
when the service rates change.)

4DN4 - WFQ, pg 20

Prof. T Szymanski

Weighted GPS and Packet Fair Queueing

(Fig. 7.48, pg 546)

Prof. T Szymanski

4DN4 - WFQ, pg 21

GPS Finishing Time: Empty Queue

"
"

"

"

"

Finishing times are expressed in rounds of service.

Suppose packet k from flow f arrives at an empty queue at time t
(f,k) and round R( t(f,k) )=>
the packet arrives when R( t(f,k) ) rounds of service have already
finished.
The Packet has length P(f, k) bits. Therefore, it will complete its
transmission after P(f,k)/Wf rounds of additional service (Wf
expressed as an integer), since each round services Wf bits
The finishing time is clearly:
F(f,k)= R( t(f,k) ) + P(f,k)/Wf

4DN4 - WFQ, pg 22

(1)

Prof. T Szymanski

GPS Finishing time: Non-Empty Queue

"

"
"

If packet k from flow f arrives at a nonempty queue, it must wait

for all packets ahead of it (in particular packet k-1 from flow f) to
be serviced before it is serviced.
The finishing time of the packet k-1 ahead if it is F(f, k-1).
The current packet needs P(f,k)/Wf additional rounds of service,
therefore, the finishing time of this packet is clearly:
F(f,k) = F(f,k-1) + P(f,k)/Wf

(2)

Prof. T Szymanski

4DN4 - WFQ, pg 23

Comparing Both Finishing Times:

"

Recall the finishing times using GPS fluid-flow fair queuing,

when all flows have the same weight:
F(f,k) = max{F(f,k-1), R( t(f,k) } + P(f,k)

"

When each flow has its own weight, the equation for the
finishing times using GPQ fluid-flow fair queueing becomes:
F(f,k) = max{F(f,k-1), R( t(f,k) } + P(f,k)/Wf

"

Each flows weight determines the flows share of the

bandwidth.

4DN4 - WFQ, pg 24

Prof. T Szymanski

Example
"

"
"
"

Consider 3 flows being serviced by a link with capacity C = 1 Mbps (ie 1 usec
per bit). Assume each flow has its own queue with weights 5, 10, and 20
(bits/round).
Flow 1: pkt with 100 bits arrives at time t = 0 usec, weight = 5
Flow 2: pkt with 300 bits arrives at time t = 50 usec, weight = 10
Flow 3: pkt with 200 bits arrives at time t = 100 usec, weight = 20

"
"

"

"

(a) Plot the # Queued bits versus the round, for each queue in a fluid-flow
GPS queueing system. (A real router does not need plot this. We ask this
question to make sure you understand why the VFT equations are valid.)
(b) Plot the # Queued bits versus the real-time, for each queue in a fluid-flow
GPS queueing system. (A real router does not need to plot this. We ask this
question to make sure you understand why the VFT equations are valid.)
(c) Label the slope of each piecewise linear segment. Label the (x,y)
coordinates of each end point of a piece-wise linear segment

4DN4 - WFQ, pg 25

Prof. T Szymanski

Example
"

"

"

(d) What are the departure times of the last bit in each flow, first expressed
in Rounds (Virtual-Time), and then expressed in Real-Time ?
(e) Now consider a packet-by-packet WFQ approximation of the fluid-flow
GPS system. Only one whole (entire) packet is transmitted at a time.
Find the start time and end time for each packet in each flow, expressed in
real-time. (A real router needs to know these numbers.)

4DN4 - WFQ, pg 26

Prof. T Szymanski

Solution Summary
Flo
w

Bits

Weight Arr. Time

* (usec)

Arr. Time
(round)

Exit Time
(round)

ExitTime
* (usec)

100

0 us

30

320

300

10

50 us

10

40

600

200

20

100 us

14

24

440

Nill

Assume new flows arrive at the beginning of a round, for

simplicity.

Prof. T Szymanski

4DN4 - WFQ, pg 27

Bits vs Virtual Time (Round)

This slide helps explain the

GPS equations.
The bits in a GPS queue
drain at a smooth rate,
when the x-axis is the round
of service

4DN4 - WFQ, pg 28

Prof. T Szymanski

Bits vs Real Time

This slide helps explain the

GPS equations.
The bits in a GPS queue
drain at a variable rate, when
the x-axis is real-time.
That is why we use the virtual
time in the GPS equations It is
easier to compute the virtual
finishing times using virtual
time (round), than real-time.

4DN4 - WFQ, pg 29

Prof. T Szymanski

Queued Bits vs Real-Time, WFQ

A real WFQ server in a router

implements this behaviour:
The green packet arrives at time
100 usec, just as the red packet
finishes transmission. It has a
smaller VFT than the blue packet
(see slide 28), so the WFQ server
sends the green packet before the
blue packet, even though the blue
packet arrived before the green
packet.

4DN4 - WFQ, pg 30

Prof. T Szymanski

What Does a Real Router Do?

"

"

"

"

A real router must implement the 4 GPS equations, to compute the virtual
finishing time of each packet that arrives at the router.
A router must have 2 clocks, a real-time clock, and a virtual-time clock (i.e.,
the current round of service).
Every time a packet arrives at the router, the router must compute the
packets VFT in rounds (virtual time).
A real router has a WFQ server, which transmits outgoing packets in the
order of increasing packet VFTs, so that packets leave the real router in the
same order as if a perfectly fair fluid-flow GPS server was used.

4DN4 - WFQ, pg 31

Prof. T Szymanski

Jitter Comparison, TCP vs WFQ

Compare: TCP vs WFQ traffic sources, generating packets at a

average rate of 50 Mbp, for a 1 Gbps transmission line.
A TCP source could generate bursts of packets, at the full
transmission rate of 1 Gbps, every time a sequence of ACKs is received
(roughly every round-trip time): bursts could occur 1/20 of the time, idle
periods could occupy 19/20 of the time, and if the network is not too
loaded, the bursts will travel without much dispersion => potentially
overloading queues
4DN4 - WFQ, pg 32

Prof. T Szymanski

Inter-Departure Time Distribution, TCP vs WFQ

"

"

"
"

Assume packets are fixed-size; the TCP departures vs. Time will be
bursty:
An unconstrained TCP source produces bursts of packets(red),
consuming the full 1Gbps bandwidth for the duration of the burst.
Bursts also be caused by the uneven delays in the reception of ACKs.
TCP can cause instantaneous congestion at intermediate nodes ,
causing potential dropping of packets when Random Early Detection
(RED) is used, introducing wasted bandwidth and resource usage.

4DN4 - WFQ, pg 33

Prof. T Szymanski

Departure Time Distribution , TCP vs WFQ

! The WFQ jitter is much less bursty

! With WFQ, each flow has a fair share of the 1 Gbps bandwidth. At a
50 Mb/sec sustained rate, WFQ packet departures are evenly spaced,
with one packet in every 20 packet time-slots, followed by 19 empty
time-slots.
! The traffic is much less bursty and will reduce/eliminate the
instantaneous congestion at intermediate routers that TCP bursts
may experience

4DN4 - WFQ, pg 34

Prof. T Szymanski

Commercial Implementations
http://nsssc.superb.net/img/algx-usmap.gif

! Consider a national network:

! WFQ can be used per individual flow.
! Very expensive when used for individual flows, since we may have over
1 billion flows for a national backbone network.
Prof. T Szymanski

4DN4 - WFQ, pg 35

Class-Based Weighted Fair Queuing

[2]

! A commercial implementation of WFQ is Class Based Weighted Fair

Queuing (CBWFQ) , used by Cisco Systems
! CBWFQ extends the WFQ functionality by providing support for userdefined traffic classes instead of individual flows.
! IP doesnt have any easy way to specify flows. Therefore, classes are
defined by a match criteria on packet headers which include:
! Protocols (TCP, UDP, RTP, etc)
! IP addresses
! TCP and UDP ports
! A queue is reserved for each traffic class. Incoming packets are filtered
and IP packets that meet the match criteria for a class are placed in their
corresponding class queue. Network management software allows for IP
routers to specify class-match criteria (class filters)

4DN4 - WFQ, pg 36

Prof. T Szymanski

Class-Based Weighted Fair Queuing

! After defining a class according to its filter, the characteristics of the
class must be assigned.
! Class characteristics include:
! Bandwidth guaranteed amount to be delivered to a class
determined by class weight.
! Maximum Packet limit maximum amount of packets allowed
to accumulate in a queue for that class.
! The weight for each packet is derived from class weight and CBWFQ
uses the class weights to ensure that each class is serviced fairly.
! More information available at Cisco Systems website:
www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/
120t/120t5/cbwfq.htm

4DN4 - WFQ, pg 37

Prof. T Szymanski

Other scheduling methods

"

There are many other proposed fair queue scheduling algorithms,

and many more variations of Weighted Fair Queueing, including:

"

Deficit Round Robin

"

Self-Clocked Fair Queuing

"

Rate Controlled Service Discipline

"

Worst case fair queuing

"

GPS and WFQ define the ideal fair bandwidth-sharing performance.

In practice, one cannot implement fluid-flow GPS, so an algorithm
which approximates the ideal scheme, such as packet-based WFQ,
can be used.

4DN4 - WFQ, pg 38

Prof. T Szymanski

Appendix
1.

Communication Networks: Fundamental Concepts and Key

Architectures Second Edition. Alberto Leon-Garcia and Indra
Widjaja. McGraw-Hill.

2.

Class-Based Weighted Fair Queuing. Cisco Systems Inc.

www.cisco.com/univercd/cc/td/doc/product/software/
ios120/120newft/120t/120t5/cbwfq.htm

3.

A. Parekh and R.G. Gallager, A Generalized Processor Sharing

Approach to Flow Control in Integrated Services Networks: The
Single-Node Case IEEE/ACM Trans. Networking, vol. 1, no. 3,
pp. 344-357, June 1993. (Based upon A. Parekhs PhD Thesis at
MIT).

Prof. T Szymanski

4DN4 - WFQ, pg 39

Example

4DN4 - WFQ, pg 40

(Ignore in 2013)

Prof. T Szymanski

Example

(Ignore in 2013)

Prof. T Szymanski

4DN4 - WFQ, pg 41

Example

4DN4 - WFQ, pg 42

(Ignore in 2013)

Prof. T Szymanski

Example

(Ignore in 2013)

Prof. T Szymanski

4DN4 - WFQ, pg 43

Example

4DN4 - WFQ, pg 44

(Ignore in 2013)

Prof. T Szymanski

Some Fun With Matlab

! Lets use Matlab to implement WFQ as a real router would
! Most Internet routers implement WFQ in real time to achieve some
type of fairness, processing millions of packets per second
! We generally have around a microsecond per packet to compute
the fair GPS virtual finishing times
! Lets (1) generate one flow = random stream of arriving packets
! Lets (2) plot the flow so we can visualize the packets and their
sizes
! Lets (3) generate 4 flows
! Lets (4) plot the 4 flows on one graph, so we can see the
competition for bandwidth
! Let (5) figure out how to implement WFQ in a router !

4DN4 - WFQ, pg 45

Prof. T Szymanski

Generate One Flow

4DN4 - WFQ, pg 46

Prof. T Szymanski

Plot the Flow on One Graph

4DN4 - WFQ, pg 47

Prof. T Szymanski

The Main program : Generate 1 Flow

4DN4 - WFQ, pg 48

Prof. T Szymanski

The Figure : One Flow (Packet Stream)

4DN4 - WFQ, pg 49

Prof. T Szymanski

Generate 4 Flows (Packet Streams)

4DN4- WFQ, pg 50

Prof. T Szymanski

Plot All 4 Flows on One Graph

Prof. T Szymanski

4DN4 - WFQ, pg 51

The Result: 4 Arriving Flows

Now lets use WFQ to achieve

fairness in a router.
4DN4 - WFQ, pg 52

Prof. T Szymanski

Next: In Class Example for WFQ

! Ill ask the class for ideas on how to use Matlab to implement
WFQ as a real router would
! Lets experiment with the Matlab, and see what we can achieve
!

Assignment #2 will cover WFQ

4DN4 - WFQ, pg 53

Prof. T Szymanski