A New End-to-End Probing and Analysis Method for Estimating Bandwidth Bottlenecks

Bob Melander, Mats Bjorkman, Per Gunningberg Dept. of Computer Systems Uppsala University Box 325, SE-75 1 05 Uppsala, Sweden. E-mail:{Bob.Melander, M a t s . B j o r k m a n , Per.Gunningberg}@docs.uu.se
Abswoct- We present a network friendly bandwidth measurement method, TOPP, that is based on active probing and includes analysis by segmented regression. This method can estimate two complementing available bandwidth metrics in addition to the link bandwidth of the congested link. Contrary to traditional packet pair estimates of the bottleneck link bandwidth, our estimate is not limited by the rate at which we can inject probe packets into the network. We also show that our method is able to detect bottlenecks that are invisible to methods such as C-probe. Furthermore, we describe scenarios where our analysis method is able to calculate bandwidth estimates for several congested hops based on a single end-toend probe session.

ing methods. We define three bandwidth metrics that are useful when establishing the somewhat elusive notion of available bandwidth. We show by measurements and simulations how TOPP overcomes the hidden bottleneck problem.
11. BANDWIDTH DEFINITIONS

In this section we define and discuss three complementing metrics that are useful when characterizing the bandwidth along a network path.
A. Bottleneck link bandwidth

I. INTRODUCTION
In this paper, we present a new end-to-end probing and analysis method, TOPP,that overcomes some of the problems with several existing bandwidth probing methods. These problems include the inability to identify the situation when different types of bandwidth bottlenecks occur on separate links. We call this problem the hidden bottleneckproblem, and we present measurements and ns simulations of probing in these situations. In order to clarify the difference between various types of bandwidth bottlenecks, we present definitions of surplus bandwidth, proportional share bandwidth, and protocol dependent available bandwidth. TOPP probing is an extension to the packet pair probing technique for estimating bandwidth bottlenecks. This technique has been studied by a number of researchers [4], [2]. Carter and Crovella [3] propose two packet-pair based techniques, B-probe and C-probe, to measure bottleneck link bandwidth and available bandwidth, respectively. Another variant of the packet-pair technique have been suggested by Lai and Baker [6]. They use a receiver-only method based on existing traffic together with a potential bandwidth filtering scheme. In order to quickly adapt to bandwidth changes they introduce a packet window. When estimating the bandwidth, only packets that arrive within the time interval given by the window will be used. Paxson [7] has pointed out a number of weaknesses of the packet pair method, such as the problem with multi-channel links, limitations due to clock resolution and out of order packet delivery. To deal with these shortcomings, he introduces an extension of the packet pair technique called the PBM probing technique. There, estimates for a range of probe bunch sizes (i.e. train lengths) are formed and multiple bottleneck values are allowed. The contributions made in this paper are the following: We present a probing and analysis method, TOPP, that overcomes some of the problems common for many types of existing prob-

With the bottleneck link bandwidth, we mean the data transmission rate of the slowest forwarding element of a network path between a sender and a receiver. A sender is seldom alone on a path, but is sharing hops with cross traffic from other sources. Of interest to an application is then the available bandwidth on the path, i.e. the bandwidth the application can achieve when sharing hops with cross traffic. The available bandwidth for an application depends on the behavior of many factors including the application, the protocols, the characteristics of the cross traffic and the routers. The cross traffic will also vary in time, making the available bandwidth an even more elusive goal to measure. We have identified three metrics that can be used to characterize the available bandwidth of a path: the proportional share, the surplus, and the protocol dependent available bandwidth.
B. Proportional share

When a link on a path has link bandwidth 1 and a cross traffic with rate mi and when a sender wants to introduce traffic with rate o such that m < 1 < m + 0, we have an overload situation. What share of the link bandwidth the sender will get depends on the sharing policy of the link multiplexer. The common policy of backbone routers is to use First Come First Serve or some similar policy. Under the assumption of a proportional stateless scheduling policy, such as FCFS, and a random dropping policy of packets at buffer overflow,the connections will get a share of the link bandwidth proportional to their offered rate. The share for new traffic with offered load o is then:

f =o 1 O m+

(1)

This means that f will increase when o increases. An aggressive sender (e.g. a sender using UDP) could therefore momentarily

0-7803-6451-1/00/$10.00 0 ZOO0 IEEE

415

grab a substantial part of the link bandwidth of a link by increas- The observed proportional share bandwidth at the receiver, f , is thus f = oj+,+l. This model of the observed bandwidth, f ing the load. as a function of the offered bandwidth, 0, will later be used to C. Surplus bandwidth estimate the link bottleneck bandwidth, surplus bottleneck bandWith the surplus bottleneck bandwidth of a path we mean the width and the proportional share bandwidth of a path. We call a link i congestible for a particular load o if oi > highest possible sending rate of a new sender without affecting cross traffic on the path. With affecting we mean that the si. Congestible links are detectable by our probing method. cross traffic should not be forced to lower their rate or start to Whether a link is congestible or not depends not only on the lose packets when the sender starts to send with the surplus rate. surplus bandwidth but also on all links upstream. For a particAssume that the sum of the cross traffic on link i has a sus- ular load o there is a set of congestible links. Assume the set is tained rate of mi. Then the surplus bandwidth on the link is ordered in the order they are detected when raising the offered s i = li -mi ( l i 2 si for all i). The surplus bottleneck bandwidth load. We call this order Smallest Surplus First, SSF for short. is then: SI, = min(s1, s2,. . . ,s, The surplus bandwidth is a ,} Iv. THEHIDDEN BOTTLENECK PROBLEM lower bound on the available bandwidth. The surplus bandwidth is an important metric, since, with We have discovered a fundamental problem with probing knowledge about the link and surplus bandwidth of a link, a promethods based on sequences of back-to-back packets, e.g. Cportional share can be calculated for an arbitrary offered bandprobe. They can only find the surplus bottleneck if it coincides width. with the link bottleneck on a path. C-probe sends a train of ICMP echo packets to the destination and measure the interarD. Protocol dependent available bandwidth rival times of the reply packets. The probe packets in a train For an application and its protocols, it is interesting to know are of equal size and are sent as close as possible. The availwhat sustainable rate the application can achieve and we call this able bandwidth is calculated as the amount of data in a train rate the protocol dependent availab[e bandwidth. This rate de- divided by the arrival time difference between the last and the pends not only on protocols and applications but also on the be- first packet in a train. havior of the cross traffic. When the sustained rate of the sender The C-probe packet train only measures the proportional reaches the surplus bottleneck rate and beyond, the cross trafshare available bandwidth at the back-to-back transmission rate. fic will be affected and may react accordingly to accommodate For estimating the surplus bandwidth it is necessary to measure the new traffic by lowering their rates or not react at all. The the link bottleneck bandwidth with some other probing tool, protocol dependent bandwidth is hard to predict from probing such as B-probe. Given the link bandwidth 1 it is possible to unless the probe behaves exactly like the application. A probing rewrite equation 1 such that: m = o(1 - f)/f and from the session may generate a considerable amount of traffic in order equation derive the surplus bandwidth as s = 1 m. to get reliable measurement values. As an example, assume a two link path, link one is a 155 Mbps link, and the link two is a 10 Mbps link, i.e. 11 = 155 111. NETWORK MODEL AND ASSUMPTIONS Mbps and 12 = 10 Mbps. Cross traffic of 152 Mbps merge with We assume proportional sharing of links in our paths. Under probe traffic into the 155 Mbps link, and the 10 Mbps link has that assumption, we will get the following proportional share a cross traffic of 3 Mbps, so ml = 152 Mbps and ma = 3 available bandwidth, on, at the receiver of hop i when the ofMbps. In this configuration the bottleneck surplus bandwidth 86 fered load of the sender is 0, 00 = o and the cross traffic at hop is 3 Mbps (at link 1) and the bottleneck link bandwidth is 10 i is mi: Mbps (at link 2). Assume that we can generate probe traffic at 10 Mbps, i.e. o = 10 Mbps. Sending a probe train with a rate of 10 Mbps will yield 01 = 10/(152 10) x 155 = 9.57 Mbps and then 02 = 9.57/(3 9.57) x 10 = 7.61Mbps. The rate after the last hop, i.e. at the receiver will be f = on. A measurement by e.g. B-probe will detect the link bottleThis means that in a scenario where we successively increase neck bandwidth as 10Mbps. Using the equation m = o(1-f)/f the offered rate 0, we will get the rate f = o until o reaches the surplus bottleneck bandwidth, S b , at some hop i. There- without discrimination then gives: s = 1 - m = 10 - ((10 after the rate at the receiver will correspond to the proportional 7.61)/7.61) x 10 = 6.86 Mbps. C-probe will hence predict a too high surplus value. We call share bandwidth of the link i. While continuing to increase o beyond 8 b we may eventually reach the second lowest surplus this phenomenon the hidden bottleneck, since the surplus bottlebandwidth on the path. Assume that the surplus links appear at neck is not visible to a number of probing tools. Our TOPP method is able to accurately find this 3Mbps "hidden bottlej and at j p , then we will get: neck", see later sections.

v.

THETOPP MEASUREMENT METHOD

To deal with the problems that have been described above we have designed a new method for measuring available bandwidth. One of the ideas in this method is to send trains of packet

416

pairs and for that reason the method goes under the name TOPP (Trains of Packet Pairs). The TOPP measurement method has two separate phases. The first one is the active probing phase where pairs of probe packets are injected into the network. The second phase is the analysis phase, where the bandwidth estimates are calculated based on the reception times of the probe packets.
A. Probing phase

B. Analysis phase

The analysis phase relies on the principle of the bottleneck spacing effect. That is, when two packets with time separation A S are transmitted over a link with a service time f > AS, & then as the packets leave the link they will separated by AR = Q b . Using the size of the packets, b, and the time separation AR, the experienced bandwidth across that link can then be estimated as
f=-

Should Q b 5 S, then R = S, indicating that the link could service (i.e. transmit) the packets at the rate they arrived. The outcome of the probing phase is a series of times stamps for all the received probe packets. Since there are n time stamp pairs for each offered rate we calculated the mean of these n Ib ATp values to get one value. As a result, there will be one ARj value corresponding to every offered rate, oi. value. Now, using Pig. I . Puts of a TOPP packet pair sequence, showing inter- and intra-pair the ARi values, the size of the probe packets, b, and equation 4, spacing. Probe packets of size b bytes are sent as pairs with decreasing can be calculated for each of n offered l intra-pair spacing, i.e. ta > tb > tC. For each intra-pair spacing, t", tb, bandwidth estimates, etc, n pairs are sent. rate levels. Under the assumptions about the network in section 1 1 then 1, for each i-,= 1,... ,121, the relationship between the bandwidth the following way. Starting at some rate omin, n well separated estimate f and the offered bandwidth oi is given by equations 2. ' pairs of equally sized probe packets are sent to the destination Probe packets sent at an offered rate level oi can only make hops host. After those n packets have been sent, the offered rate o with a surplus bandwidth s 5 oi congested. Since the sequence is increased by Ao and another set of n probe packets are sent. of offered rates [ol, . . . onJ]is increasing, several congejtible Then o is increased again (by the same amount Ao) and another hops may be detected by studying the sequence l.: ,f"']. set of n probe packets are sent. This goes on until the offered rate reaches some rate omaz which marks the end of the probe BJ Segmented regression phase, Hence, there will be nl = offered rate Given two or more ( 0 , f)pairs, it is possible to estimate the levels. Figures 2 and 2 illustrate the probe phase.The time sepunknowns I and m in equation 1. A difficulty is that f is not linearly dependent on o so traditional linear regression is not possible. Fortunately,the equation can be converted into a linear 0 0 0 . * form by applying the transformation o/ f . The resulting linear equation is then
IC

One of the aims of the TOPP probing phase is to be network friendly by avoiding to stress the network unnecessarily. This is important for (at least) two reasons; the existing traffic along the probe path will suffer less, and the risk of skewing measurements due to a heavy transient load from the probe traffic is reduced. To achieve these goals the probe traffic is generated in

AR

(4)

p,

[f' .

[omay,@nl

,.

= (1- -j-)
n d

I-m

+ -1o = 1

(1- i)

+ -1o = a + p o 1

(5)

where s is the surplus bandwidth, given by s = 1 - m. Figure 3

f r

Fig. 2. A TOPP probe sequence, showing the stepwise increase in 0. Each black dot corresponds to a pair of packets.

aration between two probe pairs, ATP,is chosen such that the likelihood of more than two probe packets being queued at a node is small. This has the benefit of making the probing phase more network friendly since the nodes along the path will not experience long bursts of probe packets. On the receiving side, the probe packets are time stamped upon reception. Once all packets have been received (or a timeout is exceeded to deal with lost packets), the time stamps are one way. The reason for that is that we want to get an estimate for one direction only and thereby avoid the problem of asymmetric paths.

*
Fig. 3. Plot of o/f a function of 0. Vertices Corresponding to SUrPlUS bandwidth points are indicated.

417

shows a plot of o / f as a function of 0. As can be seen in the of the probe packets have filled an entire Ethernet frame. One figure, the curve is only piece-wise linear. This due to the fact reason for making the probe packets as large as possible (withthat there are more than one congestible hop as explained ear- out causing fragmentation) is that packet size (together with lier. Hence, the true model is actually a segmented linear model. time resolution) set an upper limit on the rate that can be meaAn overview of this topic can be found in (81. In the general sured as dictated by equation 4. Another reason is that previous case, if the breakpoints 7, are be known, then usual linear re- work [ 11, [3] has indicated that large probe packets lead to more gression can be applied to each of the segments. Many times accurate estimates. these breakpoints can be found by simple inspection but there are more elaborate statistical methods for doing this [ 5 ] . Now, assume the ~ i i are known (e.g. by studying a plot simis lar to the one in figure 3). It is then possible to calculate the link Lan 1 r5 ? I and surplus bandwidths for the congestible hop that affected the probe traffic. However, there is one complication, manifested Fig. 4. Network topology used in the ns simulations. by equation 2. The rate dictated by one congested link will be the incoming rate to the next congested hop on-the path. As a result, an o/ f by o plot may be biased since a f i value is plotVII. MEASUREMENTS ted against the offered rate o* when in fact it corresponded to a In this section, we present results from real measurements and (lower) value oi. If the order of the congestible hops is known, then it is possi- ns simulations, together with theoretical reults. All confidence ble to compensate for the dependency of the hops. This is done intervals shown are at the 95 % level. by calculating what the rate of the incoming probe traffic to a hop j must have been and use that 01 value instead of the ini- A. One congestible link tially offered rate 0. The calculation of oil is straightforward as it must be the outgoing rate from hop j - 1. That rate will, in turn, be determined by the outgoing rate from hop j - 2. This recursive unfolding of offered rates can then be traced back to the hop with the lowest surplus bandwidth. But for this hop, the initially offered rates oi can be used to estimate its link bandwidth 1 and surplus bandwidth s = sb. Once those bandwidths have been estimated, it is possible to calculate the 1 and s estimates for the remaining congestible links. This is done by traversing them in the order they appear in the network path, i.e. the reverse of the order that they were unfolded and perform the regression for the corresponding segment using the newly obtained oi rates. Normally we will not know the order of the potentially congested hops. Therefore we make the assumption that the con0 2 4 6 8 10 gested links are in SSF order, and then proceed with the calcuoffered bandwidth [Mbpsl lations in same way as described above.

VI. EXPERIMENTAL SETUP TOstudy the performance of the TOPP method we have made measurements involving three Intel-based hosts running version 2.2 kernels of the Linux operating system. In addition to real world measurements we have performed extensive simulations using the networks simulator 11s. In the ns simulations, the topology in figure 4 has been used. Two lOMbps Ethernet LAN are connected via a WAN consisting of five routers nodes (the shaded circles). The link bandwidths and delays of the WAN links have been varied depending on simulation. To generate the cross traffic in the WAN, a number of nodes are connected to the router nodes. The ten boxes in the figure illustrate these cross traffic nodes. The nodes that generate and receive probe traffic are drawn as pentagrams in the figure. The cross traffic has been generated by traffic generators sending packets with a uniformly distributed random interpacket spacing at some given rate. In all measurements, the size

rm Fig. 5. Measurement results f o a real network with one congestible link, li = 10 Mbps and s i = 5 Mbps.

If we have a path from sender to receiver with only one congestible link, i.e. only one link i where the surplus bandwidth si is lower than our probe ceiling omCz,TOPP is able to detect and estimate the surplus and link bandwidth of this link. Figure 5 shows results from a measurement over a controlled network with one congested link having a link bandwidth l i of 10 Mbps and a surplus bandwidth s i of 5 Mbps, i.e., the cross traffic is 5 Mbps. In the figure, o/f is plotted as a function of the offered bandwidth o. Using our regression analysis method, we can estimate the link bandwidth to 9.19 Mbps and the surplus bandwidth to 4.17 Mbps. Figure 6 shows results from an ns simulation of the same network. For this simulation, regression analysis gives us an estimated link bandwidth of 10.34 Mbps and a surplus bandwidth estimation of 4.91 Mbps.Table I shows results from ns simulations with varying li and si of the congestible link i. Both the

418

congestible links. The column labeled TOPP SSFcontains numbers from SSF analysis as above. The column labeled ALTcontains the alternative results as explained above. For train probing, we present an estimation of s calculated from the measured available bandwidth given that the link bandwidth for the link bottleneck link (IO Mbps in the examples) is known. For two of the scenarios presented in the table, there are no values for TOPP estimations of link q. This is because in these scenarios, TOPP can only establish one bottleneck link, p.As
TABLE I l TOPP ESTIMATIONS OF 1 AND S FOR TWO CONGESTIBLE LINKS,
TOGETHER WITH TRAIN PROBING ESTIMATIONS.
I . "

4 6 6 Offered bandwidth [Mbps]

10

la

Correct
IllSl

121s2

Fig. 6. Measurement results from ns simulations of a network with one congestible link, l , = 10 Mbps and s, = 5 Mbps.

15513 I l0n

TOPP SSF lplSp lqlSq 15513 I lor7

ALT lqlSq

Train
S

10.3n.2

6.86

correct values li and si, and the estimated values ii and Si are shown. As can be seen, the TOPP bandwidth estimations are often quite accurate.
TABLE I TOPP ESTIMATIONS FROM ns SIMULATIONS O F ONE CONGESTIBLELINK
FOR VARYING BANDWIDTHS.

Correct li

10 10

Correct s i 3

Estimated li

Estimated &

10.27 9.89 96.9 96.9 167.3 151.4

2.90 6.29 3.05 6.30 2.25 6.03

can be seen from the figures in the table, for the two first scenarios, TOPP finds bottlenecks that are hidden from train probing. For the next three scenarios, both TOPP and train probing can estimate the lowest surplus bottleneck. For the middle one of these, TOPP is also able to detect the second bottleneck. For the last scenario, train probing results in a notable underestimation. The bottlenecks that TOPP cannot find are obscured because of upstream lower bottlenecks. Note also that when links do not appear in SSF order, TOPP SSF estimates are conservative. Figure 7 shows TOPP measurements from an ns simulation us-

100 155 155

B. Two congestible links

Should two links i and j in the probe path be congestible, both can often be detected and estimated by TOPP. Since TOPP analysis (see the analysis section above) is performed in SSF order, the true order of the links determine the accuracy of TOPP. Should the order be reversed, TOPP analysis in SSF order will introduce a calculation error in the second link. Note however (as already mentioned earlier) that the estimations of the smallest surplus link are always correct, and that the estimations of the other link tend to be conservative, i.e. we underestimate both 1 and s for that link. Also, in the case of two congestible links, we can calculate the two alternative /3 values and then pick the most conservative, i.e. the highest / value (yielding the lowest 3 I and s estimations). We therefore also present the alternative value not presented by TOPP SSF under the heading of ALT (for ALTernative) in the table below. In table 11, we see thoretical figures for TOPP and train probing estimations for a number of combinations of 1 and s for two

0.9

0.8
10

12

Ofleredbandwidth [Mbps]

Fig. 7. Results from a TOPP measurement simulation with two congestible links, see text.

ing I1 = 155 Mbps, S I = 3 Mbps, 12 = 10 Mbps and s2 = 7 Mbps, i.e. the same scenario as the top row in the table above. For this data set, TOPP segmented regression yields the estimations 1, = 147.7 Mbps, S = 2 2 Mbps, iq = 1 . Mbps and 5, = 6.82 , .1 07

419

Mbps. Figure 8 shows TOPP analysis when the links from the

09 I .

8 a Onered bandwidth [Mbpsl

10

12

-.-

6 8 Onered bandwidlh [Mbps]

10

12

Fig. 9. Measurement with three congestible links, see text.

Fig. 8. Results from a TOPP measurement with two congestible links, order reversed compared to previous figure.

VIII. CONCLUSIONS

We have demonstrated the feasibility of our TOPP end-toend probing and analysis method with real measurements and previous example are reversed, i.e. 11 = 10 Mbps, s1 = 7 Mbps, ns simulations. We have defined the terms surplus, proportional 12 = 155 Mbps, and 92 = 3 Mbps. For this order of the links, share, and protocol dependent available bandwidth in terms of TOPP analysis results in the estimations 1, = 148.0 Mbps, 8 = our network model. , 1.82 Mbps, jq = 10.20 Mbps and Sq = 6.70 Mbps. The TOPP method can estimate what we call hidden bottleThe conclusion from experiments with two congestible links necks (when the surplus bottleneck is not on the link bottleneck). are that TOPP finds several bottlenecks that are hidden to train The TOPP regression analysis can not only estimate the surprobing methods, and that TOPP estimations often can be quite plus bottleneck and link bandwidth but also the surplus and link accurate. Also, in the situations where link order assumptions bandwidths of several hops under some assumptions. TOPP is network friendly. It uses trains of packet pairs and are wrong, TOPP SSF estimates tend to be conservative and thus the distance between pairs is variable in order to avoid overunderestimate 8 and 1 bandwidths of the second link. loading the network with bursts of packets and to minimize the impact on cross traffic. C. More than two congestible links The set of experiments and simulations show that TOPP can handle single hops and multiple hops with varying bandwidths The ability to detect more than two congestible links depends and cross traffic. on the characteristics of the links and the order of them. For links in SSF order, the detection ability depends on our ability REFERENCES to detect vertices on the o f f curve. In theory, any number of [I] Mats Bjerkman and Bob Melander. Impact of the Ethernet Capture Effect congestible links can be detected as long as their surplus bandon Bandwidth Measurements. In Networking 2000 Conference Proceedings, pages 156-167, Paris, France, May 2000. widths are all different. [2] Jean-Chrysostome Bolot. End-to-end packet delay and loss behavior in the Should link surplus bandwidths not be in SSF order, the abilintemet. In SIGCOMM 93 Conference Proceedings, pages 289-298, San Francisco, CA, USA, September 13-17. 1993.ACM SIGCOMM Computer ity to detect congestible links depends on the order of the links. The same conditions basically hold as in the two-link scenar- [3] Communication Review, 23(4). Robert L.Carter and Mark E. Crovella. Measuring bottleneck link speed in ios above, only with more links involved, i.e. downstream links packet-switched networks. Technical Report TR-96-006, Boston University Computer Science Department, Boston, MA, USA, March 1996. may be shadowed by limitations in bandwidths of upstream [4] Srinivasan Keshav. A control-theoretic approach to flow control. In SIGlinks. As an example of detection of more than two congestible C O M M 91 Conference Proceedings, pages 3-15, Zirich, Switzerland, links, figure 9 shows the analysis of a TOPP probe session over September 3-6, 1991. ACM SIGCOMM Computer Communication Re14. view, 2 ( ) a simulated network having three congestible links with 11 = 155 [5] Helmut KUchenhoff. An exact algorithm for estimating breakpoints in segMbps and SI = 3 Mbps, 12 = 34 Mbps and 92 = 5 Mbps, and 13 mented generalized linear models. Computational Statistics, 12,pages 235247,1997. = 10 Mbps and 93 = 7 Mbps. In the off graph, we see the three regression lines corresponding to three detected surplus band- [a] Kevin Lni and Mary Baker. Measuring bandwidth. In Proceedings of IEEE INFOCOM 99, New York, USA, March 21-25.1999. widths. From the analysis, we obtain the following estimates: 11 [7] Vern Paxson. End-to-end internet packet dynamics. In SIGCOMM 97 Conference Proceedings, pages 139-152, Cannes, France, September 14= 143.7 Mbps, SI = 2.03 Mbps, 12 = 31.9 Mbps, 52 = 4.02 Mbps, 18,1997.ACM SIGCOMM Computer Communication Review, 2 ( ) 74. 13 = 10.1 Mbps and 93 = 6.37 Mbps. As can be seen, surplus es[8] G. A. E Seber and C. J. Wild Nonlinear regression John Wiley & Sons, timates are accurate within 1 Mbps of the correct bandwidths, New York, 1989. while link bandwidth estimates are accurate within 8 %.

420