Diffserv Bandwidth Brokers as Mini-Markets

George Fankhauser

Bernhard Plattner

fgfa, plattnerg@tik.ee.ethz.ch

Computer Engineering and Networks Lab

Swiss Federal Institute of Technology, Z rich, Switzerland
u

Abstract

also expand their geographical scope (MCI/Alternet, for example, has more than 1100 connections to other ASes).

Recent work on Bandwidth Brokers focuses mainly on service provisioning between peer ISPs. Although routing and pricing issues
are also important aspects of a contract between ASes (autonomous
systems) they are usually not treated as an integral part of Bandwidth brokers. Adding these features to a Bandwidth Broker can
transform a simple negotiation into a market-like process. In this
paper we discuss such a form of an advanced Bandwidth Broker
called Service Level Agreement Trader (SLA Trader).
Building on recent studies of temporal and spatial trafc behavior at the inter-AS level we discuss where such SLA Traders are
most benecially placed. Trafc patterns exhibit considerable longterm uctuations and spatial distribution with hot-spots. These variations in demand and the high connectivity between ASes in the
core of the Internet could form the base for a vibrant ISP market.
Keywords: Bandwidth broker, service level agreement, interdomain routing, service provisioning, differentiated services, network resource pricing, mini-market.

Increasing number of application types and protocols:

Many new applications and services require new protocols
or use standard protocols in a different way than originally
intended (e.g., video streaming is often built on top of standard web protocols). They inuence the picture when modeling trafc distributions.
First attempts at service discrimination: Diffserv as a simple, locally deployable service provisioning mechanism is
gaining ground quickly. While the applications are not very
sophisticated right now they are quite useful. For example,
the Swiss academic ISP (SWITCH) is using it to distinguish
two customer classes in servicing and pricing.
Diffserv based solutions, as we discuss them here, focus on the
core network and assume aggregated trafc owing to and from
the access networks. Figure 1 shows the basic division of the network into a core part (backbones, transit networks) and access networks connecting the individual customers. The contracts between
ISPs describing a peering agreement are called SLAs (service level

1 Introduction

agreements). They may be based on SLSs (service level specica-

The Internet is growing in size and structural complexity. While

tions). The access networks form the edge of the network.

many studies have been published on the growth itself, structural

As with traditional routing solutions or service provisioning at

changes to the network are not as well known. They mainly include:

this level, pricing was proposed to be applied in a hierarchy too

Increasing connectivity: Over the last years a steady im-

[Shenker et al. 1996]. While it is efcient to collect money at the

provement in connectivity has been observed between au-

edge of the network, large ISPs at the backbone tree (core) only

tonomous systems (AS) [Faloutsos et al. 1999]. As of June

deal with their peers.

is 4.2 (increasing from 3.4, 3.6,

Diffserv is a framework that has a narrow focus on mechanisms.

3.8 in 11-97, 4-98, 12-98 resp.). If we ignore the 35% leaf

Only the base mechanisms and some basic forwarding types are de-

ASes (only one link to another AS) its even 6.1 [Moat 1999].

ned as a standard [Blake et al. 1998; Nichols et al. 1998; Heina-

The original backbone has turned into a backbone mesh.

nen et al. 1999; Jacobson et al. 1999]. This gives a great degree of

Basic structural changes: More and more backbone

freedom to ISPs how services are provided, with which peers they

providers enter the scene and inter-connect themselves. They

interconnect, how they route trafc and what prices are applied.

1999 the average

out-degree1

Bandwidth Brokers for Diffserv, as rst mentioned in [Nichols

1

out-degree is dened as

2
edges
nodes

et al. 1997], were introduced mainly with service provisioning in

A
DS
Domain A

DS
Domain B

partly of resources that are directly owned by the network provider

DS
Domain C

Access
Network

and partly of resources that were bought by this ISP itself from

Core Network

DS
Domain E

other providers. On selling service, ISPs charge other providers for

DS
Domain F
a

99

DS
Domain D

the service through their network including the outgoing link from

gf

19

its egress nodes to the next AS. For example, consider the four ISPs
A, B, C and E from Figure 1. B may offer services with destination
E to A if B has bought service with destination E itself from C (or,

Figure 1: ISP networks at access and core network level.

alternatively from D).

From this small example we see that bilateral agreements in

mind. However, they can provide a set of useful services:

Automatic SLA negotiation

form of SLAs build up in a nested manner providing nally an endto-end4 service. Cost and delay increase at each ISP (additive met-

Inter-domain path selection (routing)

ric) along the path while the bandwidth metric is concave and stays

Service provisioning

at its minimum. As [Nichols et al. 1997] states, ...[the] observa-

Automatic SLA negotiation is very important and should (as a

tion [is] that multilateral agreements rarely work.... Of course, the

long-term goal) replace todays SLA setup solutions that are pro-

advantage of bilateral agreements comes at the expense of a possi-

cessed by humans. Path selection is interesting for ISPs with alter-

ble service setup delay. However, we can avoid such delays through

native peers for same service and same destinations. Rather than

clever and foresighted contracting. Furthermore, we argue that SLA

relying on shortest path based routing or policy routing, a decision

trading happens at a medium time scale (several minutes to hours)

is made based on available SLAs 2 . SLAs may cover a wide range

and operates on jumbo ows5 .

of levels: for service provisioning everything from best-effort to

virtual leased line (VLL) is possible.

2.1 Service Level Agreements

Assuming that a considerable chunk of the trafc is being used

The denition of an SLA provides a base for heterogeneous trad-

for business purposes with a high service expectation we focus furMoreover, customers ex-

ing systems. [Blake et al. 1998] denes the SLA term as A service

hibit geographical trafc patterns which gives ISPs a good knowl-

contract between a customer and a service provider that species

edge about preferred destination networks.

the forwarding service a customer should receive. A customer may

ther on these higher service standards

3.

Based on these considerations and the fact that resource needs

be a user organization (source domain) or another DS domain (up-

(i.e. trafc at ISP trunks) in the Internet uctuate heavily we argue

stream domain). An SLA may include trafc conditioning rules....

for a cooperative solution of service provisioning, routing and pric-

SLA trading protocols and the traders itself may change from

ing. Integrated service provisioning and routing is also known as

location to location. In our scenario, SLAs include the destination

QoS-routing (see e.g. [Crawley et al. 1998]).

of the jumbo ow to ensure end-to-end service. However, in a re-

We give an outline of the SLA Trading concept in Section 2 and

laxed form, SLAs may also describe services that do not have an

discuss aspects concerning trafc and topologies of this problem in

end-to-end signicance, i.e. provide lower assurance of QoS. In de-

Section 3. Interestingly, this will lead us directly to the problem of

tail, we dene SLAs at each ISP by the following parameters:

nding metrics for routing and service options. Market principles

A trafc description. This includes a PHB and an optional a

apply very well to this problem. We discuss properties of the mini-

QoS-vector (e.g. bandwidth and delay) for a specic trafc

markets (Section 4), give an example of a prototype system that

description. Using a specic parameters instead of a PHB has

uses a cooperative solution for service provisioning, routing and

the advantage of being a universal metric understood by all

pricing (Section 5) and nally draw conclusions in Section 6.

ISPs. It is their obligation to map service requests and offers

existing PHB in their respective domains.

2 Bandwidth Brokers and the Concept of SLA Trading

A geographical scope from the ISPs network to some other

destination network.

Trading SLAs is performed between an ISP and its neighbors. Ba-

Duration of the agreement. All SLAs expire after the time

sically, what ISPs offer to peers are network resources that consist

specied in the contract.

2

This is for example planned as the second phase for the Internet 2 Bandwidth
Brokers [Neilson et al. 1999]
3
This means in practice, we carry best-effort trafc using the same mechanisms as
today: FIFO queuing, shortest path routing and at-rate pricing

4
5

Actually, an AS-to-AS service

Jumbo ows are dened between two ASes [Fang and Peterson 1999]

Cost of the agreement. Local pricing methods and business

strategies may be used to calculate prices for new offers 6 .
2.2 Negotiations between ISPs: SLA Trading
SLA trading is performed by SLA traders situated somewhere in
(a) Internet nodes
and links

the ISPs Diffserv domain. For the time being we assume traders to

(b) ASes as routing domains.

(c) AS topology.

be centralized for each AS. SLA traders make local decisions about
what services are provided to which peers. Such decisions may be

Figure 2: Topology aggregation.

made spontaneously or they are the reaction to an external event.

Initially, SLA traders may offer services to peer ISPs only (i.e.
0 has to be cal-

What is still missing is a valid model for the spatial ow dis-

culated. It is the ISPs business whether to employ a model that

tribution at the inter-domain level. This is why real traces from

always covers its own cost or to decide to implement a long-term

ISPs are often used for simulations. However, traces provide only a

strategy where, e.g. heavy discounting may be used.

small, local snapshot of the bigger picture. A rst attempt at provid-

one inter-domain hop). For such SLAs a price p

Once offers from other partners are received and accepted as an

ing more insight into the spatial inter-AS ow distribution is given

SLA, an ISP may build new services out of the existing ones. The

[Fang and Peterson 1999]. About 90% of the so called jumbo ows

price for such a service is the sum of the SLA price offered by the

happen on only 10% of the possible paths of the inter-domain topol-

peer plus the cost of the ISPs own resource. Or, if all the nesting is

ogy. This observation is based on inter-ISP trafc measurements

uncoiled, the sum of all local prices set by all ISPs involved.

and provides a useful starting point. What it does not provide is a

clue about the temporal behavior of these hot spots (do they move,

Each time an SLA trader wants to construct a new service it

and if so, how fast?).

may compare offers made by all the peers. Usually the best offer,
compared to the tness of the service and the price will be taken.

These spatial and temporal load uctuations are eventually re-

However, this is not a necessity: by adding policy mechanisms to

sponsible for the actions taken by Bandwidth Brokers and represent

peers offers, our purely market-based approach can be distorted

the current demand at each link.

It is well known from studies on QoS routing that a fast re-

by regulation. In general, only policies about the peer ISPs are expressible

sponse to these uctuations can be counter productive or even

7.

worse, it leads to route apping and oscillations.

We see that SLA traders have a great deal of autonomy. It is

possible to follow short- or long-term strategies, to take higher or

For Bandwidth Brokers we will probably observe different

lower risks, and to exploit the ISPs very own business strategy.

problems in this eld. First, the time scale of SLA exchanges is

A detailed description of Diffserv issues, communication protocols

expected to be larger than that of routing updates (

involved in SLA Trading and security considerations are given in

ond, SLAs are destination AS specic which enables us to capture

[Fankhauser et al. 1999].

the demand of the aggregate of long-lived ows to some hot spots

30s). Sec-

of the network (promising results (even at intra-domain level) of

that approach were published in [Shaikh et al. 1999]).

3 Integrated Service Provisioning and Routing:

Trafc and Routing Behavior

3.2 Structure of the Network

3.1 Inter-domain Trafc Behavior

The Internet consists of a hierarchy of hosts grouped by network

A lot of research has been devoted to the characterization of aggre-

prexes. Sets of network prexes are grouped themselves into ASes

gated Internet trafc (e.g. [Paxson 1997b], and many others). From

and there is even the notion of AS blocks. This hierarchy provides

these models fast and plausible synthesis methods for this type of

topology aggregation (see Figure 2) which reduces the overhead of

trafc have been developed.

routing considerably. From the currently about 60000000 hosts

and 60000 network prexes were able to aggregate this to less

than 6000 ASes. Historical and current data on AS topologies is

For the sake of simplicity we assume global currency. In a practical system a

currency converter may be employed.
7
One could also think about an extension to global policies by, e.g. excluding
providers from the path of nested SLAs. Policy attributes would then become part
of the SLA itself and could be propagated to the next provider.

available form [Moat 1999]. However, no link speeds and delays

are available (they can be partially guessed from geographical information and whois database entries).

Class 4 Small metro,

campus, corp. networks
93% of ASes, dv < 7

10000

1000

Class 3 Smaller
regional networks
5% of ASes, 7 < dv < 22

data set
log interpolation

1000

Class 1 National and

international backbones
0.5% of ASes, dv > 79
100
100

# of ASes

Outdegree

Class 2 Large regional and

national networks
1.5% of ASes, 22 < dv < 79

10

10

1
Data Source: moats (12/98)

10

100

1000

0.1

200

400

1999 gfa

(a) AS classication.

600

800

1000

Outdegree

AS index

(b) Log-log scale distribution.

(c) Histogram.

Figure 3: AS connectivity classication and statistics.

In [Fang and Peterson 1999] the authors discuss statistical prop-

ISP resells resources it has bought from one of its peers to

erties of the above-mentioned data set. They nd that ASes can be

another

classied into four classes (Figure 3(a)). Class 1 and 2 networks

ISP asks peer(s) for a specic resource

form the core of the Internet where each AS has at least 20 con-

ISP is being asked for a specic resource

nections to other ASes 8 . Fortunately, this structure provides a high

abstraction level where less than 2% of the ASes are being con-

ISP buys a specic resource from a peer ISP

Following the Diffserv paradigm of local decisions Bandwidth

sidered interesting with respect to market considerations and the

Brokers have to set prices for a resource. As a very simple pric-

large majority counts as ASes with less choice on the bandwidth

ing function we chose the residual bandwidth function for our rst

market.

prototypes (cf. Figure 4).

Still, this amount of domains (about 100 ASes) is considerable

Basically, a Bandwidth Broker becomes a Mini-Market when

when exchanging global connectivity and resource availability in-

its out-degree is at least 3 (local peering partners (non-transit) do

formation. Also, current simulation techniques are not capable of

not count).10 If we look at current topologies about 1800 ISPs have

handling networks this size for reasonable complex problems.

3 and more connections. And as seen earlier, the two classes of

core-ASes count currently for less than 100 ISPs (Figure 3(b) and

4 Mini-Markets

Figure 3(c) show this distribution).

The next relevant parameter is the ISPs size: a very small one

We introduce the term Mini-Market here because of the locality in

might not add very much transit-value to the network. Size is

behavior and local economic system each provider maintains. Basi-

not equal to distance here. Any form of added network value may

cally, Bandwidth Brokers keep track of current demand and supply

count: a short congested link, a difcult-to-reach network con-

of resources, classied by service level and destination. Although

nected by a wireless link, etc.

this information offers global aspects (destination) the contractual

Finally, whats happening at the edge of the network

relationship is limited to the brokers directly reachable peers.

(outdegree

The market scope may include the following actions9 :

ISP offers local resources to one of its peers

3)? Static routing and provisioning still makes sense

in this environment. Collecting money from customers (charging)

is very efcient and maybe some advanced access scheme with ad-

ISP sells local resources to one of its peers

mission control can be applied (e.g. auctions set prices and pick

ISP offers resources it has bought from one of its peers to

eligible customers in congested access networks such as wireless

another

access networks).

Local peering exchanges are excluded from being counted as connectivity.

As we look at backbone providers we do not deal with end-users contracts that
apply to micro-ows.
9

10
Domains of out-degree 2 still have a service option in buying resources but not for
the case of selling or reselling.

1200

and the accompanying conguration of Diffserv boundary

price

nodes is considerably simplied.

Local decision and deployment: Pricing decisions, provisioning, and even routing policy can be handled all locally. Also,
Diffserv Bandwidth Brokers can be tested in limited environsold SLA capacity

ments or they can be operated in parallel with legacy routing

protocols.
Resource allocation for jumbo ows: Inter-AS trafc ows

Figure 4: Residual bandwidth price function.

are high volume and still time variant. SLA trading provides
the framework to optimize these allocations.

5 An Experimental Approach

Besides these advantages there are still many open questions

The discussed principles have been implemented as a prototype

to be answered in ongoing research. Currently we investigate the

in a simulation environment [Schweikert and Fankhauser 1999].

quantitative behavior of SLA traders in the described environment.

A detailed technical discussion and more results are given in

Depending on the degree of non-uniformity of jumbo ows we can

[Fankhauser et al. 1999]. The example in this section demonstrates

increase the efciency of the network by selecting load-sensitive

how simple pricing functions affect the path selection and service

least prices. We consider this trafc property a rather static compo-

provisioning.

nent of the network model. Depending on the trafc ows tempo-

If we congure all 5 ISPs of Figure 5(a) ff equally (same link

ral variance the exchange of trading messages is greatly inuenced.

speed, same price function) the shortest path (2 hops) is also the

With this dynamic component of the model we will be able to de-

cheapest. When demand increases the price for the shortest path in-

ne bounds for the protable operation in function of the trafcs

creases also until the initially unattractive longer path (3 hops) be-

variance and non-uniformity.

comes competitive and is nally selected too. Figure 5(a) ff. show

Protability is expressed in simple network terms like utiliza-

the selected paths, price and bandwidth of such a simple setup. De-

tion. More advanced models with differently priced applications

pending on the demand for destination 4 a price p is minimal for

and network services are a topic on the future research agenda.

the desired ow of bandwidth

u0
1

u0 .
2

What is neglected in theory

References

is a certain granularity of SLAs. Implementations exhibit quantization of demand (cf. Figure 5(e)).

[Blake et al. 1998] B LAKE , S., B LACK , D., C ARLSON , M.,

As a next step we will expand our prototype system to include

DAVIES , E., WANG , Z., AND W EISS , W. An Architecture for

synthesized, aggregated ISP trafc and AS topology data [Paxson

Differentiated Services. RFC 2475. December 1998.

1997a; Moat 1999]. Since an analytical model, even for single com-

[Crawley et al. 1998] C RAWLEY, E., NAIR , R., R AJAGOPALAN ,

ponents of the system, is out of reach, simulations are run on partial

B., AND S ANDICK , H. A Framework for QoS-based Routing

topologies. We will concentrate mainly on subsets with few con-

in the Internet. RFC 2386. August 1998.

nections to the rest of the network (islands) and varying subsets of

[Faloutsos et al. 1999] FALOUTSOS , M., FALOUTSOS , P., AND

the core backbone tree (i.e. the n best connected domains).

FALOUTSOS , C. On Power-Law Relationships of the Internet

Topology. In ACM SIGCOMM 99. September 1999.

6 Conclusion and Outlook

[Fang and Peterson 1999] FANG , W. AND P ETERSON , L. InterAS Trafc Patterns and Their Implications. In Global Internet

While the principle of Bandwidth Brokers with routing capability

99, Rio de Janeiro, Brazil. December 1999.

is simple, its evaluation is complex and can be conducted in an ex-

[Fankhauser et al. 1999] FANKHAUSER , G., S CHWEIKERT, D.,

perimental fashion at best. This approach is in its early stages and

P LATTNER , B. Service Level Agreement Trading for the

mainly used as a proof of concept. We can summarize the advan-

AND

tages of Bandwidth Brokers with integrated SLA trading functions

Differentiated Services Architecture. Swiss Federal Institute of

as follows:

Technology, Computer Engineering and Networks Lab, Technical Report No. 59. November 1999.

Efciency: selecting least-cost links and balancing the load in

[Heinanen et al. 1999] H EINANEN , J., BAKER , F., W EISS , W.,

many links at the same time increases efciency.

AND

Market managed vs. human processed contracts: SLA setup

W ROCLAWSKI , J. Assured Forwarding PHB Group. RFC

2597. June 1999.

5

(a) First, the shortest path is selected.

(b) Then, an alternate path becomes competitive.

(c) Finally, a balanced state is reached.

700000

price

Bandwidth on Link 3->4

Bandwidth on Link 2->4
600000

500000
Bandwidth [bps]

p*

3p

400000

300000

200000

2p

100000

u1

u2

u1

u2

1e+06

(d) Price function for both paths.

1.5e+06
Time [us]

2e+06

2.5e+06

(e) Bandwidth for links 3-4 and 2-4

Figure 5: Example of a competitive situation.

[Jacobson et al. 1999] JACOBSON , V., N ICHOLS , K., AND P O DURI ,

[Paxson 1997a] PAXSON , V.

Trafc. pp. 518. October 1997.

1999.
[Moat 1999] Moat.

[Paxson 1997b] PAXSON , V. Measurements and Analysis of End-

Global ISP Interconnectivity by AS

number. http://moat.nlanr.net/AS/background.

to-End Internet Dynamics. Ph. D. thesis. April 1997.

html. June 1999.

[Schweikert and Fankhauser 1999] S CHWEIKERT,

[Neilson et al. 1999] N EILSON , R., W HEELER , J., R EICH MEYER ,

Fast, Approximate Synthesis of

Fractioanl Gaussian Noise for Generating Self-Similar Network

K. An Expedited Forwarding PHB. RFC 2598. June

F., AND H ARES , S. Eds.

D.

AND

FANKHAUSER , G. owsim, A Flow-based Network Simulator.

http://www.tik.ee.ethz.ch/fs2. 1999.

A Discussion of

Bandwidth Broker Requirements for Internet2 Qbone De-

[Shaikh et al. 1999] S HAIKH , A., R EXFORD , J., AND S HIN ,

ployment. Internet2 Qbone Bandwidth Broker Advisory

K. G. Load-Sensitive Routing of Long-Lived IP Flows. In ACM

Council: http://www.merit.edu/working.groups/

SIGCOMM 99. September 1999.

i2-qbone-bb. Work in progress, August 1999.

[Shenker et al. 1996] S HENKER , S., C LARK , D., E STRIN , D.,

[Nichols et al. 1998] N ICHOLS , K., B LAKE , S., BAKER , F., AND

AND

B LACK , D. Dention of the Differentiated Services Field (DS

the Research Agenda. pp. 1943. April 1996.

Field) in the IPv4 and IPv6 Headers. RFC 2474. December

1998.
[Nichols et al. 1997] N ICHOLS ,

K.,

JACOBSON ,

V.,

H ERZOG , S. Pricing in Computer Networks: Reshaping

AND

Z HANG , L. A two-bit differentiated services architecture for

the internet. http://www-nrg.ee.lbl.gov/papers/
bitarch.pdf. November 1997.