Extreme Networks White Paper

Exploring New Data

Center Network
Architectures with
Multi-Switch Link
Aggregation (M-LAG)
Abstract
The broad adoption of virtualization has led to a flurry
of server consolidation projects. IT administrators are
looking to push the envelope when it comes to how
many virtual servers or Virtual Machines (VMs) can be
packed on a single physical server. This is a disruptive
change and impacts traditional network architectures
and best practices in many ways. This white paper examines the challenges and the different architectural
approaches to meet bandwidth, redundancy and
resiliency requirements from the server edge to the
core of the network in a virtualized environment.

Make Your Network Mobile

2011 Extreme Networks, Inc. All rights reserved. Do not reproduce.

Extreme Networks White Paper: Exploring New Data Center Network Architectures with Multi-Switch Link Aggregation (M-LAG)

Introduction
The broad adoption of virtualization has led to a flurry of
server consolidation projects. IT administrators are looking
to push the envelope when it comes to how many virtual
servers or Virtual Machines (VMs as they are commonly
referred to) can be packed on a single physical server. This
is a disruptive change and impacts traditional network
architectures and best practices in many ways. One direct
consequence of higher server virtualization ratios is that
as more VMs are packed on a single server, the bandwidth
demands from the server edge, all the way to the core of
the network, are growing at a rapid pace. Additionally with
more virtual machines on a single server, the redundancy
and resiliency requirements from the server edge to the
core of the network are increasing.
Traditionally, the approach to increasing bandwidth from
the server to the network edge has been to add more Network Interface Cards (NICs) and use Link Aggregation (LAG)
or NIC teaming as it is commonly called to bond links to
achieve higher bandwidth. If any of the links in the group
of aggregated links fails, the traffic load is redistributed
among the remaining links. Link aggregation provides a
simpler and easier way to both increase bandwidth and add
resiliency. Link aggregation is also commonly used between
two switches to increase bandwidth and resiliency. However, in both cases, link aggregation works only between
two individual devices, for example switch to switch, or
server to switch. If any one of the devices on either end of
the link aggregated group (or trunk as it is also called) fails,
then there is complete loss of connectivity.
In order to add device level redundancy various other
mechanisms have been deployed. Where Layer 3 routing
and segmentation is deployed in the network, various
router redundancy protocols such as VRRP, in conjunction
with interior gateway protocols such as OSPF, provide

adequate resiliency, failover and redundancy in the network. However, with virtualization driving the need for flatter Layer 2 topologies (since virtual machine movement
today is typically restricted to within a subnet boundary),
the drive towards a broader flatter Layer 2 data center network is gaining momentum. In this environment, protocols
such as the spanning tree protocol have typically provided
redundancy around both link and device failures. Spanning
tree protocol works by blocking ports on redundant paths
so that all nodes in the network are reachable through a
single path. If a device or a link failure occurs, based on
the spanning tree algorithm, a selective redundant path
or paths are opened up to allow traffic to flow, while still
reducing the topology to a tree structure which prevents
loops. Spanning tree protocol can be used in combination
with link aggregation where links between two nodes
such as switch to switch connections can be aggregated
using link aggregation to increase bandwidth and resiliency
between nodes or devices. Spanning tree would typically
treat the aggregated link as a single logical port in its calculations to come up with a loop free topology.
While spanning tree has served for many years as the de
facto network redundancy protocol, the changing requirements of data center networks today are forcing a
re-examination of the choice of redundancy mechanisms.
For example, one of the drawbacks of spanning tree protocol is that in blocking redundant ports and paths, spanning tree effectively reduces the available bandwidth
significantly, i.e. the bandwidth available on the redundant paths goes unused until a failure occurs. Additionally, in many situations the choice of which ports to block
can also lead to a suboptimal path of communication
between end nodes by forcing traffic to go up and down
the spanning tree. See Figure 1 below. Finally, the time
taken to recompute the spanning tree and propagate the
changes in the event of a failure can vary as well.

Traffic Path

LAG

LAG

LAG

LAG

STP Block

LAG

LAG

STP Block

5563-01

Figure 1

2
2011 Extreme Networks, Inc. All rights reserved.

Extreme Networks White Paper: Exploring New Data Center Network Architectures with Multi-Switch Link Aggregation (M-LAG)

Multi-Switch Link Aggregation

(M-LAG)
A number of new protocols and approaches have been
suggested to address some of the shortcomings of
spanning tree protocol. One approach to addressing
both the performance as well as the resiliency requirements of these highly virtualized data centers is to
extend the link-level redundancy capabilities of link
aggregation and add support for device-level redundancy. This can be accomplished by allowing one end of the
link aggregated port group to be dual-homed into two
different devices to provide device-level redundancy. The
other end of the group is still single homed into a single
device. See Figure 2 below.
In Figure 2, Device 1 treats the link aggregated ports as
a normal link aggregated trunk group, i.e. it does not
see anything different. Traffic from Device 1 is distributed across the ports in the group using traditional link
aggregation algorithms which would typically hash the
traffic across the ports in the group using a variety of
hashing algorithms. If one of the links in the group were
to go down, traffic would get redistributed across the
remaining ports in the group. However, the other end
of link aggregated group is where things now function
differently. Device 2 and Device 3 now work together
to create the perception of a common link aggregated
group so that Device 1 doesnt see anything different
from a link aggregation perspective, even though the

Device 2

Device 3

ISL

M-LAG
(Device 2 & Device 3)

link aggregated ports are now distributed across Device

2 and Device 3, thereby leading to the term Multi-Switch
Link Aggregation (M-LAG). Device 2 and Device 3 communicate information to each other over the Inter Switch
Link (ISL) so that forwarding, learning and bridging work
consistently without causing any loops. The ISL itself can
be a regular LAG. If either the link to Device 1 from Device
2 or Device 3 were to go down, or if Device 2 or Device 3
itself went down, traffic would now get forwarded across
the remaining link/device thus providing both link-level
and device-level redundancy. The intelligence that allows
the ports on Device 2 and Device 3 to present itself as a
single link aggregated trunk group to Device 1 today is
implemented using mostly proprietary mechanisms, i.e.
M-LAG technology is still largely proprietary. However,
the proprietary nature of the technology is confined to
the layer which presents itself as a distributed link
aggregated group, specifically Device 2 and Device 3
in the figure below, both of which should come from
the same vendor. Device 1 does not participate in this
proprietary protocol, and in fact Device 1 may come from
a different vendor and can in fact be a different type of
device. For example, Device 1 can be a server which has dual
NICs teamed together, while Device 2 and Device 3 can be
Ethernet switches from a single vendor. M-LAG can work at
different layers from the access to the core of the network.
M-LAG can be used in conjunction with traditional link
aggregation to increase bandwidth as well as add link-level
redundancy between devices. See Figure 3 below.

Device 2

Device 3

LAG

M-LAG

(Device 2)

(Device 2 & Device 3)

LAG (Device 1)

LAG (Device 1)
Device 1

Device 1

5565-01

5564-01

Figure 2

Figure 3

3
2011 Extreme Networks, Inc. All rights reserved.

Extreme Networks White Paper: Exploring New Data Center Network Architectures with Multi-Switch Link Aggregation (M-LAG)

using both links in an active-active manner. Creating an

M-LAG Direct Attach architecture not only helps to eliminate multiple tiers of switches thereby eliminating multiple
points of oversubscription, latency and power, but it also
adds link, device and network level resiliency to the data
center fabric. And it does this without blocking ports or links
thereby allowing full utilization of the capacity built into the
data center fabric. In effect, an M-LAG Direct Attach architecture provides a very scalable, high-performance, low latency
network fabric for highly virtualized data centers. See Figure
4 below.

Combining M-LAG with the

Direct Attach Architecture
(M-LAG Direct Attach)
M-LAG serves as a powerful mechanism and tool to
address newer architectural requirements centered on
bandwidth and resiliency. Since the proprietary
nature of M-LAG is limited to only the switches providing
the distributed link aggregation capabilities, it can be
combined with other technologies, devices and vendor
equipment to build better network architectures. For
example, M-LAG can be used in conjunction with the
Direct Attach architecture from Extreme Networks to
eliminate tiers and simplify switching in the data center.
The Direct Attach architecture allows virtual machines
to be directly switched in the aggregation or the core
of the physical network, thereby eliminating multiple
switching tiers such as the virtual switch, as well as the
blade switch and the access switch. M-LAG allows dual
homing links from the server into the network and using
both links in an active-active manner. By combining the
two, virtual machines on a single server can be dual-homed
directly into the aggregation or core of the network, while

For more information on Extreme Networks Direct Attach

architecture and how it works to reduce tiers, read the
white paper at:
www.extremenetworks.com/go/DirectAttach.
A key benefit of an M-LAG and M-LAG/Direct Attach
approach is that it can be deployed on existing data center switches using a simple software upgrade, i.e. it does
not require an infrastructure refresh. While M-LAG itself is
proprietary, it works in conjunction with standard proven
link aggregation technology commonly available across
server and switch vendors, as well as in conjunction with
different hypervisor technologies in Direct Attach mode.

96-port 10/100/1000BASE-T, MPU21

96-port 10/100/1000BASE-T, MPU21

M-LAG
Blade Chassis
Blade
Server

LAG

VM

Pass
Through
Module

Pass
Through
Module

VM
5566-01

Figure 4

4
2011 Extreme Networks, Inc. All rights reserved.

Extreme Networks White Paper: Exploring New Data Center Network Architectures with Multi-Switch Link Aggregation (M-LAG)

TRILL and SPB

M-LAG is one of several approaches to building out
modern data center network fabrics. TRILL and Shortest
Path Bridging (SPB) are two other new approaches that
are being positioned as an alternative and replacement to
the spanning tree protocol. TRILL and SPB are competing
proposals being pursued in the IETF and IEEE respectively.
Both TRILL and SPB use link state routing protocols to
compute optimal paths between nodes in the network.
However, unlike traditional link state routing protocols
that operate at the IP or Layer 3 level, both TRILL and SPB
operate at the Layer 2 level. While SPB leverages the IS-IS
link state protocol, TRILL uses a variant of IS-IS. Additionally, both TRILL and SPB use encapsulation mechanisms to
transport packets across the network. TRILL uses a form of
MAC-in-MAC encapsulation while SPB has variants for both
MAC-in-MAC as well as Q-in-Q encapsulation. Both TRILL
and SPB provide for multiple active redundant paths to
effectively fully utilize available bandwidth.
The challenge with both TRILL and SPB is that they are
new protocols that require understanding and expertise
of two new technologies in the data center, IS-IS and
MAC-in-MAC encapsulation. Additionally depending on
the protocol, multicast forwarding can require computing additional multicast trees which can add further
complexity from a troubleshooting and debugging
perspective. Lastly, both TRILL and SPB require new infrastructure due to the encapsulation mechanisms they use
for data forwarding i.e. most existing data center network
infrastructures will not support either TRILL or IS-IS.
Different vendors have expressed support for either
TRILL or SPB leading to some confusion as to the industry
direction for both TRILL and SPB. In the face of this uncertainty around TRILL and SPB, and a lack of broad support
across product lines for either TRILL or SPB, the M-LAG
Direct Attach approach provides a viable alternative
today to deploying a scalable and resilient data center
fabric.

Make Your Network Mobile

Corporate
and North America
Extreme Networks, Inc.
3585 Monroe Street
Santa Clara, CA 95051 USA
Phone +1 408 579 2800

Europe, Middle East, Africa

and South America
Phone +31 30 800 5100

Asia Pacific
Phone +65 6836 5437

Japan
Phone +81 3 5842 4011

extremenetworks.com
2011 Extreme Networks, Inc. All rights reserved. Extreme Networks, the Extreme Networks logo and Direct Attach are either registered trademarks or trademarks of Extreme Networks, Inc. in the United States and/or other countries. Specifications are subject to change without notice. 1750_02 07/11