Documente Academic
Documente Profesional
Documente Cultură
53-1004890-04
April 2018
© 2018, Extreme Networks, Inc. All Rights Reserved.
Extreme Networks and the Extreme Networks logo are trademarks or registered trademarks of Extreme Networks, Inc. in the United States and/or other
countries. All other names are the property of their respective owners. For additional information on Extreme Networks Trademarks please see
www.extremenetworks.com/company/legal/trademarks. Specifications and product availability are subject to change without notice.
© 2017, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other
countries. Other brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at www.brocade.com/en/legal/
brocade-Legal-intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,
equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without
notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade
sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the
United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this
document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license
agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and
obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.
Preface ............................................................................................................................................................................................ 7
Extreme Validated Designs....................................................................................................................................................... 7
Purpose of This Document ....................................................................................................................................................... 7
Target Audience ....................................................................................................................................................................... 7
Authors ..................................................................................................................................................................................... 8
Document History ..................................................................................................................................................................... 8
About Extreme Networks .......................................................................................................................................................... 8
Introduction .................................................................................................................................................................................... 9
Terminology .............................................................................................................................................................................. 9
References .................................................................................................................................................................................... 95
Preface
• Extreme Validated Designs. .................................................................................................................................. 7
• Purpose of This Document. ................................................................................................................................... 7
• Target Audience. .................................................................................................................................................... 7
• About the Authors. ................................................................................................................................................. 7
• Document History. ................................................................................................................................................. 8
• About Extreme Networks. ..................................................................................................................................... 8
Target Audience
This document is written for Extreme systems engineers, partners, and customers who design, implement, and support data center
networks. This document is intended for experienced data center architects and engineers. It assumes that the reader has a good
understanding of data center switching and routing features.
Authors
• Krish Padmanabhan
Sr Principal Engineer, System and Solution Engineering
• Eldho Jacob
Principal Engineer, System and Solution Engineering
The authors would like to acknowledge the following for their technical guidance in developing this validated design:
• Abdul Khader
Director, System and Solution Engineering
• Vivek Baveja
Director, Product Management
Document History
Date Part Number Description
December 8, 2016 53-1004890-01 Initial release.
April 21, 2017 53-1004890-02 Included new SLX platforms. Refer to the Hardware Matrix on page 20 for details
on the platforms and their PINs.
January 2018 53-1004890-03 Updated document to reflect Extreme's acquisition of Brocade's data center
networking business.
April 2018 53-1004890-04 Format change
Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity
and cost while enabling virtualization and cloud computing to increase business agility.
To help ensure a complete solution, Extreme Networks partners with world-class IT companies and provides comprehensive education,
support, and professional services offerings. (www.ExtremeNetworks.com)
Introduction
The IP fabric architecture is targeted for large enterprises and data centers planning to migrate from traditional Layer 2 fabrics to a
Layer 3 fabric. This document is about building IP fabrics with Extreme VDX and SLX switching platforms. The SLX series of switches
and routers are Extreme’s next-generation platforms addressing the scale requirements of MSDC customers.
The configurations and design practices documented here are fully validated and conform to the IP fabric reference
architectures. The intention of this Extreme Validated Design document is to provide reference configurations and document best
practices for building cloud-scale data-center networks using VDX and SLX switches and IP fabric architectures.
It should be noted that this document does not cover the network virtualization or Layer 2/Layer 3 multitenancy aspects of IP fabric.
Network virtualization in IP fabric is covered in the Network Virtualization in IP Fabric with BGP EVPN1 Extreme Validated Design
document.
Terminology
Term Description
ARP Address Resolution Protocol
AS Autonomous System
ASN Autonomous System Number
BGP Border Gateway Protocol
eBGP External Border Gateway Protocol
ECMP Equal Cost Multi-Path
iBGP Internal Border Gateway Protocol
IP Internet Protocol
MCT Multi-Chassis Trunk
ND Neighbor Discovery
NLRI Network Layer Reachability Information
ORF Outbound Route Filtering
PoD Point of Delivery
ToR Top of Rack switch
URIB Unicast Route Information Base
vLAG Virtual Link Aggregation Group
VLAN Virtual Local Area Network
VM Virtual Machine
IP Fabric Overview
• Evolution of Data Center Fabrics. .......................................................................................................................11
• Leaf-Spine Architecture. ......................................................................................................................................15
• IP Fabric Control Plane. ........................................................................................................................................ 17
Extreme IP fabric provides a Layer 3 Clos deployment architecture for data center sites. It is a paradigm shift from traditional Layer 2
switching fabrics. With Extreme IP fabric, all links in the Clos topology are Layer 3 links.
In comparison with the traditional 2-tier access/aggregation topologies and L2 fabrics where the L2/L3 demarcation happens on a
device typically more than a hop away, the L2/L3 boundary in IP fabric is pushed to the ToR or the leaf node itself (AKA routing to the
ToR). Leafs advertise the server subnets attached to them directly into the routing control-plane protocol. Modern data centers zeroed in
on BGP as the preferred control-plane protocol. Because the infrastructure is built on IP, advantages like the following are leveraged:
loop-free communication using industry-standard routing protocols, ECMP, a very high solution scale, and standards-based
interoperability.
FIGURE 1 L2 Aggregation
This model provides L2 extension naturally between racks. But there are trade-offs. Most of the network functions are concentrated on
the aggregation pair. The number of ToRs, VLANs, MAC entries, IP subnets, ARP/ND entries, route scale, and so on is determined by
these two devices for the entire fabric. The scale of the network and diameter (or number of racks) is limited by these two devices for the
entire fabric. The VLANs or broadcast domains must be pruned properly according to the membership or interest in various racks to
avoid large broadcast domains.
FIGURE 2 L2 Fabric
Leaf-Spine Architecture
Leaf-Spine Layer 3 Clos Topology (Two-Tier)
The leaf-spine topology has become the de facto standard for networking topologies when building medium- to large-scale data center
infrastructures. The leaf-spine topology is adapted from Clos telecommunications networks. The Extreme IP fabric within a PoD
resembles a two-tier or 3-stage folded Clos fabric. The two-tier leaf-spine topology is shown in Figure 4. The bottom layer of the IP
fabric has leaf devices (top-of-rack switches), and the top layer has spines. The role of the leaf is to provide connectivity to the endpoints
in the data center network. These endpoints include compute servers and storage devices, as well as other networking devices like
routers, switches, load balancers, firewalls, and any other physical or virtual networking endpoints. Because all endpoints connect only to
the leaf, policy enforcement, including security, traffic-path selection, QoS marking, traffic policing, and shaping, is implemented at the
leaf.
The leafs act as the L2/L3 boundary for the server segments in an IP fabric.
The role of the spine is to provide connectivity between leafs. The major role of the spine is to participate in the control-plane and data-
plane operations for traffic forwarding between leafs.
This type of topology has the predictable latency and also provides the ECMP forwarding in the underlay network. The number of hops
between two leaf devices is always two within the fabric. This topology also enables easier scale-out in the horizontal direction as the data
center expands and is limited by the port density and bandwidth supported by the spine devices.
This validated design recommends the same hardware in the spine layer. Mixing different hardware is not recommended.
In this validated design, we recommend eBGP as the control plane in the fabric. IP fabric is also referred to as routing to ToR in
comparison with the traditional multitier access and aggregation networks. In those traditional networks, the L2/L3 boundary is on the
aggregation devices. In IP fabric, this boundary is moved to the edge or leaf. The leaf nodes directly advertise their server subnets into
the BGP control plane. Reachability between the server subnets on various racks is established using control-plane learning.
Moving the intelligence to the leaf helps with scaling out the number of spines depending on the bandwidth or oversubscription issues in
the network. For instance, the ratio between the bandwidth of the server ports and the uplinks ports (from leaf to spine layer). If the
oversubscription is too high, additional spines may be added.
Pervasive eBGP
This deployment model refers to the usage of eBGP peering between the leaf and the spine in the fabric. This design using eBGP as a
routing protocol within the data center is based on the IETF draft Use of BGP for Routing in Large-Scale Data Centers2.
In this model, each leaf node is assigned its own autonomous system (AS) number. The other nodes are grouped based on their role in
the fabric, and each of these groups is assigned a separate AS number, as shown in Figure 6. Using eBGP in an IP fabric is simple and
also provides the ability to apply BGP policies for traffic engineering on a per-leaf or per-rack basis since each leaf or rack in a PoD is
assigned a unique AS number. Private AS numbers are used in the fabric. One design consideration for the AS number assignment is that
a 2-byte AS number provides a maximum of 1023 private AS numbers (ASN 64512 to ASN 65534); if the IP fabric is larger than 1023
devices, we recommend using 4-byte private AS numbers (ASN 4,200,000,000 to 4,294,967,294).
In this deployment model, each PoD and edge services PoD is configured with a unique AS number, as shown in Figure 7. The spines
and leafs in a PoD are configured with the same AS number. The iBGP design is different than the eBGP design because iBGP must be
fully meshed with all BGP-enabled devices in an IP fabric. In order to avoid the complexities of a full mesh, spines must act as route
reflectors toward the leaf nodes inside the PoD.
eBGP is used to peer between spines and super-spines. The super-spine layer is configured with a unique AS number; all super-spines
use the same AS number.
Pervasive eBGP
Figure 8 shows the design for a 5-stage IP fabric using eBGP as the control protocol. Note that the border leafs are connected to the
super-spines in this design. For small topologies, a 3-stage fabric may be sufficient. As shown in Figure 8, each POD is a 3-stage fabric.
For 3-stage fabrics, the border leafs are directly connected to the spines.
Hardware Matrix
TABLE 1 Extreme Switch Platforms Supported in IP Fabric
Places in the Network Platform Minimum Software Version
Leaf Nodes SLX 9140 SLX-OS 17s.1.02
VDX 6740 Network OS 7.2.0a
IP Fabric Configuration
This section covers the provisioning and validation of the IP fabric network topology. The configuration is given in four parts:
4. The BGP control plane, which is split into two models: 3-stage and 5-stage fabrics.
All VDX platforms are set to a VCS ID of 1 by default. Two nodes having a common VCS ID will form a VCS fabric between them. Since
these nodes will be independent in the IP fabric, we need to ensure that they do not form a VCS fabric between them. This is achieved by
configuring a unique VCS ID on each node.
In the validated design, each VDX node is configured with a unique VCS ID. The RBridge ID may be re-used. We recommend using
RBridge ID 1 for individual leafs and RBridge IDs 1 and 2 for the vLAG pair.
The vLAG pair leaf is assigned its own unique VCS ID, and each node in the vLAG pair has a separate RBridge ID. For example, in the
validated design, Leaf1 is a 2-node vLAG pair.
vLAG Peer 1
vLAG Peer 2
POD1-Leaf1-2# vcs vcsid 405 set-rbridge-id 2
In the following output, RBridge 2 is the principal switch. All configuration for both nodes in the vLAG leaf can be done from this principal
switch.
VDX Platforms
SLX Platforms
interface Ethernet 3/2
mtu 9216
ip mtu 9100
ip proxy-arp
ip address 10.2.1.0/31
ipv6 address fdf8:10:2:1::/127
ipv6 mtu 9100
no shutdown
!
rbridge-id 1
interface Loopback 2
no shutdown
ipv6 address fdf8:10:122:1::2/128
ip address 10.122.1.2/32
!
ip router-id 10.122.1.2
!
rbridge-id 2
interface Loopback 2
no shutdown
ipv6 address fdf8:10:122:1::3/128
ip address 10.122.1.3/32
!
ip router-id 10.122.1.2
!
rbridge-id 1
interface Loopback 2
no shutdown
ipv6 address fdf8:10:122:5::2/128
ip address 10.122.5.2/32
!
ip router-id 10.122.5.2
!
SLX Platforms
SLX platforms do not need an RBridge configuration as do the VDX platforms.
interface Loopback 2
ipv6 address fdf8:10:125:1::2/128
ip address 10.125.1.2/32
no shutdown
!
ip router-id 10.125.1.2
rbridge-id 1
interface Ve 101
ipv6 address fdf8:10:5:101::1/64
ipv6 mtu 9000
ip mtu 9000
ip address 10.5.101.1/24
no shutdown
!
!
rbridge-id 1 rbridge-id 2
interface Ve 101 interface Ve 101
ipv6 address fdf8:10:1:101::1/64 ipv6 address fdf8:10:1:101::2/64
ipv6 mtu 9000 ipv6 mtu 9000
ipv6 vrrp-extended-group 100 ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:101::254 virtual-ip fdf8:10:1:101::254
enable enable
preempt-mode preempt-mode
priority 150 priority 140
advertisement-interval 1 advertisement-interval 1
short-path-forwarding short-path-forwarding
! !
ip mtu 9000 ip mtu 9000
ip proxy-arp ip proxy-arp
ip address 10.1.101.1/24 ip address 10.1.101.2/24
vrrp-extended-group 10 vrrp-extended-group 10
virtual-ip 10.1.101.254 virtual-ip 10.1.101.254
advertisement-interval 1 advertisement-interval 1
enable enable
preempt-mode preempt-mode
priority 150 priority 140
short-path-forwarding short-path-forwarding
! !
no shutdown no shutdown
! !
! !
1
If there are L2 switches or bridges between the leaf and servers, spanning tree must be enabled. If there is a possibility of enabling bridges inadvertently
under the leaf nodes, we recommend enabling spanning tree and configuring the server ports as edge ports.
Rbridge 1 Rbridge 2
vLAG
interface Port-channel 1
vlag ignore-split
mtu 9022
switchport
switchport mode trunk
switchport trunk allowed vlan add 101-110
switchport trunk tag native-vlan
spanning-tree shutdown
no shutdown
!
interface TenGigabitEthernet 1/0/9
channel-group 1 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/9
channel-group 1 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
vlan 76
router-interface Ve 76
!
vlan 77
router-interface Ve 77
!
interface Ve 76
ipv6 address fdf8:10:0:4c::1/64
ipv6 mtu 9000
ip mtu 9000
ip address 10.0.76.1/24
no shutdown
!
interface Ve 80 interface Ve 80
ip mtu 9000 ip mtu 9000
ip address 10.0.80.1/24 ip address 10.0.80.2/24
ipv6 address fdf8:10:0:50::2/96 ipv6 address fdf8:10:0:50::2/96
ipv6 mtu 9000 ipv6 mtu 9000
ipv6 vrrp-extended-group 2 ipv6 vrrp-extended-group 2
virtual-ip fdf8:10:0:50::254 virtual-ip fdf8:10:0:50::254
enable enable
no preempt-mode no preempt-mode
short-path-forwarding short-path-forwarding
! !
vrrp-extended-group 1 vrrp-extended-group 1
virtual-ip 10.0.80.254 virtual-ip 10.0.80.254
enable enable
no preempt-mode no preempt-mode
short-path-forwarding short-path-forwarding
! !
no shutdown no shutdown
! !
As shown in the illustration below, the MCT pair is interconnected by two 40-Gbps Ethernet ports for ICL.
Cluster Configuration
In the configuration shown below, port channel 101 is configured as a multi-chassis LAG port channel. In the configuration shown—
“client Server1 1”—identifier 1 on both peers designates the client port channel as a dual-homed LAG. This identifier must match on
both peers.
interface Ethernet 0/49 interface Ethernet 0/49
speed 40000 speed 40000
channel-group 10 mode active type standard channel-group 10 mode active type standard
no shutdown no shutdown
! !
interface Ethernet 0/50 interface Ethernet 0/50
speed 40000 speed 40000
channel-group 10 mode active type standard channel-group 10 mode active type standard
no shutdown no shutdown
! !
interface Port-channel 10 interface Port-channel 10
speed 40000 speed 40000
mtu 9216 mtu 9216
no shutdown no shutdown
! E 0/49 !
MCT Peer1 MCT Peer2
E 0/50
E 0/1
E 0/1
mLAG
(The configuration templates shown below are applicable to both edge leafs in the MCT cluster. Where there are differences, the
configuration blocks are shown side-by-side.)
VLAN/VE/VRRP Configuration
BGP Configuration
router bgp Originate default route into the fabric.
address-family ipv4 unicast vrf DC-VRF
graceful-restart
default-information-originate
!
address-family ipv6 unicast vrf DC-VRF
graceful-restart
default-information-originate
!
There are a few minor differences in the MCT configuration for the SLX 9540 platform compared to the SLX 9140 platforms used as
leafs.
vlan 4 vlan 4
router-interface Ve 4 router-interface Ve 4
! !
interface Ve 4 interface Ve 4
ip mtu 9100 ip mtu 9100
ip address 10.55.56.0/31 ip address 10.55.56.1/31
ipv6 mtu 9100 ipv6 mtu 9100
no shutdown no shutdown
! !
interface Loopback 1 interface Loopback 1
ip address 10.55.55.1/32 ip address 10.56.56.1/32
no shutdown no shutdown
! !
ip router-id 10.55.55.1 ip router-id 10.56.56.1
ip route 10.56.56.1/32 10.55.56.1 ip route 10.55.55.1/32 10.55.56.0
! !
cluster Edge-cluster 1 cluster Edge-cluster 1
member vlan add 2-3 member vlan add 2-3
peer-interface Ve 4 peer-interface Ve 4
peer 10.56.56.1 peer 10.55.55.1
client-isolation-strict client-isolation-strict
deploy deploy
client 6720 1 client 6720 1
client-interface Port-channel 2 client-interface Port-channel 2
esi 0:0:0:0:0:0:0:1:1 esi 0:0:0:0:0:0:0:1:1
deploy deploy
! !
interface Port-channel 2 interface Port-channel 2
mtu 9216 mtu 9216
switchport switchport
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan add 2-3 switchport trunk allowed vlan add 2-3
no shutdown no shutdown
! !
interface Ethernet 0/9 interface Ethernet 0/9
channel-group 2 mode active type standard channel-group 2 mode active type standard
no shutdown no shutdown
! !
router bgp router bgp
local-as 4200007000 local-as 4200007000
capability as4-enable capability as4-enable
fast-external-fallover fast-external-fallover
neighbor 10.56.56.1 remote-as 4200007000 neighbor 10.55.55.1 remote-as 4200007000
neighbor 10.56.56.1 update-source loopback 1 neighbor 10.55.55.1 update-source loopback 1
neighbor 10.56.56.1 bfd neighbor 10.55.55.1 bfd
Spine Configuration
All spines within a PoD have a similar configuration. Peer groups are used to simplify the configurations and also for efficiency in BGP
update processing. In a 5-stage fabric, spines are connected to leafs inside their PoD and to super-spines.
• Configure the directly connected leaf IP addresses in one peer group called leaf-group.
• Configure the directly connected leaf IPv6 addresses in a peer group called leaf-group-ipv6.
• Configure the directly connected super-spine IP addresses in a peer group called superspine-group.
• Configure the directly connected super-spine IPv6 addresses into another peer group called superspine-group-ipv6.
2 Exchanging both IPv4 and IPv6 AFI over a single BGP peering session is not supported. Support is planned in the upcoming releases of NOS
and SLX-OS.
Each spine should establish IPv4 and IPv6 peering with all leafs inside the PoD and super-spines. (Note that the leaf nodes in a vLAG
pair share one common AS number between them, and super-spines belong to one AS number.)
Leaf Configuration
All leafs within a PoD have a similar configuration. Peer groups are used to simplify the configuration and also for efficiency in BGP
update processing. Leafs are connected to spines only.
• Configure the directly connected IP addresses of the spines into a peer group spine-group.
• Configure the directly connected IPv6 addresses of the spines into a peer group spine-group-ipv6.
• Advertise both IPv4 and IPv6 server subnets. Using a route map, filter the subnets of fabric links from being advertised into
BGP and allow only the server subnets and loopback IP address. Advertising loopback IP and IPv6 address helps debug
routing and node reachability issues. A sample route map is given below. This can be modified according to the deployment
requirements.
IPv4 Route-Map Configuration
ip prefix-list fabric_links_ip seq 10 permit 0.0.0.0/0 ge 31 le 31
route-map ToR-map deny 10
match ip address prefix-list fabric_links_ip
!
route-map ToR-map permit 20
!
!POD1-leaf1-1
router bgp
local-as 4259840001
capability as4-enable
fast-external-fallover
!
neighbor spine-group peer-group
neighbor spine-group remote-as 4259905537 Peer-group config grouping spines’ IPv4
neighbor spine-group description To spine addresses.
neighbor spine-group password $9$MCgKGaNt6OASX68/7TC6Lw== Enable MD5 authentication and BFD
neighbor spine-group bfd
!
neighbor 10.0.1.1 peer-group spine-group
neighbor 10.0.1.3 peer-group spine-group
neighbor 10.0.1.5 peer-group spine-group
neighbor 10.0.1.7 peer-group spine-group
!
neighbor spine-group-ipv6 peer-group
neighbor spine-group-ipv6 remote-as 4259905537 Peer-group config grouping spines’ IPv6
neighbor spine-group-ipv6 description To spine AS-65001.1 addresses.
neighbor spine-group-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw== Enable MD5 authentication and BFD
neighbor spine-group-ipv6 bfd
!
neighbor fdf8:10:0:1:1::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:1:2::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:1:3::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:1:4::2 peer-group spine-group-ipv6 Enable IPv4 Address-Family
!
Redistribute connected to advertise the
address-family ipv4 unicast
redistribute connected route-map ToR-map-ip VLAN subnets and /32 loopback
neighbor spine-group enable-peer-as-check addresses.
neighbor 10.0.1.11 route-map out BGP-med Enable graceful-restart
maximum-paths 8
graceful-restart Enable IPv6 Address-Family.
! Activate the peer-groups configured for
address-family ipv6 unicast IPv6 route exchange.
redistribute connected route-map ToR-map-ipv6 Advertise /128 loopback IPv6 address and
neighbor spine-group-ipv6 activate Host VLAN IPv6 subnets
neighbor spine-group-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
Super-Spine Configuration
This is applicable to all super-spines. Peer groups are used to simplify the configuration. Super-spines peer with the spines in each PoD
and with the border leafs. In the validated design, the super-spines connect to two PoDs. The configuration may be replicated for multiple
PoDs.
• Super-spines connect to the spines in each PoD and to the border leafs using fabric links.
• Create a peer group for each PoD for IPv4 peering, and exchange IPv4 routes:
– pod1_spine-group—Add the directly connected neighbor IP addresses of all spines in PoD1 to this group.
– pod2_spine-group—Add the directly connected neighbor IP addresses of all spines in PoD2 to this group.
• Create a peer group for each PoD for IPv6 peering, and exchange IPv6 routes:
– pod1_spine-group-ipv6—Add the directly connected neighbor IPv6 addresses of all spines in PoD1 to this group.
– pod2_spine-group-ipv6—Add the directly connected neighbor IPv6 addresses of all spines in PoD2 to this group.
• Create a separate peer group for the pair of edge leafs: edge-group and edge-group-ipv6. Add the directly connected neighbor
IPv4 and IPv6 addresses of the edge leafs to these groups.
• Enable MD5 authentication to all peer groups.
• Enable BFD on all peer groups.
router bgp
local-as 4200000010
capability as4-enable
fast-external-fallover
!
neighbor edge-group peer-group Peer-group config pointing to
neighbor edge-group remote-as 4200007000 edge leafs
neighbor edge-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor edge-group bfd
!
neighbor 10.2.1.17 peer-group edge-group
neighbor 10.2.1.19 peer-group edge-group
!
neighbor edge-group-ipv6 peer-group
neighbor edge-group-ipv6 remote-as 4200007000
neighbor edge-group-ipv6 password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor edge-group-ipv6 bfd
!
neighbor fdf8:10:2:1::17 peer-group edge-group-ipv6 Peer-group config pointing to spines in
neighbor fdf8:10:2:1::19 peer-group edge-group-ipv6 PoD1
!
neighbor pod1-spine-group peer-group
neighbor pod1-spine-group remote-as 4259905537
neighbor pod1-spine-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod1-spine-group bfd
!
neighbor 10.0.51.1 peer-group pod1-spine-group
neighbor 10.0.52.1 peer-group pod1-spine-group
neighbor 10.0.53.1 peer-group pod1-spine-group
neighbor 10.0.54.1 peer-group pod1-spine-group
!
neighbor pod1-spine-group-ipv6 peer-group
neighbor pod1-spine-group-ipv6 remote-as 4259905537
neighbor pod1-spine-group-ipv6 password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod1-spine-group-ipv6 bfd
!
neighbor fdf8:10:0:51:1::2 peer-group pod1-spine-group-ipv6
neighbor fdf8:10:0:52:1::2 peer-group pod1-spine-group-ipv6
neighbor fdf8:10:0:53:1::2 peer-group pod1-spine-group-ipv6
neighbor fdf8:10:0:54:1::2 peer-group pod1-spine-group-ipv6
! Peer-group config pointing to Spines in
neighbor pod2-spine-group peer-group PoD2
neighbor pod2-spine-group remote-as 4200002000
neighbor pod2-spine-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod2-spine-group bfd
!
neighbor 10.2.1.9 peer-group pod2-spine-group
neighbor 10.2.1.11 peer-group pod2-spine-group
neighbor 10.2.1.13 peer-group pod2-spine-group
neighbor 10.2.1.15 peer-group pod2-spine-group
!
neighbor pod2-spine-group-ipv6 peer-group
neighbor pod2-spine-group-ipv6 remote-as 4200002000
neighbor pod2-spine-group-ipv6 password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod2-spine-group-ipv6 bfd
!
neighbor fdf8:10:2:1::9 peer-group pod2-spine-group-ipv6
neighbor fdf8:10:2:1::11 peer-group pod2-spine-group-ipv6 Enable ipv4 Address-Family
neighbor fdf8:10:2:1::13 peer-group pod2-spine-group-ipv6 Advertise loopback interface’s IPv4
neighbor fdf8:10:2:1::15 peer-group pod2-spine-group-ipv6 address.
! Enable graceful-restart.
address-family ipv4 unicast
network 10.125.1.1/32
neighbor edge-group enable-peer-as-check
neighbor pod2-spine-group enable-peer-as-check
neighbor pod1-spine-group enable-peer-as-check Enable ipv6 Address-Family
maximum-paths 8
Advertise IPv6 address of the loopback
graceful-restart
interface.
!
address-family ipv6 unicast Activate the peer-groups configured for
network fdf8::10:124:1:1/128 IPv6 routes exchange.
neighbor edge-group-ipv6 activate Enable graceful-restart
neighbor edge-group-ipv6 enable-peer-as-check
neighbor pod2-spine-group-ipv6 activate
neighbor pod2-spine-group-ipv6 enable-peer-as-check
neighbor pod1-spine-group-ipv6 activate
neighbor pod1-spine-group-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
40 ! Extreme IP Fabric Architecture
IP Fabric Validated Designs
Note that the super-spine peer groups are in the global VRF in the configuration below. They may be configured under a separate VRF
as discussed in MCT on SLX 9540 Edge Leafs on page 30.
!edge-leaf1
router bgp
local-as 4200007000
capability as4-enable
!
neighbor super-spine peer-group
neighbor super-spine remote-as 4200000010
neighbor super-spine description IPv4 peering to super-spines Peer-groups config pointing to
neighbor super-spine password 2 $MlVzZCFAbg== super-spines
neighbor super-spine bfd
!
neighbor 10.2.1.16 peer-group super-spine
neighbor 10.3.1.16 peer-group super-spine
!
neighbor super-spine-ipv6 peer-group
neighbor super-spine-ipv6 remote-as 4200000010
neighbor super-spine-ipv6 description IPv6 peering to super-spines
neighbor super-spine-ipv6 password 2 $MlVzZCFAbg==
neighbor super-spine-ipv6 bfd
!
neighbor fdf8:10:2:1::16 peer-group super-spine-ipv6
neighbor fdf8:10:3:1::16 peer-group super-spine-ipv6
! Peer-groups config pointing to
neighbor wan-group peer-group WAN-edges
neighbor wan-group remote-as 4200007001
neighbor wan-group description IPv4 peering to WAN Edge
!
neighbor 192.168.1.2 peer-group wan-group
neighbor 192.168.2.2 peer-group wan-group
!
neighbor wan-group-ipv6 peer-group
neighbor wan-group-ipv6 remote-as 420007001
neighbor wan-group-ipv6 description IPv6 peering to WAN Edge
!
neighbor fdf8:192:168:1::2 peer-group wan-group-ipv6
neighbor fdf8:192:168:1::2 peer-group wan-group-ipv6
! Enable both IPv4 and IPv6
address-family ipv4 unicast Address-Families
network 10.123.3.1/32
neighbor wan-group enable-peer-as-check Activate wan-group and super-
neighbor super-spine enable-peer-as-check spine groups for IPv6 route
maximum-paths 8 exchange
graceful-restart
!
address-family ipv6 unicast
neighbor wan-group-ipv6 activate
neighbor wan-group-ipv6 enable-peer-as-check
neighbor super-spine-ipv6 activate
neighbor super-spine-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
Spine Configuration
All spines within a PoD have a similar configuration for IPv4 underlay. In a 3-stage fabric, spines peer with leafs and border leafs. Peer
groups are used to simplify the configurations and also for efficiency in BGP update processing.
• Configure the directly connected leafs' IPv4 addresses into one peer group: leaf-group.
• Configure the directly connected leafs' IPv6 addresses into one peer group: leaf-group-ipv6.
• Configure the edge leafs IPv4 fabric link addresses into one peer group: edge-group.
• Configure the edge leafs IPv4 fabric link addresses into one peer group: edge-group-ipv6.
3
Exchanging both IPv4 and IPv6 AFI over a single BGP peering session is not supported. It is planned in the upcoming releases of NOS and
SLX-OS.
!POD1-leaf1-1
router bgp
local-as 4200001001
capability as4-enable
!
neighbor spine-group peer-group
neighbor spine-group remote-as 4200001000
neighbor spine-group description connected to 4 spines Peer-group config grouping spines’ IPv4
neighbor spine-group password 2 $PVNHITJVPWQ= addresses.
! Enable MD5 authentication
neighbor 10.0.1.0 peer-group spine-group
neighbor 10.0.2.0 peer-group spine-group
neighbor 10.0.3.0 peer-group spine-group
neighbor 10.0.4.0 peer-group spine-group
!
neighbor spine-group-ipv6 peer-group
neighbor spine-group-ipv6 remote-as 4200001000
neighbor spine-group-ipv6 description connected to 4 spines
neighbor spine-group-ipv6 password 2 $MlVzZCFAbg== Peer-group config grouping spines’ IPv6
! addresses.
neighbor fdf8:10:0:1:: peer-group spine-group-ipv6 Enable MD5 authentication
neighbor fdf8:10:0:2:: peer-group spine-group-ipv6
neighbor fdf8:10:0:3:: peer-group spine-group-ipv6
neighbor fdf8:10:0:4:: peer-group spine-group-ipv6
!
address-family ipv4 unicast
redistribute connected route-map ToR-map Enable IPv4 Address-Family
neighbor spine-group enable-peer-as-check
Redistribute connected to advertise the
maximum-paths 8
graceful-restart VLAN subnets
! Enable graceful-restart
address-family ipv6 unicast
redistribute connected route-map ToR-map-ipv6 Repeat the same for IPv6 Address-Family
neighbor spine-group-ipv6 activate
neighbor spine-group-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
!
Note that the spine peer groups are in the global VRF in the configuration below. They may be configured under a separate VRF as
discussed in MCT on SLX 9540 Edge Leafs on page 30.
router bgp
local-as 4200007000
capability as4-enable
!
neighbor super-spine peer-group
neighbor super-spine remote-as 4200000010
neighbor super-spine description IPv4 peering to super-spines
neighbor super-spine password 2 $MlVzZCFAbg== Peer-groups config pointing to
neighbor super-spine bfd super-spines
!
neighbor 10.2.1.16 peer-group super-spine
neighbor 10.3.1.16 peer-group super-spine
!
neighbor super-spine-ipv6 peer-group
neighbor super-spine-ipv6 remote-as 4200000010
neighbor super-spine-ipv6 description IPv6 peering to super-spines
neighbor super-spine-ipv6 password 2 $MlVzZCFAbg==
neighbor super-spine-ipv6 bfd
!
neighbor fdf8:10:2:1::16 peer-group super-spine-ipv6
neighbor fdf8:10:3:1::16 peer-group super-spine-ipv6
!
neighbor wan-group peer-group Peer-groups config pointing to
neighbor wan-group remote-as 4200007001 WAN-edges
neighbor wan-group description IPv4 peering to WAN Edge
!
neighbor 192.168.1.2 peer-group wan-group
neighbor 192.168.2.2 peer-group wan-group
!
neighbor wan-group-ipv6 peer-group
neighbor wan-group-ipv6 remote-as 420007001
neighbor wan-group-ipv6 description IPv6 peering to WAN Edge
!
neighbor fdf8:192:168:1::2 peer-group wan-group-ipv6
neighbor fdf8:192:168:1::2 peer-group wan-group-ipv6
!
address-family ipv4 unicast Enable both IPv4 and IPv6
neighbor super-spine enable-peer-as-check Address-Families
neighbor wan-group enable-peer-as-check
maximum-paths 8 Activate wan-group and super-
graceful-restart spine groups for IPv6 route
! exchange
address-family ipv6 unicast
neighbor wan-group-ipv6 activate
neighbor wan-group-ipv6 enable-peer-as-check
neighbor super-spine-ipv6 activate
neighbor super-spine-ipv6 enable-peer-as-check
maximum-paths 8
graceful-restart
!
Illustration Examples
In this section, we illustrate the use cases using sections of the validated design network topology as appropriate. This will help the reader
to further understand the deployment scenarios.
This example shows the various CLI output required to verify the server subnet reachability between the leafs. Note that there is no
manipulation of BGP paths such as route policies and route aggregation. For additional information, refer to Design Considerations on
page 51.
Verification
Let's look at the step-by-step verification of a server subnet advertised from Leaf1 into the fabric
and the propagation of this route to other leafs.
Server VLAN
pod1-leaf1-1# show vlan 101
VLAN Name State Ports Classification
(F)-FCoE (u)-Untagged
(R)-RSPAN (c)-Converged
(T)-TRANSPARENT (t)-Tagged
================ =============== ========================== =============== ====================
101 VLAN0101 ACTIVE Po 1(t)
10.1.101.0/24, attached
*via DIRECT, Ve 101, [0/0], 22d0h, direct, tag 0
The BGP tables show whether the route is advertised to all BGP neighbors (spines).
Next hop 0.0.0.0 indicates the route being locally originated.
10.1.101.0/24,
*via 10.0.1.1, Fo 1/0/1, [20/0], 15d1h, eBgp, tag 0
*via 10.0.1.3, Fo 1/0/3, [20/0], 15d1h, eBgp, tag 0
10.1.101.0/24
*via 10.2.1.1, Eth 3/2, [20/0], 1d19h, eBgp, tag 0
*via 10.2.1.3, Eth 3/9, [20/0], 1d19h, eBgp, tag 0
*via 10.2.1.5, Eth 3/16, [20/0], 1d19h, eBgp, tag 0
*via 10.2.1.7, Eth 3/22, [20/0], 1d19h, eBgp, tag 0
Design Considerations
• Scale. ...................................................................................................................................................................52
• Recommendations for ISL Ports in a VDX vLAG Pair Leaf.................................................................................... 52
• Recommendations for ICL Ports in an SLX MCT Pair. .......................................................................................... 52
• Generalized TTL Security Mechanism for BGP (GTSM). ...................................................................................... 52
Scale
The following table gives various scale parameters and platforms used in this validated test topology. Note that this is not a measure of
the maximum scale that can be supported with Extreme switches in an IP fabric.
Parameter Value
Number of server racks/leafs (dual ToRs) 100
Number of spines 4
Number of server VLAN segments per rack 50
Number of hosts in each compute rack 2000
Number of super-spines 2
router bgp
local-as 4200001001
neighbor spine-group peer-group
neighbor spine-group ebgp-btsh
router bgp
address-family ipv4 unicast
aggregate-address 10.0.0.0/8 summary-only
aggregate-address 172.0.0.0/8 summary-only
• Aggregate all networks in a PoD at the spines, and advertise only the summary route to the leafs inside that PoD. Doing so, the
leaf does not learn every subnet attached to the other leafs in the PoD. Traffic forwarding to those subnets occurs using the
In the example shown below, the super-spine is configured to deny the default route (if any) advertised by the spines in the
PoDs.
• ORF prefix-list capability is a powerful tool that can be used by a BGP peer to tell its neighbor the list of prefixes (or routes) that
it is interested in receiving from the neighbor. This capability is a way to indicate the destinations of interest based on the
applications or services running behind the leaf. This is different from receiving all routes and then filtering unwanted prefixes.
The sender sends only the routes in the range of the prefix list.
ORF prefix-list capability can also be leveraged for receiving only the default route by, say, the super-spine from the border leaf.
In the example shown below, the super-spine indicates to the border leaf that it would like to receive only the default route and
not any others.
This appendix includes the relevant configurations of a few nodes in a 5-stage IP fabric.
Peer2
clock timezone America/Los_Angeles
!
ntp server 10.31.2.80 use-vrf mgmt-vrf
switch-attributes chassis-name SLX9140
switch-attributes host-name pod1-SLX-2
!
vrf mgmt-vrf
address-family ipv4 unicast
ip route 0.0.0.0/0 10.20.32.1
!
ipv6 protocol vrrp-extended
!
mtu 9022
ipv6 mtu 9000
ip mtu 9000
!
vlan 76
router-interface Ve 76
description non-MCT vlan on SLX-2
!
!
vlan 100
router-interface Ve 100
description non-MCT vlan on SLX-2
!
vlan 101
router-interface Ve 101
description Server vlan on MCT Po-101
!
!
vlan 200
router-interface Ve 200
description Server vlan on MCT Po-110
!
vlan 4090
router-interface Ve 4090
description MCT control vlan
!
ip prefix-list fabric_links_ip seq 10 permit 0.0.0.0/0 ge 31 le 31
!
route-map BGP-med permit 10
set metric add 100
!
route-map ToR-map permit 20
!
route-map ToR-map deny 10
match ip address prefix-list fabric_links_ip
!
route-map ToR-map-ipv6 permit 20
!
route-map ToR-map-ipv6 deny 10
match ipv6 address prefix-list fabric_links_ipv6
!
protocol vrrp-extended
!
evpn default
route-target both auto ignore-as
rd auto
vlan add 76-200
!
router bgp
local-as 4259840001
capability as4-enable
neighbor spine-group peer-group
neighbor spine-group remote-as 4259905537
neighbor spine-group description To spine AS-65001.1 from AS-65000.1
neighbor spine-group password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor spine-group bfd
neighbor spine-group-ipv6 peer-group
short-path-forwarding
!
no shutdown
!
interface Ve 200
ip mtu 9000
ip address 10.0.200.2/24
ipv6 address fdf8:10:0:c8::2/96
ipv6 mtu 9000
ipv6 vrrp-extended-group 2
virtual-ip fdf8:10:0:c8::254
enable
no preempt-mode
short-path-forwarding
!
vrrp-extended-group 1
virtual-ip 10.0.200.254
enable
no preempt-mode
short-path-forwarding
!
no shutdown
!
interface Ve 4090
ip mtu 9100
ip address 10.0.1.5/31
no shutdown
!
interface Management 0
no tcp burstrate
no shutdown
vrf forwarding mgmt-vrf
ip address dhcp
!
interface Ethernet 0/1
speed 10000
description MCT lag member to Server Racks
channel-group 101 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/2
speed 10000
description MCT lag member to Server Racks
channel-group 102 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/3
speed 10000
description MCT lag member to Server Racks
channel-group 103 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/4
speed 10000
description MCT lag member to Server Racks
channel-group 104 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/5
speed 10000
description MCT lag member to Server Racks
channel-group 105 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/6
speed 10000
description MCT lag member to Server Racks
ip address 10.0.2.2/31
ipv6 address fdf8:10:0:2::2/127
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/53
speed 40000
mtu 9216
description Link to Spines
ip mtu 9100
ip proxy-arp
ip address 10.0.2.6/31
ipv6 address fdf8:10:0:2::6/127
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/54
speed 40000
mtu 9216
description Link to Spines
ip mtu 9100
ip proxy-arp
ip address 10.0.2.8/31
ipv6 address fdf8:10:0:2::8/127
ipv6 mtu 9100
no shutdown
!
interface Port-channel 10
speed 40000
mtu 9216
description MCT peer-link LAG
no shutdown
!
interface Port-channel 101
speed 10000
mtu 9216
description MCT LAG
switchport
switchport mode trunk-no-default-native
switchport trunk allowed vlan add 101-110
no shutdown
!
interface Port-channel 102
speed 10000
mtu 9216
description MCT LAG
switchport
switchport mode trunk-no-default-native
switchport trunk allowed vlan add 111-120
no shutdown
!
interface Port-channel 103
speed 10000
mtu 9216
description MCT LAG
switchport
switchport mode trunk-no-default-native
switchport trunk allowed vlan add 121-130
no shutdown
!
interface Port-channel 104
speed 10000
mtu 9216
description MCT LAG
switchport
switchport mode trunk-no-default-native
switchport trunk allowed vlan add 131-140
no shutdown
!
interface Port-channel 105
speed 10000
mtu 9216
client SLX-1 5
client-interface Port-channel 105
deploy
!
client SLX-1 6
client-interface Port-channel 106
deploy
!
client SLX-1 7
client-interface Port-channel 107
deploy
!
client SLX-1 8
client-interface Port-channel 108
deploy
!
client SLX-1 9
client-interface Port-channel 109
deploy
!
client SLX-1 10
client-interface Port-channel 110
deploy
!
!
no shutdown
!
interface Ethernet 0/4
speed 40000
mtu 9216
description L3 link to VDX leaf
ip mtu 9100
ip proxy-arp
ip address 10.0.6.1/31
ipv6 address fdf8:10:0:6:1::2/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/5
speed 40000
mtu 9216
description L3 link to N9K Leaf
ip mtu 9100
ip proxy-arp
ip address 10.0.4.1/31
ipv6 address fdf8:10:0:4:1::2/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/6
speed 40000
mtu 9216
description L3 link to N9K Leaf
ip mtu 9100
ip proxy-arp
ip address 10.0.5.1/31
ipv6 address fdf8:10:0:5:1::2/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/7
speed 40000
mtu 9216
description L3 link to VDX leaf
ip mtu 9100
ip proxy-arp
ip address 10.0.7.1/31
ipv6 address fdf8:10:0:7:1::2/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/11
speed 100000
mtu 9216
description L3 link to SLX-9850 Super-Spine
ip mtu 9100
ip proxy-arp
ip address 10.0.51.1/31
ipv6 address fdf8:10:0:51:1::2/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/12
mtu 9216
description L3 link to SLX-9850 Super-Spine
ip mtu 9100
ip proxy-arp
ip address 10.0.51.3/31
ipv6 address fdf8:10:0:51:2::2/96
ipv6 mtu 9100
no shutdown
!
!
hardware
port-group 3/1
mode 40g
!
port-group 3/2
mode 40g
!
port-group 3/3
mode 40g
!
port-group 3/4
mode 100g
!
port-group 3/5
mode 100g
!
port-group 3/6
mode 40g
!
!
ntp server 10.31.2.80 use-vrf mgmt-vrf
switch-attributes chassis-name SLX-9850
switch-attributes host-name Super-Spine1
!
vrf mgmt-vrf
address-family ipv4 unicast
ip route 0.0.0.0/0 10.20.32.1
!
ipv6 prefix-list denydefault-fromspine seq 10 permit ::/0
ipv6 prefix-list receivedgwonly-fromborderleaf seq 10 permit ::/0
!
ip router-id 10.125.1.1
ip prefix-list allowdefaultonly seq 10 permit 0.0.0.0/0
ip prefix-list denydefault-fromspine seq 10 permit 0.0.0.0/0
ip prefix-list pfx-list-default seq 10 permit 0.0.0.0/0
ip prefix-list pfx-list-denydefault-from-spine seq 10 permit 0.0.0.0/0
!
route-map denydefault-fromspine permit 20
!
route-map denydefault-fromspine deny 10
match ip address prefix-list pfx-list-denydefault-from-spine
!
route-map denydefault-fromspine-ipv6 permit 10
match ipv6 address prefix-list denydefault-fromspine
!
route-map denydefault-fromspine-ipv6 permit 20
!
router bgp
local-as 4200000010
capability as4-enable
fast-external-fallover
neighbor edge-leaf peer-group
neighbor edge-leaf remote-as 4200007000
neighbor edge-leaf password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor edge-leaf bfd
neighbor edge-leaf-ipv6 peer-group
neighbor edge-leaf-ipv6 remote-as 4200007000
neighbor edge-leaf-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor edge-leaf-ipv6 bfd
neighbor pod1-spine-group peer-group
neighbor pod1-spine-group remote-as 4259905537
neighbor pod1-spine-group password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor pod1-spine-group bfd
neighbor pod1-spine-group-ipv6 peer-group
neighbor pod1-spine-group-ipv6 remote-as 4259905537
neighbor pod1-spine-group-ipv6 password $9$MCgKGaNt6OASX68/7TC6Lw==
neighbor pod1-spine-group-ipv6 bfd
neighbor pod2-spine-group peer-group
neighbor pod2-spine-group remote-as 4200002000
neighbor pod2-spine-group password $9$BfpeY2eMFj4uKynSwFRgWA==
neighbor pod2-spine-group bfd
ip proxy-arp
ip address 10.0.53.0/31
ipv6 address fdf8:10:0:53:1::1/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 3/49
mtu 9216
description L3 link to SLX-9240 Pod1-Spine4
ip mtu 9100
ip proxy-arp
ip address 10.0.54.0/31
ipv6 address fdf8:10:0:54:1::1/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 3/50
mtu 9216
description L3 link to SLX-9240 Pod1-Spine2
ip mtu 9100
ip proxy-arp
ip address 10.0.52.0/31
ipv6 address fdf8:10:0:52:1::1/96
ipv6 mtu 9100
no shutdown
!
no preempt-mode
short-path-forwarding
!
no shutdown
!
interface Ve 3
ip mtu 9100
ip address 192.168.3.1/24
ipv6 address fdf8:192:168:3::1/96
ipv6 mtu 9100
ipv6 vrrp-extended-group 2
virtual-ip fdf8:192:168:3::254
enable
no preempt-mode
short-path-forwarding
!
vrrp-extended-group 1
virtual-ip 192.168.3.254
enable
no preempt-mode
short-path-forwarding
!
no shutdown
!
interface Ve 4
ip mtu 9100
ip proxy-arp
ip address 10.55.56.0/31
ipv6 mtu 9100
no shutdown
!
interface Management 0
no tcp burstrate
no shutdown
vrf forwarding mgmt-vrf
ip address dhcp
!
!
interface Ethernet 0/9
description MCT lag member to FireWall
channel-group 2 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/10
description MCT peer-link LAG members
channel-group 1 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/11
description MCT peer-link LAG members
channel-group 1 mode active type standard
lacp timeout long
no shutdown
!
interface Ethernet 0/49
speed 40000
mtu 9216
description L3 link to Super-Spine1
vrf forwarding DC-VRF
ip mtu 9100
ip proxy-arp
ip address 10.0.55.11/31
ipv6 address fdf8:10:0:55:1::1/96
ipv6 mtu 9100
no shutdown
!
interface Ethernet 0/50
speed 40000
mtu 9216
description L3 link to Super-Spine2
!
interface Vlan 105
!
interface Vlan 106
!
interface Vlan 107
!
interface Vlan 108
!
interface Vlan 109
!
interface Vlan 110
!
protocol lldp
advertise dcbx-fcoe-app-tlv
advertise dcbx-fcoe-logical-link-tlv
advertise dcbx-tlv
advertise optional-tlv system-name
system-description Brocade-VDX-VCS 1
!
vlan dot1q tag native
port-profile UpgradedVlanProfile
vlan-profile
switchport
switchport mode trunk
switchport trunk allowed vlan all
!
!
port-profile default
vlan-profile
switchport
switchport mode trunk
switchport trunk native-vlan 1
!
!
port-profile-domain default
port-profile UpgradedVlanProfile
!
class-map cee
!
class-map default
!
rbridge-id 1
ip router-id 10.122.1.2
ip prefix-list fabric_links_ip seq 10 permit 0.0.0.0/0 ge 31 le 31
switch-attributes chassis-name VDX6740
switch-attributes host-name pod1-leaf1-1
vrf mgmt-vrf
address-family ipv4 unicast
ip route 0.0.0.0/0 10.20.32.1
!
address-family ipv6 unicast
!
!
route-map ToR-map permit 20
!
route-map ToR-map deny 10
match ip address prefix-list fabric_links_ip
!
route-map ToR-map-ipv6 permit 20
!
route-map ToR-map-ipv6 deny 10
match ipv6 address prefix-list fabric_links_ipv6
!
router bgp
local-as 4200001001
capability as4-enable
fast-external-fallover
neighbor spine-group peer-group
neighbor spine-group remote-as 4200001000
neighbor spine-group description connected to 4 spine-groups
neighbor spine-group password 2 $MlVzZCFAbg==
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.102.1/24
vrrp-extended-group 10
virtual-ip 10.1.102.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 103
ipv6 address fdf8:10:1:103::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:103::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.103.1/24
vrrp-extended-group 10
virtual-ip 10.1.103.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 104
ipv6 address fdf8:10:1:104::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:104::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.104.1/24
vrrp-extended-group 10
virtual-ip 10.1.104.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 105
ipv6 address fdf8:10:1:105::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:105::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.105.1/24
vrrp-extended-group 10
virtual-ip 10.1.105.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 106
ipv6 address fdf8:10:1:106::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:106::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.106.1/24
vrrp-extended-group 10
virtual-ip 10.1.106.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 107
ipv6 address fdf8:10:1:107::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:107::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.107.1/24
vrrp-extended-group 10
virtual-ip 10.1.107.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 108
ipv6 address fdf8:10:1:108::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:108::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.108.1/24
vrrp-extended-group 10
virtual-ip 10.1.108.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 109
ipv6 address fdf8:10:1:109::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:109::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.109.1/24
vrrp-extended-group 10
virtual-ip 10.1.109.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
interface Ve 110
ipv6 address fdf8:10:1:110::1/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:110::254
enable
preempt-mode
priority 150
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.110.1/24
vrrp-extended-group 10
virtual-ip 10.1.110.254
advertisement-interval 1
enable
preempt-mode
priority 150
short-path-forwarding
!
no shutdown
!
!=========================================
rbridge-id 2
ip router-id 10.122.1.3
ip prefix-list fabric_links_ip seq 10 permit 0.0.0.0/0 ge 31 le 31
switch-attributes chassis-name VDX6740
switch-attributes host-name pod1-leaf1-2
vrf mgmt-vrf
address-family ipv4 unicast
ip route 0.0.0.0/0 10.20.32.1
!
address-family ipv6 unicast
!
!
route-map ToR-map permit 20
!
route-map ToR-map deny 10
match ip address prefix-list fabric_links_ip
!
route-map ToR-map-ipv6 permit 20
!
route-map ToR-map-ipv6 deny 10
match ipv6 address prefix-list fabric_links_ipv6
!
router bgp
local-as 4200001001
capability as4-enable
fast-external-fallover
neighbor spine-group peer-group
neighbor spine-group remote-as 4200001000
neighbor spine-group description connected to 4 spine-groups
neighbor spine-group password 2 $MlVzZCFAbg==
neighbor spine-group capability as4 enable
neighbor spine-group-ipv6 peer-group
neighbor spine-group-ipv6 remote-as 4200001000
neighbor spine-group-ipv6 description connected to 4 spine-groups
neighbor spine-group-ipv6 password 2 $MlVzZCFAbg==
neighbor spine-group-ipv6 capability as4 enable
neighbor fdf8:10:0:1::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:2::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:3::2 peer-group spine-group-ipv6
neighbor fdf8:10:0:4::2 peer-group spine-group-ipv6
neighbor 10.0.1.2 peer-group spine-group
neighbor 10.0.2.2 peer-group spine-group
neighbor 10.0.3.2 peer-group spine-group
neighbor 10.0.4.2 peer-group spine-group
address-family ipv4 unicast
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.101.2/24
vrrp-extended-group 10
virtual-ip 10.1.101.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 102
ipv6 address fdf8:10:1:102::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:102::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.102.2/24
vrrp-extended-group 10
virtual-ip 10.1.102.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 103
ipv6 address fdf8:10:1:103::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:103::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.103.2/24
vrrp-extended-group 10
virtual-ip 10.1.103.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 104
ipv6 address fdf8:10:1:104::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:104::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.104.2/24
vrrp-extended-group 10
virtual-ip 10.1.104.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 105
ipv6 address fdf8:10:1:105::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:105::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.105.2/24
vrrp-extended-group 10
virtual-ip 10.1.105.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 106
ipv6 address fdf8:10:1:106::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:106::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.106.2/24
vrrp-extended-group 10
virtual-ip 10.1.106.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 107
ipv6 address fdf8:10:1:107::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:107::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.107.2/24
vrrp-extended-group 10
virtual-ip 10.1.107.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 108
ipv6 address fdf8:10:1:108::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:108::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.108.2/24
vrrp-extended-group 10
virtual-ip 10.1.108.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 109
ipv6 address fdf8:10:1:109::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:109::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.109.2/24
vrrp-extended-group 10
virtual-ip 10.1.109.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Ve 110
ipv6 address fdf8:10:1:110::2/64
ipv6 mtu 9000
ipv6 vrrp-extended-group 100
virtual-ip fdf8:10:1:110::254
enable
preempt-mode
priority 140
advertisement-interval 1
short-path-forwarding
!
ip mtu 9000
ip proxy-arp
ip address 10.1.110.2/24
vrrp-extended-group 10
virtual-ip 10.1.110.254
advertisement-interval 1
enable
preempt-mode
priority 140
short-path-forwarding
!
no shutdown
!
interface Management 1/0
no tcp burstrate
ip icmp echo-reply
no ip address dhcp
ip address 10.20.34.57/22
ipv6 icmpv6 echo-reply
ipv6 address 2620:100:0:fa48:34::57/64
no ipv6 address autoconfig
no ipv6 address dhcp
vrf forwarding mgmt-vrf
no shutdown
!
interface Management 2/0
no tcp burstrate
ip icmp echo-reply
no ip address dhcp
ip address 10.20.34.58/22
ipv6 icmpv6 echo-reply
ipv6 address 2620:100:0:fa48:34::58/64
no ipv6 address autoconfig
no ipv6 address dhcp
vrf forwarding mgmt-vrf
no shutdown
!
interface TenGigabitEthernet 1/0/1
description ISL link to the peer
fabric isl enable
fabric trunk enable
no shutdown
!
interface TenGigabitEthernet 1/0/2
description ISL link to the peer
fabric isl enable
fabric trunk enable
shutdown
!
interface TenGigabitEthernet 1/0/9
channel-group 1 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 1/0/10
channel-group 2 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 1/0/11
channel-group 3 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 1/0/12
channel-group 4 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 1/0/13
channel-group 5 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/1
description ISL link to the peer
fabric isl enable
fabric trunk enable
no shutdown
!
interface TenGigabitEthernet 2/0/2
description ISL link to the peer
fabric isl enable
fabric trunk enable
no shutdown
!
interface TenGigabitEthernet 2/0/9
channel-group 1 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/10
channel-group 2 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/11
channel-group 3 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/12
channel-group 4 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/13
channel-group 5 mode active type standard
no fabric isl enable
no fabric trunk enable
lacp timeout long
no shutdown
!
interface FortyGigabitEthernet 1/0/49
mtu 9216
description Link to Spine1
no fabric isl enable
no fabric trunk enable
ip mtu 9100
ip proxy-arp
ip address 10.0.1.1/31
ipv6 address fdf8:10:0:1::1/127
References
1. Network Virtualization in IP Fabric with BGP EVPN - Extreme Validated Design
https://cloud.kapostcontent.net/pub/e043a3a3-dded-4e23-960e-e2df94f8afe0/network-virtualization-
evd?kui=K5TiUQIwFu0s1wTdY6YFRw
2. Use of BGP for Routing in Large-Scale Data Centers
https://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/
3. Extreme VDX hardware installation guides
https://documentation.extremenetworks.com/networkos/HW/vdx6940-installguide.pdf
https://documentation.extremenetworks.com/networkos/HW/vdx6740-installguide.pdf
4. Extreme SLX 9850 Router
https://www.extremenetworks.com/product/slx-9850-router/
5. Extreme SLX 9850 hardware installation guides
https://documentation.extremenetworks.com/slxos/HW/53-1004400-07_98508SLX_IG_Dec2017.pdf
https://documentation.extremenetworks.com/slxos/HW/53-1004399-07_98504SLX_IG_Dec2017.pdf
6. Extreme SLX 9540 Switch Hardware Installation Guide
https://documentation.extremenetworks.com/slxos/HW/53-1004986-05_9540SLX_IG_Dec2017.pdf
7. Extreme SLX 9140 Switch Hardware Installation Guide
https://documentation.extremenetworks.com/slxos/HW/53-1004984-03_9140SLX_IG_Nov2017.pdf
8. Extreme SLX 9240 Switch Hardware Installation Guide
https://documentation.extremenetworks.com/slxos/HW/53-1004985-03_9240SLX_IG_Nov2017.pdf