Sunteți pe pagina 1din 98

EVPN: Network

Virtualization Solution
for Next Generation
DCs & DC Interconnect

Ali Sajassi– Distinguished Engineer, Cisco


BRKMPL-2333
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

Cisco Spark spaces will be cs.co/ciscolivebot#BRKMPL-2333


available until July 3, 2017.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• EVPN Overview
• Why EVPN
• EVPN Technology Prime
• EVPN Startup Sequence
• EVPN Operation
• A Day in Life of a Packet

• DC Fabric Evolution w/ EVPN-IRB


• EVPN-VPWS
• EVPN Deployment: DC Fabric and
WAN Integration
EVPN Overview
Service Provider Data Center
Next gen all-in-one VPN Industry defacto-standard multi-
solution to provide all the tenancy solution that provides
services previously done by flexible workload placement,
multiple solutions and more workload mobility, full cross-
sectional BW utilization and
• L2VPN (P2P, MP, P2MP) elasticity.
• L3VPN
• IRB • L2, L3, L2+L3 Overlay
• DCI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Underlay v.s. Overlay
Spines

EVI 20
EVI 10

Leafs

• Underlay: physical topology + • Overlay: Virtual topology among


associated routing/signaling protocol endpoints (NVEs)
protocol • Deals with reachability between tenant
• High redundancy in fabric (link/node) routes and MAC addresses
• IP underlay: IGP (or BGP) • EVPN uses BGP as routing protocol for
• MPLS underlay: LDP, RSVP-TE, SR distribution of tenants IP
• Leaf/Spine (CLOS) Topology prefixes/addresses and MACs
• Uses IP or MPLS as underlay
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
EVPN in a Nut Shell
MAC & IP learning in control plane (via BGP)

MAC Routing: Control plane (BGP) Optimum forwarding,


advertise the learnt MACs from CE
ECMP, Multi-pathing
PE1 PE3 Common L2/L3 VPN
All active Single active Operational Mode
multi-homing multi-homing
Same operational
CE1 CE3
principles as IP-VPN
C-MAC: M1 IP or MPLS
Extensive multi-homing
capabilites

PE2 PE4 Flexible Policy Control

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
EVPN Family of Solutions RFC 7432

EVPN

P2P Multipoint

EVPN-VPWS EVPN-FXC

RFC 7623

EVPN-IRB EVPN-Overlay EVPN-Etree PBB-EVPN

draft-ietf-bess-evpn-inter-subnet-forwarding draft-ietf-bess-evpn-overlay draft-ietf-bess-evpn-etree


draft-ietf-bess-evpn-prefix-advertisement

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
What’s the big deal about EVPN?
EVPN is next generation all-in-one VPN solution
It not only does the job of many other VPN technologies but it does it better !!
E-LAN E-LINE E-TREE DC Fabric IRB DCI
(MP2MP (P2P (P2MP L3VPN (IntraDC (L2/L3 (InterDC)
L2VPN) L2VPN) L2VPN) Overlay) Overlay)
VPLS- VxLAN
VPLS PW 4364 VPLS,OTV
ETREE TRILL

EVPN

PBB- EVPN EVPN EVPN- EVPN- EVPN- EVPN-


EVPN VPWS ETREE L3VPN Overlay IRB DCI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Example: EVPN Does it Better than VPLS !!

Service Additional Capabilities


E-LAN • Provides All-Active multi-homing
• Prevents loop for both all-active & single-active even in transient
state
• Efficient utilization of network cross-sectional bandwidth (via
optimum forwarding, ECMP, multi-pathing on a per flow basis)
• Flexible policy control per MAC and per Site

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
EVPN – Control and Data plane

EVPN
Control-
(MP-BGP)
Plane
RFC7432

Network Virtualization Overlay


Multi-Protocol Label Switching Provider Backbone Bridges
(VXLAN, NVGRE, Geneve,
(MPLS) (PBB+MPLS)
RFC7432 RFC7623
GUE, GPE)
Data- draft-ietf-bess-evpn-overlay
Plane

LDP, RSVP-TE, or SR IPv4/IPv6, or SRv6

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
2006 2010 2011 2013 2015 +

Following drafts were Following drafts were Enhancements


- OPEN project was started at - Introduced to IETF as introduced: introduced - Virtual ES
Cisco Routed-VPLS - EVPN - EVPN IRB - Optimized ingress replication
- OPEN = Optimum Ethernet - Merged with Juniper’s - PBB-EVPN - EVPN DCI - IGMP aggregation between
Network MAC-VPN and was - EVPN-VPWS PODs
introduced an EVPN - EVPN-Overlay - mcast tunnels between DCs
- EVPN-ETREE - Inter-AS for IRB
- L3VPN multi-homing
- EVPN-IRB Multicast
- DF Election algorithmes
- Mobility Enhancements
- FXC
- VPLS seamless integration for
all-active multi-homing
- Etc.

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Why EVPN ?
Today’s DC Requirements
• Efficient and Flexible Multi-Homing
• Efficient fabric bandwidth utilization
• Flexible workload placement
• Seamless workload mobility: inter-POD, intra-
DC, inter-DC
• High scale multi-tenancy
• Optimal forwarding of intra & inter-subnet
traffic
• Traffic engineering (SPDC)
• Seamless integration w/ existing L2 and
L3VPN solutions

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Efficient and Flexible Multi-Homing
Prior to EVPN EVPN
CE1 PE1 CE1 PE1

CE2 PE2 CE2 PE2


CE3
PE3
CE3 PE3

CE4
PE4
CE4 PE4
• N-way redundancy
• Limited to Dual-homing
• No inter-chassis links + fast convergence – enables geo-
• Increased cost because of inter-chassis links redundancy

• Inflexible configuration • Works for both MPLS & IP transport

• Proprietary (vPC, Virtual Switch, Cluster) • Standard based with multitude of applications: SP access, SP
inter-domain, intra-DC, and inter-DC

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Efficient Fabric Bandwidth Utilization
Prior to EVPN EVPN

MAC1 CE1 CE2 MAC2


MAC1 CE1 CE2 MAC2 PE1 PE3
PE1 PE3

PE2 PE4
PE2 PE4

• Enabling multiple paths through the fabric between a pair of


• Limited number of paths among workloads via the fabric
workloads with each path having multiple ECMP itself
• For MPLS/IP fabric, data plane & control plane Scale
• MAC learning in BGP -> removing PWs & targeted LDP sessions
issues: O(n^2) full-mesh of PWs, learning every MAC
over PW, full-mesh of targeted LDP sessions • Enabling policy for individual MACs & ingress filtering
• No MAC-based policy • Provides Aliasing for multiple-path enablement even in the
absence of MAC learning on a PE!!
• Single-active multi-homing support only
• Simplifying operation via autosensing & auto-config of ES

• Enabling path selection


BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Flexible Workload Placement & Seamless Mobility
EVPN
Prior to EVPN SDN
Controller

Spine Spine

Core Core

TOR1 TOR2 TOR3 TOR4

SW1 SW2 SW3 SW4 App


VNF VNF
App
VNF VNF
VNF VNF App

• Enable flexible and efficient workload placement and


• L2 fabric with centralized routing mobility across the whole fabric
– Inefficient L2 forwarding & LB in the fabric (MSTP)
• Distributed Anycast GW
– Inefficient L3 forwarding
- Optimum L2 & L3 forwarding
• Sub optimal application performance (flooding of traffic,
delayed due to centralized routing) • Consistent and seamless solution for intra-POD, inter-
POD, inter-DC, inter-domain (SDN vs. virtual vs physical)
• Along w/ n-way active/active redundancy
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
High Scale Multi-Tenancy & Optimum Forwarding
Centralized Routing Distributed Routing

L3 Centralized GW Boarder
Leaf
L2

Fabric Fabric
L3
Leaf Leaf
L2

Subnet 1/VLAN1 Subnet 2/VLAN2 Subnet 1/VLAN1 Subnet 2/VLAN2

• All east<->west routed traffic traverses to centralized • Optimized forwarding of east-west traffic
gateways • ARP/MAC state localized to Leafs - Helps with
• Centralized gateways have full ARP/MAC state in the DC horizontal scaling of DC
- Scale challenge

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
EVPN Efficient Cross-Sectional BW Utilizaiton

Flow Based Load-balancing – PE to PE direction


• EVPN provides per-flow load-
Vlan X - F1 P P
balancing among egress PEs Vlan X – F2 E E
using BGP multi-pathing

P
E

• Per-flow load balancing Vlan X - F1 Flow Based Multi-Pathing in the Core


Vlan X – F2
between ingress and egress Vlan X – F3 P
E
P
E
Vlan X – F4 P P
PEs are provided using IGP
ECMP (ingress PE still needs
to add entropy field in the P P
P
packet). E

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
EVPN Technology Prime
EVPN Technology Prime

• EVPN Concepts
• EVPN Startup Sequence
• EVPN Operation
• A Day in Life of a Packet
EVPN Instance (EVI) & MAC-VRF
E-VPN Instance
Spine
(EVI) & MAC-VRF

MAC-VRF EVI is an EVPN instance


spanning Leafs that participate
EVI 20 in that EVPN
MAC-VRF
EVI 10
PE Leaf
MAC- MAC- MAC- MAC- MAC- MAC-
EVPN uses BGP route target
• EVI identifies a VPN in the VRF VRF VRF VRF VRF VRF filtering to enable leafs that don’t
network belong to a specific EVI, not import
• Encompass one or more any MACs for that EVI, providing
bridge-domains, depending efficient scalability
on service interface type
Port-based VM
VM VM VM VM
VLAN-based (shown above) EVPN can use BGP RT constraint
VLAN-bundling (RTC) and Outbound Route
VLAN aware bundling (NEW) Filtering (ORF) for further filtering
of MAC routes at RR

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Ethernet Segment (ES)
Multi-home (MHD) Multi-home (MHD) All-
Ethernet Segment Single Home Device
Single-Active (Per- Active (Per-Flow) LB
(SHD)
VLAN) LB
SHD CE1 PE1
ESI1

MHD CE2

ESI2
PE2
• Represents a ‘site’
connected to one or more ESI-0 ESI-0 ESI-1 ESI-1
ESI-1 ESI-1
PEs
• Uniquely identified by a 10-
byte global Ethernet VM
VM VM
Segment Identifier (ESI)
• Could be a single device or
an entire network • Typically used for MHD in DCs
• Ethernet Segment • Typically used for MHN in
Single-Homed Device (SHD)
Multi-Homed Device (MHD)
Identifier (ESI) of ‘0’ SPs • Per-flow LB for known unicast
Single-Homed Network (SHN) traffic
• No DF election • Per VLAN DF election for
Multi-Homed Network (MHN)
all traffic • Per-VLAN DF for BUM traffic
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Split-Horizon Filtering, DF Election, and
Aliasing
MAC1 MAC2
• Split-horizon: BUM traffic doesn’t CE1 PE1 PE3 CE2

get loopback to the originating CE


Echo !
device.

PE2 PE4

• DF selection: Either PE3 or PE4 MAC1 CE1 CE2 MAC2


PE1 PE3
forward the broadcast traffic to the
far-end dual-homed device CE2.
Duplicate !

PE2 PE4

• Aliasing: When PE3 wants to


MAC1 CE1 CE2 MAC2
forward a packet with destination PE1 PE3

address MAC1, it needs to send it


to both PE1 and PE2 even though Load balancing

it only learned MAC1 from PE1.


PE2 PE4

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Integrated Routing & Bridging (IRB)

IP-VPN Instance represents a


Spine collection of IP-VRFs and their
associated connectivity for a
given tenant

Leaf IP-VRF IP-VRF IP-VRF IP-VRF IRB Interface: a virtual interface


MAC- MAC- MAC- MAC- MAC- MAC-
connecting a MAC-VRF to an IP-
MAC-
VRF VRF VRF VRF VRF VRF VRF VRF

VM VM VM VM VM
Inter-subnet - EVPN can use BGP RT constraint
Routed (RTC) and Outbound Route
Intra-subnet - Filtering (ORF) for further filtering
Bridged of IP routes at RR
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Distributed Anycast Gateway with BGP-EVPN
Optimal intra and inter-subnet connectivity with seamless workload mobility

Spine

Identical Anycast Gateway Virtual IP


Distributed Anycast Gateway serves and MAC address are configured on
as the gateway for connected hosts all the Leafs

BVI BVI BVI BVI


GW MAC GW MAC GW MAC GW MAC

Leaf

All the BVIs perform active forwarding


in contrast to active/standby like FHRP VM VM

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
EVPN Route Types & Benefits
Route Type Usage Benefits
Ethernet A-D Route • Aliasing • Loop avoidance – even
(Type 1) • Mass Withdraw of addresses transient
• SH/AA MH Indication • Fast convergence
• Advertising Split-Horizon Label • Efficient load balancing
• Per-site policy

MAC/IP Advertisement Route (Type 2) • Advertise MAC (and IP) reachability • Per MAC policy
• Advertise MAC/IP binding • ARP suppression
• MAC mobility • Workload Mobility
Inclusive Multicast Route • Auto discovery of multicast tunnel • Support multicast even
(Type 3) endpoints & mcast tunnel type when core doesn’t
Ethernet Segment Route • Auto discovery of redundancy group • A/A and S/A MHD & MHN
(Type 4) support
IP Prefix Route • IP Prefix advertisement (not for IP host • IP route aggregation
(Type 5) advertisement) • Interop w/ L3VPN

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
EVPN BGP Routes RFC7432
• EVPN defines a new BGP NLRI used to carry all EVPN routes
• BGP Capabilities Advertisement used to ensure that two speakers support
EVPN NLRI (per RFC4760)
• AFI 25: L2VPN, SAFI 70: EVPN [1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
1 byte Route Type [3] Inclusive Multicast Route
[4] Ethernet Segment Route
1 byte Length [5] IP Prefix Route

Variable Route type specific

EVPN NLRI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
EVPN Technology Prime

• EVPN Concepts
• EVPN Startup Sequence
• EVPN Operation
• A Day in Life of a Packet
EVPN Startup Sequence

Segment Auto-Discovery VPN Auto-Discovery

Multicast Tunnel Endpoint


ESI Auto-Sensing
Discovery

Redundancy Group Membership


Auto-Discovery

DF Election & VLAN Carving

ESI Label & MH type Discovery

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
ESI Auto-Sensing
ESI (10B) can be auto-generated1
from CE’s LACP information ->
concatenation of CE’s LACP
System Priority + Sys ID + Port Key System System MAC
Segment Auto-Discovery Example:
Priority Address
Port Key

0000. 0011.0022.0033.0018 2 bytes 6 bytes 2 bytes


PE1 PE3

ESI Auto-Sensing3 LACP PDU


exchange
CE LACP info:
LACP System ID (MAC) (6B)
e.g. 0011.0022.0033 CE1 CE3
LACP System Priority (2B)
e.g. 0000 MPLS
LACP Port Key (2B)
e.g. 0018

PE2 PE4

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Redundancy Group Membership Auto-Discovery

PE 1 Eth Segment Route


RD = RD10 RD – RD unique per
ESI = ESI1 adv. PE
Segment Auto-Discovery
MAC address portion ES-Import Route Target
of ESI (6B) e.g. 0011.0022.0033

PE1 PE3

ESI Auto-Sensing

PE4
CE1
Redundancy Group Membership
MPLS
Auto-Discovery PE1000

PE2
Exchange ofPE4
Ethernet
PE 2 Eth Segment Route Segment Routes
RD = RD20
ESI = ESI1
ES-Import Route Target
e.g. 0011.0022.0033
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ordered List of discovered PEs
DF Election & VLAN Carving Modulo Operation
starting from zero (lowest IP add)

VID mod N PE Ordered List


VID (N = # of PEs)
(e.g. VID mod 2) Position PE
Result of modulo 0 PE1
100 0
Segment Auto-Discovery operation is used to
1 PE2
determine DF and 101 1
BDF status 102 0 Example:
103 1 PE1 DF for VIDs 100, 102
PE1 BDF for VIDs 101, 103
ESI Auto-Sensing PE1 PE3

Redundancy Group Membership CE1 CE3


Auto-Discovery
MPLS

DF Election & VLAN Carving PE Ordered List


Position PE
Modulo Operation 0 PE1
PE2 PE4
VID (VID mod 2) 1 PE2
100 0
Example:
101 1
PE2 DF for VIDs 101, 103
102 0 PE2 BDF for VIDs 100, 102
103 1 BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
ESI Label & MH type Discovery
PE1 Eth A-D per ES
RD = RD-1a
ESI1
Eth Tag = MAX-ET
Label = 0
Segment Auto-Discovery ESI Label ext. com
L1

PE1 PE3

ESI Auto-Sensing

CE1
Redundancy Group Membership ESI1
MPLS
Auto-Discovery

DF Election and VLAN Carving PE2 Eth A-D per ES


RD = RD-1b PE2 PE4
ESI1

ESI Label & MH type Discovery Eth Tag = MAX-ET


Label = 0
ESI Label ext. com
L2
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multicast Tunnel Endpoint Discovery
PE 1 Inclusive Multicast Route
Tunnel Type – Ingress RD = RD-1a
RD – RD unique per
Replication or P2MP LSP PMSI Tunnel Attribute adv. PE per EVI

Mcast MPLS Label – used to Tunnel Type (e.g. Ing. Repl.)


transmit BUM traffic - Label (e.g. L1)
downstream assigned (ing.
VPN Auto-Discovery repl.) or upstream assigned
RT ext. community
(Aggregate Inclusive P2MP RT-a
LSP2)
PE1 PE3
RT – RT associated with a
Multicast Tunnel Endpoint given EVI
Discovery

CE1 CE3

MPLS

PE 2 Inclusive Multicast Route


RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.) PE2 PE4
Label (e.g. L2)
RT ext. community
RT-a PMSI - P-Multicast Service Interface
BUM – Broadcast / Unknown Unicast / Multicast
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
EVPN BGP route 0x4 - Ethernet Segment Route
• Usage:
• Auto-discovery of multi-homed Ethernet Segments
• Designated Forwarder election

• Tagged with ES-Import Extended Community


• PEs apply route filtering based on ES-Import community. Thus, Ethernet Segment route is imported
only by the PEs that are multi-homed to the same Ethernet segment

Unique per Advertising PE


8 bytes RD
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier
IP address length
1 bytes IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Router’s IP add.

Route Type specific encoding of E-VPN NLRI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
ES-Import RT Extended Community
Usage:
• Sent with Ethernet Segment route
• Limits the scope of Ethernet Segment routes distribution to PEs connected to the same multi-homed
Segment

0x06

0x02 MAC Address portion of the ESI


6 bytes ES-Import

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
EVPN BGP Route 0x1 – Ethernet Auto-discovery Route
This route has two flavors:

Per-ES Ethernet A-D route Per-EVI Ethernet A-D route

• Advertise the Split-Horizon Label associated with an • Advertise VPN label used for Aliasing or Backup-Path
Ethernet Segment

• For AA or SA MH indication

• Used for MAC Mass-Withdraw

Unique per Advertising PE per EVI


Unique per Advertising PE
8 bytes RD ESI of Ethernet Segment
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-Aware
MUST be set to MAX-ET Bundling Service interface, otherwise 0
4 bytes Ethernet Tag ID
(0xFFFFFFFF)
3 bytes MPLS Label VPN (Aliasing) Label per (ESI,
MUST be set to 0 Ethernet Tag)

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
ESI Label extended community ESI-2

MAC1 MAC2
Agg1 PE1 PE3 Agg2
Usage: ESI-1

• Sent with Ethernet AD Route per ES


• Advertises the Split-Horizon Label for the Ethernet Segment
PE4
PE2
• Indicates the Redundancy Mode: Single Active vs. All-Active

0x06  PE1 advertises in BGP a split-horizon label associated


with the ESI-1 (in the Ethernet AD route)
0x01 Bit 0: Redundancy Mode  Split-horizon label is only used for multi-destination
(single active vs. all active) frames (unknown unicast, mcast, bcast)
Flags
 When PE1 wants to forward a multi-destination frame, it
Reserved Set to 0 appends this SH label to the packet
 PE2 uses this label to perform split-horizon filtering
ESI MPLS Label Ethernet Segment Split- for frames destined to ESI-1 - e.g., a frame originated by
Horizon Label a segment must not be received by the same segment

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
EVPN BGP route 0x3 – Inclusive Multicast
• Usage:
• Multicast tunnels used to transport Broadcast, Multicast and Unknown Unicast frames
(BUM)

Unique per Advertising PE per EVI


8 bytes RD Set to VLAN or I-SID for VLAN-Aware
4 bytes Ethernet Tag ID Bundling Service interface, otherwise 0
1 bytes
IP address length
IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Router’s IP add.

Route Type specific encoding of E-VPN NLRI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
PMSI Tunnel Attribute – RFC6514

Flags based on RFC6514


1 bytes Flags Ingress Replication/mLDP etc.
1 bytes Tunnel Type
Multicast MPLS Label
3 bytes MPLS Label
When the Tunnel Type is set to Ingress Replication, the Tunnel Identifier
variable Tunnel Identifier carries the unicast tunnel endpoint IP address of the local PE that is to be this
PE's receiving endpoint address for the tunnel.
Route Type specific encoding of E-VPN NLRI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
EVPN Technology Prime

• EVPN Concepts
• EVPN Startup Sequence
• EVPN Operation
• A Day in Life of a Packet
MAC Address Reachability
Route Type Usage Benefits

MAC/IP Advertisement • Advertise MAC (and IP) • Per MAC (and IP)
Route (Type 2) reachability policy RR
• Advertise MAC/IP binding • ARP suppression
• MAC Mobility • Workload Mobility

PE1 PE3

CE3
ESI1 PE4 CE4
MAC1 CE1
PE2

• PE1 & PE2 learns MAC1 from CE1 and advertises in BGP to all other PEs with ES field in the MAC/IP
advertisement set to ESI1
• PE3 and PE4 learn that MAC1 sits behind ESI1 which in turn sits behind PE1 & PE2
• PE3 and PE4 now know for packets destined to CE1, they can load balanced between PE1 and PE2

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
ARP Broadcast Suppression
Route Type Usage Benefits Challenge:
How to reduce ARP broadcasts over the
MAC/IP Advertisement • Advertise MAC (and IP) • Per MAC (and IP) MPLS/IP network, especially in large
Route (Type 2) reachability policy
• Advertise MAC/IP binding • ARP suppression scale virtualized server deployments?
• MAC Mobility • Workload Mobility

MAC3, IP3
PE1 PE3
MAC1, IP1
CE1 CE3
PE4 CE4

PE2 3. ARP Request (IP1)


4. ARP Reply (IP1)
Act as ARP
proxy for IP1.
• CE1 sends out an ARP request for CE3’s IP3
• PE1 snoops the ARP packet and learns (MAC1, IP1). It adds MAC1 to its MAC-VRF, MAC1/IP1 binding to its ARP
cache. It also advertises this binding to all other PEs in BGP and floods this initial ARP request.
• All other PEs learn of (MAC1, IP1). They add the MAC1 to their MAC-VRFs and add (MAC1, IP1) to their ARP cache.
• Now, when CE4 sends an ARP request for IP1, PE4 has the binding info and can provide an ARP response (e.g., ARP
proxy).

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
MAC Mobility
Challenge:
Route Type Usage Benefits How to handle MAC move ?
MAC/IP Advertisement • Advertise MAC (and IP) • Per MAC (and IP)
Route (Type 2) reachability policy
• Advertise MAC/IP binding • ARP suppression
• MAC Mobility • Workload Mobility PE1 PE3
MAC1, IP1

CE1 CE3
PE4
MAC1, IP1
PE2
• At T0, PE1 learn the MAC1, and advertise to all other PEs
• At T1, MAC1 move to the PE3. PE1 is not aware of this
• PE3 learn the MAC1. It will overwrite the MAC route learnt from PE1
• PE3 will advertise MAC1 to all other PEs with sequence number +1
• All other PE will overwrite the MAC route
• Original PE1 will withdraw its old route
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Split Horizon Filtering Challenge:
How to prevent flooded traffic from
echoing back to a multi-homed
Route Type Usage Benefits Ethernet Segment?
Ethernet A-D Route • Advertising Split-Horizon • Loop avoidance – ESI-1 ESI-2
(Type 1) Label even transient
• Aliasing • Efficient load CE1 PE1 PE3 CE3
• Mass Withdraw of • Fast convergence
addresses • balancing Echo !
CE5
• SH/AA MH Indication • Per-site policy
CE4 PE2 PE4

• PE advertises in EVPN Ethernet AS route with a split-horizon label (ESI MPLS Label) associated with
each multi-homed Ethernet Segment
• Split-horizon label is only used for multi-destination frames (Unknown Unicast, Multicast & Broadcast)
• When an ingress PE floods multi-destination traffic, it encodes the Split-Horizon label identifying the
source Ethernet Segment in the packet
• Egress PEs use this label to perform selective split-horizon filtering over the attachment circuit

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Challenge:
Aliasing How to load-balance traffic towards a multi-
homed device across multiple PEs when MAC
addresses are learnt by only a single PE?
Route Type Usage Benefits

Ethernet A-D Route • Advertising Split-Horizon • Loop avoidance – I can reach


(Type 1) Label even transient MAC1 via ESI1
• Aliasing • Efficient load I can
MAC1 ESI1 PE1
• Mass Withdraw of • Fast convergence reach ESI1
PE2
addresses • balancing (All-Active)

• SH/AA MH Indication • Per-site policy MAC1 PE1 PE3


MAC1 CE1
CE3

ESI-1 PE4 CE4


PE2 I can
reach ESI1
• PEs advertise in BGP the ESIs of local multi-homed Ethernet Segments. (All-Active)

• All-Active Redundancy Mode indicated

• When PE learns MAC address on its AC, it advertises the MAC in BGP along
with the ESI of the Ethernet Segment from which the MAC was learnt.
• Remote PEs can load-balance traffic to a given MAC address across all PEs
advertising the same ESI.

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Challenge:
MAC Mass Withdraw How to inform remote PEs of a failure
affecting many MAC addresses quickly while
the control-plane re-converges?
Route Type Usage Benefits

Ethernet A-D Route • Advertising Split-Horizon • Loop avoidance –


(Type 1) Label even transient
• Aliasing • Efficient load
MAC1, MAC2, .. MACn  ESI1  PE1

X
• Mass Withdraw of • Fast convergence I lost ESI1  PE2
addresses • balancing MAC1,
MAC2,… PE1 PE3
• SH/AA MH Indication • Per-site policy MACn
MAC1
CE1 CE3

ESI-1 PE4 CE4


PE2
• PEs advertise two sets of information:
• MAC addresses along with the ESI from the address was learnt
• Connectivity to ESI(s)

• If a PE detects a failure impacting an Ethernet Segment, it withdraws the route for the
associated ESI.
• Remote PEs remove failed PE from the path-list for all MAC addresses associated with an ESI.
• This effectively is a MAC ‘mass-withdraw’ function.
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
EVPN BGP Route 0x2 – MAC Advertisement
Unique per Advertising PE per EVI
ESI of Ethernet Segment on which MAC
Address was learnt. All 1s ESI for PBB-EVPN
8 bytes RD
Set to VLAN or I-SID for VLAN-Aware
10 bytes Ethernet Segment Identifier
Bundling Service interface, otherwise 0
4 bytes Ethernet Tag ID
Allows for MAC Address ‘summarization’, i.e.
1 byte MAC Address Length hierarchical MAC Addresses. Typically set to 48
6 bytes MAC Address
Could be C-MAC Address (EVPN) or B-MAC
1 byte IP Address Length Address (PBB-EVPN)
To distinguish IPv4 vs. IPv6 addresses.
4 or 16 IP Address

3 bytes Used for ARP flood suppression or for


MPLS Label1
Integrated Routing and Bridging (IRB).
3 bytes MPLS Label2
MAC & IP Labels - downstream assigned

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
MAC Mobility extended community
• Used to tag the MAC Advertisement route
• EVPN: Indicates that a MAC address has moved from one PE to another

0x06

0x00
Set to 0
2 bytes Reserved
Indicates the count of MAC address mobility
4 bytes Sequence Number events

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
EVPN BGP Route 0x1 – Ethernet Auto-discovery Route

This route has two flavors:


Per-EVI Ethernet A-D route
• Advertise VPN label used for Aliasing or Backup-Path

Unique per Advertising PE per EVI


RD ESI of Ethernet Segment
Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-Aware
Bundling Service interface, otherwise 0
Ethernet Tag ID

MPLS Label VPN (Aliasing) Label per (ESI,


Ethernet Tag)

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
EVPN Technology Prime

• EVPN Concepts
• Startup Sequence
• Operation
• A Day in Life of a Packet
Life of a Packet
Ingress Replication – Multi-destination Traffic Forwarding
PE1 receives broadcast Mcast MPLS
During start-up traffic from CE1. PE1 PSN MPLS label Label assigned by
sequence, PE1, PE2, forwards it using ingress to reach PE3 PE3 for incoming
PE3, PE4 sent Inclusive replication – 3 copies BUM traffic on a
Multicast route which created given EVI PE3 – as DF, it
include Mcast label
During start-up forwards BUM
sequence, PE2 sent Per- PE1 PE3 PE1 PE3 traffic towards
ESI Ethernet AD route segment
with ESI MPLS label VID 100
(split-horizon) (see SMAC: M1
below) DMAC: F.F.F L3

CE1 CE3 CE1 CE3


L2 L5
MPLS MPLS
PE 4 Inclusive Multicast
PE 2 Eth A-D Route (Per-ESI)
Route
L4
RD = RD20
RD = RD-4a
ESI = ESI1
PMSI Tunnel Attribute
ESI MPLS Label ext. comm.
Tunnel Type = Ing. Repl.
Redund. Flag = All-Active PE2 PE4 PE2 ESI (split-horizon) PE4
Label = L4 MPLS label
Label = L5 allocated by PE2
ESI MPLS Label – used by RT ext. community
PE2 – drops BUM PE4 – non-DF for
RT ext. community local PEs for split-horizon - for segment ES1
Mcast MPLS Label – used to RT-a traffic originated given EVI drops
RT-a, RT-b, RT-c, RT-d downstream assigned (for on ES1 BUM traffic
transmit BUM traffic -
ingress replication)
downstream assigned (for
ingress replication) BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Life of a Packet (cont.)
Unicast Traffic Forwarding
PE1 MAC Route
MP2P VPN Label – PE3 forwards
downstream allocated label RD = RD-1a
MAC advertised PSN MPLS label MP2P VPN Label traffic destined to
used by other PEs to send ESI = ESI1 assigned by PE1 M1 based on RIB
by route to reach PE1
traffic to advertised MAC for incoming traffic information (PE1)
MAC = M1
for the target EVI
Label = L1
RT ext. community
PE1 PE3 PE1 PE3
RT-a VID 100
SMAC: M2
VID 100 DMAC: M1
SMAC: M1
DMAC: F.F.F
L1
CE1 CE3 CE1 CE3

MPLS MPLS

PE2 PE4 PE2 PE4

PE3 RIB Path List


VPN MAC ESI NH
RT-a M1 ES1 PE1

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Life of a Packet (cont.)
Unicast Forwarding and Aliasing PE3 forwards PE3 forwards
traffic on a flow traffic on a flow
PE1 MAC Route MP2P VPN
MP2P VPN Label – (flow 1) based on (flow 2) based on
RD = RD-1a Label
downstream allocated label PSN MPLS label RIB information RIB information
MAC advertised assigned by
used by other PEs to send ESI = ESI1 to reach PE1 (towards PE1) (towards PE2)
by route PE1 for
traffic to advertised MAC
MAC = M1 incoming for
target EVI
Label = L1
During start-up
sequence, PE1 sent Per- RT ext. community VID 100
EVI Ethernet AD route PE1 PE3 PE1 PE3 SMAC: M3
RT-a
DMAC: M1
VID 100
VID 100 SMAC: M4
SMAC: M1 DMAC: M1
DMAC: F.F.F L1
CE1 CE3 CE1 CE3

During start-up sequence, MPLS MPLS


PE2 sent Per-EVI Ethernet L2
AD route (see below)

PE 2 Eth A-D Route (Per-EVI)


RD = RD-2a PE2 PE4 PE2 PE4
ESI = ESI1
Aliasing MPLS Label – used PE3, PE4 RIB Path List
Label = L2 Aliasing MPLS
by remote PEs to load- VPN MAC ESI NH PSN MPLS label Label assigned by
RT ext. community balance among local PEs to reach PE2 PE2 for (ES1, EVI)
RT-a M1 ES1 PE1
RT-a pair
PE2
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
E-VPN Operational Scenarios
PE3 MAC Route
MAC Mobility PE1 MAC Route RD = RD-3a
RD = RD-1a ESI = ESI2
1 4
ESI = ESI1 3 MAC = M1
PE1 advertises MAC Host M1 moves After host sends traffic
route for M1. Route MAC = M1 Label = L3
from CE1 to CE3’s at new location, PE2
may include MAC Label = L1 location MAC Mobility ext. now adv MAC route
mobility community community. for M1 incrementing
MAC Mobility ext.
community sequence # in MAC
Seq. Num = 2
mobility community
PE1 Seq. Num = 1 PE3 PE1 RT ext. community PE3
RT ext. community RT-a VID 100
VID 100 SMAC: M1
SMAC: M1
RT-a
DMAC: M2
DMAC: M2

CE1 CE3 CE1 CE3


M1
MPLS M1 MPLS M1
5
PE1 withdraws its
2 M1 route and
PE3 / PE4 install installs a new one
M1 route towards pointing to PE3
PE2 PE4 PE1 PE2 PE4

PE3 / PE4 RIB Path List PE1 / PE2 RIB Path List
VPN MAC ESI NH VPN MAC ESI NH
RT-a M1 ES1 PE1 RT-a M1 ES2 PE3

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
E-VPN Failure Scenarios / Convergence
Link / Segment Failure – Active/Active per Flow

2 7
PE1 withdraws
PE1 withdraws Per-ESI individual MAC
Ethernet AD route for advertisement routes
1 failed segment related to failed
PE1 detects failure segment
of one of its
attached segments PE1 PE3 PE1 PE3

CE1 CE3 CE1 CE3

3 MPLS 4 MPLS
PE1 withdraws Ethernet Mass withdrawal - PE3
Segment Route / PE4 remove PE1 from
path list for all MAC
addresses of failed
segment (ES1)
5 PE2 PE4 6 PE2 PE4
PE2 recalculates
DF/BDF. Becomes DF PE3, PE4 RIB Path List PE2 adv. M1 MAC route PE3, PE4 RIB Path List
for all EVIs on segment VPN MAC ESI NH after CE traffic is VPN MAC ESI NH
hashed towards PE2
RT-a M1 ES1 PE1 RT-a M1 ES1 PE2
PE2
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
E-VPN Failure Scenarios / Convergence
PE Failure

2
BGP RR / PE3 detects
1 BGP session time-out
PE1 experiences a with PE1
node failure (e.g.
power failure) PE1 PE3 PE1 PE3

CE1 CE3 CE1 CE3

MPLS MPLS
6
2 3 PE3 / PE4 will forward
2 BGP RR / PE4 PE3 / PE4 invalidate M1 traffic towards PE2
BGP RR / PE2 detects detects BGP routes from PE1
BGP session time-out session time-
with PE1
PE2 out with PE1 PE4
5 PE2 PE4
PE2 adv. M1 MAC route
4 PE3, PE4 RIB Path List after CE traffic is PE3, PE4 RIB Path List
VPN MAC ESI NH hashed towards PE2 VPN MAC ESI NH
PE2 reruns DF election.
Becomes DF for all RT-a M1 ES1 PE1
PE1 RT-a M1 ES1 PE2
EVIs on segment
PE2
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
DC Fabric Evolution w/ EVPN-IRB
The Evolution of the DC Fabric

L2 Overlay over IP L2/L3 Overlay over IP Fabric:


Fabric:
ASR9K) EVPN-IRB w/ VxlAN encap
VXLAN, DP Learning • Spine-leaf
L2 Fabric: • Virtual L2 & L3 overlay across
FabricPath/Trill physical boundary
• VXLAN: Ultra-high scale
L2 Fabric: Legacy VLAN, STP • Efficient forwarding: L3 ECMPs
• L2/L3 boundary: limited • EVPN control plane
mobility • Policy-based forwarding
• 4K VLAN: Limited scale
• Inefficient forwarding: STP
• Complex VLAN provisioning
• Vendor specific L2
enhancement

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Centralized v.s. Distributed Routing
Distributed Routing Centralized Routing

Boarder L3 Centralized GW
Leaf
L2

Fabric Fabric
L3
Leaf Leaf
L2

Subnet 1/VLAN1 Subnet 2/VLAN2 Subnet 1/VLAN1 Subnet 2/VLAN2

• Optimized forwarding of east-west traffic • All east<->west routed traffic traverses to centralized gateways
• ARP/MAC state localized to Leafs • Centralized gateways have full ARP/MAC state in the DC
• Helps with horizontal scaling of DC • Scale challenge

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Distributed Routing: EVPN IRB
Symmetric IRB Asymmetric IRB

Boarder Boarder
Leaf Leaf

Fabric Fabric
IP-VRF IP-VRF
IP-VRF IP-VRF
Leaf Leaf
MAC- MAC- MAC- MAC-
MAC- MAC- MAC- MAC-
VRF VRF VRF VRF
VRF VRF VRF VRF

• Ingress Leaf needs ARP/MAC state for every egress leaf


• ARP/MAC state localized to Leafs
• Limits scale
• Helps with horizontal scaling of DC
• Inefficient encap: requires Ethernet header for inter-
• More efficient encapsulation for MPLS and
subnet IP routing
VxLAN with GPE
• Asymmetric processing: Imposition PE performs more
• Symmetric processing
work and lookups than disposition PE

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
EVPN IRB: Distributed Anycast Gateway at Leafs
The same anycast gateway virtual IP
address and MAC address are
configured on all VTEPs in the VNI.

Leafs

Host-1 Host-2 Host-6 Host-4 Host-5


IP-H1, MAC-H1 IP-H2, MAC-H2 IP-H6, MAC-H6 IP-H1, MAC-H1 IP-H2, MAC-H2
Bridge-domain 1 Bridge-domain 2 Bridge-domain 3 Bridge-domain 1 Bridge-domain 2

Mobility zone

Tenant A:
• Distributed anycast default gateway for hosts
IP-VRF Blue • IRB provides intra and inter-subnet forwarding
Bridge domain-1
Bridge domain-2
Bridge domain-3
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
EVPN IRB Intra-Subnet: Control Plane Operation
Intra-subnet/bridged traffic between two remote end hosts : host-1 to host-3

DCI
Add IP-H3 to IP-VRF table. 3
Learn IP-
H3/MAC-H3
MAC Route:
R Spine binding via
2 R
RD: VPN RD ARP snooping
IP: IP-H3 1 on bridge port
MAC: MAC-H3
Label1: MAC VRF Label
Label2: IP VRF Label Leaf
Ext. Community/Attributes:
Route Target: MAC-VRF GARP
Route Target : IP-VRF 1
Next Hop: IP-L3 3
3
Host-1 Host-2 Host-3 Host-4
IP-H1, MAC-H1 IP-H2, MAC-H2 IP-H3, MAC-H3 IP-H4, MAC-H4
BD-1 BD-2 BD-1 BD-2

1. Add MAC-H3 to MAC table: MAC-H3 via IP-L3, Label1.


2. Add {IP-H3, MAC-H3} to ARP suppression cache Add IP-H3 to IP VRF table: IP-H3 via IP-L3, Label2
3. Add IP route: IP-H3 via IP-L3, Encap: Label2 L4: IP VRF Table
L1: ARP Suppression
L1: MAC Table L1: IP VRF Table
Cache IP-H3-> IP-L3, Label2
MAC-H3 -> IP-L3, Label1 IP-H3-> IP-L3, Label2
IP-H3 -> MAC-3

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
EVPN IRB Intra-Subnet: Packet Flow (1)
Intra-subnet/bridged traffic between two remote end hosts : host-1 to host-3

DCI
1. Host-1 sends an ARP request for
Host-3 , with target IP-H3.
Spine
2. The ARP request is intercepted at the
L1: ARP Suppression Cache
leaf L1 and punted to the control
IP-H3 -> MAC-3
plane.
2 CPU Leaf
3. L1 looks up IP-H3 in ARP 1
suppression cache, if an entry is
found, ARP response is locally 3
generated with the actual host MAC Host-3
Host-1 H1 ARP Cache IP-H3, MAC-H3
address MAC-H3. If the entry is not BD-1
IP-H1, MAC-H1 IP-H3 -> MAC-3
found in the ARP suppression cache, BD-1
then ARP is flooded in the bridge
domain.

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
EVPN IRB Intra-Subnet: Packet Flow (2)
Intra-subnet/bridged traffic between two remote end hosts : host-1 to host-3

4. Host-1 sends data packet with MAC-H3 DCI


as the destination MAC.
5. L1 receives the packet, does a MAC
DMAC: MAC--H3
lookup in MAC table for MAC-H3. It finds 7
Spine
an entry for MAC-H3 pointing to IP-L3 SMAC: MAC-H1
with Label1 as encap. DIP: IP-H3 6
8
6. L1 encapsulates packet with MPLS SIP: IP-H1 5 Leaf
label1, and a transport label
corresponding to the IP-L3.
7. Spines forward the MPLS packet based 4 9
on the transport label towards L3.
DMAC: MAC-H3
8. L3 receives MPLS packet, uses Label1 to Host-1
IP-H1, MAC-H1 SMAC: MAC-H1 Host-3
determine the MAC table to do the inner BD-1 IP-H3, MAC-H3
DMAC lookup and finds the egress bridge DIP: IP-H3 BD-1
port to forward the packet.
SIP: IP-H1
9. Host-3 receives data packet with MAC-H3
as the destination MAC and consumes it.

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
EVPN IRB Inter-Subnet: Control Plane
Inter-subnet routed traffic between two remote end hosts : host-1 to host-4

DCI
1. Add IP-H4 to IP VRF table. 3
Learn IP-
H4/MAC-H4
MAC Route: RR Spine binding via
ARP snooping
RD: VPN RD 2 on bridge port
IP: IP-H4 1
MAC: MAC-H4
Label1: MAC VRF Label
Label2: IP VRF Label
Ext. Community/Attributes: Leaf
Route Target: MAC-VRF
Route Target : IP-VRF GARP
Next Hop: IP-L4
1
3
Host-1 Host-2 Host-3 Host-4
IP-H1, MAC-H1 IP-H2, MAC-H2 IP-H3, MAC-H3 IP-H4, MAC-H4
BD-1 BD-2 BD-1 BD-2

1. Add MAC-H4 to MAC table: MAC-H4 via IP-L4, Label1.


2. Add {IP-H4, MAC-H4} to ARP suppression cache
3. Add IP route: IP-H4 via IP-L4, Encap: Label2

L1: MAC Table L1: ARP Suppression Cache L4: IP VRF Table

MAC-H4 -> IP-L4, Label1 IP-H4 -> MAC-4 IP-H4-> IP-L4, Label2

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Inter-Subnet: Packet Flow (1)
Inter-subnet traffic between two remote end hosts : host-1 to host-4

1. Host-1 sends an ARP request for its default DCI


gateway, with target as its gateway IP
address (BVI1’s IP address : IP-BVI1).
2. The ARP request is received at the ToR L1 Spine
and punted to the control plane.
L3
3. Target IP address in ARP request is the BVI-

BVI’s IP address, L1 sends ARP reply with L2 2 CPU 1


BVI
-2 Leaf
1
MAC-BVI1 as the MAC address.
3

Host-1 Host-4
IP-H1, MAC-H1 IP-H4, MAC-H4
BD-1 BD-2
H1 ARP Cache

IP-BVI1 -> MAC-BVI-1

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
EVPN IRB Inter-Subnet: Packet Flow (2)
Inter-subnet traffic between two remote end hosts : host-1 to host-4
4. Host-1 sends data packet with MAC-BVI1 as
the destination MAC and IP-H4 as the DCI
destination IP address.
7
5. L1 receives the packet, since the dMAC is its
BVI MAC, it does IP lookup and finds route to Spine
IP-H4, that points to IP-L4 as next hop and 8
6
Label2 as the VPN label. DMAC: MAC-BVI1
6. L1 encapsulates packet with MPLS header SMAC: MAC-H1 Leaf
DIP: IP-H4 5
with Label2 as the VPN label and outer label SIP: IP-H1
as the transport label corresponding to IP-L4.
9
7. Resulting packet is forwarded towards L4. 4
DMAC: MAC-H4
SMAC: MAC-BVI2
8. L4 receives MPLS packet, uses Label2 to Host-1
DIP: IP-H4
IP-H1, MAC-H1 Host-4
determine the VPN table to do the inner IP SIP: IP-H1 IP-H4, MAC-H4
BD-1
lookup, finds the egress BVI to forward the BD-2
packet.
9. Host-H4 receives data packet with MAC-H4 as
the destination MAC and consumes it.

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
EVPN IRB Inter-Subnet: Control Plane
Inter-subnet traffic between two local end hosts : host-1 to host-2

1. Host-1 sends an ARP request for its default DCI


gateway, with target as its gateway IP
address (BVI1’s IP address : IP-BVI1).
2. The ARP request is received at the ToR L1 Spine
and punted to the control plane.
3. Target IP address in ARP request is the 2
BVI’s IP address, L1 sends ARP reply with
BVI- BVI
1
CPU -2 Leaf
MAC-BVI1 as the MAC address for the 1
gateway IP.
3

Host-1 Host-2
IP-H1, MAC-H1 IP-H2, MAC-H2
BD-1 BD-2
H1 ARP Cache

IP-BVI1 -> MAC-BVI-1

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
EVPN IRB Inter-Subnet: Packet Flow
Inter-subnet traffic between two local end hosts : host-1 to host-2
1. Host-1 sends data packet with MAC-
BVI1 as the destination MAC and IP-H2 DCI
as the destination IP address.
2. L1 receives the packet, since the dMAC Spine
DMAC: MAC-BVI1
is its BVI MAC, it does IP lookup and
finds route to IP-H2, that points to BVI2 SMAC: MAC-H1
5
as egress interface. And routes it DIP: IP-H2
locally, there is no MPLS encapsulation BVI- BVI
-2
Leaf
SIP: IP-H1 1

needed. It is possible that the ARP for


6
IP-H2 is not resolved, in which case an
4
ARP request is triggered on BVI2 to DMAC: MAC-H2
resolve it. Host-1 Host-2 SMAC: MAC BVI2
IP-H1, MAC-H1 IP-H2, MAC-H2
3. Host-H4 receives data packet with BD-1 BD-2 DIP: IP-H2
MAC-H4 as the destination MAC and SIP: IP-H1
consumes it.

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
EVPN-VPWS
EVPN-VPWS Does it Better than Legacy VPWS !!

Service Additional Capabilities


E-Line • All-active & single-active multi-homing support
• Both single-segment & multi-segment support
• Discovery & signaling via single protocol – BGP

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
EVPN BGP route type
Route type Usage EVPN EVPN VPWS
0x1 Ethernet Auto-Discovery • MAC Mass-Withdraw
(A-D) Route • Aliasing (load balancing)  
• Split-Horizon
“Tagged with ESI Label Extended Community”
0x2 MAC Advertisement Route • Advertise MAC addresses
• Provide MAC / IP address bindings for ARP  NOT used
broadcast suppression
“Tagged with MAC Mobility Extended
Community”
0x3 Inclusive Multicast Route • Multicast tunnels used to transport
Broadcast, Multicast and Unknown Unicast NOT used
frames (BUM) 
“Tagged with PMSI tunnel attribute” (P tunnel
type & ID) – RFC6514
0x4 Ethernet Segment Route • Auto discovery of Multi-homed Ethernet
Segments, i.e. redundancy group discovery
• Designated Forwarder (DF) Election  
“Tagged with ES-Import Extended Community”

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
EVPN BGP Extended Community

Attribute Usage Tagged BGP EVPN EVPN VPWS


route

ESI label Extended • Split-Horizon for Ethernet Ethernet A-D


Community Segment. Route
• Indicate Redundancy Mode  
(Single Active vs. All-Active)
ES-Import Extended • Limit the import scope of the Ethernet
Community Ethernet Segment routes. Segment Route  
MAC Mobility Extended • E-VPN: Indicate that a MAC MAC
Community address has moved from one Advertisement  Not used
segment to another across PEs. Route
• PBB-EVPN: Signal C-MAC
address flush notification

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
EVPN VPWS
Control-plane
attachment circuit
• Benefits of EVPN applied to point-to-point advertisement over the
services Core

• No signaling of PWs. Instead signals MP2P LSPs


instead (ala L3VPN) VPWS Service Config:
EVI = 100
• All-active CE multi-homing (per-flow LB) Local AC ID = AC2
PE1 PE2 Remote AC ID = AC1
• Single-active CE multi-homing (per-service LB)
CE1 CE2

• Relies on a sub-set of EVPN routes to


ES1 MPLS ES2
advertise Ethernet Segment and AC
reachability VPWS Service Config:
• PE discovery & signaling via a single protocol – EVI = 100
Local AC ID = AC1
BGP
Remote AC ID = AC2
• Per-EVI Ethernet Auto-Discovery route
• Handles double-sided provisioning with remote PE
auto-discovery
I have a P2P service that BGP Eth. Auto-
needs to communicate Discovery Route
• Under standardization: draft-ietf-bess-evpn- with the PE(s) that own EVPN NLRI
vpws of AC = AC2 AC AC1 via PE1

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
EVPN VPWS Operation – Single-homed
ESI – 10 bytes ESI as specify
by EVPN Ethernet segment
IETF draft – zero for single- 3
homed RD – RD unique per adv. PE
PE 1 Eth A-D Route
per EVI
RD = RD-1a
Eth.Tag ID – 4-bytes local
ESI = ES1 (0)
AC-ID
Eth.Tag ID = AC1 RT – RT associated with a
given EVI
Label (e.g. X)
MPLS Label – (downstream
assigned) used by remote RT ext. community VPWS Service Config:
PEs to reach segment EVI = 100
RT-a
Local AC ID = AC2
PE1 PE2 Remote AC ID = AC1 ES2 – Since CE2 is single
CE1 CE2 homed to PE2, ES2 = 0

1 ES1 MPLS ES2


2
VPWS Service Config:
EVI = 100
PE 2 Eth A-D Route
Local AC ID = AC1
Remote AC ID = AC2 RD = RD-2a
5 ESI = ES2 (0) 6
PE1 RIB Path List Path List
Eth.Tag ID = AC2 PE2 RIB
NH
VPN MAC ESI Eth.TAG Label (e.g. Y) VPN MAC ESI Eth.TAG
NH
PE2
RT ext. community PE1
RT-a - 0 AC2 4 RT-a - 0 AC1
RT-a

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
EVPN VPWS Operation – Single-active
PE 1 Eth A-D Route RD – RD unique per adv. PE
ESI – 10 bytes ESI as specify
per EVI
by EVPN Ethernet segment RD = RD-1a
IETF draft
ESI = ES1
3
Eth.Tag ID = AC1
Eth.Tag ID – 4-bytes local Label (e.g. X) RT – RT associated with a
AC-ID given EVI
RT ext. community
RT-a
MPLS Label – (downstream
assigned) used by remote PE1 VPWS Service Config: Only one PE (PE1)
PEs to reach segment EVI = 100 shows as next hop for
Single-Active == per-vlan
Local AC ID = AC2 the remote AC
load-balancing CE-PEs
ES1 PE3 Remote AC ID = AC1
Two bundles on CE VPWS Service Config:
CE1 CE2
device EVI = 100
Local AC ID = AC1 ES2 – Since CE2 is single
Remote AC ID = AC2 MPLS ES2
2 homed to PE2, ES2 = 0

1
VPWS Service Config: ES1
EVI = 100 PE 3 Eth A-D Route 6
Local AC ID = AC1 RD = RD-2a
PE2
5 Remote AC ID = AC2
ESI = ES2 (0)
PE3 RIB
Path List
PE1 & PE2 RIB Path List VPN MAC ESI Eth.TAG
Eth.Tag ID = AC2 NH
NH
VPN MAC ESI Eth.TAG Label (e.g. Y) RT-a - ES1 PE1
PE3
RT ext. community
RT-a - 0 AC2 4 RT-a - ES1 PE2
RT-a RT-a - ES1 AC1 PE1

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
EVPN VPWS Operation – All-active
PE 1 Eth A-D Route RD – RD unique per adv. PE
ESI – 10 bytes ESI as specify
per EVI
by EVPN Ethernet segment RD = RD-1a
IETF draft
ESI = ES1
3
Eth.Tag ID = AC1
Eth.Tag ID – 4-bytes local Label (e.g. X) RT – RT associated with a
AC-ID given EVI
RT ext. community
RT-a
MPLS Label – (downstream
assigned) used by remote PE1 VPWS Service Config: Both PEs (PE1/PE2)
PEs to reach segment EVI = 100 shows as next hop for
ALL-Active == per-flow
Local AC ID = AC2 the remote AC
load-baancing CE-PEs
ES1 PE3 Remote AC ID = AC1
Single bundle on CE VPWS Service Config:
CE1 CE2
device EVI = 100
ES2 – Since CE2 is single
Local AC ID = AC1
Remote AC ID = AC2 MPLS ES2
2 homed to PE2, ES2 = 0

1
VPWS Service Config: ES1
EVI = 100 PE 3 Eth A-D Route 6
Local AC ID = AC1 RD = RD-2a PE3 RIB
PE2
5 Remote AC ID = AC2
ESI = ES2 (0)
Path List
VPN MAC ESI Eth.TAG
PE1 & PE2 RIB Path List NH
Eth.Tag ID = AC2
NH RT-a - ES1 PE1
VPN MAC ESI Eth.TAG Label (e.g. Y)
PE3 RT-a - ES1 PE2
RT ext. community
RT-a - 0 AC2 4 RT-a - ES1 AC1 PE1,PE2
RT-a

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
EVPN Deployment:
DC Fabric and WAN Integration
The Solution Must be End-to-End

• Legacy 802.1q handoff


APIC • It means multi-pathing is out the
door between DC & WAN
• VLANs and sub-interfaces
DC creation
• No policy level integration
• Small FIB/MAC table size on
Spine
border Leaf, create bottleneck
WAN/DCI

Leaf Leaf bLeaf bLeaf

802.1q?
SDN-DC, VXLAN overlay SDN-WAN, MPLS/Segment Routing

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
DC Gateway – Seamless DC and WAN Integration
• L3 Gateway
• L2 Gateway
• IRB with Anycast
Gateway
• Common MP-BGP (EVPN AF) control plane
• VXLAN to MPLS data plane interworking
APIC • SDN based auto provisioning: OpFlex,
APIC/VTS
DC • Integrated Policy control: WAN optimization

Spine

WAN/DCI

Leaf Leaf bLeaf bLeaf

Integration
SDN-DC, VXLAN overlay Interworking SDN-WAN, MPLS/SR

Scalable, Resilient, Optimized, End-to-End © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
DC Gateway – L3 Gateway • EVPN/VXLAN to IP-VPN/MPLS interworking
• EVPN/VXLAN to global internet
S-N Routing: all active • L3 EVPN between Leaf and GW (per-vrf VNI)
• Leaf is the L3 default gateway for VMs (with
EVPN IRB anycast GW) and does inter-vxlan
routing

Internet VPN client


Client Client
Internet
Branch IP-VPN
Internet

DC DC-1 Gateway
WAN DC-2
Gateway

Spine L3 EVPN
(per-VRF VNI)

Leaf
Leaf
(L3 anycast
VM1 VM2 L2/L3 DC fabric L2/L3 DC fabric GW)
VM3 VM4

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
DC Gateway – L2 Gateway • L2 EVPN/VXLAN in the DC
L2 Stretch: E-W and S-N • L2 DCI (E-W): EVPN/VXLAN with EVPN/VPLS
all-active or single-active interworking
• L2 to client (S-N): EVPN/VXLAN with VPLS/PW
interworking

VPN client
Client

Branch
VPLS/PW/EV
PN

DC Gateway DC-1 Gateway


WAN DC-2

Spine
L2 EVPN
(per-BD VNI)

Leaf
Leaf
VM1 VM2
L2/L3 DC fabric L2/L3 DC fabric VM3 VM4

Leaf Gateway Gateway


Leaf
L2 EVPN/VXLAN L2 stretch: EVPN/VPLS (MPLS) L2 ©EVPN/VXLAN
BRKMPL-2333 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
DC Gateway – IRB Anycast Gateway
Integrated Routing and Bridging • DC fabric is L2 only. All routing on DC gateway
• DC gateway is the L3 default gateway for VMs via
EVPN IRB anycast gateway
• Support both L2 and L3 for the same VNI at the
same time

Internet VPN client


Client Client
Internet IP-VPN
Branch
Internet
VPLS/PW/EVP
IRB N
DC DC-1 IRB IRB Gateway
IRB DC-2
Gateway WAN (L3 anycast GW)

Spine L2 EVPN
(per-BD VNI)

Leaf Leaf
VM1 VM2 L2 only DC fabric (L2 only)
L2 only DC fabric VM3 VM4

Leaf Gateway Gateway


Leaf
L2 EVPN/VXLAN L2 stretch: EVPN/VPLS (MPLS) L2 ©EVPN/VXLAN
2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Anycast Gateway Deployment Options
Distributed vs. Centralized Anycast Gateway

client
Internet VPN client
Client Client
IP-VPN
Internet IP-VPN
Internet Branc
VPLS/PW/EVP
Centralized Internet
h VPLS/PW/EVP
N Anycast GW
N
DC Gateway IRB
(IRB anycast GW) DC-1 DC Gateway
IRB DC-2
WAN (L3 GW)

L2 EVPN
(per-BD VNI) L3 EVPN
(per-VRF VNI)

Leaf IRB IRB IRB IRB


Leafonly)
(L2 IRB IRB Leaf
VM1 VM2 L2 only DC fabric L2/L3 DC fabric (IRB anycast GW)
VM3 VM4 Distributed
Anycast GW
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
IRB Anycast Gateway Deployment Option
Comparison
Option 1 Option 2
Distributed IRB Anycast Gateway Centralized IRB Anycast Gateway

DC Gateway Router • L2 or L3 gateway function • EVPN IRB for integrated L2 and L3


• Doesn’t require IRB function • IRB anycast gateway for VM’s default
gateway

Leaf • EVPN IRB for integrated L2 and L3 • L2 EVPN peering across DC


• IRB anycast gateway for VM’s • Cross-DC underlay IP routing is required
default gateway

Pros • Optimized E-W inter-vxlan routing • Simple DC fabric design: L2 only


• Large ARP table on the DC gateway
router

Cons • EVPN IRB function across all Leaf • Sub-optimal E-W inter-vxlan routing
nodes
• Require both L2 and L3 function on
the leaf

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
DC Gateway – the Policy Integration and Auto-
Provisioning: Application-engineered Routing
DC: classification PBTS: steering packet to
and marking SR-TE in the WAN

DC policy domain WAN policy domain DC policy domain

WAN Segment
Routing
VTS/APIC VTS/APIC

DC DC-1
WAN DC-2
Gateway

Spine

Leaf
VM1 VM2
VM3 VM4

High bandwidth flow


Low latency flow 89
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example: Application has a preference for disjoint path in dual-plane
WAN networks

ACI Fabric

Customer has the requirement that traffic from


WEB WEB applications RED and BLUE should be
Policy

transported across disjoint paths in the WAN.


Segment - Policy expressed on APIC and delivered by
Routing SR-enabled +WAE WAN
WAN
with WAE

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Example: Application requires the lowest latency path in the WAN

ACI Fabric

WAN has cheap capacity via US with higher latency.


WEB WEB
Scarce, expensive capacity via Russia, with lower
Policy

latency.
Tokyo
Segment
Routing Russia Customer identify the applications that require the
US lowest possible latency path on APIC, integration
WAN steers traffic on the path via Russia.
with WAE

Brussels

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Summary
Service Provider Data Center
Next gen all-in-one VPN Industry defacto-standard multi-
solution to provide all the tenancy solution that provides
services previously done by flexible workload placement,
multiple solutions and more workload mobility, full cross-
sectional BW utilization and
• L2VPN (P2P, MP, P2MP) elasticity.
• L3VPN
• IRB • L2, L3, L2+L3 Overlay
• DCI

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
References
• [FabricPath]: FabricPath http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html

• [LISP]: Locator/ID Separation Protocol https://datatracker.ietf.org/wg/lisp/charter/

• [802.1Qbp] ECMP http://www.ieee802.org/1/files/public/docs2011/new-ashwood-sajassi-ecmp-par-0111-v04.pdf

• [EVPN]: BGP MPLS Based Ethernet VPN http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04

• [TRILL]: Transparent Interconnection of Lots of Links https://datatracker.ietf.org/wg/trill/charter/


http://tools.ietf.org/wg/trill/draft-ietf-trill-rbridge-protocol/

• [VL2]: VL2: A Scalable and Flexible Data Center Network http://ccr.sigcomm.org/online/?q=node/502

• [MOOSE]: Addressing the Scalability of Ethernet with MOOSE http://www.cl.cam.ac.uk/~mas90/MOOSE/MOOSE.pdf

• [PORTLAND]: PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric http://ccr.sigcomm.org/online/?q=node/503

• [SEATTLE]: Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises


http://www.cs.princeton.edu/~chkim/Research/SEATTLE/seattle.pdf

• [MONSOON]: Towards a Next Generation Data Center Architecture: Scalability and Commoditization
http://research.microsoft.com/apps/pubs/default.aspx?id=79348

• [VLB]: Valiant Load Balancing in Backbone Networks http://www.stanford.edu/~ashishg/network-algorithms/rui.pdf

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.

Don’t forget: Cisco Live sessions will be


available for viewing on demand after the
event at www.CiscoLive.com/Online.

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions

BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Thank you