BRKMPL 2333

EVPN: Network
Virtualization Solution
for Next Generation
DCs & DC Interconnect
Ali Sajassi– Distinguished Engineer, Cisco

BRKMPL-2333
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
Cisco Spark spaces will be cs.co/ciscolivebot#BRKMPL-2333

available until July 3, 2017.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• EVPN Overview
• Why EVPN
• EVPN Technology Prime
• EVPN Startup Sequence
• EVPN Operation
• A Day in Life of a Packet
• DC Fabric Evolution w/ EVPN-IRB

• EVPN-VPWS
• EVPN Deployment: DC Fabric and
WAN Integration
EVPN Overview
Service Provider Data Center
Next gen all-in-one VPN Industry defacto-standard multi-
solution to provide all the tenancy solution that provides
services previously done by flexible workload placement,
multiple solutions and more workload mobility, full cross-
sectional BW utilization and
• L2VPN (P2P, MP, P2MP) elasticity.
• L3VPN
• IRB • L2, L3, L2+L3 Overlay
• DCI
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Underlay v.s. Overlay
Spines
EVI 20
EVI 10
Leafs
• Underlay: physical topology + • Overlay: Virtual topology among

associated routing/signaling protocol endpoints (NVEs)
protocol • Deals with reachability between tenant
• High redundancy in fabric (link/node) routes and MAC addresses
• IP underlay: IGP (or BGP) • EVPN uses BGP as routing protocol for
• MPLS underlay: LDP, RSVP-TE, SR distribution of tenants IP
• Leaf/Spine (CLOS) Topology prefixes/addresses and MACs
• Uses IP or MPLS as underlay
EVPN in a Nut Shell
MAC & IP learning in control plane (via BGP)
MAC Routing: Control plane (BGP) Optimum forwarding,

advertise the learnt MACs from CE
ECMP, Multi-pathing
PE1 PE3 Common L2/L3 VPN
All active Single active Operational Mode
multi-homing multi-homing
Same operational
CE1 CE3
principles as IP-VPN
C-MAC: M1 IP or MPLS
Extensive multi-homing
capabilites
PE2 PE4 Flexible Policy Control
EVPN Family of Solutions RFC 7432
EVPN
P2P Multipoint
EVPN-VPWS EVPN-FXC
RFC 7623
EVPN-IRB EVPN-Overlay EVPN-Etree PBB-EVPN
draft-ietf-bess-evpn-inter-subnet-forwarding draft-ietf-bess-evpn-overlay draft-ietf-bess-evpn-etree

draft-ietf-bess-evpn-prefix-advertisement
What’s the big deal about EVPN?
EVPN is next generation all-in-one VPN solution
It not only does the job of many other VPN technologies but it does it better !!
E-LAN E-LINE E-TREE DC Fabric IRB DCI
(MP2MP (P2P (P2MP L3VPN (IntraDC (L2/L3 (InterDC)
L2VPN) L2VPN) L2VPN) Overlay) Overlay)
VPLS- VxLAN
VPLS PW 4364 VPLS,OTV
ETREE TRILL
EVPN
PBB- EVPN EVPN EVPN- EVPN- EVPN- EVPN-

EVPN VPWS ETREE L3VPN Overlay IRB DCI
Example: EVPN Does it Better than VPLS !!
Service Additional Capabilities

E-LAN • Provides All-Active multi-homing
• Prevents loop for both all-active & single-active even in transient
state
• Efficient utilization of network cross-sectional bandwidth (via
optimum forwarding, ECMP, multi-pathing on a per flow basis)
• Flexible policy control per MAC and per Site
EVPN – Control and Data plane
EVPN
Control-
(MP-BGP)
Plane
RFC7432
Network Virtualization Overlay

Multi-Protocol Label Switching Provider Backbone Bridges
(VXLAN, NVGRE, Geneve,
(MPLS) (PBB+MPLS)
RFC7432 RFC7623
GUE, GPE)
Data- draft-ietf-bess-evpn-overlay
Plane
LDP, RSVP-TE, or SR IPv4/IPv6, or SRv6
2006 2010 2011 2013 2015 +
Following drafts were Following drafts were Enhancements

- OPEN project was started at - Introduced to IETF as introduced: introduced - Virtual ES
Cisco Routed-VPLS - EVPN - EVPN IRB - Optimized ingress replication
- OPEN = Optimum Ethernet - Merged with Juniper’s - PBB-EVPN - EVPN DCI - IGMP aggregation between
Network MAC-VPN and was - EVPN-VPWS PODs
introduced an EVPN - EVPN-Overlay - mcast tunnels between DCs
- EVPN-ETREE - Inter-AS for IRB
- L3VPN multi-homing
- EVPN-IRB Multicast
- DF Election algorithmes
- Mobility Enhancements
- FXC
- VPLS seamless integration for
all-active multi-homing
- Etc.
Why EVPN ?
Today’s DC Requirements
• Efficient and Flexible Multi-Homing
• Efficient fabric bandwidth utilization
• Flexible workload placement
• Seamless workload mobility: inter-POD, intra-
DC, inter-DC
• High scale multi-tenancy
• Optimal forwarding of intra & inter-subnet
traffic
• Traffic engineering (SPDC)
• Seamless integration w/ existing L2 and
L3VPN solutions
Efficient and Flexible Multi-Homing
Prior to EVPN EVPN
CE1 PE1 CE1 PE1
CE2 PE2 CE2 PE2

CE3
PE3
CE3 PE3
CE4
PE4
CE4 PE4
• N-way redundancy
• Limited to Dual-homing
• No inter-chassis links + fast convergence – enables geo-
• Increased cost because of inter-chassis links redundancy
• Inflexible configuration • Works for both MPLS & IP transport
• Proprietary (vPC, Virtual Switch, Cluster) • Standard based with multitude of applications: SP access, SP
inter-domain, intra-DC, and inter-DC
Efficient Fabric Bandwidth Utilization
Prior to EVPN EVPN
MAC1 CE1 CE2 MAC2

MAC1 CE1 CE2 MAC2 PE1 PE3
PE1 PE3
PE2 PE4
PE2 PE4
• Enabling multiple paths through the fabric between a pair of

• Limited number of paths among workloads via the fabric
workloads with each path having multiple ECMP itself
• For MPLS/IP fabric, data plane & control plane Scale
• MAC learning in BGP -> removing PWs & targeted LDP sessions
issues: O(n^2) full-mesh of PWs, learning every MAC
over PW, full-mesh of targeted LDP sessions • Enabling policy for individual MACs & ingress filtering
• No MAC-based policy • Provides Aliasing for multiple-path enablement even in the
absence of MAC learning on a PE!!
• Single-active multi-homing support only
• Simplifying operation via autosensing & auto-config of ES
• Enabling path selection

Flexible Workload Placement & Seamless Mobility
EVPN
Prior to EVPN SDN
Controller
Spine Spine
Core Core
TOR1 TOR2 TOR3 TOR4
SW1 SW2 SW3 SW4 App

VNF VNF
App
VNF VNF
VNF VNF App
• Enable flexible and efficient workload placement and

• L2 fabric with centralized routing mobility across the whole fabric
– Inefficient L2 forwarding & LB in the fabric (MSTP)
• Distributed Anycast GW
– Inefficient L3 forwarding
- Optimum L2 & L3 forwarding
• Sub optimal application performance (flooding of traffic,
delayed due to centralized routing) • Consistent and seamless solution for intra-POD, inter-
POD, inter-DC, inter-domain (SDN vs. virtual vs physical)
• Along w/ n-way active/active redundancy
High Scale Multi-Tenancy & Optimum Forwarding
Centralized Routing Distributed Routing
L3 Centralized GW Boarder
Leaf
L2
Fabric Fabric
L3
Leaf Leaf
L2
Subnet 1/VLAN1 Subnet 2/VLAN2 Subnet 1/VLAN1 Subnet 2/VLAN2
• All east<->west routed traffic traverses to centralized • Optimized forwarding of east-west traffic
gateways • ARP/MAC state localized to Leafs - Helps with
• Centralized gateways have full ARP/MAC state in the DC horizontal scaling of DC
- Scale challenge
EVPN Efficient Cross-Sectional BW Utilizaiton
Flow Based Load-balancing – PE to PE direction

• EVPN provides per-flow load-
Vlan X - F1 P P
balancing among egress PEs Vlan X – F2 E E
using BGP multi-pathing
P
E
• Per-flow load balancing Vlan X - F1 Flow Based Multi-Pathing in the Core

Vlan X – F2
between ingress and egress Vlan X – F3 P
E
P
E
Vlan X – F4 P P
PEs are provided using IGP
ECMP (ingress PE still needs
to add entropy field in the P P
P
packet). E
EVPN Technology Prime
• EVPN Concepts
• EVPN Operation
EVPN Instance (EVI) & MAC-VRF
E-VPN Instance
Spine
(EVI) & MAC-VRF
MAC-VRF EVI is an EVPN instance

spanning Leafs that participate
EVI 20 in that EVPN
MAC-VRF
EVI 10
PE Leaf
MAC- MAC- MAC- MAC- MAC- MAC-
EVPN uses BGP route target
• EVI identifies a VPN in the VRF VRF VRF VRF VRF VRF filtering to enable leafs that don’t
network belong to a specific EVI, not import
• Encompass one or more any MACs for that EVI, providing
bridge-domains, depending efficient scalability
on service interface type
Port-based VM
VM VM VM VM
VLAN-based (shown above) EVPN can use BGP RT constraint
VLAN-bundling (RTC) and Outbound Route
VLAN aware bundling (NEW) Filtering (ORF) for further filtering
of MAC routes at RR
Ethernet Segment (ES)
Multi-home (MHD) Multi-home (MHD) All-
Ethernet Segment Single Home Device
Single-Active (Per- Active (Per-Flow) LB
(SHD)
VLAN) LB
SHD CE1 PE1
ESI1
MHD CE2
ESI2
PE2
• Represents a ‘site’
connected to one or more ESI-0 ESI-0 ESI-1 ESI-1
ESI-1 ESI-1
PEs
• Uniquely identified by a 10-
byte global Ethernet VM
VM VM
Segment Identifier (ESI)
• Could be a single device or
an entire network • Typically used for MHD in DCs
• Ethernet Segment • Typically used for MHN in
Single-Homed Device (SHD)
Multi-Homed Device (MHD)
Identifier (ESI) of ‘0’ SPs • Per-flow LB for known unicast
Single-Homed Network (SHN) traffic
• No DF election • Per VLAN DF election for
Multi-Homed Network (MHN)
all traffic • Per-VLAN DF for BUM traffic
Split-Horizon Filtering, DF Election, and
Aliasing
MAC1 MAC2
• Split-horizon: BUM traffic doesn’t CE1 PE1 PE3 CE2
get loopback to the originating CE

Echo !
device.
PE2 PE4
• DF selection: Either PE3 or PE4 MAC1 CE1 CE2 MAC2

PE1 PE3
forward the broadcast traffic to the
far-end dual-homed device CE2.
Duplicate !
PE2 PE4
• Aliasing: When PE3 wants to

MAC1 CE1 CE2 MAC2
forward a packet with destination PE1 PE3
address MAC1, it needs to send it

to both PE1 and PE2 even though Load balancing
it only learned MAC1 from PE1.

PE2 PE4
Integrated Routing & Bridging (IRB)
IP-VPN Instance represents a

Spine collection of IP-VRFs and their
associated connectivity for a
given tenant
Leaf IP-VRF IP-VRF IP-VRF IP-VRF IRB Interface: a virtual interface

MAC- MAC- MAC- MAC- MAC- MAC-
connecting a MAC-VRF to an IP-
MAC-
VRF VRF VRF VRF VRF VRF VRF VRF
VM VM VM VM VM
Inter-subnet - EVPN can use BGP RT constraint
Routed (RTC) and Outbound Route
Intra-subnet - Filtering (ORF) for further filtering
Bridged of IP routes at RR
Distributed Anycast Gateway with BGP-EVPN
Optimal intra and inter-subnet connectivity with seamless workload mobility
Spine
Identical Anycast Gateway Virtual IP

Distributed Anycast Gateway serves and MAC address are configured on
as the gateway for connected hosts all the Leafs
BVI BVI BVI BVI

GW MAC GW MAC GW MAC GW MAC
Leaf
All the BVIs perform active forwarding

in contrast to active/standby like FHRP VM VM
EVPN Route Types & Benefits
Route Type Usage Benefits
Ethernet A-D Route • Aliasing • Loop avoidance – even
(Type 1) • Mass Withdraw of addresses transient
• SH/AA MH Indication • Fast convergence
• Advertising Split-Horizon Label • Efficient load balancing
• Per-site policy
MAC/IP Advertisement Route (Type 2) • Advertise MAC (and IP) reachability • Per MAC policy
• Advertise MAC/IP binding • ARP suppression
• MAC mobility • Workload Mobility
Inclusive Multicast Route • Auto discovery of multicast tunnel • Support multicast even
(Type 3) endpoints & mcast tunnel type when core doesn’t
Ethernet Segment Route • Auto discovery of redundancy group • A/A and S/A MHD & MHN
(Type 4) support
IP Prefix Route • IP Prefix advertisement (not for IP host • IP route aggregation
(Type 5) advertisement) • Interop w/ L3VPN
EVPN BGP Routes RFC7432
• EVPN defines a new BGP NLRI used to carry all EVPN routes
• BGP Capabilities Advertisement used to ensure that two speakers support
EVPN NLRI (per RFC4760)
• AFI 25: L2VPN, SAFI 70: EVPN [1] Ethernet Auto-Discovery (AD) Route
[2] MAC Advertisement Route
1 byte Route Type [3] Inclusive Multicast Route
[4] Ethernet Segment Route
1 byte Length [5] IP Prefix Route
Variable Route type specific
EVPN NLRI
• EVPN Concepts
• EVPN Operation
EVPN Startup Sequence
Segment Auto-Discovery VPN Auto-Discovery
Multicast Tunnel Endpoint

ESI Auto-Sensing
Discovery
Redundancy Group Membership

Auto-Discovery
DF Election & VLAN Carving
ESI Label & MH type Discovery
ESI Auto-Sensing
ESI (10B) can be auto-generated1
from CE’s LACP information ->
concatenation of CE’s LACP
System Priority + Sys ID + Port Key System System MAC
Segment Auto-Discovery Example:
Priority Address
Port Key
0000. 0011.0022.0033.0018 2 bytes 6 bytes 2 bytes

PE1 PE3
ESI Auto-Sensing3 LACP PDU

exchange
CE LACP info:
LACP System ID (MAC) (6B)
e.g. 0011.0022.0033 CE1 CE3
LACP System Priority (2B)
e.g. 0000 MPLS
LACP Port Key (2B)
e.g. 0018
PE2 PE4
Redundancy Group Membership Auto-Discovery
PE 1 Eth Segment Route

RD = RD10 RD – RD unique per
ESI = ESI1 adv. PE
Segment Auto-Discovery
MAC address portion ES-Import Route Target
of ESI (6B) e.g. 0011.0022.0033
PE1 PE3
ESI Auto-Sensing
PE4
CE1
Redundancy Group Membership
MPLS
Auto-Discovery PE1000
PE2
Exchange ofPE4
Ethernet
PE 2 Eth Segment Route Segment Routes
RD = RD20
ESI = ESI1
ES-Import Route Target
e.g. 0011.0022.0033
BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ordered List of discovered PEs
DF Election & VLAN Carving Modulo Operation
starting from zero (lowest IP add)
VID mod N PE Ordered List

VID (N = # of PEs)
(e.g. VID mod 2) Position PE
Result of modulo 0 PE1
100 0
Segment Auto-Discovery operation is used to
1 PE2
determine DF and 101 1
BDF status 102 0 Example:
103 1 PE1 DF for VIDs 100, 102
PE1 BDF for VIDs 101, 103
ESI Auto-Sensing PE1 PE3
Redundancy Group Membership CE1 CE3

Auto-Discovery
MPLS
DF Election & VLAN Carving PE Ordered List

Position PE
Modulo Operation 0 PE1
PE2 PE4
VID (VID mod 2) 1 PE2
100 0
Example:
101 1
PE2 DF for VIDs 101, 103
102 0 PE2 BDF for VIDs 100, 102
103 1 BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
ESI Label & MH type Discovery
PE1 Eth A-D per ES
RD = RD-1a
ESI1
Eth Tag = MAX-ET
Label = 0
Segment Auto-Discovery ESI Label ext. com
L1
PE1 PE3
ESI Auto-Sensing
CE1
Redundancy Group Membership ESI1
MPLS
Auto-Discovery
DF Election and VLAN Carving PE2 Eth A-D per ES

RD = RD-1b PE2 PE4
ESI1
ESI Label & MH type Discovery Eth Tag = MAX-ET

Label = 0
ESI Label ext. com
L2
Multicast Tunnel Endpoint Discovery
PE 1 Inclusive Multicast Route
Tunnel Type – Ingress RD = RD-1a
RD – RD unique per
Replication or P2MP LSP PMSI Tunnel Attribute adv. PE per EVI
Mcast MPLS Label – used to Tunnel Type (e.g. Ing. Repl.)

transmit BUM traffic - Label (e.g. L1)
downstream assigned (ing.
VPN Auto-Discovery repl.) or upstream assigned
RT ext. community
(Aggregate Inclusive P2MP RT-a
LSP2)
PE1 PE3
RT – RT associated with a
Multicast Tunnel Endpoint given EVI
Discovery
CE1 CE3
MPLS
PE 2 Inclusive Multicast Route

RD = RD-2a
PMSI Tunnel Attribute
Tunnel Type (e.g. Ing. Repl.) PE2 PE4
Label (e.g. L2)
RT ext. community
RT-a PMSI - P-Multicast Service Interface
BUM – Broadcast / Unknown Unicast / Multicast
EVPN BGP route 0x4 - Ethernet Segment Route
• Usage:
• Auto-discovery of multi-homed Ethernet Segments
• Designated Forwarder election
• Tagged with ES-Import Extended Community

• PEs apply route filtering based on ES-Import community. Thus, Ethernet Segment route is imported
only by the PEs that are multi-homed to the same Ethernet segment
Unique per Advertising PE

8 bytes RD
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier
IP address length
1 bytes IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Router’s IP add.
Route Type specific encoding of E-VPN NLRI
ES-Import RT Extended Community
Usage:
• Sent with Ethernet Segment route
• Limits the scope of Ethernet Segment routes distribution to PEs connected to the same multi-homed
Segment
0x06
0x02 MAC Address portion of the ESI

6 bytes ES-Import
EVPN BGP Route 0x1 – Ethernet Auto-discovery Route
This route has two flavors:
Per-ES Ethernet A-D route Per-EVI Ethernet A-D route
• Advertise the Split-Horizon Label associated with an • Advertise VPN label used for Aliasing or Backup-Path
Ethernet Segment
• For AA or SA MH indication
• Used for MAC Mass-Withdraw
Unique per Advertising PE per EVI

Unique per Advertising PE
8 bytes RD ESI of Ethernet Segment
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-Aware
MUST be set to MAX-ET Bundling Service interface, otherwise 0
4 bytes Ethernet Tag ID
(0xFFFFFFFF)
3 bytes MPLS Label VPN (Aliasing) Label per (ESI,
MUST be set to 0 Ethernet Tag)
ESI Label extended community ESI-2
MAC1 MAC2
Agg1 PE1 PE3 Agg2
Usage: ESI-1
• Sent with Ethernet AD Route per ES

• Advertises the Split-Horizon Label for the Ethernet Segment
PE4
PE2
• Indicates the Redundancy Mode: Single Active vs. All-Active
0x06  PE1 advertises in BGP a split-horizon label associated

with the ESI-1 (in the Ethernet AD route)
0x01 Bit 0: Redundancy Mode  Split-horizon label is only used for multi-destination
(single active vs. all active) frames (unknown unicast, mcast, bcast)
Flags
 When PE1 wants to forward a multi-destination frame, it
Reserved Set to 0 appends this SH label to the packet
 PE2 uses this label to perform split-horizon filtering
ESI MPLS Label Ethernet Segment Split- for frames destined to ESI-1 - e.g., a frame originated by
Horizon Label a segment must not be received by the same segment
EVPN BGP route 0x3 – Inclusive Multicast
• Usage:
• Multicast tunnels used to transport Broadcast, Multicast and Unknown Unicast frames
(BUM)

8 bytes RD Set to VLAN or I-SID for VLAN-Aware
4 bytes Ethernet Tag ID Bundling Service interface, otherwise 0
1 bytes
IP address length
IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Router’s IP add.
PMSI Tunnel Attribute – RFC6514
Flags based on RFC6514

1 bytes Flags Ingress Replication/mLDP etc.
1 bytes Tunnel Type
Multicast MPLS Label
3 bytes MPLS Label
When the Tunnel Type is set to Ingress Replication, the Tunnel Identifier
variable Tunnel Identifier carries the unicast tunnel endpoint IP address of the local PE that is to be this
PE's receiving endpoint address for the tunnel.
• EVPN Concepts
• EVPN Operation
MAC Address Reachability
MAC/IP Advertisement • Advertise MAC (and IP) • Per MAC (and IP)
Route (Type 2) reachability policy RR
• MAC Mobility • Workload Mobility
PE1 PE3
CE3
ESI1 PE4 CE4
MAC1 CE1
PE2
• PE1 & PE2 learns MAC1 from CE1 and advertises in BGP to all other PEs with ES field in the MAC/IP
advertisement set to ESI1
• PE3 and PE4 learn that MAC1 sits behind ESI1 which in turn sits behind PE1 & PE2
• PE3 and PE4 now know for packets destined to CE1, they can load balanced between PE1 and PE2
ARP Broadcast Suppression
Route Type Usage Benefits Challenge:
How to reduce ARP broadcasts over the
MAC/IP Advertisement • Advertise MAC (and IP) • Per MAC (and IP) MPLS/IP network, especially in large
Route (Type 2) reachability policy
• Advertise MAC/IP binding • ARP suppression scale virtualized server deployments?
• MAC Mobility • Workload Mobility
MAC3, IP3
PE1 PE3
MAC1, IP1
CE1 CE3
PE4 CE4
PE2 3. ARP Request (IP1)

4. ARP Reply (IP1)
Act as ARP
proxy for IP1.
• CE1 sends out an ARP request for CE3’s IP3
• PE1 snoops the ARP packet and learns (MAC1, IP1). It adds MAC1 to its MAC-VRF, MAC1/IP1 binding to its ARP
cache. It also advertises this binding to all other PEs in BGP and floods this initial ARP request.
• All other PEs learn of (MAC1, IP1). They add the MAC1 to their MAC-VRFs and add (MAC1, IP1) to their ARP cache.
• Now, when CE4 sends an ARP request for IP1, PE4 has the binding info and can provide an ARP response (e.g., ARP
proxy).
MAC Mobility
Challenge:
Route Type Usage Benefits How to handle MAC move ?
MAC/IP Advertisement • Advertise MAC (and IP) • Per MAC (and IP)
Route (Type 2) reachability policy
• MAC Mobility • Workload Mobility PE1 PE3
MAC1, IP1
CE1 CE3
PE4
MAC1, IP1
PE2
• At T0, PE1 learn the MAC1, and advertise to all other PEs
• At T1, MAC1 move to the PE3. PE1 is not aware of this
• PE3 learn the MAC1. It will overwrite the MAC route learnt from PE1
• PE3 will advertise MAC1 to all other PEs with sequence number +1
• All other PE will overwrite the MAC route
• Original PE1 will withdraw its old route
Split Horizon Filtering Challenge:
How to prevent flooded traffic from
echoing back to a multi-homed
Route Type Usage Benefits Ethernet Segment?
Ethernet A-D Route • Advertising Split-Horizon • Loop avoidance – ESI-1 ESI-2
(Type 1) Label even transient
• Aliasing • Efficient load CE1 PE1 PE3 CE3
• Mass Withdraw of • Fast convergence
addresses • balancing Echo !
CE5
• SH/AA MH Indication • Per-site policy
CE4 PE2 PE4
• PE advertises in EVPN Ethernet AS route with a split-horizon label (ESI MPLS Label) associated with
each multi-homed Ethernet Segment
• Split-horizon label is only used for multi-destination frames (Unknown Unicast, Multicast & Broadcast)
• When an ingress PE floods multi-destination traffic, it encodes the Split-Horizon label identifying the
source Ethernet Segment in the packet
• Egress PEs use this label to perform selective split-horizon filtering over the attachment circuit
Challenge:
Aliasing How to load-balance traffic towards a multi-
homed device across multiple PEs when MAC
addresses are learnt by only a single PE?
Ethernet A-D Route • Advertising Split-Horizon • Loop avoidance – I can reach

(Type 1) Label even transient MAC1 via ESI1
• Aliasing • Efficient load I can
MAC1 ESI1 PE1
• Mass Withdraw of • Fast convergence reach ESI1
PE2
addresses • balancing (All-Active)
• SH/AA MH Indication • Per-site policy MAC1 PE1 PE3

MAC1 CE1
CE3
ESI-1 PE4 CE4

PE2 I can
reach ESI1
• PEs advertise in BGP the ESIs of local multi-homed Ethernet Segments. (All-Active)
• All-Active Redundancy Mode indicated
• When PE learns MAC address on its AC, it advertises the MAC in BGP along
with the ESI of the Ethernet Segment from which the MAC was learnt.
• Remote PEs can load-balance traffic to a given MAC address across all PEs
advertising the same ESI.
Challenge:
MAC Mass Withdraw How to inform remote PEs of a failure
affecting many MAC addresses quickly while
the control-plane re-converges?
Ethernet A-D Route • Advertising Split-Horizon • Loop avoidance –

(Type 1) Label even transient
• Aliasing • Efficient load
MAC1, MAC2, .. MACn  ESI1  PE1
X
• Mass Withdraw of • Fast convergence I lost ESI1  PE2
addresses • balancing MAC1,
MAC2,… PE1 PE3
• SH/AA MH Indication • Per-site policy MACn
MAC1
CE1 CE3
ESI-1 PE4 CE4

PE2
• PEs advertise two sets of information:
• MAC addresses along with the ESI from the address was learnt
• Connectivity to ESI(s)
• If a PE detects a failure impacting an Ethernet Segment, it withdraws the route for the
associated ESI.
• Remote PEs remove failed PE from the path-list for all MAC addresses associated with an ESI.
• This effectively is a MAC ‘mass-withdraw’ function.
EVPN BGP Route 0x2 – MAC Advertisement
ESI of Ethernet Segment on which MAC
Address was learnt. All 1s ESI for PBB-EVPN
8 bytes RD
Set to VLAN or I-SID for VLAN-Aware
10 bytes Ethernet Segment Identifier
Bundling Service interface, otherwise 0
4 bytes Ethernet Tag ID
Allows for MAC Address ‘summarization’, i.e.
1 byte MAC Address Length hierarchical MAC Addresses. Typically set to 48
6 bytes MAC Address
Could be C-MAC Address (EVPN) or B-MAC
1 byte IP Address Length Address (PBB-EVPN)
To distinguish IPv4 vs. IPv6 addresses.
4 or 16 IP Address
3 bytes Used for ARP flood suppression or for

MPLS Label1
Integrated Routing and Bridging (IRB).
3 bytes MPLS Label2
MAC & IP Labels - downstream assigned
MAC Mobility extended community
• Used to tag the MAC Advertisement route
• EVPN: Indicates that a MAC address has moved from one PE to another
0x06
0x00
Set to 0
2 bytes Reserved
Indicates the count of MAC address mobility
4 bytes Sequence Number events
EVPN BGP Route 0x1 – Ethernet Auto-discovery Route
This route has two flavors:

Per-EVI Ethernet A-D route
• Advertise VPN label used for Aliasing or Backup-Path

RD ESI of Ethernet Segment
Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-Aware
Bundling Service interface, otherwise 0
Ethernet Tag ID
MPLS Label VPN (Aliasing) Label per (ESI,

Ethernet Tag)
• EVPN Concepts
• Startup Sequence
• Operation
Life of a Packet
Ingress Replication – Multi-destination Traffic Forwarding
PE1 receives broadcast Mcast MPLS
During start-up traffic from CE1. PE1 PSN MPLS label Label assigned by
sequence, PE1, PE2, forwards it using ingress to reach PE3 PE3 for incoming
PE3, PE4 sent Inclusive replication – 3 copies BUM traffic on a
Multicast route which created given EVI PE3 – as DF, it
include Mcast label
During start-up forwards BUM
sequence, PE2 sent Per- PE1 PE3 PE1 PE3 traffic towards
ESI Ethernet AD route segment
with ESI MPLS label VID 100
(split-horizon) (see SMAC: M1
below) DMAC: F.F.F L3
CE1 CE3 CE1 CE3

L2 L5
MPLS MPLS
PE 4 Inclusive Multicast
PE 2 Eth A-D Route (Per-ESI)
Route
L4
RD = RD20
RD = RD-4a
ESI = ESI1
PMSI Tunnel Attribute
ESI MPLS Label ext. comm.
Tunnel Type = Ing. Repl.
Redund. Flag = All-Active PE2 PE4 PE2 ESI (split-horizon) PE4
Label = L4 MPLS label
Label = L5 allocated by PE2
ESI MPLS Label – used by RT ext. community
PE2 – drops BUM PE4 – non-DF for
RT ext. community local PEs for split-horizon - for segment ES1
Mcast MPLS Label – used to RT-a traffic originated given EVI drops
RT-a, RT-b, RT-c, RT-d downstream assigned (for on ES1 BUM traffic
transmit BUM traffic -
ingress replication)
downstream assigned (for
ingress replication) BRKMPL-2333 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Life of a Packet (cont.)
Unicast Traffic Forwarding
PE1 MAC Route
MP2P VPN Label – PE3 forwards
downstream allocated label RD = RD-1a
MAC advertised PSN MPLS label MP2P VPN Label traffic destined to
used by other PEs to send ESI = ESI1 assigned by PE1 M1 based on RIB
by route to reach PE1
traffic to advertised MAC for incoming traffic information (PE1)
MAC = M1
for the target EVI
Label = L1
RT ext. community
PE1 PE3 PE1 PE3
RT-a VID 100
SMAC: M2
VID 100 DMAC: M1
SMAC: M1
DMAC: F.F.F
L1
CE1 CE3 CE1 CE3
MPLS MPLS
PE2 PE4 PE2 PE4
PE3 RIB Path List

VPN MAC ESI NH
RT-a M1 ES1 PE1
Life of a Packet (cont.)
Unicast Forwarding and Aliasing PE3 forwards PE3 forwards
traffic on a flow traffic on a flow
PE1 MAC Route MP2P VPN
MP2P VPN Label – (flow 1) based on (flow 2) based on
RD = RD-1a Label
downstream allocated label PSN MPLS label RIB information RIB information
MAC advertised assigned by
used by other PEs to send ESI = ESI1 to reach PE1 (towards PE1) (towards PE2)
by route PE1 for
traffic to advertised MAC
MAC = M1 incoming for
target EVI
Label = L1
During start-up
sequence, PE1 sent Per- RT ext. community VID 100
EVI Ethernet AD route PE1 PE3 PE1 PE3 SMAC: M3
RT-a
DMAC: M1
VID 100
VID 100 SMAC: M4
SMAC: M1 DMAC: M1
DMAC: F.F.F L1
CE1 CE3 CE1 CE3
During start-up sequence, MPLS MPLS

PE2 sent Per-EVI Ethernet L2
AD route (see below)
PE 2 Eth A-D Route (Per-EVI)

RD = RD-2a PE2 PE4 PE2 PE4
ESI = ESI1
Aliasing MPLS Label – used PE3, PE4 RIB Path List
Label = L2 Aliasing MPLS
by remote PEs to load- VPN MAC ESI NH PSN MPLS label Label assigned by
RT ext. community balance among local PEs to reach PE2 PE2 for (ES1, EVI)
RT-a M1 ES1 PE1
RT-a pair
PE2
E-VPN Operational Scenarios
PE3 MAC Route
MAC Mobility PE1 MAC Route RD = RD-3a
RD = RD-1a ESI = ESI2
1 4
ESI = ESI1 3 MAC = M1
PE1 advertises MAC Host M1 moves After host sends traffic
route for M1. Route MAC = M1 Label = L3
from CE1 to CE3’s at new location, PE2
may include MAC Label = L1 location MAC Mobility ext. now adv MAC route
mobility community community. for M1 incrementing
MAC Mobility ext.
community sequence # in MAC
Seq. Num = 2
mobility community
PE1 Seq. Num = 1 PE3 PE1 RT ext. community PE3
RT ext. community RT-a VID 100
VID 100 SMAC: M1
SMAC: M1
RT-a
DMAC: M2
DMAC: M2
CE1 CE3 CE1 CE3

M1
MPLS M1 MPLS M1
5
PE1 withdraws its
2 M1 route and
PE3 / PE4 install installs a new one
M1 route towards pointing to PE3
PE2 PE4 PE1 PE2 PE4
PE3 / PE4 RIB Path List PE1 / PE2 RIB Path List
VPN MAC ESI NH VPN MAC ESI NH
RT-a M1 ES1 PE1 RT-a M1 ES2 PE3
E-VPN Failure Scenarios / Convergence
Link / Segment Failure – Active/Active per Flow
2 7
PE1 withdraws
PE1 withdraws Per-ESI individual MAC
Ethernet AD route for advertisement routes
1 failed segment related to failed
PE1 detects failure segment
of one of its
attached segments PE1 PE3 PE1 PE3
CE1 CE3 CE1 CE3
3 MPLS 4 MPLS
PE1 withdraws Ethernet Mass withdrawal - PE3
Segment Route / PE4 remove PE1 from
path list for all MAC
addresses of failed
segment (ES1)
5 PE2 PE4 6 PE2 PE4
PE2 recalculates
DF/BDF. Becomes DF PE3, PE4 RIB Path List PE2 adv. M1 MAC route PE3, PE4 RIB Path List
for all EVIs on segment VPN MAC ESI NH after CE traffic is VPN MAC ESI NH
hashed towards PE2
RT-a M1 ES1 PE1 RT-a M1 ES1 PE2
PE2
E-VPN Failure Scenarios / Convergence
PE Failure
2
BGP RR / PE3 detects
1 BGP session time-out
PE1 experiences a with PE1
node failure (e.g.
power failure) PE1 PE3 PE1 PE3
CE1 CE3 CE1 CE3
MPLS MPLS
6
2 3 PE3 / PE4 will forward
2 BGP RR / PE4 PE3 / PE4 invalidate M1 traffic towards PE2
BGP RR / PE2 detects detects BGP routes from PE1
BGP session time-out session time-
with PE1
PE2 out with PE1 PE4
5 PE2 PE4
PE2 adv. M1 MAC route
4 PE3, PE4 RIB Path List after CE traffic is PE3, PE4 RIB Path List
VPN MAC ESI NH hashed towards PE2 VPN MAC ESI NH
PE2 reruns DF election.
Becomes DF for all RT-a M1 ES1 PE1
PE1 RT-a M1 ES1 PE2
EVIs on segment
PE2
DC Fabric Evolution w/ EVPN-IRB
The Evolution of the DC Fabric
L2 Overlay over IP L2/L3 Overlay over IP Fabric:

Fabric:
ASR9K) EVPN-IRB w/ VxlAN encap
VXLAN, DP Learning • Spine-leaf
L2 Fabric: • Virtual L2 & L3 overlay across
FabricPath/Trill physical boundary
• VXLAN: Ultra-high scale
L2 Fabric: Legacy VLAN, STP • Efficient forwarding: L3 ECMPs
• L2/L3 boundary: limited • EVPN control plane
mobility • Policy-based forwarding
• 4K VLAN: Limited scale
• Inefficient forwarding: STP
• Complex VLAN provisioning
• Vendor specific L2
enhancement
Centralized v.s. Distributed Routing
Distributed Routing Centralized Routing
Boarder L3 Centralized GW
Leaf
L2
Fabric Fabric
L3
Leaf Leaf
L2
Subnet 1/VLAN1 Subnet 2/VLAN2 Subnet 1/VLAN1 Subnet 2/VLAN2
• Optimized forwarding of east-west traffic • All east<->west routed traffic traverses to centralized gateways
• ARP/MAC state localized to Leafs • Centralized gateways have full ARP/MAC state in the DC
• Helps with horizontal scaling of DC • Scale challenge
Distributed Routing: EVPN IRB
Symmetric IRB Asymmetric IRB
Boarder Boarder
Leaf Leaf
Fabric Fabric
IP-VRF IP-VRF
IP-VRF IP-VRF
Leaf Leaf
MAC- MAC- MAC- MAC-
MAC- MAC- MAC- MAC-
VRF VRF VRF VRF
VRF VRF VRF VRF
• Ingress Leaf needs ARP/MAC state for every egress leaf

• ARP/MAC state localized to Leafs
• Limits scale
• Helps with horizontal scaling of DC
• Inefficient encap: requires Ethernet header for inter-
• More efficient encapsulation for MPLS and
subnet IP routing
VxLAN with GPE
• Asymmetric processing: Imposition PE performs more
• Symmetric processing
work and lookups than disposition PE
EVPN IRB: Distributed Anycast Gateway at Leafs
The same anycast gateway virtual IP
address and MAC address are
configured on all VTEPs in the VNI.
Leafs
Host-1 Host-2 Host-6 Host-4 Host-5

IP-H1, MAC-H1 IP-H2, MAC-H2 IP-H6, MAC-H6 IP-H1, MAC-H1 IP-H2, MAC-H2
Bridge-domain 1 Bridge-domain 2 Bridge-domain 3 Bridge-domain 1 Bridge-domain 2
Mobility zone
Tenant A:
• Distributed anycast default gateway for hosts
IP-VRF Blue • IRB provides intra and inter-subnet forwarding
Bridge domain-1
Bridge domain-2
Bridge domain-3
EVPN IRB Intra-Subnet: Control Plane Operation
Intra-subnet/bridged traffic between two remote end hosts : host-1 to host-3
DCI
Add IP-H3 to IP-VRF table. 3
Learn IP-
H3/MAC-H3
MAC Route:
R Spine binding via
2 R
RD: VPN RD ARP snooping
IP: IP-H3 1 on bridge port
MAC: MAC-H3
Label1: MAC VRF Label
Label2: IP VRF Label Leaf
Ext. Community/Attributes:
Route Target: MAC-VRF GARP
Route Target : IP-VRF 1
Next Hop: IP-L3 3
3
Host-1 Host-2 Host-3 Host-4
IP-H1, MAC-H1 IP-H2, MAC-H2 IP-H3, MAC-H3 IP-H4, MAC-H4
BD-1 BD-2 BD-1 BD-2
1. Add MAC-H3 to MAC table: MAC-H3 via IP-L3, Label1.

2. Add {IP-H3, MAC-H3} to ARP suppression cache Add IP-H3 to IP VRF table: IP-H3 via IP-L3, Label2
3. Add IP route: IP-H3 via IP-L3, Encap: Label2 L4: IP VRF Table
L1: ARP Suppression
L1: MAC Table L1: IP VRF Table
Cache IP-H3-> IP-L3, Label2
MAC-H3 -> IP-L3, Label1 IP-H3-> IP-L3, Label2
IP-H3 -> MAC-3
EVPN IRB Intra-Subnet: Packet Flow (1)
DCI
1. Host-1 sends an ARP request for
Host-3 , with target IP-H3.
Spine
2. The ARP request is intercepted at the
L1: ARP Suppression Cache
leaf L1 and punted to the control
IP-H3 -> MAC-3
plane.
2 CPU Leaf
3. L1 looks up IP-H3 in ARP 1
suppression cache, if an entry is
found, ARP response is locally 3
generated with the actual host MAC Host-3
Host-1 H1 ARP Cache IP-H3, MAC-H3
address MAC-H3. If the entry is not BD-1
IP-H1, MAC-H1 IP-H3 -> MAC-3
found in the ARP suppression cache, BD-1
then ARP is flooded in the bridge
domain.
EVPN IRB Intra-Subnet: Packet Flow (2)
4. Host-1 sends data packet with MAC-H3 DCI

as the destination MAC.
5. L1 receives the packet, does a MAC
DMAC: MAC--H3
lookup in MAC table for MAC-H3. It finds 7
Spine
an entry for MAC-H3 pointing to IP-L3 SMAC: MAC-H1
with Label1 as encap. DIP: IP-H3 6
8
6. L1 encapsulates packet with MPLS SIP: IP-H1 5 Leaf
label1, and a transport label
corresponding to the IP-L3.
7. Spines forward the MPLS packet based 4 9
on the transport label towards L3.
DMAC: MAC-H3
8. L3 receives MPLS packet, uses Label1 to Host-1
IP-H1, MAC-H1 SMAC: MAC-H1 Host-3
determine the MAC table to do the inner BD-1 IP-H3, MAC-H3
DMAC lookup and finds the egress bridge DIP: IP-H3 BD-1
port to forward the packet.
SIP: IP-H1
9. Host-3 receives data packet with MAC-H3
as the destination MAC and consumes it.
EVPN IRB Inter-Subnet: Control Plane
Inter-subnet routed traffic between two remote end hosts : host-1 to host-4
DCI
1. Add IP-H4 to IP VRF table. 3
Learn IP-
H4/MAC-H4
MAC Route: RR Spine binding via
ARP snooping
RD: VPN RD 2 on bridge port
IP: IP-H4 1
MAC: MAC-H4
Label1: MAC VRF Label
Label2: IP VRF Label
Ext. Community/Attributes: Leaf
Route Target: MAC-VRF
Route Target : IP-VRF GARP
Next Hop: IP-L4
1
3
Host-1 Host-2 Host-3 Host-4
IP-H1, MAC-H1 IP-H2, MAC-H2 IP-H3, MAC-H3 IP-H4, MAC-H4
BD-1 BD-2 BD-1 BD-2
1. Add MAC-H4 to MAC table: MAC-H4 via IP-L4, Label1.

2. Add {IP-H4, MAC-H4} to ARP suppression cache
3. Add IP route: IP-H4 via IP-L4, Encap: Label2
L1: MAC Table L1: ARP Suppression Cache L4: IP VRF Table
MAC-H4 -> IP-L4, Label1 IP-H4 -> MAC-4 IP-H4-> IP-L4, Label2
EVPN IRB Inter-Subnet: Packet Flow (1)
Inter-subnet traffic between two remote end hosts : host-1 to host-4
1. Host-1 sends an ARP request for its default DCI

gateway, with target as its gateway IP
address (BVI1’s IP address : IP-BVI1).
2. The ARP request is received at the ToR L1 Spine
and punted to the control plane.
L3
3. Target IP address in ARP request is the BVI-
BVI’s IP address, L1 sends ARP reply with L2 2 CPU 1

BVI
-2 Leaf
1
MAC-BVI1 as the MAC address.
3
Host-1 Host-4
IP-H1, MAC-H1 IP-H4, MAC-H4
BD-1 BD-2
H1 ARP Cache
IP-BVI1 -> MAC-BVI-1
EVPN IRB Inter-Subnet: Packet Flow (2)
Inter-subnet traffic between two remote end hosts : host-1 to host-4
4. Host-1 sends data packet with MAC-BVI1 as
the destination MAC and IP-H4 as the DCI
destination IP address.
7
5. L1 receives the packet, since the dMAC is its
BVI MAC, it does IP lookup and finds route to Spine
IP-H4, that points to IP-L4 as next hop and 8
6
Label2 as the VPN label. DMAC: MAC-BVI1
6. L1 encapsulates packet with MPLS header SMAC: MAC-H1 Leaf
DIP: IP-H4 5
with Label2 as the VPN label and outer label SIP: IP-H1
as the transport label corresponding to IP-L4.
9
7. Resulting packet is forwarded towards L4. 4
DMAC: MAC-H4
SMAC: MAC-BVI2
8. L4 receives MPLS packet, uses Label2 to Host-1
DIP: IP-H4
IP-H1, MAC-H1 Host-4
determine the VPN table to do the inner IP SIP: IP-H1 IP-H4, MAC-H4
BD-1
lookup, finds the egress BVI to forward the BD-2
packet.
9. Host-H4 receives data packet with MAC-H4 as
the destination MAC and consumes it.
EVPN IRB Inter-Subnet: Control Plane
Inter-subnet traffic between two local end hosts : host-1 to host-2
1. Host-1 sends an ARP request for its default DCI

gateway, with target as its gateway IP
address (BVI1’s IP address : IP-BVI1).
2. The ARP request is received at the ToR L1 Spine
and punted to the control plane.
3. Target IP address in ARP request is the 2
BVI’s IP address, L1 sends ARP reply with
BVI- BVI
1
CPU -2 Leaf
MAC-BVI1 as the MAC address for the 1
gateway IP.
3
Host-1 Host-2
BD-1 BD-2
H1 ARP Cache
IP-BVI1 -> MAC-BVI-1
EVPN IRB Inter-Subnet: Packet Flow
Inter-subnet traffic between two local end hosts : host-1 to host-2
1. Host-1 sends data packet with MAC-
BVI1 as the destination MAC and IP-H2 DCI
as the destination IP address.
2. L1 receives the packet, since the dMAC Spine
DMAC: MAC-BVI1
is its BVI MAC, it does IP lookup and
finds route to IP-H2, that points to BVI2 SMAC: MAC-H1
5
as egress interface. And routes it DIP: IP-H2
locally, there is no MPLS encapsulation BVI- BVI
-2
Leaf
SIP: IP-H1 1
needed. It is possible that the ARP for

6
IP-H2 is not resolved, in which case an
4
ARP request is triggered on BVI2 to DMAC: MAC-H2
resolve it. Host-1 Host-2 SMAC: MAC BVI2
3. Host-H4 receives data packet with BD-1 BD-2 DIP: IP-H2
MAC-H4 as the destination MAC and SIP: IP-H1
consumes it.
EVPN-VPWS
EVPN-VPWS Does it Better than Legacy VPWS !!
Service Additional Capabilities

E-Line • All-active & single-active multi-homing support
• Both single-segment & multi-segment support
• Discovery & signaling via single protocol – BGP
EVPN BGP route type
Route type Usage EVPN EVPN VPWS
0x1 Ethernet Auto-Discovery • MAC Mass-Withdraw
(A-D) Route • Aliasing (load balancing)  
• Split-Horizon
“Tagged with ESI Label Extended Community”
0x2 MAC Advertisement Route • Advertise MAC addresses
• Provide MAC / IP address bindings for ARP  NOT used
broadcast suppression
“Tagged with MAC Mobility Extended
Community”
0x3 Inclusive Multicast Route • Multicast tunnels used to transport
Broadcast, Multicast and Unknown Unicast NOT used
frames (BUM) 
“Tagged with PMSI tunnel attribute” (P tunnel
type & ID) – RFC6514
0x4 Ethernet Segment Route • Auto discovery of Multi-homed Ethernet
Segments, i.e. redundancy group discovery
• Designated Forwarder (DF) Election  
“Tagged with ES-Import Extended Community”
EVPN BGP Extended Community
Attribute Usage Tagged BGP EVPN EVPN VPWS

route
ESI label Extended • Split-Horizon for Ethernet Ethernet A-D

Community Segment. Route
• Indicate Redundancy Mode  
(Single Active vs. All-Active)
ES-Import Extended • Limit the import scope of the Ethernet
Community Ethernet Segment routes. Segment Route  
MAC Mobility Extended • E-VPN: Indicate that a MAC MAC
Community address has moved from one Advertisement  Not used
segment to another across PEs. Route
• PBB-EVPN: Signal C-MAC
address flush notification
EVPN VPWS
Control-plane
attachment circuit
• Benefits of EVPN applied to point-to-point advertisement over the
services Core
• No signaling of PWs. Instead signals MP2P LSPs

instead (ala L3VPN) VPWS Service Config:
EVI = 100
• All-active CE multi-homing (per-flow LB) Local AC ID = AC2
PE1 PE2 Remote AC ID = AC1
• Single-active CE multi-homing (per-service LB)
CE1 CE2
• Relies on a sub-set of EVPN routes to

ES1 MPLS ES2
advertise Ethernet Segment and AC
reachability VPWS Service Config:
• PE discovery & signaling via a single protocol – EVI = 100
Local AC ID = AC1
BGP
Remote AC ID = AC2
• Per-EVI Ethernet Auto-Discovery route
• Handles double-sided provisioning with remote PE
auto-discovery
I have a P2P service that BGP Eth. Auto-
needs to communicate Discovery Route
• Under standardization: draft-ietf-bess-evpn- with the PE(s) that own EVPN NLRI
vpws of AC = AC2 AC AC1 via PE1
EVPN VPWS Operation – Single-homed
ESI – 10 bytes ESI as specify
by EVPN Ethernet segment
IETF draft – zero for single- 3
homed RD – RD unique per adv. PE
PE 1 Eth A-D Route
per EVI
RD = RD-1a
Eth.Tag ID – 4-bytes local
ESI = ES1 (0)
AC-ID
Eth.Tag ID = AC1 RT – RT associated with a
given EVI
Label (e.g. X)
MPLS Label – (downstream
assigned) used by remote RT ext. community VPWS Service Config:
PEs to reach segment EVI = 100
RT-a
Local AC ID = AC2
PE1 PE2 Remote AC ID = AC1 ES2 – Since CE2 is single
CE1 CE2 homed to PE2, ES2 = 0
1 ES1 MPLS ES2

2
VPWS Service Config:
EVI = 100
PE 2 Eth A-D Route
Local AC ID = AC1
Remote AC ID = AC2 RD = RD-2a
5 ESI = ES2 (0) 6
PE1 RIB Path List Path List
Eth.Tag ID = AC2 PE2 RIB
NH
VPN MAC ESI Eth.TAG Label (e.g. Y) VPN MAC ESI Eth.TAG
NH
PE2
RT ext. community PE1
RT-a - 0 AC2 4 RT-a - 0 AC1
RT-a
EVPN VPWS Operation – Single-active
PE 1 Eth A-D Route RD – RD unique per adv. PE
per EVI
by EVPN Ethernet segment RD = RD-1a
IETF draft
ESI = ES1
3
Eth.Tag ID = AC1
Eth.Tag ID – 4-bytes local Label (e.g. X) RT – RT associated with a
AC-ID given EVI
RT ext. community
RT-a
assigned) used by remote PE1 VPWS Service Config: Only one PE (PE1)
PEs to reach segment EVI = 100 shows as next hop for
Single-Active == per-vlan
Local AC ID = AC2 the remote AC
load-balancing CE-PEs
ES1 PE3 Remote AC ID = AC1
Two bundles on CE VPWS Service Config:
CE1 CE2
device EVI = 100
Local AC ID = AC1 ES2 – Since CE2 is single
Remote AC ID = AC2 MPLS ES2
2 homed to PE2, ES2 = 0
1
VPWS Service Config: ES1
EVI = 100 PE 3 Eth A-D Route 6
Local AC ID = AC1 RD = RD-2a
PE2
5 Remote AC ID = AC2
ESI = ES2 (0)
PE3 RIB
Path List
PE1 & PE2 RIB Path List VPN MAC ESI Eth.TAG
Eth.Tag ID = AC2 NH
NH
VPN MAC ESI Eth.TAG Label (e.g. Y) RT-a - ES1 PE1
PE3
RT ext. community
RT-a - 0 AC2 4 RT-a - ES1 PE2
RT-a RT-a - ES1 AC1 PE1
EVPN VPWS Operation – All-active
PE 1 Eth A-D Route RD – RD unique per adv. PE
per EVI
by EVPN Ethernet segment RD = RD-1a
IETF draft
ESI = ES1
3
Eth.Tag ID = AC1
Eth.Tag ID – 4-bytes local Label (e.g. X) RT – RT associated with a
AC-ID given EVI
RT ext. community
RT-a
assigned) used by remote PE1 VPWS Service Config: Both PEs (PE1/PE2)
PEs to reach segment EVI = 100 shows as next hop for
ALL-Active == per-flow
Local AC ID = AC2 the remote AC
load-baancing CE-PEs
ES1 PE3 Remote AC ID = AC1
Single bundle on CE VPWS Service Config:
CE1 CE2
device EVI = 100
ES2 – Since CE2 is single
Local AC ID = AC1
Remote AC ID = AC2 MPLS ES2
2 homed to PE2, ES2 = 0
1
VPWS Service Config: ES1
EVI = 100 PE 3 Eth A-D Route 6
Local AC ID = AC1 RD = RD-2a PE3 RIB
PE2
5 Remote AC ID = AC2
ESI = ES2 (0)
Path List
VPN MAC ESI Eth.TAG
PE1 & PE2 RIB Path List NH
Eth.Tag ID = AC2
NH RT-a - ES1 PE1
VPN MAC ESI Eth.TAG Label (e.g. Y)
PE3 RT-a - ES1 PE2
RT ext. community
RT-a - 0 AC2 4 RT-a - ES1 AC1 PE1,PE2
RT-a
EVPN Deployment:
DC Fabric and WAN Integration
The Solution Must be End-to-End
• Legacy 802.1q handoff

APIC • It means multi-pathing is out the
door between DC & WAN
• VLANs and sub-interfaces
DC creation
• No policy level integration
• Small FIB/MAC table size on
Spine
border Leaf, create bottleneck
WAN/DCI
Leaf Leaf bLeaf bLeaf
802.1q?
SDN-DC, VXLAN overlay SDN-WAN, MPLS/Segment Routing
DC Gateway – Seamless DC and WAN Integration
• L3 Gateway
• L2 Gateway
• IRB with Anycast
Gateway
• Common MP-BGP (EVPN AF) control plane
• VXLAN to MPLS data plane interworking
APIC • SDN based auto provisioning: OpFlex,
APIC/VTS
DC • Integrated Policy control: WAN optimization
Spine
WAN/DCI
Leaf Leaf bLeaf bLeaf
Integration
SDN-DC, VXLAN overlay Interworking SDN-WAN, MPLS/SR
Scalable, Resilient, Optimized, End-to-End © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
DC Gateway – L3 Gateway • EVPN/VXLAN to IP-VPN/MPLS interworking
• EVPN/VXLAN to global internet
S-N Routing: all active • L3 EVPN between Leaf and GW (per-vrf VNI)
• Leaf is the L3 default gateway for VMs (with
EVPN IRB anycast GW) and does inter-vxlan
routing
Internet VPN client

Client Client
Internet
Branch IP-VPN
Internet
DC DC-1 Gateway
WAN DC-2
Gateway
Spine L3 EVPN
(per-VRF VNI)
Leaf
Leaf
(L3 anycast
VM1 VM2 L2/L3 DC fabric L2/L3 DC fabric GW)
VM3 VM4
DC Gateway – L2 Gateway • L2 EVPN/VXLAN in the DC
L2 Stretch: E-W and S-N • L2 DCI (E-W): EVPN/VXLAN with EVPN/VPLS
all-active or single-active interworking
• L2 to client (S-N): EVPN/VXLAN with VPLS/PW
interworking
VPN client
Client
Branch
VPLS/PW/EV
PN
DC Gateway DC-1 Gateway

WAN DC-2
Spine
L2 EVPN
(per-BD VNI)
Leaf
Leaf
VM1 VM2
L2/L3 DC fabric L2/L3 DC fabric VM3 VM4
Leaf Gateway Gateway

Leaf
L2 EVPN/VXLAN L2 stretch: EVPN/VPLS (MPLS) L2 ©EVPN/VXLAN
BRKMPL-2333 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
DC Gateway – IRB Anycast Gateway
Integrated Routing and Bridging • DC fabric is L2 only. All routing on DC gateway
• DC gateway is the L3 default gateway for VMs via
EVPN IRB anycast gateway
• Support both L2 and L3 for the same VNI at the
same time
Internet VPN client

Client Client
Internet IP-VPN
Branch
Internet
VPLS/PW/EVP
IRB N
DC DC-1 IRB IRB Gateway
IRB DC-2
Gateway WAN (L3 anycast GW)
Spine L2 EVPN
(per-BD VNI)
Leaf Leaf
VM1 VM2 L2 only DC fabric (L2 only)
L2 only DC fabric VM3 VM4
Leaf Gateway Gateway

Leaf
L2 EVPN/VXLAN L2 stretch: EVPN/VPLS (MPLS) L2 ©EVPN/VXLAN
2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
EVPN IRB Anycast Gateway Deployment Options
Distributed vs. Centralized Anycast Gateway
client
Internet VPN client
Client Client
IP-VPN
Internet IP-VPN
Internet Branc
VPLS/PW/EVP
Centralized Internet
h VPLS/PW/EVP
N Anycast GW
N
DC Gateway IRB
(IRB anycast GW) DC-1 DC Gateway
IRB DC-2
WAN (L3 GW)
L2 EVPN
(per-BD VNI) L3 EVPN
(per-VRF VNI)
Leaf IRB IRB IRB IRB

Leafonly)
(L2 IRB IRB Leaf
VM1 VM2 L2 only DC fabric L2/L3 DC fabric (IRB anycast GW)
VM3 VM4 Distributed
Anycast GW
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
IRB Anycast Gateway Deployment Option
Comparison
Option 1 Option 2
Distributed IRB Anycast Gateway Centralized IRB Anycast Gateway
DC Gateway Router • L2 or L3 gateway function • EVPN IRB for integrated L2 and L3

• Doesn’t require IRB function • IRB anycast gateway for VM’s default
gateway
Leaf • EVPN IRB for integrated L2 and L3 • L2 EVPN peering across DC

• IRB anycast gateway for VM’s • Cross-DC underlay IP routing is required
default gateway
Pros • Optimized E-W inter-vxlan routing • Simple DC fabric design: L2 only

• Large ARP table on the DC gateway
router
Cons • EVPN IRB function across all Leaf • Sub-optimal E-W inter-vxlan routing
nodes
• Require both L2 and L3 function on
the leaf
DC Gateway – the Policy Integration and Auto-
Provisioning: Application-engineered Routing
DC: classification PBTS: steering packet to
and marking SR-TE in the WAN
DC policy domain WAN policy domain DC policy domain
WAN Segment
Routing
VTS/APIC VTS/APIC
DC DC-1
WAN DC-2
Gateway
Spine
Leaf
VM1 VM2
VM3 VM4
High bandwidth flow

Low latency flow 89
Example: Application has a preference for disjoint path in dual-plane
WAN networks
ACI Fabric
Customer has the requirement that traffic from

WEB WEB applications RED and BLUE should be
Policy
transported across disjoint paths in the WAN.

Segment - Policy expressed on APIC and delivered by
Routing SR-enabled +WAE WAN
WAN
with WAE
Example: Application requires the lowest latency path in the WAN
ACI Fabric
WAN has cheap capacity via US with higher latency.

WEB WEB
Scarce, expensive capacity via Russia, with lower
Policy
latency.
Tokyo
Segment
Routing Russia Customer identify the applications that require the
US lowest possible latency path on APIC, integration
WAN steers traffic on the path via Russia.
with WAE
Brussels
Summary
Service Provider Data Center
Next gen all-in-one VPN Industry defacto-standard multi-
solution to provide all the tenancy solution that provides
services previously done by flexible workload placement,
multiple solutions and more workload mobility, full cross-
sectional BW utilization and
• L2VPN (P2P, MP, P2MP) elasticity.
• L3VPN
• IRB • L2, L3, L2+L3 Overlay
• DCI
References
• [FabricPath]: FabricPath http://www.cisco.com/en/US/prod/switches/ps9441/fabric_path_promo.html
• [LISP]: Locator/ID Separation Protocol https://datatracker.ietf.org/wg/lisp/charter/
• [802.1Qbp] ECMP http://www.ieee802.org/1/files/public/docs2011/new-ashwood-sajassi-ecmp-par-0111-v04.pdf
• [EVPN]: BGP MPLS Based Ethernet VPN http://tools.ietf.org/html/draft-raggarwa-sajassi-l2vpn-evpn-04
• [TRILL]: Transparent Interconnection of Lots of Links https://datatracker.ietf.org/wg/trill/charter/

http://tools.ietf.org/wg/trill/draft-ietf-trill-rbridge-protocol/
• [VL2]: VL2: A Scalable and Flexible Data Center Network http://ccr.sigcomm.org/online/?q=node/502
• [MOOSE]: Addressing the Scalability of Ethernet with MOOSE http://www.cl.cam.ac.uk/~mas90/MOOSE/MOOSE.pdf
• [PORTLAND]: PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric http://ccr.sigcomm.org/online/?q=node/503
• [SEATTLE]: Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises

http://www.cs.princeton.edu/~chkim/Research/SEATTLE/seattle.pdf
• [MONSOON]: Towards a Next Generation Data Center Architecture: Scalability and Commoditization
http://research.microsoft.com/apps/pubs/default.aspx?id=79348
• [VLB]: Valiant Load Balancing in Backbone Networks http://www.stanford.edu/~ashishg/network-algorithms/rui.pdf
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

BRKMPL 2333

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

BRKMPL 2333

Încărcat de

Drepturi de autor:

Formate disponibile

EVPN: Network

Ali Sajassi– Distinguished Engineer, Cisco

Cisco Spark spaces will be cs.co/ciscolivebot#BRKMPL-2333

• DC Fabric Evolution w/ EVPN-IRB

• Underlay: physical topology + • Overlay: Virtual topology among

MAC Routing: Control plane (BGP) Optimum forwarding,

PE2 PE4 Flexible Policy Control

EVPN-IRB EVPN-Overlay EVPN-Etree PBB-EVPN

draft-ietf-bess-evpn-inter-subnet-forwarding draft-ietf-bess-evpn-overlay draft-ietf-bess-evpn-etree

PBB- EVPN EVPN EVPN- EVPN- EVPN- EVPN-

Service Additional Capabilities

Network Virtualization Overlay

LDP, RSVP-TE, or SR IPv4/IPv6, or SRv6

Following drafts were Following drafts were Enhancements

CE2 PE2 CE2 PE2

• Inflexible configuration • Works for both MPLS & IP transport

MAC1 CE1 CE2 MAC2

• Enabling multiple paths through the fabric between a pair of

• Enabling path selection

TOR1 TOR2 TOR3 TOR4

SW1 SW2 SW3 SW4 App

• Enable flexible and efficient workload placement and

Subnet 1/VLAN1 Subnet 2/VLAN2 Subnet 1/VLAN1 Subnet 2/VLAN2

Flow Based Load-balancing – PE to PE direction

• Per-flow load balancing Vlan X - F1 Flow Based Multi-Pathing in the Core

MAC-VRF EVI is an EVPN instance

get loopback to the originating CE

• DF selection: Either PE3 or PE4 MAC1 CE1 CE2 MAC2

• Aliasing: When PE3 wants to

address MAC1, it needs to send it

it only learned MAC1 from PE1.

IP-VPN Instance represents a

Leaf IP-VRF IP-VRF IP-VRF IP-VRF IRB Interface: a virtual interface

Identical Anycast Gateway Virtual IP

BVI BVI BVI BVI

All the BVIs perform active forwarding

Variable Route type specific

Segment Auto-Discovery VPN Auto-Discovery

Multicast Tunnel Endpoint

Redundancy Group Membership

DF Election & VLAN Carving

ESI Label & MH type Discovery

0000. 0011.0022.0033.0018 2 bytes 6 bytes 2 bytes

ESI Auto-Sensing3 LACP PDU

PE 1 Eth Segment Route

VID mod N PE Ordered List

Redundancy Group Membership CE1 CE3

DF Election & VLAN Carving PE Ordered List

DF Election and VLAN Carving PE2 Eth A-D per ES

ESI Label & MH type Discovery Eth Tag = MAX-ET

Mcast MPLS Label – used to Tunnel Type (e.g. Ing. Repl.)

PE 2 Inclusive Multicast Route

• Tagged with ES-Import Extended Community

Unique per Advertising PE

Route Type specific encoding of E-VPN NLRI

0x02 MAC Address portion of the ESI

Per-ES Ethernet A-D route Per-EVI Ethernet A-D route

• Used for MAC Mass-Withdraw

Unique per Advertising PE per EVI

• Sent with Ethernet AD Route per ES

0x06  PE1 advertises in BGP a split-horizon label associated

Unique per Advertising PE per EVI

Route Type specific encoding of E-VPN NLRI