Brkaci 3101

ACI Under the Hood - How Your
Configuration is Deployed
Phillip Ferrell, Technical Leader Insieme BU Escalation Team
Andy Gossett, Technical Leader Insieme BU Escalation Team
BRKACI-3101
Agenda
• Introduction
• Building the Overlay
• Access Policies
• VRFs, Bridge Domains, and Endpoint Groups
• L2Outs and Loop Prevention
• Traversing the Overlay

• Learning, Forwarding, and Policy Enforcement
• Shared Services and Route Leaking
• L3outs and Routing Protocols
Recommended Sessions
BRKACI-2008 - A Technical Introduction into ACI
BRKACI-2004 - How to setup an ACI fabric from scratch
BRKACI-2102 - ACI Troubleshooting
BRKACI-2003 - Deployment Options for Interconnecting Multiple ACI Fabrics
BRKACI-3503 - Extending ACI to Multiple Sites - Dual Site Deployment Deep Dive
BRKACI-2020 - Understanding Cisco ACI Architecture and Scalable Layer-3 DCI / WAN integration with
OPFLEX
BRKACI-2001 - Integration and Interoperation of Existing Nexus Networks into an ACI Architecture
BRKACI-2121 - Making the best of Services Automation with ACI Service Graph and Python
BRKSEC-3004 - Deep Dive on Cisco Security in ACI
CCSACI-3000 - ACI Real World Deployment
BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Acronyms/Definitions
Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure LPM Longest Prefix Match
ACL Access Control List MDT Multicast Distribution Tree
APIC/IFC Application Policy Infrastructure Controller/ MST Multiple Spanning Tree

Insieme Fabric Controller
BD Bridge Domain pcTag Policy Control Tag
COOP Council of Oracle Protocol PL Physical Local
ECMP Equal Cost Multipath SVI Switch Virtual Interface
EP Endpoint TC Topology Change

EPG Endpoint Group VL Virtual Local
FTEP/VTEP Fabric/Virtual or VXLAN Tunnel Endpoint VNID Virtual Network Identifier
GIPo Outer Group IP Address VXLAN/iVXLA Virtual Extensible LAN / Insieme VXLAN
N
ISIS Intermediate System to Intermediate System XR VXLAN Remote
 Reference Slide
Introduction
Introduction
What are our basic network requirements?
1) Provide paths for endpoints to communicate at

Layer2(MAC) and Layer3(IP) 4) Communication to external L2 networks (DCI)
2) Provide separation of endpoint into Layer2 5) Communication to external L3 networks (WAN)
forwarding domains (vlan or BD)
3) Routing between IP/IPv6 subnets and allow
separation of these into multiple VRFs
L2 L3
EP1 VLAN EP2 EP3 VLAN EP4
1 2 External External
VRF-1
Introduction
What are our basic network requirements?
6) Allow security policies in order to limit communication to between endpoints to allowed protocols
ip access-list web-in
VLAN 1 VRF1 VLAN 2 permit tcp Subnet1 Subnet2 eq 80
Subnet1 Subnet2 ip access-list web-out
permit tcp Subnet2 eq 80 Subnet1
EP1
80 EP3
ip access-group web1 in
ip access-group web2 out

EP2
22 EP4
What physical topology is required?
Physical topology must support our endpoint communication (layer-2 / layer-3)
and the location of endpoints within the physical network will affect the supporting
design/configuration.
L2 L3
EP1 VLAN EP2 EP3 VLAN EP4
VRF-1
Traditional Topology – Routing at Core/Spine
STP results in unused links / limits scale / slower convergence
Layer2 – STP forwarding

Layer2 – STP blocked
Layer3 – ECMP
L2 L3
EP1 VLAN EP2 EP3 VLAN
VRF-1
Traditional Topology – Routing at Access
Restricts L2 endpoint locations / requires separate links for L2 / segmented STP
Layer2 – STP forwarding

Layer2 – STP blocked
Layer3 – ECMP
L2 L3
EP1 VLAN EP2 EP3 VLAN
VRF-1
ACI Infrastructure
Physical links
ISIS is run on links between spines / leaves
ISIS / MDT
L2 L3
EP1 EP2 EP3
External External
ACI Infrastructure
Physical links
APICs communicate to fabric over infra vlan
ISIS / MDT
L2 L3
EP1 EP2 EP3 APIC
External External
ACI Infrastructure
Physical links
Leaves/spines advertise TEP via ISIS
ISIS / MDT
T Tunnel Endpoint (TEP)

L2 v4 v6 L2 v4 v6
L2 v4 v6 Anycast Spine Proxy TEPs
T T T T T
T
L2 L3
EP1 EP2 EP3 APIC
External External
ACI Infrastructure
Physical links
Leaves advertise learned EP to spines via COOP
ISIS / MDT
COOP Oracles T Tunnel Endpoint (TEP)
L2 v4 v6 L2 v4 v6
L2 v4 v6 Anycast Spine Proxy TEPs
10.1.1.57
COOP Citizens
T T T T T
10.1.1.57
L2 L3
EP1 EP2 EP3 APIC
External External
10.1.1.57
ACI Infrastructure
Physical links
BL advertises external routes to fabric through MP-BGP
ISIS / MDT
MP-BGP RRs T Tunnel Endpoint (TEP)
L2 v4 v6 L2 v4 v6
0.0.0.0/0 L2 v4 v6 Anycast Spine Proxy TEPs
RR-Clients
0.0.0.0/0
T T T T T
0.0.0.0/0
L2 L3
EP1 EP2 EP3 APIC
External External
ACI Infrastructure
APIC provisions BD/VRF VXLAN overlays based on EPG attachments
EPG1 EPG1 EPG2 EPG-L2Ext l3extSubnet EPG2 101/1/5

L2Out
EPG1 103/1/3
104/1/8
102/1/1
l3extInstP 105/1/10
L2 L3
EP1 EP2 EP3 APIC
BD-1 BD-2 External External
VRF-1 VRF-1 VRF-1
VXLAN
VXLAN differentiates tunneled traffic based on VNID field.
OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
Flags
Reserved
I
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Virtual Network Identifier (VNID) Reserved
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
iVXLAN
In addition to differentiating traffic based on VNID, iVXLAN allows the source EPG of traffic to be identified
by the source group (PCTAG) bits and to determine if policy was applied by source (SP) / destination (DP).
OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header
iVXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
Flags S D
Reserved Source Group
I P P
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Virtual Network Identifier (VNID) Reserved
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
ACI Infrastructure
Policy is implemented through contracts / filters specifying allowed traffic
EPGs have a consumer / provider

relationship to a contract.
cons prov
EPG1 EPG2
HTTP (80)
EPG1 EPG2 EPG-L2Ext l3extSubnet

L2 L3
EP1 EP2 EP3 APIC
BD-1 BD-2 External External
VRF-1
MAC Header
Ethernet Frame
MAC Header PAYLOAD FCS
Destination MAC Address (DMAC)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
… Source MAC Address (SMAC)
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
EtherType
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
MAC w/802.1Q Header
Ethernet Frame
802.1Q
Destination MAC Address (DMAC)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
… Source MAC Address (SMAC)
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
PCP / D
Tag Protocol Identifier (0x8100) E VLAN Identifier
COS I
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
EtherType
128 129 130 131 132 133 134 135
IPv4 Header
Ethernet frame containing IP packet
MAC Header IPv4 Header PAYLOAD FCS
802.1Q
Version Header Length DSCP ECN Total Length
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
D M
Identification R Fragment Offset
F F
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
Time to Live (TTL) Protocol Header Checksum
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
Source IP Address
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
Destination IP Address
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
TCP Header
Ethernet Frame containing TCP packet
MAC Header IPv4 Header TCP Header FCS
PAYLOAD FCS
802.1Q
Source Port Destination Port
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Sequence Number
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
Acknowledgement Number
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
Header Length Reserved Flags / Control bits Window Size

N C E U A P R S F
96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127
Checksum Urgent Pointer
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
UDP Header
Ethernet Frame containing UDP packet
MAC Header IPv4 Header UDP Header FCS
PAYLOAD FCS
802.1Q
Source Port Destination Port
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Length Checksum
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
Access / 802.1q Trunked Hosts
Ethernet Frame
Ethernet Frame
802.1Q
802.1Q
Trunk
Ethernet frame containing IP packet
MAC Header IPv4 Header PAYLOAD FCS
802.1Q
Ethernet Frame containing TCP packet

MAC Header IPv4 Header TCP Header FCS
PAYLOAD FCS
802.1Q
Ethernet Frame containing UDP packet

MAC Header IPv4 Header UDP Header FCS
PAYLOAD FCS
802.1Q
Hypervisor Host w/AVS
AVS or Openstack connected hosts can

be configured to use VXLAN
encapsulation.
This traffic must be received on the infra
vlan and destined to the FTEP address
VXLAN Tunnel
of the leaf.
OUTER INNER
MAC Header IPv4 Header UDP Header VXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
ACI Intra-fabric
All tenant traffic is iVXLAN encapsulated

when forwarded between leaves/spines.
802.1Q
Trunk
VXLAN Tunnel
OUTER INNER
MAC Header IPv4 Header UDP Header iVXLAN Header MAC Header IPv4 Header UDP Header PAYLOAD FCS
802.1Q
Access Policies
Access Policies
Access policies refer to the configuration that is applied for physical and virtual
(hypervisors/VMs) devices attached to the fabric.
Broken into a few major areas:
Global Policy Switch Policy Interface Policy

• Pools • Policies • Policies
• Domains • Policy Groups • Policy Groups
• Attachable Access Entity Profiles • Profiles • Profiles
Global Policy
Pools (Vlan / VXLAN) Pool1 Pool2
A resource pool of encapsulations that
can be allocated within the fabric.
Domains (Physical / External Bridged / External Routed)

Administrative domain which selects a vlan/vxlan pool for DomPhy1 DomL2Ext1
allocation of encaps within the domain
Attachable Access Entity Profiles (AEP)

AEP
Selects one or more domains and is referenced/applied TenantA
by interface policy groups.
Global Policy - Attachable Entity Profiles
Configuration:
• Create a VLAN/VXLAN pool with a range
of encapsulations Pool1 Pool2 Pool3 Pool4
• Create a domain (physical, l2/l3 external, DomPhy1 DomVm1 DomL2 DomL3
or VMM) and associate pool
• Associate domain to AEP AEP AEP AEP
• Associate interface policy group to AEP Statics VMs External
switch/interface selectors will apply the
config through the interface policy group
assign to specific ports
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
What have we accomplished?
• Specified what domains and
corresponding pools are allowed per
interface in the fabric!
Access Policies SWITCH POLICY
Policies define protocol / feature configurations
Policy Groups select which policies should be applied
Profiles associate policy groups to switches or interfaces,

through the use of selectors
Switch Policy Types: Interface Policy Types:

VPC Domain Link-level Storm Control
Spanning-tree (MST) CDP Data plane policing
BFD LLDP MCP
Fibre-channel SAN / Node Port-channel / LAG L2 (Vlan local / global) INTERFACE POLICY
Port-channel member Firewall
Spanning-tree
Interface Policy Groups
Used to specify which interface policies to be applied to a particular interface type.
It also associates an AEP (which defines which domains are allowed on the interface).
Types:
VPC Domain 1
Access port (EP1)
Access Bundle Groups
• Virtual Port-channel (EP2)
• Port-channel (EP3)
EP1 EP2 EP3
Note: Separate policy groups should be created for each port-channel (standard or VPC) that you
need to configure. All interfaces on leaf that are associated with a particular access bundle group reside
in same channel.
Port-Channel Policies
Classical vPC Domain configuration ACI Port-Channel Policies

Required configuration of domain, peer-link, and Specify mode, minimum / maximum links, and related
peer-keepalive link on both devices in domain protocol options (relating to LACP)
interface Ethernet1/5-6
lacp port-priority 32768
lacp rate normal
channel-group 10 mode on
interface Ethernet1/10-11
lacp port-priority 32768
lacp rate fast
channel-group 20 mode active
Access Policy Example
General Configuration (reused for many interfaces): AEP Pool1
1) Configure a physical domain and vlan pool Vandalay
DomPhy1
2) Create an AEP and associate physical domain
3) Create switch/interfaces profiles for leaf (LEAF101) Switch Profile
• very easy to apply configurations if you create a LEAF101

switch/interface profile for each leaf and one for
each VPC domain pair blk_101
4) Configure Interface policies (LACP / LLDP)

Interface Profile
LACP Active LEAF101
Policies
LLDP Rx / Tx enabled
Creating Physical Domain / AEP / Vlan Pool
In dropdown:
Click Create Attachable Entity Profile
Click + to add vlan range
In dropdown:
Click Create VLAN Pool
Specify start and

end vlans in
range
Create Interface Profile for each leaf / VPC domain
Enter name and submit
Create Switch Profile for each leaf / VPC domain
Create Switch Profile for each leaf / VPC domain
Enter name
Click + to add selector
Select the Interface Profile

created for this leaf earlier
Enter a name and choose

appropriate leaf or leaves
(for vpc pair)
Create common protocol configurations
Example demonstrates a common lacp port-channel policy
Use a descriptive name
Select the protocol
Configure options/knobs
Access Policy Example
Interface specific (each time you add a new interface): AEP Pool1
1) Create policy group for device (VPC / PC / Access) Vandalay
DomPhy1
2) Within the policy group, select the desired policies /
AEP Switch Profile
3) Associate interfaces to policy group via desired leaf LEAF101
profile
• use specific leaf profile if access or PC blk_101
• use VPC leaf profile if policy group is VPC

Interface Profile
LACP Active PC_Server_1 LEAF101
Policies Policy Groups blk_1/1-2
LLDP Rx / Tx enabled Access_Servers blk_1/47-48
Create policy groups
Note:
A separate policy
Descriptive name
group should be
created for each
PC/VPC that you will
deploy
Associate your desired
interface policies
(otherwise default)
Associate your AEP to select

which domains this interface
can deploy
Create interface selectors / associate policy group
Click + to add selector
Choose interface profile

to add selectors
Use a descriptive name
Specify interface/range
Associate the policy group to

deploy on interfaces
Example policy scheme
Switch Profile Leaf101 Leaf101_102
Interface Profile Leaf101 Leaf101_102
Interface Selector linux windows n7k_pc10 asa_cl1_pc1 n7k1_pc10 n7k2_pc10
Interface Block 1/20-25 1/30-35 1/10-11 1/45-48 1/10 1/20
1/1-4
Interface Policy Group linux-access windows-access asa_vpc_ccl asa_vpc_data n7k_vpc10
vPC Protection Group Policy
vPC Domain 1
vPC Domain 1 vPC Domain 2

Classical vPC Domain configuration ACI vPC Domain configuration
Required configuration of domain, peer-link, and Specify the Domain ID and the two Leaf switch IDs that
peer-keepalive link on both devices in domain form the domain pair
vpc domain 1
peer-keepalive destination 172.168.1.2 / VPC Protection Group
source 172.168.1.1 vrf vpc-keepalive
peer-gateway
Name: vPC-Domain100
ip arp synchronize
ID: 100
interface port-channel 20 Switch1: 101
vpc peer-link
Switch2: 102
VPC Protection Group (example configuration)
GUI sequence:
Tabs:
Fabric -> Access Policies
Navigation Tree:
Switch Policies -> Policies ->
VPC Domain -> Default
VRFs, Bridge Domains, and
Endpoint Groups
VRF/BD/EPG Logical Configuration
VRF-Vandalay
BD-Importers BD-Exporters
Importer-1 Importer-2 Exporters
IM1 IM2 IM3 EX1 EX2
Classical configuration steps ACI Logical configuration

• Create VRF • Create Tenant
• Create Vlans • Create VRF
• Create Vlan interfaces • Create BDs
• Associate to VRF • Associate to VRF
• Assign Subnets / configure gateway • Define a Subnet (optional)
redundancy • Create App Profile
• Assign encapsulation to interfaces • Create EPGs
• Associate to Domain
• Define a Subnet (optional)
Each node must be individually configured with the
Classical VRF/BD config VRF, associated vlans/BDs, and an SVI with unique
IP. For gateway redundancy, HSRP must also be
configured.
VRF-Vandalay vrf context vandalay
vlan 100
BD-Importers BD-Exporters name importers
vlan 200
Importer-1 Importer-2 Exporters name exporters
feature interface-vlan
IM1 IM2 IM3 EX1 EX2 feature hsrp
interface Vlan100
vrf member vandalay
ip address 10.10.0.2/24
ip address 10.20.0.2/24 secondary
vlan-100 vlan-100 vlan-100 vlan-200 vlan-200 hsrp 100
ip 10.10.0.1
interface Vlan200
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 vrf member vandalay
ip address 10.30.0.2/24
hsrp 200
ip 10.30.0.1
5 6 6 5 7 5 6 2
interface Ethernet1/1
switchport trunk vlan allowed 100
vlan-100 interface Port-channel1
vPC Domain vlan-100 vlan-200 switchport access vlan 200
ACI Logical Configuration
Tenant: Vandalay Industries • Create Tenant

• Create VRF
Networking App Profile: Operations • Create BDs
VRF: Vandalay EPG: Importer-1 • Associate to VRF
Subnet: 10.20.0.1/24 • Create an App Profile
BD: Importers • Create EPGs
EPG: Importer-2 • Associate to Domain
Subnet: 10.10.0.1/24
EPG: Exporters
BD: Exporters
Subnet: 10.30.0.1/24
What have we accomplished?
Specified the logical configuration that should be
deployed on each leaf where EPG is deployed.
We also restricted which interfaces can deploy
Domain: DomPhy1 the EPG through Domain associations.
Overlay Fabric Allocations
VRF-VNID – allocated per VRF
Tenant: Vandalay Industries • (unique within fabric)
Networking App Profile: Operations
BD-VNID – allocated per BD
VRF: Vandalay EPG: Importer-1 • (unique within fabric)
Subnet: 10.20.0.1/24
PCTAG – allocated per EPG
BD: Importers • FABRIC-global if shared service
EPG: Importer-2 provider
Subnet: 10.10.0.1/24
• VRF-local otherwise
EPG: Exporters
BD: Exporters EPG-VNID – allocated from vlan pool
Subnet: 10.30.0.1/24 (domain specific) and is unique within
fabric
• Used for STP BPDU flooding and
Domain: DomPhy1 flood in encap for unknown unicast
traffic
Creating a Tenant
Create a tenant by clicking the
Tenant Tab and ‘Add Tenant’
icon.
Provide a name for the new
Provide a name for the new tenant
tenant.
Click ‘Add Tenant’
Creating a VRF in the Tenant
Right click on the VRF’s under
the networking folder and
choose ‘Create VRF’. Provide a name for
the new VRF
Provide a name for the new
VRF
Under the networking tab,

on the VRF folder, Right
click and choose ‘Create
VRF’
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creating a BD and associate with VRF
Create a new BD by right clicking on the ‘Bridge Domain’ folder under the Networking tab and choose
‘Create Bridge Domain’
Provide a name for the new BD and associate it to the previously created VRF.
Click ‘Next’ and leave the L3 Configurations and Advanced/Troubleshooting with default values
Under the networking tab, on the

Bridge Domain folder, Right
click and choose ‘Create Bridge
Domain’
Provide a name for the

new Bridge Domain
Associate the BD to the

VRF
Adding a Subnet to a BD
Create a new Subnet under the
bridge domain by right clicking
the subnets folder and choose
‘Create Subnet’
Configure the subnet

address and mask
More details on the Subnet

Scope flags in the Routing
section. For now, leave default
scope of ‘private’
Under the BD, on the Subnet

folder, Right click and choose
‘Create Subnet’
Creating an Application Profile
Create a new application profile by right
clicking the folder and choose ‘Create
Application Profile’

application profile
Under Application Profiles, Right

click and choose ‘Create
Application Profile’
Optionally, configure new EPGs
with associated BDs, Domains, and
static paths
Creating an Application EPG
Create the EPG and associate it
with the correct BD.

EPG
Associate the EPG to

the correct BD
Under Application EPGs folder,

click ‘Create Application EPG’
Adding a Domain to the EPG
After the EPG has been creating, associate a physical
domain by right clicking on the Domains folder and
choosing ‘Add a Physical Domain’.
Associate the EPG to

the appropriate
domain
Right click and choose ‘Add

Physical Domain’
Adding a Static Path to the EPG
To add a static path, under the Static Bindings folder
right click and choose ‘Deploy Static EPG’.
Specify the static path port, port-channel, or VPC
along with the VLAN encap
Select the static path to

deploy the EPG along
with the VLAN encap
Under the EPG on the Static

Bindings folder, right click and
choose ‘Deploy static EPG’
EPG Static Path Deployment
EPG are deployed through:
VRF-Vandalay • Static binding to port/PC/VPC
BD-Importers BD-Exporters • Static binding to node
• VM attachment
IM1 IM2 IM3 EX1 EX2 To successfully deploy an EPG
configuration on a leaf:
1. AEP of target interface must allow
same domain as assigned to EPG
2. encapsulation/vlan must be allowed
in the target domain
vPC Domain
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
DomPhy1
IM1 IM2 IM3 EX1 EX2
vlan 100-200
EPG Static Path Deployment Leaf101 BD-Importers vlan-101
VRF-Vandalay VRF-Vandalay 10.10.0.1/24
BD-Importers BD-Exporters 10.20.0.1/24
Leaf102 BD-Importers vlan-101

vlan-102
IM1 IM2 IM3 EX1 EX2
VRF-Vandalay 10.10.0.1/24
10.20.0.1/24
Leaf103 BD-Exporters vlan-102

vlan-101 vlan-102 vlan-500 vlan-102 vlan-200
vPC Domain
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
VRF-Vandalay DomPhy1
10.30.0.1/24
vlan 100-200 IM1 IM2 IM3 EX1 EX2
vlan 100-200
EPG Static Path Deployment Leaf101 BD-Importers vlan-101
VRF-Vandalay VRF-Vandalay 10.10.0.1/24
BD-Importers BD-Exporters 10.20.0.1/24
Leaf102 BD-Importers vlan-101

vlan-102
IM1 IM2 IM3 EX1 EX2
10.20.0.1/24

vlan-101 vlan-102 vlan-110 vlan-102 vlan-200
vPC Domain VRF-Vandalay 10.30.0.1/24

10.10.0.1/24
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 BD-Importers vlan-110

AEP
VPC1 102/1/2 103/1/1 PC1 104/1/3 Pool1
Statics
VRF-Vandalay DomPhy1
10.30.0.1/24
vlan 100-200 IM1 IM2 IM3 EX1 EX2
vlan 100-200
Common Network Faults
vlan 100-200
Suppose Pool1 contain VLAN
block of 100-200 Pool1 Pool2 Pool3 Pool4
• EPG-E1 and EPG-E3 are associated DomPhy1 DomVm1 DomL2 DomL3
to domain DomPhy1
• A static path is added for each EPG
AEP AEP AEP
Network Faults: Statics VMs External
• EPG-2: Invalid Path, Invalid VLAN
Domain DomPhy1 not associated to
EPG-E2.
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
• EPG-3: Invalid Path
vlan-101 vlan-151 vlan-201
Interface AEP is not associated with
domain DomPhy1
E1 E2 E3
Add EPG-E1 and EPG-E3 to DomPhy1

L2Outs and Loop Prevention
Extending Layer-2 domain outside of ACI
Extend the bridge domain with L2Out
L2 Outside network extends the bridge domain

to legacy devices through an external EPG.
All traffic for the extended BD will be

encapsulated using the specified vlan tag (one vlan-500
vlan-100 vlan-100
per L2Out) on the specified interfaces and is
always tagged.
EP3 EP4
Policy is enforced between the external EPG EP1 EP2
and all other EPGs in the fabric.
External EPG –
LegacyApache EPG – Apache
BD – Webservers
BD – Webservers
Extending Layer-2 domain outside of ACI
Extend an EPG to legacy switches
EPG is extended to external devices using

regular static-path bindings to ports (along
with desired encap/vlan).
The leaf will learn the endpoint information

and assign the traffic (by matching the port vlan-500
vlan-100 vlan-100
and VLAN ID) to the proper EPG, and then
enforce the policy.
EP3 EP4
The endpoint learning, data forwarding, and EP1 EP2
policy enforcement remain the same whether
the endpoint is directly attached to the leaf
port or if it is behind a L2 network (provided
the proper VLAN is enabled in the L2 EPG – Apache
EPG – Apache
network).
BD – Webservers
BD – Webservers
Role Description
Spanning Tree R
D
Root port
Designated port
Classical behavior B (Blk) Blocking port
• STP BDPUs (PVST or MST) are
generated by each switch in the topology. Root Bridge
• STP root is elected and interface
forwarding is calculated to prevent loops
by blocking some interfaces. D D
• All interfaces with best-path (highest
bandwidth) towards root bridge will be
forwarding.
• Backup paths will be put in a blocking R R
state by the switch with worst path
towards root on the affected path
(usually based on either the bridge
identifier or port priority) D B
• Topology changes (TC) trigger MAC
addresses to be flushed in received vlan,
allowing traffic reconvergence based on
new topology
Spanning Tree
ACI floods BPDUs in the fabric encap
• ACI leaves don’t participate in spanning
tree (generate BPDUs or block any ports)
• STP BDPUs (PVST or MST) are flooded
within the fabric/EPG encap (allocated per
vlan encap in a domain)
• Leaves flush endpoints in the EPG if a TC
BPDU is received. EPG - Web
• Spanning Tree Domain policy
determines which EPGs to flush for
MST domain TCs BPDU BPDU
D D
NOTE: MST BPDUs are untagged and
require an untagged/native EPG to be Root
Root Bridge
deployed on all interfaces connected to MST Bridge
domain (this includes L3outs using SVIs)
Spanning-Tree Policy
Classical MST Configuration
Requires configuration of STP mode, MST region,
MST revision, and vlan assignments to MST instances
Root Bridge
Note: MST configuration must match for all switches
within a specified region. If they do not, any port
receiving conflicting or legacy BPDUs will be treated as D D
part of the IST instance.
switch(config)# show run spanning-tree

spanning-tree mode mst R R
spanning-tree mst configuration
name Region1
revision 1
instance 1 vlan 1-9,20-29 D B
instance 2 vlan 10-19,30-39
Spanning Tree Domain Policy
ACI MST Configuration
Configuration is fabric-wide and supports multiple
regions for use within different tenants/domains.
Any ports connecting to MST switches within the

same region MUST have untagged static-path.
Each MST region should have it’s own EPG for

BPDU flooding.
Fabric -> Access Policies -> Switch Policies ->

Spanning Tree -> default
• Add a Region Policy
• Add a Domain Policy for each MST instance
within the region (instance 0 is implicit)
• Add vlan blocks
Common mistakes that cause loops
Missing untagged/native EPG in MST region
MST BPDUs are sent untagged by switches

and will only be accepted by leaf if an EPG is
deployed with an untagged/native EPG path
binding.
EPG - Web
All interfaces connected to a common MST
vlan-100 vlan-100
region should have the same EPG deployed
(this is to ensure BPDU is flooded to all of the D D
MST switches connected to fabric).
LOOP!!
BPDUD R BPDU
BPDU BPDU
Root
Bridge
Common mistakes that cause loops
Multiple fabric encaps used for same EPG
BPDUs are flooded within the fabric encap of

an EPG (allocated based on domain/vlan
pool). Domain Domain
A B
In order for BDPUs to be flooded properly, all EPG - Web
interfaces within the EPG that are connected
vlan-100 vlan-100
to external bridges MUST reside in the same
physical or L2 external domain and vlan D D
encapsulation.
LOOP!!
BPDUD R BPDU
BPDU BPDU
Root
Bridge
Agenda
• Introduction
• Access Policies

Learning, Forwarding, and Policy
Enforcement
Classical Learning and Forwarding
Encap + Interface => VLAN
VLAN => VRF
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC
L2 Forwarding for (VLAN, DMAC)

L2 Learning for (VLAN, SMAC) => (Interface)
L3 Forwarding for (VRF, DIP)
L2 Forwarding: L3 Forwarding (Longest Prefix Match)

(VLAN, DMAC) Miss => Flood (VRF, DIP) Miss => Drop
(VLAN, DMAC) Gateway MAC => Route (VRF, DIP) Hit=> Adjacency
(VLAN, DMAC) Hit => Destination Port
config on destination port + VLAN
Might be Glean or packet rewrite (SMAC, DMAC,
determines egress encap
VLAN, etc…), may include destination port in adjacency
(tagged or untagged)
or require second L2 lookup on new DMAC
ARP Packet
Classical Learning and Forwarding
DMAC
SMAC
LPM Routes
Eth: 0x0806
• Connected/direct routes manually Route Adj
configured 10.1.1.101/32 … Hdr/Opcode
• Static/dynamic routing protocols to 10.1.1.0/24
20.1.1.101/32 …
Glean Sender MAC
learn prefixes 20.1.1.0/24 Glean
Sender IP
Host Routes
A Target MAC
• Glean adjacency for connected routes P
to punt frame and generate ARP A Target IP
request
• ARP/ND used to create MAC to IP
binding and install host route into 10.1.1.101/24 20.1.1.101/24
routing table
ACI Learning and Forwarding (Physical Local - PL)
NEW
Encap + Interface => EPG
EPG => BD EPGs and L3
BD => VRF Learning
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC
L2 Forwarding for (BD, DMAC)

L2 Learning for (BD, SMAC) => (EPG, Interface)
L3 Learning for (VRF, SIP) => (EPG, Interface)
L2 Forwarding: L3 Forwarding (Longest Prefix Match)

(BD, DMAC) Miss => (Flood/Proxy+Drop) (VRF, DIP) Miss => Drop
(BD, DMAC) Gateway MAC => Route Proxy/Glean for BD subnets
(BD, DMAC) Hit => Adjacency (VRF, DIP) Hit=> Adjacency
Adjacency contains dst EPG, encap

information, dst VTEP or port, etc…
More in upcoming
© 2016slides
Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Optimize Forwarding
ACI Learning and Forwarding (ARP) (ARP Flooding disabled)
Encap + Interface => EPG

EPG => BD
Target Target Sender Sender Hdr/ ethtype
802.1Q SMAC DMAC BD => VRF
IP MAC IP MAC Opcode ARP
L2 Learning for (BD, SMAC) => (EPG, Interface)
L2 Learning for (BD, ARP SMAC) => (EPG, Interface)

L3 Learning for (VRF, ARP Sender IP) => (EPG, Interface)
L3 Forwarding for (VRF, ARP Target IP)
ARP L3 Forwarding
L3 forwarding based on ARP target IP field
(VRF, ARP Target IP) Miss => Proxy
with miss sent to spine proxy 
(VRF, ARP Target IP) Hit=> Adjacency
ACI Learning (Virtual Local - VL) Fabric TEP
AVS VTEP
Inner Header VXLAN Outer Header
Infra VLAN
Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID Rsvd DIP SIP 802.1Q SMAC DMAC
UDP
External VNID => EPG Infra BD MAC

EPG => BD AVS MAC
BD => VRF
L2 Learning for (BD, SMAC) => (EPG, Tunnel)
L3 Learning for (VRF, SIP) => (EPG, Tunnel)
VXLAN Tunnel
ACI Learning (Remote - XR) Dst Leaf VTEP
Src Leaf VTEP
Inner Header iVXLAN Outer Header
Fabric QoS
flags Proto
L4/Payload Proto DIP SIP ethtype SMAC DMAC VNID DIP SIP 802.1Q SMAC DMAC
EPG UDP
EPG (pcTag) Internal MAC
BD or VRF VNID (based on routed or switched)

L2 Learning for (BD, SMAC) => (EPG, Tunnel)
L3 Learning for (VRF, SIP) => (EPG, Tunnel)
ACI Learning (COOP vs. EP Sync)
COOP sync between oracles (Spines)

Spines learns all
endpoints through Coop
COOP citizen(leaf) update to oracle
remote learn on leaf (spine) for local EP learn
from dataplane packet
vPC Domain 1 vPC Domain 2

local learn on leaf
EP sync between vPC peers from dataplane packet
EP sync between vPC peers
for remote learns for local learns
(both orphan and vPC ports)
ACI Learning (EP) Leaf Endpoint Database
Endpoint Entry
What is an EP (Endpoint)? - EPG (pcTag)
• MAC Remote IP Entries - Interface/Tunnel
VRF (VRF, IP)
• IPv4 (/32) or IPv6 (/128) host route - Control flags
Frame Forwarding Learn

Operation
Remote MAC Entries
BD
Non- Bridged MAC (VRF, BD, MAC)
IP/IP
ARP - MAC (sender-HW), Local MAC and IP Entries
IP (sender-IP) (VRF, BD, VLAN/VXLAN, MAC)
Encap
IPv4 Unicast MAC, IP (VRF, BD, VLAN/VXLAN, IP)
Routed
IPv6 Unicast MAC, IP IP
Routed IP
Entry
Mac IP
Entry
IPv6 Neighbor MAC, IP Entry IP
Entry
Discovery Relationship to Entry
multiple IPs
ACI Learning
Learning Exceptions
• No IP EP learning if routing is disabled
on the BD
• No IP EP learning on external BD’s
(Layer-3 Outside interfaces)
• **No IP EP learning on Infra VLAN
VXLAN/Opflex traffic
VXLAN Tunnel
• No IP learning of shared service
prefixes outside of our VRF between AVS and
fabric on Infra VLAN
LPM Routes (Same as Classical)
• Pervasive SVI Routes (BD Subnets) Static/Dynamic WAN/Inte
Routing on L3Out rnet
• Static and dynamic routing protocols
on L3Out AVS
ARP has resolved on both hosts.
L1 does not have H2 in EP database
ACI Forwarding Hardware Proxy enabled on BD-B1
Unknown Layer2 Unicast: Hardware Proxy 1. H1 sends layer2 unicast frame to H2.
Layer2 Spine Proxy
2. L1 performs layer2 lookup on H2
Policy Applied destination MAC and misses. Frame is
S1 S2 on egress L3 sent to Spine Anycast MAC Proxy
VTEP. EPG-E1 and BD-B1 VNID set in
2 3 VXLAN header. No policy applied since
destination EPG is unknown
4 3. Spine performs EP lookup on H2

L1 L2 L3 L4 L5 L6 destination MAC. If unknown drops the
packet. Else forward to VTEP of L3
5
1 4. L3 performs layer2 lookup on H2
destination MAC. Hit in local EP
H1 H2 H3 database and derives destination EPG-E2
BD-B1 VRF-V1 BD-B2 L3 applies policy between EPG-E1 and
EPG-E1 EPG-E2 EPG-E3 EPG-E2
5. If permitted, traffic forwarded to H2 with
appropriate encap
L1 does not have H2 in EP database
ACI Forwarding Layer2 flood enabled on BD-B1
Unknown Layer2 Unicast: Layer2 Flood 1. H1 sends layer2 unicast frame to H2.
2. L1 performs layer2 lookup on destination
Implicit policy MAC and misses. Flood frame sent on
S1 S2 permit on BD for BD-B1 GIPo. EPG-E1 and BD-B1 VNID
flooded traffic set in VXLAN header.
2 3 Implicit permit rule for flooded traffic on
BD-B1
3. Spine forwards flood frame on BD-B1
L1 L2 L3 L4 L5 L6 GIPo and FTAG (multicast tree) to all leafs
containing BD-B1
4
1 4. L3 floods packet on BD-B1
H1 H2 H3 Same behavior for ARP

BD-B1 VRF-V1 BD-B2 broadcast when ARP flooding
EPG-E1 EPG-E2 EPG-E3 is enabled
Hardware Proxy Enabled
under the BD
L2 Unknown Unicast flood
with ARP flooding enabled
ACI Forwarding
BD Multicast Settings
Layer 2 Multicast Layer 3 Multicast (IANA range)
• Flood in BD • Known multicast traffic will have
flood to all ports in bridge domain IGMP/MLD snooping entry and
• Flood in Encapsulation forwarded to appropriate ports
flood to all ports matching ingress
• Unknown multicast
encapsulation. This may be a subset of
ports in the bridge domain • Flood
flood to ports in bridge domain
• Drop • Optimize Flood
send only to router ports detected by
PIM hellos
L1 has learned H2 from L3
ACI Forwarding
1. H1 sends layer2 unicast frame to H2.
Known Layer2 Unicast
destination MAC and finds endpoint with
Policy Applied destination EPG-E2 and VTEP of L3.
S1 S2 on ingress L1 L1 applies policy between EPG-E1 and
EPG-E2. If permitted, frame is sent to L3
2 3 VTEP with EPG-E1 and BD-B1 VNID set
in VXLAN header.
4 3. Spine receives frame with outer

L1 L2 L3 L4 L5 L6 destination IP of L3 and routes packet.
5 4. L3 does layer2 lookup on H2 destination

1 MAC in BD-B1. Hit in local EP database
and derives destination EPG-E2. Since
H1 H2 H3 policy already applied on L1, no policy
BD-B1 VRF-V1 BD-B2 check on L3.
EPG-E1 EPG-E2 EPG-E3
5. L3 forwards traffic to H2 with appropriate
encap
ARP has resolved on hosts for ACI GW
L1 has learned H3 from L6
ACI Forwarding
1. H1 sends layer3 unicast frame to H3
Known Layer3 Unicast (destination MAC of BD-B1).
Policy Applied destination IP and finds endpoint with
S1 S2 on ingress L1 destination EPG-E3 and VTEP of L6.
L1 applies policy between EPG-E1 and
2 3 EPG-E3. If permitted, frame is sent to L6
VTEP with EPG-E1 and VRF-V1 set in
VXLAN header.
4
L1 L2 L3 L4 L5 L6 3. Spine receives frame with outer
destination IP of L6 and routes packet.
5
1 4. L6 does layer3 lookup on H3 destination
IP in VRF-V1.Hit in local EP database and
H1 H2 H3 derives destination EPG-E3. Since policy
BD-B1 VRF-V1 BD-B2 already applied on L1, no policy check
EPG-E1 EPG-E2 EPG-E3 on L6.
5. L6 forwards traffic to H3 with appropriate
encap
No endpoints initially learned
ACI Forwarding 1. H1 sends ARP broadcast request for H2.
L1 learns MAC and IP for H1
ARP/IP Unknown Layer3 Endpoint (glean) L1 performs layer3 forwarding lookup
Layer3 Spine Proxy Miss based on ARP target IP address for H2
2. H2 not present on L1, send to Spine
S1 S2 Anycast IPv4 Proxy VTEP. VRF-V1
VNID set in VXLAN header.
2 3 No policy applied since destination EPG
is unknown
3. Spine does not have H2 IP, sends special
L1 L2 L3 L4 L5 L6 glean packet to all leafs on reserved GIPo
with VRF-V1 VNID set in VXLAN header
4 4
5
1 4. L1 and L3 have BD-B1 subnet present,
generate ARP request for H2 (sourced
H1 H2 H3 from fabric pervasive SVI)
BD-B1 VRF-V1 BD-B2
EPG-E1 EPG-E2 EPG-E3 5. H2 sends ARP response, L3 learns H2
MAC and IP and syncs to Spines
Broken Traffic Flow Example
• A Layer3 gateway device (GW) is connected to the fabric via a
normal BD/EPG. Host H3 is using GW as its gateway for a
L3Out subset of traffic.
• The initial EP database show the IP’s and MACs learned in the
correct locations.
Subnet E1 E2 Subnet
int-S1 int-S2 MAC EP Database
BD-B1 BD-B2
BD MAC EPG Port
1/1 1/2 1/3
BD-B1 mac:G1 E1 1/1
BD-B2 mac:G2 E2 1/2
GW H3
IP:G1 IP:G2 BD-B2 mac:H3 E2 1/3
mac:G1 mac:G2 IP:H3
IP EP Database
H3 gateway mac:H3
FW, LB, Router, etc. Vrf IP MAC EPG Port
v1 IP:G1 mac:G1 E1 1/1
v1 IP:H3 mac:H3 E2 1/3
• H3 sends a frame to GW on BD-B2 (L2 switched through the
fabric). GW routes the frame and sends it toward the fabric to
L3Out be routed out.
• Fabric performs IP learning on routed traffic, IP:H3 moves to
mac:G1 on EGP E1, port 1/1
Subnet E1 E2 Subnet
int-S1 int-S2 MAC EP Database
BD-B1 BD-B2
BD MAC EPG Port
1/1 1/2 1/3
BD-B1 mac:G1 E1 1/1
BD-B2 mac:G2 E2 1/2
GW H3
IP:G1 IP:G2 BD-B2 mac:H3 E2 1/3
mac:G1 mac:G2 IP:H3
IP EP Database
H3 gateway mac:H3
FW, LB, Router, etc. Vrf IP MAC EPG Port
v1 IP:H3 mac:H3
mac:G1 E2
E1 1/3
1/1
ARP for IP:H3 What’s Broken?
sent out EPG-E1
L3Out • ARP to IP:H3 may fail since the IP is pointing to the wrong port
• Routed traffic to IP:H3 may be policy dropped since it’s
classified in EPG-E1 instead of EPG-E2
Subnet E1 E2 Subnet
• IP:H3 may rapidly move within the fabric.
int-S1 int-S2
BD-B1 BD-B2
1/1 1/2 1/3 IP EP Database
ARP for Vrf IP MAC EPG Port
IP:H3H3
GW v1 IP:G1 mac:G1 E1 1/1
IP:G1 IP:G2
mac:G1 mac:G2 IP:H3 v1 IP:G2 mac:G2 E2 1/2
H3 gateway mac:H3 v1 IP:H3 mac:H3
mac:G1 E2
E1 1/3
1/1
FW, LB, Router, etc.
Solutions
L3Out 1. Connect devices that perform routing functionality to

L3Outs.
2. Disable unicast routing on BD-B2 and enable ARP
Subnet E1 E2 Subnet flooding so only MAC is examined when forwarding ARP
int-S1 int-S2 instead of performing (VRF,IP) lookup on ARP target-IP
BD-B1 BD-B2
1/3 3. Enable IP subnet prefix check on BD-B1. This will
1/1 1/2
prevent learning of IP’s outside of the subnets configured
under the BD.
GW H3 4. Enable NAT on routed device connected to internal BD.

IP:G1 IP:G2 In this way, source IP address will be translated
mac:G1 mac:G2 IP:H3 preventing fabric from learning IP address in wrong
H3 gateway mac:H3 location.
FW, LB, Router, etc.
Broken Traffic Flow Example #2 • H1 in EPG-E1 with gateway
configured on BD-B1
ext2
• H2 in EPG-E2 is in layer2 only BD-B2
L3Out with gateway outside the fabric via an
subnet L2Out
L1 L2 ext-S2 H2 subnet is not configured in fabric
Common for during brownfield migration
L2Out
• Traffic from H1 to H2 is routed outside
the fabric via the L3Out and then bridged
H1 H2
BD-B1 BD-B2 back in from an external router via the
subnet L2Out
EPG-E1 int-S1 EPG-E2
• A contract C1 is configured to allow
traffic from EPG-E1 to the L3Out.
VRF-V1
• A contract C2 is configured to allow
traffic from EPG-E2 to its gateway on the
L2Out.
EPG-E1 ext2 L2Out EPG-E2 • Traffic from EPG-E1 to EPG-E2 works
C1 subnet C2 fine but return traffic fails, why?
ext-S2
Broken Traffic Flow Example #2 1. H2 sends ARP request for
external gateway. L2 learns IP
No contract between ext2 from ARP for H2 in EPG-E2
EPG-E2 and EPG-E1 When traffic is received from L3Out on
L3Out 2.
subnet L2 with source IP of H2, L2 derives
L1 L2 ext-S2 source EPG of EPG-E2 instead of the
L3Out external EPG-ext2
L2Out
3. Policy enforcement on L2 is between
EPG-E2 and EPG-E1 instead of L3Out
H1 H2
BD-B1 BD-B2 EPG-Ext2 and EPG-E1.
subnet Since there is no contract defined
EPG-E1 int-S1 EPG-E2 between these EPGs, traffic is dropped
How to fix this issue?
VRF-V1
• Disable Unicast Routing on BD-B2. This
will prevent Layer2 only BD’s from
Enabling ‘Enforce Subnet Check’ on the learning endpoint IP’s from host ARP
BD is recommended for preventing the • OR, enable ‘Enforce Subnet Check’ on
fabric from learning rogue/misconfigured BD-B2
hosts on layer3 BD’s
enable/disable unicast
routing under the BD
enable/disable subnet
check under the BD
Classical Policy Enforcement
Ingress Egress Type Access Control Entry (ACE) Format
Pipeline Pipeline MAC action src/mask dst/mask ethertype [PD filters]
ARP action opcode srcIp/mask dstIp/mask srcMac/mask
1 2 3 4 5 dstMac/mask [PD filters]
IP/IPv6 action protocol srcIp/mask srcPort/mask dstIp/mask
dstPort/mask [PD filters]
• Multiple logical locations where ACLs can

be applied depending on what type of traffic
Egress VLAN ACL and what type of filters are needed (very
Egress Routed ACL flexible)
Ingress Routed ACL • ACE primarily based on src and dst values
within frame (may be hard to maintain)
Ingress VLAN ACL
Ingress Port ACL • ACLs often need to be configured and
maintained on multiple devices in the
network
• Policy is created based on contract between EPGs
ACI Policy Enforcement with support for L2/L3/L4 filters similar to traditional
ACLs.
• Leaf derives source EPG pcTag based on:
Scope Access Control Entry (ACE) Format • match in EP database
VRF action src-EPG dst-EPG [filters] src MAC for L2 traffic or src IP for L3 traffic
VRF permit any any (unenforced mode) • longest-prefix match against src IP
(IP-based EPG or L3Out external EPG)
• ingress port + encap
1
• Leaf derives destination EPG pcTag based on:
• match in EP database
dst MAC for L2 traffic or dst IP for L3 traffic
Apply Policy • longest-prefix match against dst IP
(L3Out external EPG or shared-services)
Derive destination EPG pcTag
EP lookup, IP Prefix • Rules are programmed with scope of VRF. Policy
lookup is always (VRF, src-EPG, dst-EPG, filter).
Derive source EPG pcTag
• Allow traffic between all EPGs without a contract
local EP, IP Prefix, or Encap
by setting the VRF to unenforced mode
ACI Policy Enforcement SYN
Web Server
(S1)
Reference TCP Packet SYN+ACK
H1
Data
Seq#, Ack# Dst Src Proto
DIP SIP ethtype SMAC DMAC
ACK
flags, etc.. Port Port TCP
port x data… port 80
Classical Switch ACL ACI Contract

Generally applied at one or more L3 boundaries
assuming H1 and S1 are in different subnets
H1
ip access-list web
permit tcp host H1 host S1 eq 80
permit tcp host S1 eq 80 host H1 EPG-Client EPG-Web
BD-X VRF-V1 BD-Y
ACI Desired Behavior
EPG-Web is Providing
Scope Access Control Entry
a service on port 80
VRF-V1 permit tcp EPG-Client EPG-Web eq 80
VRF-V1 permit tcp EPG-Web eq 80 EPG-Client
How do we get here?
Option 1 – Unidirectional filters
ACI Policy Enforcement Apply both flt-1 and flt-2 to subject
 Identify Provider (P) EPG and Consumer (C) EPG
flt-1 (C to P) and flt-2 (P to C)
src-port dst-port
permit tcp Consumer Provider eq 80
H1 C P
permit tcp Provider eq 80 Consumer
EPG-Client EPG-Web Option 2 – Bidirectional filters with reverse ports

BD-X VRF-V1 BD-Y
flt-1 (C to P implied)
•
With a bidirectional contract, the ‘provider’ will be permit tcp Consumer Provider eq 80
the dst-port filters and the ‘consumer’ will be the
src-port filters (opposite of contract arrows) flt-1 + apply both directions
 Create Filters permit tcp Consumer Provider eq 80
Name EthType Proto Src Port Dst Port permit tcp Provider Consumer eq 80 Only flt-1
flt-1 IP TCP Any 80 needed!
flt-2 IP TCP 80 Any flt-1 + apply both directions + reverse ports
 Create a contract, subject, and filter(s). Apply to EPGs permit tcp Consumer Provider eq 80
EGP-Web as provider and EPG-Client as consumer permit tcp Provider eq 80 Consumer
filter flt-1 created matching
TCP port with any source port
to destination port 80 (http)
filter flt-2 created matching
TCP port with source port 80
(http) to any destination port
Contract Scope
Create a contract The contract scope will limit which providers and
consumers can participate within the same contract.
Specify contract name • VRF
The contract can be applied between EPGs within
Contract Scope the same VRF.
(default to VRF)
• Application Profile
the same application profile
• Tenant
the same tenant.
Click + to add subject to • Global

contract The contract can be applied between any EPGs
within the fabric. Note, global contracts not in
common tenant need to be exported in order to be
consumed by EPG in a different tenant.
Consumers of global contracts will use the
‘Consumer Contract Interface’ Option
Option 1
Unidirectional filters to
Unidirectional requires specifying explicitly specify rule from
both Consumer to Provider AND consumer to provider AND
provider to consumer filters from provider to consumer.
Consumer to Provider
users filter flt-1
Consumer to Provider
users filter flt-2
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
Option 2
Bidirectional filter with
reverse port enabled.
Bidirectional contract with reverse Remember, filter flt-1

filter ports enabled referenced just destination
port 80. The ‘apply both
directions’ and reverse
filter makes this logically
equivalent to option 1.
Single filter required.
Add EPG provider to contract Add EPG consumer to contract
High Policy CAM Utilization Example
• 100 EPGs all providing a basic management
E1 E1 E1 E1 E0
contract to a single consumer EPG.
E2 E2 E2 E2
E3 E3 E3 E3 mgmt- mgmt- • TCAM Utilization Calculation (Approximate)
E2 E2 E2 E2 contract EPG ~= (entries in contract)(# of Cons)(# of Providers)(2)
E1 E2 E3 E4 ~= 2 * 1 * 100 * 2
100 EPGs ~= 400 entries in hardware
Name EthType Proto Src Port Dst Port

• Policy CAM utilization increases by over 6400
flt-ssh IP TCP 1-65535 22
Why?
flt-snmp IP UDP 1-65535 161
High Policy CAM Utilization Example
Name EthType Proto Src Port Dst Port permit tcp E1 eq 1 E0 eq 22
flt-ssh IP TCP 1-65535 22 permit tcp E1 2-3 E0 eq 22
flt-snmp IP UDP 1-65535 161 permit tcp E1 4-7 E0 eq 22
Expanded permit tcp E1 8-15 E0 eq 22

• Port Ranges permit tcp E1 16-31 E0 eq 22
Policy CAM, as with any TCAM, uses a value and permit tcp E1 32-63 E0 eq 22
mask to perform matching.
permit tcp E1 64-127 E0 eq 22
• Matching a single port utilizes only one entry in TCAM.
• Using a range of ports may need to be expanded to permit tcp E1 128-255 E0 eq 22
multiple entries in hardware depending on the start and permit tcp E1 256-511 E0 eq 22
end values. permit tcp E1 512-1023 E0 eq 22
How to fix this issue? permit tcp E1 1024-2047 E0 eq 22
• Use port 0-65535 or ‘unspecified’ source port permit tcp E1 2048-4095 E0 eq 22

=> utilization down from 6400 to 400 entries permit tcp E1 4096-8191 E0 eq 22
• Consider using VzAny if all EPGs in the VRF need it permit tcp E1 8192-16383 E0 eq 22
=> utilization down from 400 to 4 entries permit tcp E1 16384-32767 E0 eq 22
E0 permit tcp E1 32768-65535 E0 eq 22
Any
mgmt- mgmt-
contract EPG BRKACI-3101 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
VzAny
A contract can be provided or
consumed at the VRF level.
Associating a contract to a VRF
is referred to as a VzAny
contract as it allows
communication between
source of contract and all
EPG’s within the VRF.
VRF v1 consuming contract C1.

Logically all EPGs within the VRF are
now consuming contract C1
ACI Contracts and Resource Utilization
Contract created between E2 and E3 • BD-B1 and BD-B2 each have a subnet
defined. Subnet int-S1 on BD-B1 exists on
E2 E3 L1 and L3, while subnet int-S2 for BD-B2
exists on L6
Add contract and S1 S2 Add contract and When creating the contract between E2 and E3:
route to int-S2 route to int-S1 • Program contract rule between E2 and E3
in TCAM. Add Static route for int-S1
created on L6 pointing to spine proxy.
L1 L2 L3 L4 L5 L6 • Program contract rule between E2 and E3

in TCAM. Add Static route for int-S2
created on L3 pointing to spine proxy.
• Contracts are only programmed on leafs that
H1 H2 H3 have provider/consumer EPGs. BD routes
BD-B1 VRF-V1 BD-B2 are only programmed on leafs that need
Subnet Subnet them!
EPG-E1 int-S1 EPG-E2 int-S2 EPG-E3 Contracts contribute to both
policy AND routing entries
on leafs!
Shared Services and Route Leaking
export to T2
Consume
ACI Shared Services Provide
Interface
C1 C1-export
What is a shared service? scope: global
• Shared Service (Route Leaking) enables traffic
between endpoints in different VRFs.
E1 E2 E3 E4
• A shared service EPG provider is an EPG that
provides a contract consumed by an EPG in a BD-B1 BD-B2
different VRF
VRF-V1 VRF-V2
Restrictions
• Provider Subnet must be defined under the Tenant-T1 Tenant-T2
provider EPG EPG-E1 Subnet: S1 BD-B2 Subnet: S2
• Both provider and consumer subnets scope: shared scope: shared
must have scope set to shared
• contract needs correct scope VRF Route pcTag Flags VRF EPG pcTag
• VzAny not supported as provider V1 S1 1 proxy V1 E1 49155
V2 S2 1 proxy V1 E2 49156
Scope: V2 E3 16387
 Private to VRF
V2 E4 49155
 Advertise Externally
 Share Between VRFs
export to T2
Consume
Interface
C1 C1-export
What happens in the fabric? scope: global
• EPG-E1 is now a shared service provider.
It is reallocated a fabric unique pcTag E1 E2 E3 E4
(<16384)
BD-B1 BD-B2
• All subnets on consumer BD
programmed in provider VRF VRF-V1 VRF-V2
• Provider subnet programmed in
Tenant-T1 Tenant-T2
consumer VRF with pcTag of provider
EPG EPG-E1 Subnet: S1 BD-B2 Subnet: S2
scope: shared scope: shared
VRF Route pcTag Flags VRF EPG pcTag
V1 S1 1 proxy V1 E1 49155
17
V1 S2 1 proxy, rewrite V1 E2 49156
VNID(V2)
V2 E3 16387
V2 S2 1 proxy
V2 E4 49155
V2 S1 1717 proxy, rewrite
VNID(V1)
export to T2
Consume
Interface
C1 C1-export
What happens in the fabric? scope: global
• EPG-E1 is now a shared service provider.
It is reallocated a fabric unique pcTag E1 E2 E3 E4
(<16384)
BD-B1 BD-B2
• All subnets on consumer BD
programmed in provider VRF VRF-V1 VRF-V2
• Provider subnet programmed in
Tenant-T1 Tenant-T2
consumer VRF with pcTag of provider
EPG EPG-E1 Subnet: S1 BD-B2 Subnet: S2
scope: shared scope: shared
• Policy enforcement always performed in
consumer VRF. Therefore, contracts Contract VRF Action Src Dst Filter
are always programmed in consumer
C1 V2 permit E4 E1 flt1
VRF.
V2 permit E1 E4 *flt1
V1 - - - -
No Rule added in provider VRF
Shared Service
Provider
subnet scope must have The shared service provider
shared flag enabled requires that the subnet be
configured under the EPG.
Note, stating in 1.2, the subnet
can be defined under both the
For the shared service provider, define BD and the EPG
the subnet under the EPG
The shared flag is set under
the subnet.
Create Global Contract and Export to
Tenant-T2
Create a contract C1 with Global scope in Tenant-T1.
Contract defined in Tenant-T1 Create a subject with appropriate filters.
with scope of Global.
epg-E1 provides this contract Ensure EPG-E1 is a provider for C1
Right-click the contract folder and click export

contract. Specify a name for the exported contract and
the destination Tenant.
The contract now appears as an imported-contract in

Tenant-T2
Right-click Contract folder and

choose Export Contract
Choose the contract to export, the The contract now shows up in

name of the exported contract, and the Tenant-T2 as an imported contract
destination tenant
Shared Service
Consumer
The shared service consumer
subnet scope must have EPG can have the subnet
shared flag enabled defined under the BD or the
EPG. In this example, the
consumer subnet is defined
under the BD
For the shared service The shared flag is set under
consumer, subnet can be the subnet.
defined under EPG or BD.
C1-export present in
Tenant-T2
choose the imported
contract
Shared Service
Consumer
Consume the imported contract via
‘Consumed Contract Interface’
Add Consumed Contract
Interface to consume Routes are now leaked between
imported contract VRFs.
fab2-leaf101# show ip route vrf Tenant-T1:VRF-V1 fab2-leaf101# show ip route vrf Tenant-T2:VRF-V2
IP Route Table for VRF "Tenant-T1:VRF-V1" IP Route Table for VRF "Tenant-T2:VRF-V2"
'*' denotes best ucast next-hop '*' denotes best ucast next-hop
'**' denotes best mcast next-hop '**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric] '[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string> '%<string>' in via output denotes VRF <string>
10.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive 10.1.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.1.1%overlay-1, [1/0], 00:06:06, static *via 10.0.1.1%overlay-1, [1/0], 00:06:12, static
10.1.1.1/32, ubest/mbest: 1/0, attached, pervasive 10.2.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.1.1.1, vlan24, [1/0], 00:38:56, local, local *via 10.0.1.1%overlay-1, [1/0], 00:10:40, static
10.2.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive 10.2.1.1/32, ubest/mbest: 1/0, attached, pervasive
*via 10.0.1.1%overlay-1, [1/0], 00:06:06, static *via 10.2.1.1, vlan26, [1/0], 00:10:40, local, local
Tenant-T2:VRF-V2 route present in Tenant-T1:VRF-V1 route present in

Tenant-T1:VRF-V1 routing table Tenant-T2:VRF-V2 routing table
1. H1 sends packet toward gateway in
EPG-E1 with destination IP of H3
Shared Service Forwarding
2. L1 performs layer3 lookup for H3 in
From Provider E1 to Consumer E4 VRF-V1 and hits LPM entry for H3 subnet.
LPM entry points to proxy with VNID
rewrite info for VRF-V2.
Policy Applied on Packet is sent to Spine Anycast IPv4
S1 S2 egress L6 Proxy VTEP with VRF-V2 VNID and
(consumer VRF) EPG-E1 set in VXLAN header.
2 3 No policy applied in provider VRF
4 3. Spine performs proxy lookup for H3 IP in

L1 L2 L6 VRF-V2. Normal Proxy behavior to
L3 L4 L5
forward packet to VTEP of L6
5
1 destination IP in VRF-V2. Hit in local EP
database and derives destination EPG-E4
H1 H2 H3 L6 applies policy between EPG-E1 and
BD-B1 BD-B2
EPG-E6
EPG-E1 EPG-E4
5. If permitted, traffic forwarded to H3 with
appropriate encap
VRF-V1 VRF-V2
1. H3 sends packet toward gateway in
EPG-E4 with destination IP of H1
Shared Service Forwarding
2. L6 performs layer3 lookup for H1 in
From Consumer E4 to Provider E1 VRF-V2 and hits LPM entry for H1 subnet.
LPM entry points to proxy with VNID
rewrite info for VRF-V1 and pcTag of
Policy Applied on EPG-E1.
S1 S2 ingress L6 L6 applies policy between EPG-E4 and
(consumer VRF) EPG-E1 in consumer VRF-V2.
3 If permitted, packet is sent to Spine
2 Anycast IPv4 Proxy VTEP with
4 VRF-V1 VNID and EPG-E4 set in VXLAN
L1 L2 L3 L4 L5 L6
3. Spine performs proxy lookup for H1 IP in
VRF-V1. If unknown drops the packet.
5 1 Else forward to VTEP of L1
H1 H2 H3 destination IP in VRF-V1. Hit in local EP
BD-B1 BD-B2
database and derives destination EPG-E1
EPG-E1 EPG-E4 Policy already applied by L6
5. Traffic is forwarded to H1 with appropriate
VRF-V1 VRF-V2
encap
export to T2
Consume
Interface
C1 C1-export
Subnet on Provider BD scope: global
Can the shared service provider subnet be
defined under the BD? E1 E2 E3 E4
 EPG-E1 successfully updated with unique BD-B1 BD-B2
pcTag
 Contract pushed in consumer VRF VRF-V1 VRF-V2
 Consumer subnet pushed into Provider VRF Tenant-T1 Tenant-T2

BD-B1 Subnet: S1 BD-B2 Subnet: S2
 Provider subnet NOT pushed scope: shared scope: shared
in Consumer VRF
VRF Route pcTag Flags VRF EPG pcTag
…Why? V1 S1 1 proxy V1 E1 49155
17
V1 S2 1 proxy, rewrite V1 E2 49156
VNID(V2)
V2 E3 16387
V2 S2 1 proxy
V2 E4 49155
- - - missing …
ACI Shared Services
Subnet on Provider BD
• From the previous packet walk, route leaking with shared services is similar to IP prefix-based EPGs.
The consumer VRF needs to map the provider subnet to a pcTag. Since there is no subnet
explicitly defined under the provider EPG, this mapping cannot occur.
Workaround?
1. Configure subnet under the provider EPG and the provider BD
Supported in 1.2 and above
2. Force one-way leaking in both directions by making both EPGs shared service providers and shared
service consumers
export to T2
Consume
Interface
C1 C1-export
Subnet on Provider BD scope: global
Workaround 2
E1 E2 E3 E4
Both EPG-E1 and EPG-E4 are shared service
providers and shared service consumers BD-B1 BD-B2
Contract C1
VRF-V1 VRF-V2
• leaks subnet S2 into VRF-V1
Tenant-T1 Tenant-T2
• programs policy into VRF-V2 BD-B1 Subnet: S1 BD-B2 Subnet: S2
Contract C2 scope: shared scope: shared
export to T1
• leaks subnet S1 into VRF-V2
Consume Provide
• programs policy into VRF-V1 Interface C2
C2-export
scope: global
Advantages/Disadvantages?
Policy TCAM
Contract Review Contract VRF Action Src Dst Filter
C1 V2 permit E2 E1 flt1
• Shared Service EPGs V2 permit E1 E2 *flt1
EPGs that provide contract consumed by C2 V2 permit E4 E3 flt2
EPG in a different VRF: E1, E2*
V2 permit E3 E4 *flt2
• Application EPGs V2 permit ext2 E3 flt2
E1, E2, E3, E4 V2 permit E3 ext2 *flt2
C3 V2 permit ext1 any flt3
• External EPGs
configured on L3Out and classified based on V2 permit any ext1 *flt3
IP prefix: ext1, ext2
• VzAny E1 E2 E3 E4 Any L3Out

Represents all EPGs in a single VRF: Any subnet
int-S1
ext1 ext2
Contract Assumptions for this Example :
• All contract subjects have both directions C3 subnet subnet
C1 C2 ext-S1 ext-S2
and reverse filters enabled.
VRF-V1 VRF-V2
L3outs and Routing Protocols
Basic Connectivity node-103 node-104
RID: # RID: #
IP: A IP: B
Layer3 Out: L3Out-1

VRF: VRF-V1
Layer-3 Domain: DomL3 vlan-x
Logical Node Profile: node-103-104
node: node-103 node: node-104 L3Out-1

Router-ID: # Router-ID: #
VRF-V1
Logical Interface Profile: ipv4-lif
Create the L3Out
• Associate VRF and L3 Domain
path: topology/pod-1/…vpcX • Create Logical Node Profile and associate fabric
type: ext-svi, encap: vlan-x nodes to the L3Out.
IP-A, IP-B, MTU, MAC • Create Logical Interface Profile
• Specify Path attributes containing physical interface,
encapsulation, and IPs
Configure L3Out Basic
Connectivity
• Right click the External Routed
Networks and choose Create
Routed Outside
• Specify the name for the L3Out,

Specify name, VRF, and the VRF, and external routed
routed domain domain.
Right click ‘External
• Note, the routing protocols would
Routed Networks’
be enabled directly under the
and choose Create L3Out. In this example, we are not
Routed Outside Click + to add Logical Node enabling any routing protocol.
and Logical Interface Profiles
• Click the ‘+’ icon beside the Node
and Interface Protocol Profiles to
create a logical Node profile
Configure L3Out Basic
Name of Logical Node Profile Click + to add Nodes to Connectivity
this Logical Node Profile
• Configure the Logical Node Profile
by first providing the name.
• Click the ‘+’ icon beside Nodes to

add node-103 and node-104 to the
logical Node Profile.
• When adding a node, specify the

node and router-ID. If static routes
Specify the Node (node-103 in this are required under the L3Out,
Click + to add specify them under the Logical
example) along with the Router ID Logical Interface Node profile
Profile
• Repeat for Node-104
• Click the ‘+’ icon beside Interface

Profiles to add a Logical Interface
Profile
Select vpc path and
Name of Logical vlan encap for L3Out
Interface Profile
Configure IP address for

each node
Click + to add Path
Configuring an
external SVI in Configure path
this example as Trunk
• Configure the name of the Logical Interface Profile and add a path attribute. In this example, the path is configured as a vPC with encap of vlan-151.
For vPC path, the IP address will be configured on both leafs in the vPC pair. Therefore, there’s a side-A and a side-B IP address.
• Multiple other interface options can be configured under the LIF such as MTU, secondary IP’s, IPv6 Link local, etc…
Verify L3Out Basic
L3Out ‘L3Out-1’ Connectivity Configuration
• Verify that the logical node profile
Node profile ‘node- is correctly configured
103-104’
• Verify that the logical interface
profile is configured with correct
Interface profile IP’s, path, and VLAN
external SVI path configured with encapsulation
‘ipv4-lif’
IP’s assigned to both nodes on the
vPC with encap of vlan-151
Basic Connectivity Pool4
DomL3
Layer3 Out: L3Out-1
VRF: VRF-V1
Layer-3 Domain: DomL3 AEP
External
node: node-103 node: node-104

Router-ID: # Router-ID: # 1 2 3 4 1 2 3 4
path: topology/pod-1/…vpcX
topology/pod-1/…vpcX Remember AEP!
path:
SVI encap: vlan-x
IP-A, IP-B, MTU, MAC, mode • interfaces from path must be member of DomL3 AEP.
• The vlan encapsulation must be within encap blocks
defined in DomL3 vlan pool
Verify L3Out Path
Path attribute from logical Configuration
interface profile references
interface policy group In this example, a vpc is
configured as the path attribute
on the L3out. The vpc is is
configured on node-103 and
node-104 with a VLAN encap
of vlan-151.
In this slide:
• Validate the interface policy

group contains the correct
AEP
AEP that is tied to external

routed domain
Verify L3Out Path
Configuration
In this example, a vpc is
Domain is associated to configured as the path attribute
the correct AEP on the L3out. The vpc is is
configured on node-103 and
Domain is associated to
node-104 with a VLAN encap
VLAN pool4 of vlan-151.
In this slide:
• Validate the AEP

references the correct
physical domain
• Validate the physical

VLAN pool4 contains domain contains a VLAN
encap 151 pool with the correct
encap(vlan-151)
Configuring Routing Protocols
Layer3 Out: L3Out-1 Enable Protocol

VRF: VRF-V1 BGP (fabric ASN configured in fabric pod policy)
Layer-3 Domain: DomL3 OSPF - area, area-type, area-configuration
EIGRP - ASN
node: node-103 node: node-104 Static Routes configured per node

Router-ID: # Router-ID: # BGP Peer Profiles tied to Loopback

BGP Peer Profiles tied to Interface
path: topology/pod-1/…vpcX OSPF Interface Policy and Authentication
type: ext-svi, encap: vlan-x EIGRP Interface Policy
IP-A, IP-B, MTU, MAC BFD Interface Policy
Majority of configuration under the interface/peer policies…
Enabling BGP
1. Under the L3Out, enable
the BGP process.
Note, the ASN for this process

will be the same as that
configured under the pod fabric
policy.
Enable BGP under the L3Out
Enabling BGP
2. Under the logical Node
profile, create BGP peer
connectivity profile under
loopback or previously
configured logical interface
profile.
Under the Peer Connectivity

Profile configure several
options including:
• BGP controls
• BGP Credentials
• EBGP multi-hop
• Remote ASN along with

local ASN and local AS
configuration
etc.…
Enable BGP on Loopback or

logical interface
Ensure BGP RR
Types of Fabric Routes is configured to
enable MP-BGP
MP-BGP
overlay-1
L3Out-1 L3Out-2
E1 E2
ext1 ext2
BD-B1 BD-B2
subnet subnet subnet subnet
ext-S1 int-S1 int-S2 ext-S2
• Internal Routes: Subnets defined under BD are internal routes and create static pervasive routes
within the fabric.
• External Routes: Routes learned via a routing protocol or static routes configured under an L3Out.
These routes are redistributed into MP-BGP and advertise to all leafs that contain the VRF
• Transit Routes – Routes advertised between L3Outs.
Configure Fabric MP-BGP
To enable MP-BGP through the fabric
• Configure Fabric ASN

Configure ASN for fabric
• Configure one or more spines as
Route Reflectors
Configure one or more spines

as route reflectors
Types of Fabric Routes – Internal Routes
Subnet: int-S2 3
Subnet int-S2 installed on border leaf
Scope: MP-BGP
when creating contract between EPG
 Private to VRF E2 and external overlay-1
EPG ext2
 Advertise Externally
L3Out-1 L3Out-2
1
E1 E2
ext1 ext2
BD-B1 BD-B2 2
subnet subnet subnet C1 subnet
ext-S1 int-S1 int-S2 ext-S2
There are three requirements to advertise Internal Routes out an L3Out:
1. The BD must be associated with the L3Out*
The association adds prefix entry to route map controlling advertised routes
2. A contract must exists between an EPG within the BD and an external EPG on the L3Out.
The contract creates internal BD route on border leaf (cannot advertise route until it exists locally)
3. The subnet must have a public scope (Advertise Externally)
Advertise Internal BD’s
Under L3 Configurations for BD,
ensure
• BD is associated to L3Out
• Subnets that need to be advertised

Ensure Subnets in BD have must have ‘Advertise External’ flag
‘Advertise External’ flag set set.
Note shown in this slide, at least one

EPG in BD must have a contract
consumed/provided by an external-
EPG in the desired L3Out.
Associate BD to L3Out
Types of Fabric Routes – External Routes
ext-S1
ospf, eigrp, static route RD: L1:V1
redistributed into bgp and RT: ASN:V1 MP-BGP
exported into mp-bgp
overlay-1
ext-S1
L3Out-1 L3Out-2
VRF-V1 VRF-V1 VRF-V1
ext1 import RT from mp- ext-S1 ext2

ext-S1
bgp and install route
subnet via:L1(bgp) into vrf as bgp learn via:L1(bgp) subnet
ext-S1 ext-S2
• External Routes from ospf, eigrp, or static are redistributed on the border leaf into the local bgp process.
• The bgp route is exported into MP-BGP with a route-target (RT) of the corresponding VRF. Each leaf in
the fabric with the VRF present will import the RT and install the route. External routes on the non-
originating border leaf will be seen as bgp learned routes.
• External Routes are controlled via Import Route Control flag
Types of Fabric Routes – Transit Routes
MP-BGP
overlay-1
ext-S1
ext-S1
L3Out-1 L3Out-2
VRF-V1 VRF-V1 VRF-V1
ext1 ext-S1 ext-S1 ext2
subnet via:L1(bgp) via:L1(bgp) subnet

ext-S1 ext-S2
• In this example, external route ext-S1 is a Transit Route when advertised out L3Out-2.
• If OSPF or EIGRP on L3Out-2, ext-S1 is redistributed from BGP into the IGP and advertised.
• Transit Routes are controlled via Export Route Control flag
Configure L3Out External Network node-103
RID: #
node-104
RID: #
IP: A IP: B
Define an External Network, ext1 in this example
• Note: At least one external network required to
bring up L3Out interfaces on border leaf
vlan-x
• Add Subnet to External Network
Prefix-based EPG for Contracts: Scope:

 Export Route Control Subnet L3Out-1
• External Subnets for the
External EPG  Import Route Control Subnet ext1
• Shared Security Import  External Subnets for the VRF-V1
External EPG subnet
Route Control  Shared Route Control Subnet ext-S1
• Export Route Control  Shared Security Import Subnet
• Import Route Control Categorize Subnet
• Shared Route Control options Aggregate: options
• Aggregate Export  Aggregate Export
• Aggregate Import  Aggregate Import
• Aggregate Shared Routes  Aggregate Shared Routes
Subnet Prefix
Add a subnet
Specify
scope/controls
Right click Networks and choose

‘Create External Network’
Creating an external EPG

• Right click Networks and choose ‘Create External Network’. Provide a name for the external EPG and the click ‘+’ icon to add a Subnet
• Configure the subnet prefix and choose appropriate scope flags. Further definition of each flag in upcoming slides.
External Subnets for the External EPG
Previously: Import-Security Subnet: ext-S2/mask
Scope:
External Subnet for the External EPG is used to classify  External Subnets for the External EPG
dataplane packets into external EPG for policy enforcement.
EPG to pcTag
• An IP prefix is installed into leaf TCAM to classify traffic VRF EPG pcTag
to/from the external network and assign correct pcTag for
policy enforcement V1 E1 49156
Host Table LPM Table V2 E2 16387

V2 ext2 49155
VRF EP PcTag Dst VRF Subnet PcTag Dst
V1 Host1
Host1 49156
49156 Leaf1 V1 int-S1 1 Proxy
E1 L3Out
V1 Host2 16387 Leaf2 V1 ext-S2 49155 L3Out
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC ext2

C1 subnet
• Apply policy between src E1(49156) and dst ext2(49155)
ext-S2
External Subnets for the External EPG
Previously: Import-Security
int-S1 != ext-S2
• BD subnets are also installed into LPM table.
• The border leaf needs the BD route to forward traffic back
from L3Out to endpoint.
• It also needs the route present before it can advertise to
any neighbors
Do NOT configure External EPG subnets with same prefix as

a internal BD subnet. This may result in:
1. Unexpected traffic flow

sent out L3Out instead of routed within the fabric E1 L3Out
2. Incorrect EPG classification which may cause
unexpected policy drops BD-B1
3. Failure to program prefix faults subnet
4. Failure to program zoning-rule faults. int-S1
Since aclqos programs prefixes and security rules in a ext2
batch, a single failed prefix may cause security rules to
also fail. subnet
ext-S2
Import Route Control Subnet: ext-S1/mask
*Import Route Control supported only for BGP Scope:
Import Route Control is used to filter External  Import Route Control Subnet
Routes received on an L3Out
• A route-map is created per BGP neighbor to filter

incoming routes. Subnets defined with the import
flag will be added to corresponding prefix list to
allow in remote routes.
• The import flag must be enabled on the L3Out to
set import flag per external subnet.
• By default, import is disabled on the L3Out
neighbor neighbor-1
Inbound route-map imp-l3out-vrf
L3Out-1
Outbound route-map exp-l3out-vrf
Allow
Advertisement:
route-map imp-l3out-peer-vrf permit - ext-S1/mask
match: prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst - ext-S2/mask
- ext-S3/mask
ip prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst BGP
permit ext-S1/mask
Ignore
Neighbor-1
Aggregate Import Subnet: 0.0.0.0/0
Scope:
*Aggregate Import supported only for 0.0.0.0/0 or ::/0  Import Route Control Subnet
Aggregate:
Import Route Control allows fabric to permit a  Aggregate Import
specific prefix. Instead of creating each prefix
advertised by a neighbor, multiple prefixes can be
aggregated together by using the Aggregate Import
flag.
neighbor neighbor-1
Inbound route-map imp-l3out-vrf
Outbound route-map exp-l3out-vrf
route-map imp-l3out-peer-vrf permit

match prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst L3Out-1 Allow All
ip prefix-list IPv4-network-vrf-exc-ext-inferred-import-dst Advertisement:

permit 0.0.0.0/0 le 32 - ext-S1/mask
- ext-S2/mask
- ext-S3/mask
BGP
Neighbor-1
Export Route Control & Aggregate Export
*Aggregate Export supported only for 0.0.0.0/0 or ::/0
Subnet: ext-S1/mask
Export Route Control allows Transit Routes to be Scope:
advertised out of the fabric.  Export Route Control Subnet
• Export control does NOT affect pervasive BD SVIs,
they are only advertised when the BD is associated
with the L3Out.
• Similar to import route control subnet, a prefix list
with corresponding exported subnets is created to
allow routes to be advertised out
Aggregate Export is identical concept to aggregate

import, allowing prefixes to be aggregated together in
export direction. L3Out-1 L3Out-2
Subnet: 0.0.0.0/0
Scope: Export all Advertisement: Export:
 Export Route Control Subnet Transit Routes - ext-S1/mask - ext-S1/mask
Aggregate: within VRF - ext-S2/mask
- ext-S3/mask
 Aggregate Export
Shared L3Out
Similar to Shared Services, a Shared L3Out uses
contracts to leak routes between VRFs. The leaked
routes can be:
• int-S1 subnet from VRF-V1 to VRF-V2
• ext-S2 subnet from VRF-V2 into VRF-V1
Similar Restrictions as Shared Services
E1 L3Out-1
• If the application EPG is providing the contract for
shared L3Out, the internal subnet must be defined
under the EPG. BD-B1
• If the external EPG is providing the contract for shared ext2
VRF-V1
L3Out, then internal subnet can be defined either under C1
the EPG or the BD EPG-subnet subnet
• Internal subnet must have shared and Advertise int-S1 ext-S2
Externally(public) scope. VRF-V2
• Contract must be appropriately scoped. Scope:
• For shared L3Out, shared subnet must be globally  Private to VRF
unique within the entire ACI fabric.  Advertise Externally
Shared L3Out E1 L3Out-1
What happens in the fabric when contract is added? BD-B1

VRF-V1 ext2
• Internal Route int-S1 leaked into VRF-V2
ext-S2 route not leaked into VRF-V1 yet… EPG-subnet C1 subnet
int-S1 ext-S2
• Shared-Service prefix list added to route-map
permitting advertisement of int-S1. External routers VRF-V2
can now learn int-S1 through OSPF, EIGRP, or BGP
on VRF-V2.
Assume: VRF-V2 has a route to ext-S2
No need to associate BD to shared L3Out, route
through static or dynamic route
controlled by contract!
Forwarding Table
• Shared-Service contract programmed onto leaf to
allow traffic flow. VRF Route pcTag Flags
V1 int-S1 1 proxy
• Problems:
• VRF-V1 does not have return route to ext-S2 V2 ext-S2 ext2 L3Out
• Even though rule is programmed, return traffic from V2
V2 int-S1
int-S1 E1 proxy, leak->V1
VRF-V1 can’t derive destination pcTag so no policy
available to enforce
Shared L3Out E1 L3Out-1
Completing the Configuration BD-B1

VRF-V1 ext2
Shared Route Control flag allows external route to be
leaked into EPG context. EPG-subnet C1 subnet
int-S1 ext-S2
• In this example, adding shared route control to the
external subnet allows ext-S2 to be leaked into VRF-V1, Subnet: ext-S2/mask VRF-V2
but pcTag set to reserved drop value. Scope:
 Shared Route Control
Shared Security Import is used to classify dataplane
 Shared Security Import
packets into external EPG for policy enforcement for
shared prefixes Forwarding Table
• In this example, adding shared security import to the VRF Route pcTag Flags
external subnet created a prefix-based EPG in any- V1 int-S1 1 proxy
VRF* for the external subnet ext-S2 with pcTag of EPG-
ext2. V2 ext-S2 ext2 L3Out
V2 int-S1 E1 proxy, leak->V1
V1
V1 ext-S2 ext2
deny-tag L3Out, leak->V2
Subnet: 8.8.0.0/16
Scope:
Aggregate Shared  Shared Route Control
Supported for any prefix, not just 0.0.0.0!  Shared Security Import
 Aggregate Shared
Aggregate Shared flag allows multiple prefixes
from L3Out to be shared/leaked into another
VRF.
In this example, a /16 prefix is configured with
aggregate shared flag set. The external router
advertised multiple /24 subnets within the
range. Each are leaked into VRF-V1
E1 L3Out-1
Advertisement:
Restrictions - 8.8.8.0/24
Shared Route control subnets cannot be a BD-B1 - 8.8.9.0/24
subset of Shared Security import. For ext2 - 8.8.10.0/24
VRF-V1
example: C1
8.8.0.0/16 VRF-V2
• shared security import + shared route Forwarding Table
control + aggregate shared VRF Route pcTag Flags
8.8.10.0/24
• shared route control (only) V1 8.8.8.0/24 ext2 L3Out, leak->V2
Traffic on VRF-V1 toward 8.8.10.0/24 dropped V1 8.8.9.0/24 ext2 L3Out, leak->V2
V1 8.8.10.0/24 ext2© 2016 CiscoL3Out, leak->V2
and/or its affiliates. All rights reserved. Cisco Public 159
8.8.8.0/24
Aggregate Shared 8.8.9.0/24
RD:
RD:
L4:V2
8.8.10.0/24
L4:V2
RT: ASN:V2
RD: L4:V2
RT: ASN:V2
How does this work? MP-BGP RT: ASN:V2
vpnv4 VRF-V1 overlay-1
• Leaf4 exports routes into MP-BGP with import
route-target for VRF V2 RT: ASN:V1
L1 vpnv4 VRF-V2
• Leaf1 imports routes with route-targets RT: ASN:V2 L4
export
from both VRF-V1 and VRF-V2 into RT: ASN:V2
V1 vrf. Routes are filtered with route-map
based on subnet control flags
E1 L3Out-1
Advertisement:
leaf101# show bgp process vrf V1
- 8.8.8.0/24
Import route-map V1-shared-svc-leak BD-B1 - 8.8.9.0/24
Import RT list: ext2 - 8.8.10.0/24
ASN:V1 VRF-V1
ASN:V2 C1
... VRF-V2
route-map V1-shared-svc-leak, permit, sequence 1000*
Match clauses:
ip address prefix-lists: IPv4-V2-V1-shared-svc-leak
ip prefix-list IPv4-V2-V1-shared-svc-leak
seq 3 permit 8.8.0.0/16 le 32
L3 External Subnet Review
o External Subnets for the External EPG (Security Import)
Used to classify dataplane packets into external EPG for policy enforcement
o Export Route Control

filter Transit Routes advertised out of the fabric.
o Import Route Control

filter External Routes received on an L3Out
o Shared Security Import

used to classify dataplane packets into external EPG for policy enforcement for shared/leaked prefixes
o Shared Route Control

Allows external route to be leaked into another VRF
o Aggregate Export - allows prefixes to be aggregated together in export direction (0/0 or ::/0 only)
o Aggregate Import - allows prefixes to be aggregated together in import direction (0/0 or ::/0 only)
o Aggregate Shared Route - allows prefixes to be aggregated together for shared route control
Extra L3Out FAQ
How to Advertise Transit Static Route MP-BGP
ext-S1 overlay-1
In this example, a static route ext-S1 is configured on leaf L1 with
next-hop out L3Out-1. A second L3Out-2 is running OSPF. The
intention is to advertise the static route out L3Out-2.
L1 L4
An external network is configured under L3Out-2 with export flag for
0.0.0.0/0 along with aggregate to allow ALL routes to be advertised
In this topology, L4 advertises the static route while L1 does not. Static Route on L1
ext-S1
with next-hop on
Why? L3Out-1 L3Out-1 L3Out-2
By default, static routes configured within the fabric are not OSPF
advertised out L3Outs and will not match aggregate 0/0 prefixes. Static Static Route is
On L4, route to ext-S1 is technically a BGP route and internal route- advertised out L4
map will match 0/0 aggregate prefix for static route. On L1, route to
but not L1
ext-S1 is a static route that will not match aggregate 0/0 (by design).
ext-S1
Subnet: 0.0.0.0/0
 Export Route Control Subnet
Fix:  Aggregate Export
To properly advertise a static route, create an external network
matching the static route prefix and enable the export flag
Under L3Out-2, create an external Network:
Subnet: ext-S1/mask
 Export Route Control Subnet
How to use the Route Tag Policy
To prevent potential route loops, transit routes are sent with the
VRF route-tag. External routes that are advertised with the same
route-tag are denied.
By default, all VRFs resolve to the same Route Tag Policy with
default value of 4294967295. As a result, transit routes advertised
between VRFs may be denied.
In this example, a transit route ext-S1 is received on L3Out-1 in

VRF-V1 and exported out L3Out-2 in the same VRF. The exported
route is tagged with default Route Tag Policy of 4294967295. The
external router maintains this tag when advertising to L3Out-3 in L3Out-1 L3Out-2 L3Out-3
VRF-V2. Since VRF-V2 has the same Route Tag policy, the route
is denied.
ext-S1
How to fix this issue? VRF-V1 VRF-V2
Create different Route Tag Policies for each VRF. For example, on VRF-V1
VRF-V2, create a policy with tag set to 5. ext-S1 ext-S1
tag:4294967295 tag:4294967295
Under the VRF, create

new route Tag policy
Set custom Tag per

VRF
How to use the Route Control Policy
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Transit_Routing.html
Route Control Policies can set various attributes on routes received or

advertised out of the fabric. Different protocols have different attributes that
can be set:
Attribute OSPF EIGRP BGP Comments
community Yes Regular and Extended Communities
route tag Yes Yes Supported only for BD (internal) BD-B1 BD-B2 L3Out-1
subnets. Transit prefixes are always
set according to VRF route-tag policy subnet subnet
preference Yes BGP local preference
int-S1 int-S2
int-S1
metric Yes Yes Sets MED for BGP. Will change the community: 65535:2
metric for EIGRP but you cannot
specify the EIGRP composite metric.
metric type Yes OSPF Type-1 and OSPF Type-2

int-S2
community: 65535:2
MED: 150
In this example, all routes advertised out L3Out-1 need to have a community
65535:2 set. In addition, the internal route int-S2 needs to set MED 150.
For fabric routes, the route attributes can be set at the L3Out level, at the BD
level, or at the subnet level. If a route control policy is set at each level, the
most specific policy will be applied. I.e.,
• Tenant BD Subnet
• Tenant BD
• L3Out
There are two reserved policies that can be used at the L3Out level:
• default-import
• default-export
In this example, the default-export route control policy will set the community
and an additional route control policy applied at the BD will be configured to
set both the community and the MED.
Steps
1. Create an action rule to set
the community
2. Create a second action rule
that sets both the Under External Routed
community and the MED Networks, create action
rule
3. (continue to next slide)
Steps
3. Right-click the ‘Route Profiles’ under the L3Out-1 and create a new route-
profile
4. From the drop-down list, chose the reserved route profile default-export
5. Add a route control context to set the community
At the point, all traffic advertised out L3Out-1 will have the community set. A
second route profile is needed to tag BD-B2 subnets differently.
Chose the reserved

‘default-export’ from drop Select the set-attribute
down that sets community to
65535:2
Under the L3Out’s Route

Profiles, right-click to Add a Route Control
create Route Profile Context to the default-
export policy
Steps
6. Right-click the ‘Route Profiles’ under the L3Out-1 and create a new route-
profile
7. Chose a unique name for the route control profile
8. Add a route control context to set both the community and the MED
NOTE, only the default-export policy affects routes advertised at the L3Out level.
Custom route profiles still need to be applied at the BD or BD-subnet level.
Create a new route profile Select the set-attribute

with unique name that sets community to
65535:2 and MED to 150
Under the L3Out’s Route

Profiles, right-click to Add a Route Control
create Route Profile Context to the default-
export policy
Steps
9. Set the L3Out for
Route Profile under
the BD to L3Out-1
10. Set the Route Profile
to the previously
configured profile
11. Complete!
A BD can be associated to multiple

L3Outs. However, it can only have
a route profile from one. Set L3Out
To Review for route profile to L3Out-1
We used the default-export route control policy under L3Out-1
to set the BGP community for all routes advertised out.
Apply the BD-B2-export-
To fulfill the second requirement, we created another route
control policy under L3Out-1 that set both the BGP community control route profile to the
and MED. This policy was applied only on BD-B2. BD
How to use Default Information Originate
Originate a default route and advertise out L3Out. Right click the L3Out and choose ‘Create Default Route Leak Policy’.
Always: advertise a default route even if one is not currently present (not applicable for BGP)
Criteria: Leak default route in addition to currently advertised route OR leak default route only and suppress internal and transit routes
Scope: choose Outside for BGP. Choose Context for OSPF normal area or Outside for OSPF NSSA area
Right click Networks and choose

‘Create External Network’
Agenda
• Introduction
• Access Policies

Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk

• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or

watch the broadcast on cisco.com
Thank you

Brkaci 3101

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Brkaci 3101

Încărcat de

Drepturi de autor:

Formate disponibile

ACI Under the Hood - How Your

• Traversing the Overlay

APIC/IFC Application Policy Infrastructure Controller/ MST Multiple Spanning Tree

EP Endpoint TC Topology Change

1) Provide paths for endpoints to communicate at

ip access-group web2 out

Layer2 – STP forwarding

Layer2 – STP forwarding

T Tunnel Endpoint (TEP)

EPG1 EPG1 EPG2 EPG-L2Ext l3extSubnet EPG2 101/1/5

VRF-1 VRF-1 VRF-1

Virtual Network Identifier (VNID) Reserved

Virtual Network Identifier (VNID) Reserved

EPGs have a consumer / provider

EPG1 EPG2 EPG-L2Ext l3extSubnet

Destination MAC Address (DMAC)

… Source MAC Address (SMAC)

Destination MAC Address (DMAC)

… Source MAC Address (SMAC)

128 129 130 131 132 133 134 135

Version Header Length DSCP ECN Total Length

Time to Live (TTL) Protocol Header Checksum

Source Port Destination Port

Header Length Reserved Flags / Control bits Window Size

Checksum Urgent Pointer

Source Port Destination Port

Ethernet Frame containing TCP packet

Ethernet Frame containing UDP packet

AVS or Openstack connected hosts can

All tenant traffic is iVXLAN encapsulated

Broken into a few major areas:

Global Policy Switch Policy Interface Policy

Domains (Physical / External Bridged / External Routed)

Attachable Access Entity Profiles (AEP)

Policies define protocol / feature configurations

Policy Groups select which policies should be applied

Profiles associate policy groups to switches or interfaces,

Switch Policy Types: Interface Policy Types:

EP1 EP2 EP3

Classical vPC Domain configuration ACI Port-Channel Policies

• very easy to apply configurations if you create a LEAF101

4) Configure Interface policies (LACP / LLDP)

Click + to add vlan range

Specify start and

Enter name and submit

Click + to add selector

Select the Interface Profile

Enter a name and choose

Use a descriptive name

Select the protocol

• use VPC leaf profile if policy group is VPC

LLDP Rx / Tx enabled Access_Servers blk_1/47-48

Associate your AEP to select

Click + to add selector

Choose interface profile

Use a descriptive name

Associate the policy group to

Interface Profile Leaf101 Leaf101_102

Interface Selector linux windows n7k_pc10 asa_cl1_pc1 n7k1_pc10 n7k2_pc10

Interface Block 1/20-25 1/30-35 1/10-11 1/45-48 1/10 1/20

Interface Policy Group linux-access windows-access asa_vpc_ccl asa_vpc_data n7k_vpc10

vPC Domain 1 vPC Domain 2

Classical configuration steps ACI Logical configuration