BRKACI-2003

Cisco ACI Multi-Pod

Design and Deployment

John Weston – Technical Marketing Engineer

Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKACI-2003

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives

At the end of the session, the participants should be able to:

 Articulate the different deployment options to interconnect
Cisco ACI networks (Multi-Pod vs. Multi-Site)
 Understand the functionalities and specific design
considerations associated to the ACI Multi-Pod Fabric option
Initial assumption:
 The audience already has a good knowledge of ACI main
concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• ACI Network and Policy Domain
Evolution
• ACI Multi-Pod Deep Dive
Overview, Use Cases and Supported
Topologies
APIC Cluster Deployment Considerations
Inter-Pod Connectivity Deployment
Considerations
Control and Data Planes
Connecting to the External Layer 3 Domain
Network Services Integration
Migration Scenarios
ACI Network and Policy Domain
Evolution
 Cisco ACI
Fabric and Policy Domain Evolution

ACI Single Pod Fabric ACI Stretched Fabric ACI Multi-Pod Fabric

IPN
Pod ‘A’ Pod ‘n’

DC1 APIC Cluster DC2 MP-BGP - EVPN

…
APIC Cluster

ACI 1.0 - ACI 1.1 - Geographically ISE 2.1 & ACI 1.2 ACI 2.0 - Multiple Networks ACI 3.0 – Multiple Availability ACI 3.1/3.2 - Remote Leaf
Leaf/Spine Single Stretch a single Pod Federation of Identity (Pods) in a single Availability Zones (Fabrics) in a Single and vPod extends an
Pod Fabric and Interconnect Zone (Fabric) Region ’and’ Multi-Region Availability Zone (Fabric) to
TrustSec and ACI using Policy Management remote locations
IP based EPG/SGT
IP
Fabric ‘A’ Fabric ‘n’

MP-BGP - EVPN

…
ISE

ACI Multi-Site
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 Fabric and Policy Domain Evolution
Deployment Options
Single APIC Cluster/Single Fabric Multiple APIC Clusters/Multiple Fabrics
Stretched Fabric Multi-Fabric (with L2 and L3 DCI)
ACI Fabric Fabric ‘A’ Fabric ‘n’
DC1 APIC Cluster DC2
Inter-Site
App

L2/L3
DCI

Multi-Pod (from 2.0 Release) Multi-Site (3.0 Release, Q3CY17)

IPN
Pod ‘A’ Pod ‘n’ Fabric ‘A’ IP Fabric ‘n’

MP-BGP - EVPN MP-BGP - EVPN

… …
ACI
APIC Cluster Multi-Site
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Terminology
 Pod – A Leaf/Spine network sharing a common control plane (ISIS, BGP,
COOP, …)
Pod == Network Fault Domain
 Fabric – Scope of an APIC Cluster, it can be one or more Pods
Fabric == Availability Zone (AZ) or Tenant Change Domain
 Multi-Pod – Single APIC Cluster with multiple leaf spine networks
Multi-Pod == Multiple Networks within a Single Availability Zone (Fabric)
 Multi-Fabric – Multiple APIC Clusters + associated Pods (you can have
Multi-Pod with Multi-Fabric)*
Multi-Fabric == Multi-Site == a DC infrastructure Region with multiple AZs

* Available from ACI release 3.1

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 For More Information on
ACI Multi-Site VXLAN ACI Multi-Site:
Overview IP Network BRKACI-2125

MP-BGP - EVPN

Multi-Site Orchestrator

Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’

• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• ACI Multi-Site Orchestrator pushes cross-fabric • Data Plane VXLAN encapsulation across
configuration to multiple APIC clusters providing sites
scoping of all configuration changes • End-to-end policy definition and enforcement

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Typical Requirement
Creation of Two Independent Fabrics/AZs

Fabric ‘A’ (AZ 1)

Fabric ‘B’ (AZ 2)

Application
workloads
deployed across
availability zones BRKACI-2125 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Typical Requirement
Creation of Two Independent Fabrics/AZs

Multi-Pod Fabric ‘A’ (AZ 1)

‘Classic’ Active/Active

Pod ‘1.A’ Pod ‘2.A’

ACI Multi-Site

Multi-Pod Fabric ‘B’ (AZ 2)

‘Classic’ Active/Active
Application
Pod ‘1.B’workloads Pod ‘2.B’
deployed across
availability zones BRKACI-2125 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
ACI Multi-Pod Deep Dive
Overview, Use Cases and Supported
Topologies
ACI Multi-Pod
Overview VXLAN
Inter-Pod Network
Pod ‘A’ Pod ‘n’

MP-BGP - EVPN

APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP

Availability Zone

 Multiple ACI Pods connected by an IP Inter-Pod  Forwarding control plane (IS-IS, COOP)
L3 network, each Pod consists of leaf and spine fault isolation
nodes  Data Plane VXLAN encapsulation between
 Managed by a single APIC Cluster Pods
 Single Management and Policy Domain  End-to-end
© 2018policy
Cisco and/or enforcement
its affiliates. All rights reserved. Cisco Public
Single Availability Zone with Maintenance & Configuration Zones
Scoping ‘Network Device’ Changes
Maintenance Zones – Groups of
switches managed as an “upgrade”
group Inter-Pod Network

ACI Multi-Pod
Fabric
APIC Cluster

Configuration Zone ‘A’ Configuration Zone ‘B’

 Configuration Zones can span any required set of switches, simplest approach may be to map a
configuration zone to an availability zone, applies to infrastructure configuration and policy only

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Reducing the Impact of Configuration Errors
Introducing Configuration Zones

 Three different zone deployment modes:

 Enabled (default): updates are immediately sent
to all nodes part of the zone
Note: a node not part of any zone is equivalent Change the deployment
to a node part of a zone set to enabled. mode
Select entire Pod
 Disabled: updates are postponed until the zone
deployment mode is changed (or a node is Select specific Leaf Switches
removed from the zone)
 Triggered: send postponed updates to the nodes
part of the zone
Show the changes not applied yet
 The deployment mode can be configured for to a Disabled zone
an entire Pod or for a specified set of leaf
switches

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Single Availability Zone with Tenant Isolation
Isolation for ‘Virtual Network Zone and Application’ Changes

Inter-Pod Network

ACI Multi-Pod
Fabric
APIC Cluster

Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain

 The ACI ‘Tenant’ construct provide a domain of application and associated virtual
network policy change
 Domain of operational change for an application (e.g. production vs. test)
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 ACI Multi-Pod
Supported Topologies
Intra-DC Two DC sites directly connected

10G/40G/100G
10G*/40G/100G 10G*/40G/100G
POD 1 10G*/40G/100G 10G*/40G/100G
POD n POD 1 Dark fiber/DWDM POD 2
(up to 50 msec RTT**)
…

APIC Cluster APIC Cluster

3 (or more) DC Sites directly connected Multiple sites interconnected by a

10G/40G/100G
generic L3 network
10G*/40G/100G
POD 1 10G*/40G/100G POD 2
Dark fiber/DWDM 10G*/40G/100G 10G*/40G/100G
(up to 50 msec RTT**)
L3
10G*/40G/100G
10G*/40G/100G (up to 50msec RTT**)
10G*/40G/100G

POD 3 * 10G only with QSA adapters on EX/FX spines ** ©50 msec
2018 Ciscosupport addedAllin
and/or its affiliates. SW
rights release
reserved. 2.3(1)
Cisco Public
ACI Multi-Pod
SW/HW Support and Scalability Values

 All existing Nexus 9000 HW supported as leaf and spine nodes

 Maximum number of supported ACI leaf nodes (across all Pods)

 Up to 80 leaf nodes supported with a 3 nodes APIC cluster
 300 leaf nodes (across Pods) with a 5 nodes APIC Cluster
 400 leaf nodes (across Pods) with a 7 nodes APIC Cluster (from ACI release 2.2(2e))
 Maximum 200 leaf nodes per Pod
 Up to 6 spines per Pod

 Maximum number of supported Pods

 4 in 2.0(1)/2.0(2) releases
 6 in 2.1(1) release
 10 in 2.2(2e) release
 12 in 3.0(1) release

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
APIC Cluster Deployment
Considerations
 APIC – Distributed Multi-Active Data Base

One copy is ‘active’ for every

The Data Base is replicated specific portion of the Data
across APIC nodes Base
Shard 1 Shard 1 Shard 1
APIC APIC APIC
Shard 2 Shard 3 Shard 2 Shard 3 Shard 2 Shard 3

 Processes are active on all nodes (not active/standby)

 The Data Base is distributed as active + 2 backup instances (shards) for every attribute

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 APIC Cluster Deployment Considerations
Single Pod Scenario

X X
APIC APIC APIC Shards in
‘read-only’
mode
X X
APIC APIC APIC APIC APIC

 APIC will allow read-only access to the DB
when only one node remains active (standard
DB quorum)
 Hard failure of two nodes cause all shards to
be in ‘read-only’ mode (of course reboot etc.
heals the cluster after APIC nodes are up)
Shards in Shards in
‘read-only’ ‘read-write’ mode
 mode
Additional APIC will increase the system scale (up to
7* nodes supported) but does not add more
redundancy
 Hard failure of two nodes would cause inconsistent
behaviour across shards (some will be in ‘read-only’
mode, some in ‘read-write’ mode)

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

Pod 1 Pod 2

Up to 50 msec

APIC APIC APIC

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

Pod 1 Pod 2
X

Up to 50 msec

APIC APIC APIC

Read/Write Read Only

 Pod isolation scenario: changes still possible

on APIC nodes in Pod1 but not in Pod2

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

Pod 1 Pod 2

Up to 50 msec

APIC APIC APIC

 Pod isolation scenario: changes still possible

on APIC nodes in Pod1 but not in Pod2

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

X
Pod 1 Pod 2

Up to 50 msec

X X
APIC APIC APIC

 Pod isolation scenario: changes still possible

on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

X
Pod 1 Pod 2

Up to 50 msec

X X
APIC APIC APIC APIC

 Pod isolation scenario: changes still possible

on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

X
Pod 1 Pod 2

Up to 50 msec

X X
APIC APIC APIC APIC

 Pod isolation scenario: changes still possible

on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario
Pod 2

X
Pod 1 Pod 2 Pod 1

Up to 50 msec Up to 50 msec

X X
APIC APIC APIC APIC APIC APIC APIC APIC APIC

 Pod isolation scenario: changes still possible

on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario
Pod 2

X
Pod 1 Pod 2 Pod 1
X

Up to 50 msec Up to 50 msec

X X
APIC APIC APIC APIC APIC APIC APIC APIC APIC

 Pod isolation scenario: same considerations as with

 Pod isolation scenario: changes still possible single Pod (different behaviour across shards)
on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario
Pod 2

X
Pod 1 Pod 2 Pod 1

Up to 50 msec Up to 50 msec

X X
APIC APIC APIC APIC APIC APIC APIC APIC APIC

 Pod isolation scenario: same considerations as with

 Pod isolation scenario: changes still possible single Pod (different behaviour across shards)
on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 APIC Cluster Deployment Considerations
Multi-Pod – 2 Pods Scenario

X
Pod 2

X
Pod 1 Pod 2 Pod 1

Up to 50 msec Up to 50 msec

X X
APIC APIC APIC APIC
X X X
APIC APIC APIC APIC APIC

 Pod isolation scenario: same considerations as with

 Pod isolation scenario: changes still possible
on APIC nodes in Pod1 but not in Pod2
 P
od hard failure scenario: recommendation is to
activate a standby node to make the cluster
fully functional again
single Pod (different behaviour across shards)
 P
Possible to restore the whole fabric state to the latest taken
configuration snapshot (‘ID Recovery’ procedure – needs BU
and TAC involvement)
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
APIC Cluster Deployment Considerations Q2CY18
What about a 4 Nodes APIC Cluster?

Pod 1 Pod 2

Pending internal validation,

scoped for Q2CY18
Up to 50 msec

APIC APIC APIC APIC

 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to

170 leaf nodes supported)
 Pod isolation scenario: same considerations as with 5 nodes (different behaviour
across shards)
 Pod hard failure scenario
• No chance of total loss of information for any shard
• Can bring up a standby node in the second site to regain full majority for all the shards
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
APIC Cluster Deployment Considerations Q2CY18
What about a 4 Nodes APIC Cluster?

Pod 1 Pod 2
X
Pending internal validation,
scoped for Q2CY18
Up to 50 msec

APIC APIC APIC APIC

 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to

170 leaf nodes supported)
 Pod isolation scenario: same considerations as with 5 nodes (different behaviour
across shards)
 Pod hard failure scenario
• No chance of total loss of information for any shard
• Can bring up a standby node in the second site to regain full majority for all the shards
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
APIC Cluster Deployment Considerations Q2CY18
What about a 4 Nodes APIC Cluster?

Pod 1 Pod 2

Pending internal validation,

scoped for Q2CY18
Up to 50 msec

APIC APIC APIC APIC

 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to

170 leaf nodes supported)
 Pod isolation scenario: same considerations as with 5 nodes (different behaviour
across shards)
 Pod hard failure scenario
• No chance of total loss of information for any shard
• Can bring up a standby node in the second site to regain full majority for all the shards
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
APIC Cluster Deployment Considerations Q2CY18
What about a 4 Nodes APIC Cluster?

X
Pod 1 Pod 2

Pending internal validation,

scoped for Q2CY18
Up to 50 msec

X X
APIC APIC APIC APIC

 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to

170 leaf nodes supported)
 Pod isolation scenario: same considerations as with 5 nodes (different behaviour
across shards)
 Pod hard failure scenario
• No chance of total loss of information for any shard
• Can bring up a standby node in the second site to regain full majority for all the shards
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
APIC Cluster Deployment Considerations Q2CY18
What about a 4 Nodes APIC Cluster?

X
Pod 1 Pod 2

Pending internal validation,

scoped for Q2CY18
Up to 50 msec

X X
APIC APIC APIC APIC APIC

 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to

170 leaf nodes supported)
 Pod isolation scenario: same considerations as with 5 nodes (different behaviour
across shards)
 Pod hard failure scenario
• No chance of total loss of information for any shard
• Can bring up a standby node in the second site to regain full majority for all the shards
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
APIC Cluster Deployment Considerations Q2CY18
What about a 4 Nodes APIC Cluster?

X
Pod 1 Pod 2

Pending internal validation,

scoped for Q2CY18
Up to 50 msec

X X
APIC APIC APIC APIC APIC

 Intermediate scalability values compared to a 3 or 5 nodes cluster scenario (up to

170 leaf nodes supported)
 Pod isolation scenario: same considerations as with 5 nodes (different behaviour
across shards)
 Pod hard failure scenario
• No chance of total loss of information for any shard
• Can bring up a standby node in the second site to regain full majority for all the shards
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
APIC Cluster Deployment Considerations
Deployment Recommendations

 Main recommendation: deploy a 3 nodes APIC cluster when less than 80 leaf nodes
are deployed across Pods
 From Q2CY18 can deploy 4 nodes if the scalability requirements are met
 When 5 (or 7) nodes are really needed for scalability reasons, follow the rule of thumb
of never placing more than two APIC nodes in the same Pod (when possible):
Pod1 Pod2 Pod3 Pod4 Pod5 Pod6

2 Pods* APIC APIC APIC APIC APIC

3 Pods APIC APIC APIC APIC APIC

4 Pods APIC APIC APIC APIC APIC

5 Pods APIC APIC APIC APIC APIC

6+ Pods APIC APIC APIC APIC APIC

*’ID Recovery’ procedure possible for recovering of lost information

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Inter-Pod Connectivity Deployment
Considerations
ACI Multi-Pod
Inter-Pod Network (IPN) Requirements

Pod ‘A’ Pod ‘B’

MP-BGP - EVPN

DB Web/App APIC Cluster Web/App

 Not managed by APIC, must be separately configured (day-0 configuration)

 IPN topology can be arbitrary, not mandatory to connect to all spine nodes
 Main requirements:
 Multicast BiDir PIM  needed to handle Layer 2 BUM* traffic
 OSPF to peer with the spine nodes and learn VTEP reachability
 Increase MTU support to handle VXLAN encapsulated traffic
 DHCP-Relay

* Broadcast, Unknown unicast, Multicast BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Inter-Pod Connectivity
Frequently Asked Questions
What platforms can or should I
deploy in the IPN?
Can I use a 10G connection
between the spines and the IPN
network?
 Nexus 9200s, 9300-EX, but also any other
switch or router supporting all the IPN
requirements
 First generation Nexus 9300s/9500s not
supported as IPN nodes
 Yes, with QSA adapters supported on the ACI
spine devices
Available from 2.1(1h) release on EX/FX based
HW
No plans to introduce support for first generation
spines (including 9336-PQ ‘baby spine’)
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Inter-Pod Connectivity
Frequently Asked Questions (2)

I have two sites connected with

POD 1
X POD 2

dark fiber/DWDM circuits, can I

connect the spines back-to- APIC Cluster

back?
 No, because of multicast requirement for L2 multi-
destination inter-Pod traffic

10G*/40G/100G
IPN Devices
connections

POD 1 POD 2

Do I need a dedicated pair of

IPN devices in each Pod?
APIC Cluster

 Can use a single pair of IPN devices, but before 2.1(1h) release
mandates the use of 40G/100G inter-Pod links
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Control and Data Planes
 For more information on how to
ACI Multi-Pod setup an ACI Fabric from scratch:
BRKACI-2004, BRKACI-2820
Auto-Provisioning of Pods
DHCP requests are relayed
by the IPN devices back to
Provisioning interfaces on the spines the APIC in Pod 1 Spine 1 in Pod 2 connects to
facing the IPN and EVPN control the IPN and generates DHCP
plane configuration 5 requests

3 4
1
6
DHCP response reaches Spine 1
allowing its full provisioning

2 7

Discovery and provisioning Discovery and provisioning

of all the devices in the 1 Single APIC Cluster of all the devices in the
local Pod 8 local Pod
9
APIC Node 1 connected to a APIC Node 2 connected to
APIC Node 2 joins the a Leaf node in Pod 2
Leaf node in ‘Seed’ Pod 1
‘Seed’ Pod 1 Cluster Pod 2

10 Discover other Pods following the same procedure

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 ACI Multi-Pod IPN Network Routing Table

IPN Control Plane

10.0.0.0/16
10.1.0.0/16
 Separate IP address pools for VTEPs
assigned by APIC to each Pod
Summary routes advertised toward the IPN OSPF OSPF
via OSPF routing IPN

IS-IS convergence events local to a Pod not

propagated to remote Pods IS-IS to OSPF
10.0.0.0/16 mutual redistribution 10.1.0.0/16
 Spine nodes redistribute other Pods
summary routes into the local IS-IS APIC Cluster
process
Leaf Node Underlay VRF
Needed for local VTEPs to communicate with
remote VTEPs IP Prefix Next-Hop
10.1.0.0/16 Pod1-S1, Pod1-S2, Pod1-S3, Pod1-S4

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 ACI Multi-Pod
Inter-Pod MP-BGP EVPN Control Plane
 MP-BGP EVPN to sync Endpoint (EP)
and Multicast Group information EP1 Leaf 1 EP1 Proxy A
Leaf 3 MP-BGP - EVPN
All remote Pod entries associated to a Proxy EP2 EP2 Proxy A

VTEP next-hop address (not part of local EP3 Proxy B EP3 Leaf 4
TEP Pool) EP4 Proxy B IPN EP4 Leaf 6

Same BGP AS across all the Pods

Proxy A Proxy B

 iBGP EVPN sessions between spines in COOP

separate Pods
Full mesh MP-iBGP EVPN sessions EP2 APIC Cluster EP3 EP4
EP1
between local and remote spines (default
behavior)
Optional RR deployment (recommended Single BGP ASN
one RR in each Pod for resiliency)

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 ACI Multi-Pod Policy and network
information carried = VXLAN Encap/Decap
across Pods
Inter-Pod Data Plane
VTEP IP VNID Class-ID Tenant Packet

Spine encapsulates
EP1 Leaf 4 EP2 Leaf 4
traffic to remote
EP2 Proxy B EP1 Proxy A
Proxy B Spine VTEP IPN Spine encapsulates
traffic to local leaf
3 4
Proxy A Proxy B

EP2 e1/1
EP1 e1/3 EP1 Pod1 L4

5 * Proxy B
* Proxy A
Leaf learns remote EP1
EP2 unknown, traffic is 2 location and enforces policy
EP1 EP2
encapsulated to the local Proxy APIC Cluster
A Spine VTEP (adding S_Class 1 6
information) VM1 sends traffic destined If policy allows it, EP2
to remote EP2 receives the packet
EP1 EP2
EPG
C EPG
Configured on APIC
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 ACI Multi-Pod = VXLAN Encap/Decap
Inter-Pod Data Plane (2)

IPN

Proxy A Proxy B
EP1 e1/3
EP2 Pod2 L4 EP1 Pod1 L4

** Proxy A
8 * Proxy B

Leaf learns remote VM2 location 9 Leaf enforces policy in ingress

(no need to enforce policy) and, if allowed, encapsulates
EP1 EP2 traffic to remote Leaf node L4
APIC Cluster
10 7
VM1 receives the packet VM2 sends traffic back to
remote VM1
EP1 EP2
EPG
C EPG
Configured on APIC
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
ACI Multi-Pod = VXLAN Encap/Decap
Inter-Pod Data Plane (3)

IPN

Proxy A Proxy B
EP1 e1/3
EP2 Pod2 L4 EP1 Pod1 L4

** Proxy A
* Proxy B

EP1 EP2
APIC Cluster

From this point EP1 to EP2 communication is encapsulated

Leaf to Leaf (VTEP to VTEP) and policy always applied at the
11 ingress leaf (applies to both L2 and L3 communication)

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 ACI Multi-Pod
Use of Multicast for Inter-Pod Layer 2 BUM Traffic
BUM traffic originated in
the local Pod IPN1
 Ingress replication for BUM* traffic not
IGMP Join for (*, GIPo1) supported with Multi-Pod
IPN2  PIM Bidir is the only validated and
Spine 1 elected
supported option
authoritative for BD1 BUM traffic
originated from a
Scalable: only a single (*,G) entry is created in
remote Pod the IPN for each BD
Fast-convergent: no requirement for data-
driven multicast state creation
 A spine is elected authoritative for each
Bridge Domain:
Generates an IGMP Join on a specific link
toward the IPN
Always sends/receives BUM traffic on that link
BD1 GIPo1: 225.1.1.128

BUM: Broadcast, Unknown Unicast, Multicast BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 ACI Multi-Pod
Use of Multicast for Inter-Pod BUM Traffic
IPN replicates traffic to all the
4 PODs that joined MG1
(optimized delivery to Pods)

Spine 2 is designated to send

MG1 traffic toward the IPN

BUM frame is flooded

5 along one of the trees
* 2 associated to MG1

BD1 has associated MG1,

traffic is flooded intra-Pod via
one multi-destination tree EP1 EP2
APIC Cluster
1 6
VM1 in BD1 generates a VM2 receives the BUM
BUM* frame frame

BUM: Layer 2 Broadcast, Unknown Unicast, Multicast

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 ACI Multi-Pod
PIM Bidir for BUM – Supported Topologies
Full Mesh between remote IPN devices
IPN1 IPN3  Create full-mesh connections between IPN devices
 More costly for geo-dispersed Pods, as it requires
IPN2 IPN4
more links between sites
 Alternatively, connect local IPN devices with a port-
*
*
channel interface (for resiliency)
APIC Cluster EP2
 In both cases, it is critical to ensure that the
preferred path toward the RP from any IPN devices
is not via a spine
Directly connect local IPN devices
IPN1 IPN3  Recommendation is to increase the OSPF cost of
the interfaces between IPN and spines
IPN2 IPN4
interface Ethernet1/49.4 IPN1
description L3 Link to Pod1-Spine1
mtu 9150
encapsulation dot1q 4 e1/49
* ip address 192.168.1.1/31
* ip ospf cost 100
ip ospf network point-to-point
APIC Cluster EP2 ip router ospf IPN area 0.0.0.0 IPN2
ip pim sparse-mode
ip dhcp relay address 10.1.0.2
ip dhcp relay address 10.1.0.3

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Connecting to the External Layer 3
Domain
Connecting ACI to Layer 3 Domain
‘Traditional’ L3Out on the BL Nodes
Client
PE
PE
WAN
PE
L3Out PE

 Connecting to WAN Edge devices at

Border Leaf nodes
Definition of a L3Out logical construct
 VRF-lite hand-off for extending L3 multi-
Border Leafs
tenancy outside the ACI fabric
Each tenant defines one (or more) L3Out with
a set of Logical Nodes, Logical Interfaces,
peering protocol

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 Connecting Multi-Pod to Layer 3 Domain
‘Traditional’ L3Out on the BL Nodes

 A Pod does not need to have a dedicated WAN

connection (i.e. can offer transit services to other
Pods)
MP-BGP - EVPN
 Multiple WAN connections can be deployed across
Pods
 Outbound traffic: by default VTEPs always select
WAN connection in the local Pod based on preferred Pod 1 Pod 2
metric

WAN WAN

Pod 3

By default traffic flows

are hashed across
L3Outs of remote
Pods

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 Connecting Multi-Pod to Layer 3 Domain
‘Traditional’ L3Out on the BL Nodes (2)

 Asymmetric traffic paths creates issues when

independent active perimeter FWs are
deployed across Pods
 Tuning routing is possible to ensure MP-BGP - EVPN

ingress/egress traffic leverages always the

same Pod’s L3Out
• Reverting to an “Active/Standby” mode of operation for
Pod 1 Pod 2
the deployed FWs Active FW Active FW

 Host routes advertisement is a best option to

ensure all the deployed FWs are actively WAN WAN

utilized
Pod 3
• Support for host route advertisement on BL nodes
planned for a future ACI release
• Requires an L3Out connection in each Pod

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 For More Information on
Connecting ACI to Layer 3 Domain GOLF Deployment:
LABACI-2101
‘GOLF’ Design
= VXLAN Encap/Decap Client
PE
PE
WAN
PE
PE
VXLAN Data
Plane
GOLF Routers (ASR 9000, ASR
DCI 1000, Nexus 7000)
OTV/VPLS
 Direct or indirect connection from spines to WAN Edge
routers
 Better scalability, one protocol session for all VRFs, no longer
constraint by border leaf HW table
 VXLAN handoff with MP-BGP EVPN
 Simplified tenant L3Out configuration
 Support for host routes advertisement out of the ACI Fabric
 VRF configuration automation on GOLF router through
OpFlex exchange
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
 GOLF and Multi-Pod Integration
Centralized and Distributed Models
Centralized WAN Edge Devices Distributed WAN Edge Devices
WAN

WAN Edge
Routers
WAN
WAN Edge WAN Edge
Routers Routers
IPN MP-BGP
EVPN IPN IPN

MP-BGP
EVPN

 Common when Pods represent rooms/halls in the

same physical DC  Pods usually represent separate physical DCs

 MP-BGP EVPN peering required from spines in  Full mesh of EVPN peerings between Pods
each Pod and the centralized WAN Edge devices and WAN Edge routers
For more info on GOLF and Multi-Pod integration:
https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=94038&backBtn=true
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
GOLF and Multi-Pod Integration
Inter-DC Scenario
WAN Edge devices inject host
routes into the WAN or register
Host routes for endpoint belonging them in the LISP database
Host routes for endpoint belonging
to public BD subnets in Pod ‘A’
to public BD subnets in Pod ‘B’

MP-BGP EVPN Control Plane

MP-BGP EVPN Control Plane

Pod ‘A’ IPN Pod ‘B’

APIC Cluster

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
GOLF and Multi-Pod Integration
Inter-DC Scenario (2)
Remote Router Table Granular inbound path
10.10.10.10/32 optimization( host route
G1,G2
advertisement into the WAN or
10.10.10.11/32 G3,G4
integration with LISP)

G1,G2 Routing Table

10.10.10.0/24 A WAN G3,G4 Routing Table
10.10.10.10/32 A 10.10.10.0/24 B
10.10.10.11/32 B

IPN

Proxy A Proxy B

10.10.10.10 10.10.10.11

APIC Cluster

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Network Services Integration
ACI Multi-Pod
Network Services Integration Models

 Active and Standby pair deployed across Pods

 No issues with asymmetric flows but may cause
traffic hair-pinning across the IPN
Active Standby
 Works in all scenarios from release 2.3

 Independent Active/Standby pair deployed in

each Pod
 Use PBR (managed or unmanaged mode)
 Only for perimeter FW use case assuming proper
solution is adopted to keep symmetric
Active/Standby Active/Standby ingress/egress flows

 FW cluster deployed across Pods

 Not currently supported (scoped for 1HCY18)
Cluster

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Active/Standby Pair across Pods
Option 1: FW in L2 Mode

IPN

APIC Cluster

L3Out-1 L3Out-2
WAN WAN

Active MAC G Standby

L2 Mode L2 Mode
WAN = East-West
= North-South

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Active/Standby Pair across Pods
Option 2: FW in L3 Mode and PBR

IPN

PBR Policy APIC Cluster

Applied Here

L3Out- L3Out-
1 2
WAN WAN

L3 Mode L3 Mode
Active Standby
WAN = East-West
= North-South

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Active/Standby Pair across Pods
Option 2: FW in L3 Mode and PBR

IPN

PBR Policy APIC Cluster

Applied Here

L3Out- L3Out-
1 2
WAN WAN

L3 Mode L3 Mode
Active Standby
WAN = East-West
= North-South

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
FW in L3 Mode and L3Outs
Single L3Out Defined across Pods

IPN

APIC Cluster

L3Out
ASA In/Out
Web VM1 Web VM2
BDs associated to L3Outs
L3 Mode L3 Mode
Active
extended via Multi-Pod Standby

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 FW in L3 Mode and L3Outs
Single L3Out Defined across Pods (Dynamic Routing)

IPN

Routing table (VRF1)

Routing table (VRF1)
APIC Cluster External IP prefix via Pod2-sLeaf1/2
External IP prefix via Pod1-sLeaf1/2

Secondary IP: 192.168.3.254

L3Out
ASA In/Out
Web VM1 Traffic Bounced Web VM2
Peering across Pods
192.168.1.201 192.168.1.202
L3 Mode L3 Mode
Active Standby
Active ASA: 192.168.3.1 Standby ASA: 192.168.3.2

Note: supported from ACI SW releases 2.1(3), 2.2(3), 2.3(1) and 3.0(1)
and deploying EX/FX HW for ACI service leaf nodes
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
 FW in L3 Mode and L3Outs
Single L3Out Defined across Pods (Static Routing)

IPN

Routing table (VRF1)

Routing table (VRF1)
APIC Cluster External IP prefix via Pod2-sLeaf1/2
External IP prefix via Pod1-sLeaf1/2

Secondary IP: 192.168.3.254 Secondary IP: 192.168.3.254

Secondary IP: 192.168.3.254

L3Out
ASA In/Out
Web VM1 Static route Traffic Bounced Web VM2
injected into MP- across Pods
192.168.1.201 BGP VPNv4 Fabric 192.168.1.202
L3 Mode control plane L3 Mode
Active Standby
Active ASA: 192.168.3.1 Standby ASA: 192.168.3.2

Note: supported from ACI SW releases 2.1(3), 2.2(3), 2.3(1) and 3.0(1)
and deploying EX/FX HW for ACI service leaf nodes
BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Migration Scenarios
Migration Scenarios
Adding Pods to an Existing ACI
Add connections to the Connect and auto-provision
1 IPN network the other Pod(s)

Pod1 Pod2
MP-BGP - EVPN

Distribute the APIC nodes

across Pods

2 Add connections to the

IPN network
Connect and auto-provision
the other Pod(s)

MP-BGP EVPN

Pod1 Distribute the APIC nodes Pod2

across Pods

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Migration Scenarios
Converting Stretched Fabric to Multi-Pod

3
Pod1 Pod2

MP-BGP EVPN

 Re-cabling of the physical interconnection (especially when using

DWDM circuits that must be reused)
 Re-addressing the VTEP address space for the second Pod 
disruptive procedure as it requires a clean install on the second Pod
 Not internally QA-validated or recommended

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Conclusions and Q&A
ACI Multi-Pod & Multi-Site
A Reason for Both

Multi-Pod Fabric ‘A’ (AZ 1)

‘Classic’ Active/Active

Pod ‘1.A’ Pod ‘2.A’

ACI Multi-Site

Multi-Pod Fabric ‘B’ (AZ 2)

‘Classic’ Active/Active
Application
Pod ‘1.B’workloads Pod ‘2.B’
deployed across
availability zones BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Conclusions

 Cisco ACI offers different multi-fabric

options that can be deployed today
 There is a solid roadmap to evolve
those options in the short and mid term
 Multi-Pod represents the natural
evolution of the existing Stretched
Fabric design
 Multi-Site will replace the Dual-Fabric
MP-BGP EVPN MP-BGP EVPN
approach

 Cisco will offer migration options to

drive the adoption of those new
solutions

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Where to Go for More Information

 ACI Stretched Fabric White Paper

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-aci-
stretched-fabric.html#concept_524263C54D8749F2AD248FAEBA7DAD78
 ACI Multi-Pod White Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-
centric-infrastructure/white-paper-c11-737855.html?cachemode=refresh
 ACI Dual Fabric Design Guide
http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737077.html?cachemode=refresh
 ACI and GOLF High Level Integration Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/application-centric-infrastructure/white-paper-c11-736899.html

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session

How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKACI-2003

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
www.ciscolive.com/global/on-demand-library/.

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions

BRKACI-2003 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Thank you