Brkaci 2003

Deployment Options for
Interconnecting Multiple ACI Fabrics
Max Ardica – Principal Engineer

BRKACI-2003
Session Objectives
At the end of the session, the participants should be able to:

 Articulate the different Multi-Fabric deployment options
offered with Cisco ACI
 Understand the design considerations associated to those
options
Initial assumption:
 The audience already has a good knowledge of ACI main
concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)
BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• ACI Multi-Fabric Design Options

• ACI Stretched Fabric Overview
• ACI Multi-Pod Deep Dive
• ACI Multi-Site Solutions Overview
• Conclusions and Q&A
Introducing: Application Centric Infrastructure (ACI)
Web App DB
Outside QoS QoS QoS
(Tenant
Filter Service Filter
VRF)
APIC
Application Policy
ACI Fabric Infrastructure Controller
Integrated GBP VXLAN Overlay
ACI MultiPod/MultiSite Use Cases
 Single Site Multi-Fabric
Multiple Fabrics connected within the same DC (between halls, buildings, … within the
same Campus location)
Cabling limitations, HA requirements, Scaling requirements
 Single Region Multi-Fabric (Classic Active/Active scenario)

Scoped by the application mobility domain of 10 msec RTT
BDs/IP Subnets can be stretched between sites
Desire is reducing as much as possible fate sharing across sites, yet maintaining
operational simplicity
 Multi Region Multi-Fabric

Creation of separate Availability Zones
Disaster Recovery – Minimal Cross Site Communication
Deployment of applications not requiring Layer 2 adjacency
Interconnecting ACI Fabrics
Design Options
Single APIC Cluster/Single Domain Multiple APIC Clusters/Multiple Domains
Stretched Fabric Dual-Fabric Connected (L2 and L3 Extension)
ACI Fabric ACI Fabric 1 ACI Fabric 2
Site 1 Site 2
L2/L3
Multi-Pod (2.0 release) Multi-Site (Future)
Pod ‘A’ IP Network Pod ‘n’ Site ‘A’ Site ‘n’
MP-BGP - EVPN
… MP-BGP - EVPN
APIC Cluster
Agenda

For more information on ACI Stretched
Stretched ACI Fabric Fabric Deployment:
BRKACI-3503
ACI Stretched Fabric
DC Site 1 DC Site 2
vCenter
Transit leaf Transit leaf
 Fabric stretched between two or three sites  works as  Work with one or more transit leaf per site  any leaf
a single fabric deployed within a DC node can be a transit leaf
 One APIC cluster  one management and configuration  Number of transit leaf and links dictated by
point redundancy and bandwidth capacity decision
 Anycast GW on all leaf switches  Different options for Inter-site links (dark fiber, DWDM,
EoMPLS PWs)
Stretched ACI Fabric
Option 1 – Dark Fiber
DC Site 1 DC Site 2
vCenter
Transit leaf Transit leaf
 40G links currently required between Transit Leafs and remote Spines
 QSA adapters required for deploying 10G connections
Roadmap item for CY17
Option 2 – DWDM Circuits
DC Site 1 DC Site 2
10ms RTT
QSFP-40G-SR4
40G DWDM 40G

40G
DWDM
40G 4x10G
40G
MTP-LC
breakout cable
 DWDM system provides connectivity between two sites

 SR with MTP-LC breakout cable between ACI node and DWDM system
 1.0(3f) release or later, Max 10ms RTT between sites
 QSA and 10G inter-site DWDM links (roadmap, CY16)
Option 3 – Ethernet over MPLS (EoMPLS) Port mode
DC Site 1 DC Site 2
10 ms RTT
800 KM / 500 miles
QSFP-40G-SR4
40G
10G/40G/100G
40G
EoMPLS Pseudowire
10G/40G/100G
40G 40G
WAN
 Port mode EoMPLS used to stretch the ACI  1.0(3f) release or later, 10ms max RTT between
fabric over long distance sites
DC Interconnect links could be 10G (minimum) or Under normal conditions 10 ms allows us to support two
higher with 40G facing the Leaf/Spine nodes DCs up to 800 Km apart
DWDM or Dark Fiber provide connectivity between Other ports on the Router used for connecting to the WAN
two sites via L3Out
Support for 3 Interconnected Sites (Q2CY16) Site 2
 Transit leafs in all sites connect to the

local and remote spines
Site 1
Site 3
Transit Leaf
2x40G or 4x40G
Agenda

Overview, Use Cases and Supported
Topologies
ACI Multi-Pod Solution
Overview
Inter-Pod Network
Pod ‘A’ Pod ‘n’
MP-BGP - EVPN
Single APIC Cluster

IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP
 Multiple ACI Pods connected by an IP Inter-Pod  Forwarding control plane (IS-IS, COOP)
L3 network, each Pod consists of leaf and spine fault isolation
nodes  Data Plane VXLAN encapsulation between
 Managed by a single APIC Cluster Pods for seamless L2 or L3 connectivity
 Single Management and Policy Domain  End-to-end policy enforcement 16
Use Cases
 Handling 3-tiers physical Pod
cabling layout Inter-Pod
Leaf Nodes Network
Cable constrain (multiple
buildings, campus, metro)
requires a second tier of “spines” Spine Nodes
Preferred option when compared
to ToR FEX deployment
 Evolution of Stretched Fabric Pod 1 Pod 2

design
Metro Area (dark fiber, DWDM),
L3 core
APIC Cluster
DB Web/App Web/App
>2 interconnected sites
Supported Topologies
Intra-DC Two DC sites directly connected
10G/40G/100G
40G/100G 40G/100G
POD 1 40G/100G 40G/100G
POD n POD 1 Dark fiber/DWDM POD 2
(up to 10 msec RTT)
…
DB APIC Cluster APIC Cluster

Web/App Web/App DB Web/App Web/App
3 DC Sites directly connected Multiple sites interconnected by a

10G/40G/100G
generic L3 network
40G/100G 40G/100G
POD 1 POD 2
Dark fiber/DWDM 40G/100G 40G/100G
(up to 10 msec RTT)
L3
40G/100G
40G/100G (up to 10msec RTT) 40G/100G
POD 3 BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
SW/HW Roadmap and Scalability
Values
SW and HW Requirements
 Software  Software
2.0 Release 2.0 MR Release (Q3CY16)
 Hardware  Hardware
1st Generation Leafs 2nd Generation Leafs
9396PX/TX, 9372PX/TX, 93120TX, 93128TX, 93108-EX, 93180-EX
9332PQ
2nd Generation Spines
1st Generation Spines 9732C-EX Line Card
9736PQ Line Card, 9336PQ
Scalability Considerations
Those scalability values may change without warning before the Multi-Pod
solution gets officially released
 Maximum number of supported ACI leaf nodes (across all Pods)

 Up to 80 leaf nodes supported with a 3 nodes APIC cluster
 300 with a 5 nodes APIC Cluster
 Maximum 200 leaf nodes per Pod
 Up to 6 spines per Pod
 Maximum number of supported Pods

 4 in Congo/Congo-MR releases (Q3CY16)
 6 in Crystal release (Q4CY16)
 No current plans to increase those values before end of CY16
Inter-Pod Connectivity Deployment
Considerations
Inter-Pod Network (IPN) Requirements
Pod ‘A’ 40G/100G 40G/100G

Pod ‘B’
MP-BGP - EVPN
DB Web/App
APIC Cluster Web/App
 Not managed by APIC, must be pre-configured

 IPN topology can be arbitrary, not mandatory to connect to all spine nodes
 Main requirements:
Multicast BiDir PIM  needed to handle BUM traffic
OSPF to peer with the spine nodes and learn VTEP reachability
Increased MTU support to handle VXLAN encapsulated traffic
Inter-Pod Connectivity
Frequently Asked Questions
 Nexus 9200s, 9300-EX, but also any other
switch or router supporting all the IPN
requirements
What platforms can or should I
deploy in the IPN?  First generation Nexus 9300s/9500s not initially
supported as IPN nodes
SW fix is being scoped for 2HCY16 timeframe
 Yes, once QSA adapters will be supported on

Can I use a 10G connection the ACI spine devices
between the spines and the IPN Planned for 2.1 release (Q4CY16) on EX based
network? HW
Scoped for CY17 for first generation spines
Frequently Asked Questions (2)
I have two sites connected with

POD 1
X POD 2
dark fiber/DWDM circuits, can I

connect the spines back-to- APIC Cluster
back?
 No, because of multicast requirement for L2 multi-
destination inter-Pod traffic
40G/100G
IPN Devices
connections
POD 1 POD 2
Do I need a dedicated pair of

IPN devices in each Pod?
APIC Cluster
 Can use a single pair of IPN devices, but initially

mandates the use of 40G/100G inter-Pod links
Frequently Asked Questions (3)
Can I use a Layer 2 only  No, the IPN nodes should be deployed as L3
infrastructure as IPN? network devices
Any Protocol
OSPF OSPF
L3
POD 1 POD 2
IPN
Is OSPF the only protocol Devices
supported in the IPN network?
APIC Cluster
 OSPF mandatory but only between the spines and

the firstBRKACI-2003
L3 hop IPN device
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
APIC Cluster Deployment
Considerations
APIC – Distributed Multi-Active Data Base
One copy is ‘active’ for every

The Data Base is specific portion of the Data
replicated across APIC Base
nodes
Shard 1 Shard 1 Shard 1
APIC APIC APIC
Shard 2 Shard 3 Shard 2 Shard 3 Shard 2 Shard 3
 Processes are active on all nodes (not active/standby)

 The Data Base is distributed as active + 2 backup instances (shards) for every attribute
APIC Cluster Deployment Considerations
Single Pod Scenario
X X
APIC APIC APIC Shards in
‘read-only’
mode
X X
APIC APIC APIC APIC APIC
Shards in Shards in
‘read-only’ ‘read-write’ mode
 APIC will allow read-only access to the DB  mode
Additional APIC will increase the system scale (today
when only one node remains active (standard up to 5 nodes supported) but does not add more
DB quorum) redundancy
 Hard failure of two nodes cause all shards to  Hard failure of two nodes would cause inconsistent
be in ‘read-only’ mode (of course reboot etc. behaviour across shards (some will be in ‘read-only’
heals the cluster after APIC nodes are up) mode, some in ‘read-write’ mode)
Multi-Pod Scenario
X X
Pod 1 Pod 2 Pod 1 Pod 2
X X
Up to 10 msec Up to 10 msec
X X
APIC APIC APIC APIC
X X X
APIC APIC APIC APIC APIC
 Pod isolation scenario: changes still possible  Pod isolation scenario: same considerations as with
on APIC nodes in Pod1 but not in Pod2. single Pod (inconsistent behaviour across shards)
Cluster nodes fully re-join once Pods are  Pod failure scenario: Pod1 major failure may cause
reconnected the loss of information for the shards replicated across
 Pod failure scenario: under major Pod failure, the 3 local nodes
recommendation is to activate a standby node Possible to restore the whole fabric state to the latest taken
to make the cluster fully functional again configuration snapshot (‘ID Recovery’ procedure – needs BU
and TAC involvement)
Deployment Recommendations
 Main recommendation: deploy a 3 nodes APIC cluster, independently from the total
number of Pods (plus 1 backup node in a 2 Pods scenario)
Avoiding loss of information under any failure scenario
Pods can be centrally managed even without a locally connected APIC node
 When 5 nodes are needed for scalability reasons, follow the recommendations below:
Pod1 Pod2 Pod3 Pod4 Pod5 Pod6
2 Pods* APIC APIC APIC APIC APIC APIC

Standby
3 Pods APIC APIC APIC APIC APIC
6+ Pods APIC APIC APIC APIC APIC
*’ID Recovery’ procedure possible for recovering of lost information

Reducing the Impact of Configuration Errors
Introducing Configuration Zones
 Common concern with Multi-Pod: the end-to-end propagation of configuration

mistakes
 If a policy is configured to be deployed on many nodes, a change to the policy is
immediately propagated to all nodes by default
 It would be desirable to assess the impact of a policy change on a sub-set of
nodes before propagating the change on all remaining nodes
 Configuration Zones can be used for this purpose
 Infra policies are currently the only policies following zone deployment
configuration
 Tenant polices are immediately propagated to all nodes independently of zone
configuration
Reducing the Impact of Configuration Errors
Introducing Configuration Zones
 Three different zone deployment modes:

 Enabled (default): updates are immediately sent
to all nodes part of the zone
Note: a node not part of any zone is equivalent Change the deployment
to a node part of a zone set to enabled. mode
Select entire Pod
 Disabled: updates are postponed until the zone
deployment mode is changed (or a node is Select specific Leaf Switches
removed from the zone)
 Triggered: send postponed updates to the nodes
part of the zone
Show the changes not applied yet
 The deployment mode can be configured for to a Disabled zone
an entire Pod or for a specified set of leaf
switches
Multi-Pod Control and Data Planes
For more information on how to setup an
ACI Multi-Pod Solution ACI Fabric from scratch:
Auto-Provisioning of Pods BRKACI-2004
DHCP requests are relayed

by the IPN devices back to
Provisioning interfaces on the the APIC in Pod 1 Spine 1 in Pod 2 connects
spines facing the IPN and EVPN to the IPN and generates
control plane configuration 51 DHCP requests
3
1 4
1
6
1
DHCP response reaches Spine 1
allowing its full provisioning
2
1 7
1
Discovery and Discovery and

provisioning of all the 1 Single APIC Cluster provisioning of all the
devices in the local Pod 8
1 devices in the local Pod
9
1
APIC Node 1 connected to a APIC Node 2 connected
APIC Node 2 joins the to a Leaf node in Pod 2
Leaf node in ‘Seed’ Pod 1
‘Seed’ Pod 1 Cluster Pod 2
10
1 Discover other Pods following the same procedure
IPN Control Plane
IPN Global VRF
IP Prefix Next-Hop
10.0.0.0/16 Pod1-S1, Pod1-S2, Pod1-S3, Pod1-S4

 Separate IP address pools for VTEPs
assigned by APIC to each Pod
OSPF OSPF
Summary routes advertised toward the IPN
via OSPF routing
IS-IS to OSPF
 Spine nodes redistribute other Pods mutual redistribution
10.0.0.0/16 10.1.0.0/16
summary routes into the local IS-IS
process DB APIC Cluster
Web/App Web/App
Needed for local VTEPs to communicate with
remote VTEPs Leaf Node Underlay VRF
IP Prefix Next-Hop
ACI Fabric – Integrated Overlay
Decoupled Identity, Location & Policy
APIC
VTEP VXLAN IP Payload
VTEP VTEP VTEP VTEP VTEP VTEP
 ACI Fabric decouples the tenant end-point address, it’s “identifier”, from the location of that end-
point which is defined by it’s “locator” or VTEP address
 Forwarding within the Fabric is between VTEPs (ACI VXLAN tunnel endpoints) and leverages an
extender VXLAN header format referred to as the ACI VXLAN policy header
 The mapping of the internal tenant MAC or IP address to location is performed by the VTEP using
a distributed mapping database
Host Routing - Inside
Inline Hardware Mapping DB - 1,000,000+ hosts
10.1.3.35 Leaf 3
10.1.3.11 Leaf 1
Global Station Table Proxy Proxy Proxy Proxy fe80::8e5e Leaf 4
contains a local cache fe80::5b1a Leaf 6
of the fabric endpoints
10.1.3.35 Leaf 3
Proxy Station Table contains
addresses of ‘all’ hosts attached
* Proxy A to the fabric
10.1.3.11 Port 9
10.1.3.11 10.1.3.35 fe80::462a:60ff:fef7:8e5e fe80::62c5:47ff:fe0a:5b1a
Local Station Table  The Forwarding Table on the Leaf Switch is divided between local (directly attached) and
contains addresses of global entries
‘all’ hosts attached  The Leaf global table is a cached portion of the full global table
directly to the Leaf
 If an endpoint is not found in the local cache the packet is forwarded to the ‘default’
forwarding table in the spine switches (1,000,000+ entries in the spine forwarding table)
Inter-PODs MP-BGP EVPN Control Plane
 MP-BGP EVPN used to communicate

Endpoint (EP) and Multicast Group EP1 Leaf 1 EP1 Proxy A
information between Pods EP2 Leaf 3 MP-BGP - EVPN
EP2 Proxy A
All remote Pod entries associated to a Proxy EP3 Proxy B EP3 Leaf 4
VTEP next-hop address EP4 Proxy B EP4 Leaf 6
 Single BGP AS across all the Pods Proxy A Proxy B
 BGP EVPN on multiple spines in a Pod COOP
(minimum of two for redundancy) Single APIC Cluster

EP2 EP3 EP4
Full mesh MP-iBGP EVPN sessions EP1
between local and remote spines (default

behavior)
Single BGP ASN
Optional RR deployment (recommended
one RR in each Pod for resiliency)
Inter-PODs Data Plane Policy information
carried across Pods = VXLAN Encap/Decap
Group
VTEP IP VNID Tenant Packet
Policy
Spine encapsulates
EP1 Leaf 4 Leaf 4
EP2 Proxy B traffic to remote Proxy EP2
Proxy A
EP1
B Spine VTEP Spine encapsulates
traffic to local leaf
3 4
Proxy A Proxy B
EP2 e1/1
EP1 e1/3 EP1 Pod1 L4
5 * Proxy B
* Proxy A
Leaf learns remote VM1
Single APIC Cluster location and enforces policy
VM2 unknown, traffic is 2
EP1 EP2
encapsulated to the local Proxy
A Spine VTEP (adding S_Class 1 6
information) VM1 sends traffic destined If policy allows it, VM2
to remote VM2 receives the packet
Inter-PODs Data Plane (2) = VXLAN Encap/Decap
Group
VTEP IP VNID Tenant Packet
Policy
EP1 e1/3
EP2 Pod2 L4 EP1 Pod1 L4
** Proxy A
8 * Proxy B
Leaf learns remote VM2 location 9 Leaf enforces policy in ingress

(no need to enforce policy) Single APIC Cluster and, if allowed, encapsulates
EP1 EP2 traffic to remote Leaf node L4
10 7
VM1 receives the packet VM2 sends traffic back to
From this point VM1 to VM1 communication is encapsulated remote VM1
Leaf to Leaf (VTEP to VTEP) and policy always applied at the
11 ingress leaf (applies to both L2 and L3 communication)
Handling of Multi-Destination Traffic (BUM*)
IPN replicates traffic to all
the Pods that joined GIPo 1
(optimized delivery to Pods)
Spine 2 is responsible to 1
4
send GIPo 1 traffic toward
the IPN
1
3
BUM frame is flooded along the

tree associated to GIPo 1. VTEP
* 1
2 1
5
learns VM1 remote location
*
172.16.2.40 Pod1 L4
BUM frame is associated to
GIPo 1 and flooded intra-Pod Single APIC Cluster Proxy B
*
via the corresponding tree 172.16.2.40 172.16.1.20
1 1
6
VM1 generates a BUM VM2 receives the BUM
frame frame
*L2 Broadcast, Unknown Unicast and Multicast BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Establishing Layer 3 External
Connectivity
Connecting ACI to Layer 3 Domain
Traditional L3Out Design
 A Pod does not need to have a dedicated

WAN connection
Pod 1 Pod 2
 Multiple WAN connections can be deployed
across Pods MP-BGP - EVPN
 Traditional L3Out configuration

Shared between tenants or dedicated per tenant
(VRF-Lite)
 VTEPs always select WAN connection in
WAN WAN
the local Pod based on preferred metric
Inbound traffic may require “hair-pinning” across Pod 3
the IPN network
Recommended to deploy clustering technology
when stateful services are deployed
Connecting ACI to Layer 3 Domain For more information refer to:
BRKACI-2020
’GOLF’ Design (ACI 2.0 Release)
WAN
WAN Edge  Connect an ACI Fabric to the external L3
Routers
domain
Wan Edge devices functionally behave as ACI ‘border leafs’
VXLAN Data Plane L3Out at
spines Control plane and data plane scale
MP-BGP
OpFlex for automating the exchange of config parameters
EVPN
(VRF names, BGP Route-Targets, etc.)
 MP-BGP EVPN control plane between ACI

spine and WAN Edge routers
 VXLAN data plane between ACI spine and
VRF-1 VRF-2 WAN Edge routers
 Consistent policy for north-south traffic applied
at ACI leaf (both ingress and egress directions)
= VXLAN Encap/Decap BRKACI-2003 © 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
ACI Integration with WAN at Scale
Supported WAN Edge Platforms
WAN
 WAN Edge Router initial choices
Nexus 7000/7700: F3 line card in
7.3.0.DX(1)ES. M3 support in Q4CY16
MP-BGP
IP Network ASR 9000: IOS-XR 6.1.1 for platforms with
EVPN
RSP2*, RSP440 and RSP880 supervisors
ASR 1000: 16.4 release (Q4CY16), including
also CSR1Kv support
 High level whitepaper available on CCO:
http://www.cisco.com/c/en/us/solutions/collateral
/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-736899.html
*OpFlex support will be added in 6.2.1 release

ACI SW and HW Requirements
 Software  Software
2.0 Release (July 2016) 2.1 Release (Q4CY16)
 Hardware  Hardware
1st and 2nd Generation Leafs 2nd Generation Spines
9396PX/TX, 9372PX/TX, 93120TX, 93128TX, 9332PQ, 9732C-EX Line Card
93108-EX, 93180-EX
1st Gen Spines
9736PQ Line Card, 9336PQ
Supported Topologies
Directly Connected Indirectly Connected
WAN Edge Routers WAN Edge Routers Multi-Pod*
WAN WAN WAN
MP-BGP
MP-BGP
EVPN IP Network IP Network
EVPN
MP-BGP
EVPN
*Supported from 2.1 Release (Q4CY16)

Multi-Pod and Scalable WAN Integration
Centralized Scenario (Intra-DC)
WAN Edge Devices Connected to IPN WAN Edge Devices Connected to Pod Spines
WAN
WAN
MP-BGP
EVPN
MP-BGP
EVPN
 Inter-Pod VXLAN traffic exchanged via the IPN  WAN Edge devices perform a dual function:
devices  Pure L3 routing for Inter-Pod VXLAN traffic
 WAN to DC traffic VXLAN encapsulated between  VXLAN Encap/Decap for WAN to DC traffic
ACI fabric and WAN Edge devices flows
Intra-DC Deployment – Control Plane
WAN Edge
Devices WAN
WAN routes received on the Pod
spines as EVPN routes and translated
MP-BGP EVPN Control Plane to VPNv4/VPNv6 routes with the spine
proxy TEP as Next-Hop
IPN
Public BD subnets advertised to WAN
Edge devices with the external spine-
proxy TEP as Next-Hop
Multiple
Pods
...
Web/App Web/App Web/App
DB DB
Single
Single APIC Cluster
APIC Domain
Intra-DC Deployment – Control Plane
 Option to consolidate WAN Edge

and ‘IPN’ devices WAN Edge WAN
Perform pure L3 routing for Inter-Pod Devices
VXLAN traffic
VXLAN Encap/Decap for WAN to DC IPN
traffic flows
Multiple
Pods
...
Single
Single APIC Cluster
APIC Domain
Intra-DC Deployment – Data Plane
WAN Edge devices have host
routes information mapping WAN Edge
endpoints to each Pod location Devices WAN Traffic from an external user
is steered toward the WAN
Edge devices
WAN Edge devices VXLAN
encapsulate traffic and send it to
the Spine Proxy VTEP address in IPN
the ‘right’ Pod
Proxy A Proxy B Proxy n

Spine encapsulates traffic
to the destination VTEP Multiple
that can apply policy Pods
...
Single
Single APIC Cluster
APIC Domain
Intra-DC Deployment – Data Plane (2)
WAN Edge
Devices WAN
WAN Edge devices decapsulate Traffic is received by
traffic and send it to the WAN the external user
IPN
Leaf applies policy and Multiple

encapsulates traffic directly Pods
to the WAN Edge VTEP ...
address
Single
Single APIC Cluster
APIC Domain
Inter-DC Scenario
WAN Edge devices inject host

routes into the WAN or register
Host routes for endpoint belonging them in the LISP database
Host routes for endpoint belonging
to public BD subnets in Pod ‘A’
to public BD subnets in Pod ‘B’
MP-BGP EVPN Control Plane

MP-BGP EVPN Control Plane
Pod ‘A’ IPN Pod ‘B’
Single APIC Cluster
Inter-DC Scenario (2)
Remote Router Table Granular inbound path
10.10.10.10/32 optimization( host route
G1,G2
advertisement into the WAN or
10.10.10.11/32 G3,G4
integration with LISP)
G1,G2 Routing Table

10.10.10.0/24 A WAN G3,G4 Routing Table
10.10.10.10/32 A 10.10.10.0/24 B
10.10.10.11/32 B
IPN
Proxy A Proxy B
10.10.10.10 10.10.10.11
Single APIC Cluster
Migration Scenarios
Migration Scenarios
Adding Pods to an Existing ACI
Add connections to the Connect and auto-provision
1 IPN network the other Pod(s)
Pod1 Pod2
MP-BGP - EVPN
Distribute the APIC nodes

across Pods
2 Add connections to the

IPN network
Connect and auto-provision
the other Pod(s)
MP-BGP EVPN
Pod1 Distribute the APIC nodes Pod2

across Pods
Migration Scenarios
Converting Stretched Fabric to Multi-Pod
3
Pod1 Pod2
MP-BGP EVPN
 Re-cabling of the physical interconnection (especially when using

DWDM circuits that must be reused)
 Re-addressing the VTEP address space for the second Pod 
disruptive procedure as it requires a clean install on the second Pod
 Not internally QA-validated or recommended
Summary
 ACI Multi-Pod solution represents the natural evolution of the

Stretched Fabric design
 Combines the advantages of a centralized management and
policy domain with fault domain isolation (each Pod runs
independent control planes)
 Control and data plane integration with WAN Edge devices
(Nexus 7000/7700 and ASR 9000) completes and enriches the
solution
 The solution is planned to be available in Q3CY16 and will be
released with a companion Design Guide
Agenda

ACI Multi-Site Solutions
Providing Complete Isolation between ACI Fabrics
Policy Zone ‘A’ Policy Zone ‘B’

Site ‘A’ Site ‘B’
• Multi-Fabric Scenarios
Primary use case is to support multiple “Availability Zones”
Use Cases
 Multiple Fabrics within a single site (includes Multi-Floor, Multi-Room Data Centers)
 Multi-Building cross campus and metro distances (Majority of larger customers
require a dual site active/active design)
For more information on ACI Dual
ACI Multi-Site Solutions Fabric Deployment:
Current Deployment Option 1 – Dual Fabric BRKACI-3503
Orchestrator
(UCSD, etc…)
L2 Transport
Direct/vPC
OTV/VXLAN/PBB
HYPERVISOR HYPERVISOR HYPERVISOR

HYPERVISOR HYPERVISOR HYPERVISOR
WAN
L2 Outside
VRF-Lite (iBGP, OSPF, Static Routes)
L2 and L3 Traffic
ACI Dual Fabric Design Guide

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737077.pdf
Current Deployment Option 2 – ACI Toolkit Use Case
L3 Traffic Only
WAN
‘Register for EP
notification in ‘WEB1’ EPG Ext-WEB1 App
APP1 WEB1
‘Ext-WEB1 EPG created in
Subnet/BD ‘A’ ACI
Toolkit remote L3Out
 ‘Intersite’ Application of ACI Toolkit allows to peer with local and remote APIC
clusters and specifies:
What local EPG needs to be “exported” to a remote site (‘WEB1’) and its name in the
remote location (‘Ext-WEB1’)
What contracts will be consumed/provided by that EPG in the remote site
The L3Out in the remote sites where to program host routes
Current Deployment Option 2 – ACI Toolkit Use Case
IP == EPG classification on
the Border Leaf (scaling is
limited to LST 24K entries)
Policy applied on the
egress VTEP
WAN
Notifies about endpoints
APP1 WEB1 connecting to the specified Ext-WEB1 App
Subnet/BD ‘A’ ACI ‘WEB1’ EPG
Toolkit Program host routes into the
‘Ext-WEB1’ EPG in remote L3Out
 The local APIC notifies the toolkit every time an endpoint connects to the ‘WEB1’ EPG in the
“local” site
 The ACI Toolkit communicate to the remote APIC to program the host route in the L3Out of the
“remote” site (part of the ‘Ext-WEB1’ EPG)
 This allows to properly classify traffic received in DC site 2  specified contract is also applied
ACI Multi-Site Solutions Future
Future Deployment Option (CY17)
Fabric ‘A’ Fabric ‘B’

mBGP - EVPN
DB Multiple APIC Clusters

Web/App Web/App
Web1 Web2 Import Web & App Export Web & App Web1 Web2
from Fabric ‘B’ to Fabric ‘A’
App1 App2 App1 App2

Export Web, App, Import Web, App,
DB to Fabric ‘B’ DB from Fabric ‘A’
dB1 dB1 dB2
ACI Multi-Site Solutions Future
Future Deployment Option (CY17)
mBGP - EVPN Translation of VTEP, GBP-ID, VNID
(scoping of name spaces)
Fabric ‘A’ Fabric ‘B’
DB Multiple APIC Clusters

Web/App Web/App
Site to Site VTEP traffic (Class-id is

Leaf to Leaf VTEP, class-id is local to the Fabric Leaf to Leaf VTEP, class-id is local to the Fabric
mapped on Border Leaf)
VTEP Group
VNID Tenant Packet
VTEP Group IP Policy
VNID Tenant Packet VTEP Group
IP Policy VNID Tenant Packet
IP Policy
• VTEP Address is translated, VNID is translated, Group Policy is translated

Agenda

Conclusions
 Cisco ACI offers different multi-fabric
options that can be deployed today
 There is a solid roadmap to evolve those
options in the short and mid term
 Multi-Pod represents the natural evolution
of the existing Stretched Fabric design
 Multi-Site will replace the Dual-Fabric
approach
 Cisco will offer smooth and gradual

migration path to drive the adoption of
those new solutions
Where to Go for More Information
 ACI Stretched Fabric White Paper
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-
aci-stretched-fabric.html#concept_524263C54D8749F2AD248FAEBA7DAD78
 ACI Dual Fabric Design Guide

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-
centric-infrastructure/white-paper-c11-737077.pdf
 ACI and Scalable WAN Integration Whitepaper

http://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/application-centric-infrastructure/white-paper-c11-736899.html
 ACI Multi-Pod Design Guide

Coming soon!
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
• Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

Brkaci 2003

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Brkaci 2003

Încărcat de

Drepturi de autor:

Formate disponibile

Deployment Options for

Interconnecting Multiple ACI Fabrics

Max Ardica – Principal Engineer

At the end of the session, the participants should be able to:

• ACI Multi-Fabric Design Options

 Single Region Multi-Fabric (Classic Active/Active scenario)

 Multi Region Multi-Fabric

Multi-Pod (2.0 release) Multi-Site (Future)

Pod ‘A’ IP Network Pod ‘n’ Site ‘A’ Site ‘n’

• ACI Multi-Fabric Design Options

ACI Stretched Fabric

Transit leaf Transit leaf

Transit leaf Transit leaf

40G DWDM 40G

 DWDM system provides connectivity between two sites

 Transit leafs in all sites connect to the

• ACI Multi-Fabric Design Options

Pod ‘A’ Pod ‘n’

Single APIC Cluster

 Evolution of Stretched Fabric Pod 1 Pod 2

DB APIC Cluster APIC Cluster

3 DC Sites directly connected Multiple sites interconnected by a

 Maximum number of supported ACI leaf nodes (across all Pods)

 Maximum number of supported Pods

 No current plans to increase those values before end of CY16

Pod ‘A’ 40G/100G 40G/100G

 Not managed by APIC, must be pre-configured

 Yes, once QSA adapters will be supported on

I have two sites connected with

dark fiber/DWDM circuits, can I

Do I need a dedicated pair of

 Can use a single pair of IPN devices, but initially

 OSPF mandatory but only between the spines and

One copy is ‘active’ for every

 Processes are active on all nodes (not active/standby)

2 Pods* APIC APIC APIC APIC APIC APIC

3 Pods APIC APIC APIC APIC APIC

4 Pods APIC APIC APIC APIC APIC

5 Pods APIC APIC APIC APIC APIC

6+ Pods APIC APIC APIC APIC APIC

*’ID Recovery’ procedure possible for recovering of lost information

 Common concern with Multi-Pod: the end-to-end propagation of configuration

 Three different zone deployment modes:

DHCP requests are relayed

Discovery and Discovery and

10.1.0.0/16 Pod2-S1, Pod2-S2, Pod2-S3, Pod2-S4

VTEP VXLAN IP Payload

VTEP VTEP VTEP VTEP VTEP VTEP

10.1.3.11 10.1.3.35 fe80::462a:60ff:fef7:8e5e fe80::62c5:47ff:fe0a:5b1a

 MP-BGP EVPN used to communicate

 Single BGP AS across all the Pods Proxy A Proxy B

 BGP EVPN on multiple spines in a Pod COOP

(minimum of two for redundancy) Single APIC Cluster

between local and remote spines (default

Leaf learns remote VM2 location 9 Leaf enforces policy in ingress

BUM frame is flooded along the

 A Pod does not need to have a dedicated

 Traditional L3Out configuration

 MP-BGP EVPN control plane between ACI

*OpFlex support will be added in 6.2.1 release

WAN WAN WAN

*Supported from 2.1 Release (Q4CY16)

 Option to consolidate WAN Edge