BRKDCT 2328

Evolution of Network Overlays in
Data Center Clouds
Victor Moreno, Distinguished Engineer

BRKDCT-2328
Agenda
Overlay Foundational Principles and
evolution
Overlays Evolve to Meet Network
Challenges
The role of the underlay
Management and orchestration
Foundational Principles of Network
Overlays
Why Overlays?
Seek well integrated best in class Overlays and Underlays
Robust Underlay/Fabric Flexible Overlay Virtual Network

High Capacity Resilient Fabric Mobility Track end-point attach at edges
Intelligent Packet Handling Scale Reduce core state
Distribute and partition state to network edge
Programmable & Manageable
Flexibility/Programmability
Reduced number of touch points
BRKDCT-2328 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Seminal Idea: Location and Identity Separation
Traditional Behaviour
Loc/ID Overloaded Semantic
IP core
10.1.0.1 When the Device Moves, It Gets a
New IPv4 or IPv6 Address for Its New
Device IPv4 or IPv6 Identity and Location
Address Represents 20.2.0.9
Identity and Location
Overlay Behaviour
IP core
Loc/ID Split
10.1.0.1 When the Device Moves, Keeps Its
Device IPv4 or IPv6 1.1.1.1 IPv4 or IPv6 Address.
Address Represents 2.2.2.2 It Has the Same Identity
Identity Only. 10.1.0.1
Its Location Is Here!
Only the Location Changes
Overlay Taxonomy
Overlay Control Plane
Service = Virtual Network (VN)

Encapsulation
Identifier = VN Identifier (VNI)
Edge Devices
Edge Device
Hosts
Underlay Network (end-points)
Underlay Control Plane
Overlay Attributes
Service Edge Device Signalling
Layer 2 Service Host Overlays Data Plane Learning
Layer 3 Service Network Overlays Control Plane Learning
Overlay Service Type Evolution
Service
Layer 2 Service
Layer 3 Service
Types of Overlay Service
Layer 2 Overlays Layer 3 Overlays

Emulate a LAN segment Abstract IP based connectivity
Transport Ethernet Frames (IP and non-IP) Transport IP Packets
Single subnet mobility (L2 domain) Full mobility regardless of subnets
Exposure to open L2 flooding Contain network related failures (floods)
Useful in emulating physical topologies Useful in abstracting connectivity and policy
Hybrid L2/L3 Overlays offer the best of both domains

Layer 2 Overlay Considerations
Solved with
Scale of the edge devices
L2 addresses in Ethernet (MACs) use a flat space Layer 3 Overlays
which cannot be summarised
L2/L3 boundary scaling
Layer 3 Overlays
Large L2 domains require a large capacity L3
gateway to handle large ARP and MAC tables at a
frequent rate of refresh
Multi-homing sites can induce loops in the Network Overlays
network
Flooding of L2 protocols, unknown unicasts and MAC routing
broadcast in general can propagate failures
across the entire L2 domain
Multi-homing in L2 Overlays
Source learning assumes single attached sites Loop resolution
But network overlays involve edge resiliency

Core

Enhancements are required to address:
Loop resolution
Multi-pathing Multi-pathing
Broadcast/Multicast de-duplication
Core
Two Approaches:
Active-Standby (Data Plane or Control Plane)
Broadcast/Multicast de-duplication

One active device per VLAN (single attached site)

VLAN based load balancing
Active-Active (Control Plane only) Core
One active device for multi-destination traffic
Intra-VLAN load balancing for unicast
Flooding in L2 Overlays
Control Plane Signalling eliminates the need for floods
Data Plane Learning Control Protocol
DC- DC- DC- DC-

1 2 1 2
L3 L3
L2 L2
B B D A B C D A B C D
A C D A C
Pre-set flood facility No predetermined flood tree

MAC learning based on data plane MAC learning by control protocol
Flood L2 protocols and unknown unicast Contain Failures and L2 protocols
Failure propagation Rich information
Fail Open Fail Closed
Suitable for small domains (failure scope) Better suited for broad scope
Flooded L2 Overlays MAC Routing
L2 Overlay Evolution
From Data Plane Learning to Control Plane Implementations
Inter-DC (DCI) VPLS OTV & EVPN
Backbone Network
L2 L2
Intra-DC (Fabric) Fabric Path EVPN/VXLAN

L2 Overlay Flood/Learn Implementations
PE
L2 L2
MPLS
Switch-IDs NVE-IPs
PE
PE
Fabric Path VXLAN VPLS

IS-IS calculates all possible paths IP calculates all possible paths MPLS Calculates all possible LSPs
between Switch-IDs between NVE-IPs between PEs
IS-IS calculates a multicast IP multicast distribution tree for Pre-determined group of pseudo-
distribution tree for floods floods wires for flooding
MAC in MAC encapsulation MAC in IP encapsulation MAC in MPLS encapsulation
BUM Traffic Flooded over Multicast Tree
L2 Overlay Control Plane Implementations
PE ED
MPLS/IP IP
PE ED
PE ED
PE = Provider Edge ED = Edge Device
EVPN OTV
IP Calculates all possible paths between Edge
MPLS Calculates all possible LSPs between PEs
Devices (Locators)
BGP amongst PEs: advertise locators for each host IS-IS amongst EDs: advertise locators for each host
Segmentation: Route Distinguishers & Labels VLAN Tags segment control and data plane
No Flooding
Integrated Multi-homing
Layer 3 Overlay Considerations
Addressed with
Scale of the edge devices On-demand Pull

Can be improved further by using an on-demand pull
model
Layer 2 Semantics
IP Mobility for subnet disaggregation with IP routing
Members of a subnet may be distributed across
locations
Any host anywhere
Broadcast & Link-local multicast traffic to be Combined L2/L3

overlay
handled as a special case
Potentially without even learning MAC addresses
L3 Overlay Evolution
Edge Device Scale
Push Protocol Model Pull Protocol (on-demand) Model
IP/BGP MPLS VPNs are highly LISP deployments and footprint are
scalable today increasing rapidly
PE routers must: On-demand caching models ease
Hold a large number of prefixes the requirements on the edge
Maintain multiple routing protocol devices:
adjacencies Only prefixes being utilised are cached
No routing adjacencies are maintained
Mobility and cloud will add pressure
in terms of: A pull model is expected to provide
Prefix granularity and volume global scalability to enable
Increased number of PEs pervasive cloud models
L3 Overlay Implementations
1. Underlay Control Plane: IP calculates all possible paths
between Edge Devices (Locators)
Map xTR
System 2. Overlay Control Plane: All mappings registered with
LISP IP
Mapping System by xTRs
(pull) xTR 3. xTRs pull mappings on demand
xTR
1. Underlay Control Plane: MPLS calculates all possible

PE LSPs between PEs or IP Multipath Routing
BGP VPNs MPLS/IP 2. Overlay Control Plane: BGP adjacencies amongst PEs
(push) PE 3. Locators for each host pushed in BGP to all PEs
PE
Distributed Gateway Function in L3 Overlays
L3 Boundary
L2/L3 Fabric
L3 Boundary
App App
App App
OS OS OS
OS
Virtual Physical Virtual Physical
Traditional L2 - centralised L2/L3 boundary L2/L3 fabric (or overlay)

Always bridge, route only at an aggregation point Always route (at the leaves), bridge when necessary
Large amounts of state converge Distribute and disaggregate necessary state
Scale problem for large# of L2 segments Optimal scalability
Traditional L2 and L2 overlays Enhanced forwarding and L3 overlays
IP Mobility with L3 Overlays
Granular location information (host routes)
Allow subnet members to move anywhere
Layer 2 semantics
ARP proxy
Consistent default Gateway presence L3 Fabric
L3 at the Access
Access switch replies to all ARPs with the
same MAC address
Host routing for all traffic within the fabric
Summary prefix outside the fabric
The overlay control plane is enhanced to support mobility (more on this later )
L3 Overlay First Hop Routing
Routing on the Leaf Nodes
A leaf switch is assigned an IP Core
address and a gateway MAC

address for each locally defined
subnet with a connected host Aggregation
IP address of the SVIs

The same anycast IP address is L3
assigned to all leaves supporting Access
L2
attached hosts in the same SVI IP Address
SVI IP Address
subnet MAC: 0000.dead.beef vSwitch
MAC:
0000.dead.beef
vSwitch
IP: 10.1.1.1
IP: 10.1.2.1
The same gateway MAC address H1
10.1.1.10/24
can be used across all subnets H2
H3
10.1.2.10/24
10.1.1.20/24
supported on all the leaves
L3 Overlays ARP and Intra-subnet Forwarding
ARP Handling Core
1. H1 sends an ARP request for H2

10.10.10.20 L1 RIB
10.1.1.20/32 NH L4_IP Aggregation
2. The ARP request is intercepted at the L1 ARP Table
leaf L1 and punted to the Sup L4_IP L4_MAC
Access
3. A few options: 2 CPU
1. If L1 has a valid route to H2, L1 may ARP

reply with its own G_MAC
1
2. If L1 has a MAC-IP binding for H2, L1 may vSwitch
3
ARP-reply on behalf of H2 with H2s MAC
3. L1 may unicast the ARP request to the leaf
H1
where H2 is attached 10.1.1.10 H2
10.1.1.20
4. L1 may simply flood the ARP request
H1 ARP Cache
10.1.1.20 G_MAC
L3 Overlays ARP and Intra-subnet Forwarding
IP Forwarding within the Same Subnet
DSID L4
Core
SSID L1
Enhanced Forwarding: DMAC L4_MAC
6
If H1 generates a data packet SMAC L1_MAC
destined to G_MAC, then a MAC re- DIP 10.1.1.20
SIP 10.1.1.10 Aggregation

write, TTL decrement and host IP L4 RIB
5
forwarding takes place L1 RIB 10.1.1.20/32 e1/1
10.1.1.20/32 NH L4_IP
Traditional Forwarding: L1 ARP Table
If H1 generates a data packet L4_IP L4_MAC
e1/1
destined to H2_MAC, then overlay DMAC H2_MAC

7
forwarding can be done without TTL DMAC G_MAC

SMAC G_MAC
decrement based on either H2_MAC SMAC H1_MAC

DIP 10.1.1.20
vSwitch
4 DIP 10.1.1.20
or H2_IP depending on the overlay H1
SIP 10.1.1.10
SIP 10.1.1.10
implementation. 10.1.1.10
H1 ARP Cache H2
10.1.1.20
10.1.1.20 G_MAC
Combined L2/L3 Overlays
Enhanced Forwarding Mode:
Route all IP traffic including Intra-subnet
Bridge only:
Non-IP / Broadcast / Link-local multicast
L2/L3
Assumption is that most traffic is IP Fabric
Traditional Forwarding Mode:

Route inter-subnet traffic
Bridge intra-subnet and non-IP traffic
Combined L2/L3 Overlay Service Implementations

Programmable between NVE-IPs (Locators)
Fabric with
VXLAN EVPN L2/L3 2. L2+L3: MP-BGP advertisement of host locations
NVE-IPs 3. Route inter-subnet, bridge intra-subnet

Application between NVE-IPs (Locators)
Centric L2/L3
Infrastructure 2. Overlay Control Plane: Demand protocol
NVE-IPs
1. Register both IP and MACs for every host
2. Leaf nodes pull IP and/or MAC mappings on
demand
3. Forward on L3 information unless data is non-IP
Overlay Edge Device and Data Plane Evolution
Service Edge Device

Network DB
Layer 2 Service Host Overlays

App App
OS
Layer 3 Service Network Overlays OS
Virtual Physical
Overlay Network Evolution: Edge Devices
Network Overlays Host Overlays Hybrid Overlays
Protocols Flooding Network DB
V V
V V M A A
M M p p
M O O p
O p
O S S O O
S S
S S
Physical Physical Virtual Virtual Virtual Physical
Router/switch end-points Virtual end-points only Physical and Virtual

Protocols for Single admin domain Resiliency + Scale
resiliency/loops x-organisations/federation
VXLAN, NVGRE, STT
Traditional VPNs Open Standards
OTV, VPLS, LISP, FP
Tunnel End-points
Host OverlaysIP1
Elastic creation of virtual Segments
GWY
Mobile: Can be instantiated anywhere
segmentIP1
1
Move along with VMs as necessary
V GWY
web M
segment 2 segment 21 Very large number of segments
V
Do not consume resources in the network
V
app M web VSG M core
segment 22
segment
V 3
Isolated, not reachable from the IP network
VSG
db V
M
app M Front-end segment must be handled by the
fabric
segment 23
db V
M Host overlays are initiated at the hypervisor
virtual switch Virtual hosts only
Multi-tier Virtual App = VMs + vSegments + GWY GWY to connect to the non-virtualised world
Application: Cloud Services Variants: VXLAN, NVGRE, STT
IP Backbone
Hybrid Overlays
Hypervisors introduce an additional tier in the Core
network: The virtual Access (virtual Switch)
VMs connect to the virtual Access
Host overlays start at the virtual Access Aggregation
Virtualisation based resiliency: Single attached
sites
Access
Physical hosts connect to the physical
Access
Network overlays start at the physical Access Virtual Access
Network resiliency: Site multi-homing
VM VM
A hybrid overlay allows the combination of Hosts
OS OS
physical and virtual resources
Virtual Physical
Which Encapsulation?
NSH
VXLAN NVGRE
LISP MPLS
Geneve
GPE FabricPath
The Multi-protocol Router
TCP/IP SPX/IPX
AppleTalk Token Ring
ATM DECNet
The Multi-encapsulation Gateway
Multi-encapsulation Gateway:
NVGREGREEN
VXLANBLUE
MPLSGREEN
VXLAN, NVGRE, MPLS, LISP,
VLAN, OTV, Geneve, etc.
Bridging (L2 Gateway) Destination is in another segment.
Encap Packet is routed to the new segment
Router
Routing (L3 Gateway) VXLANORANGE
SVI
NVGREORANGE
Encap
Bridge VLANORANGE
Multiple TEPs in independent VRFs

Nesting of IP overlays into MPLS
VPNs
Available across the product line
Normalization: The Encapsulation Doesnt Matter
Intelligence in the Control Plane
VXLAN/FabricPath
Capabilities Exchange in Control
Plane (negotiate encapsulation)
NVGRE VXLAN
Normalize to common encapsulation
Pervasive Multi-encap Gateways for VM VM VM VM
optimal traffic patterns OS OS Physical OS OS
Data Plane and Control Plane Normalization
Federated
Network DB
VXLAN/FP
Private Network
DB
IP Network
DC-west DC-east
POD POD POD POD

NVGRE VXLAN
App App App
OS OS OS
V
M
V
M
V
M
V
M Normalized encapsulation
O O O O
S S S S
VXLAN encapsulation (MPLS or VXLAN) NVGRE (or other) Encapsulation

Normalization GWY Normalization GWY
Multi-encapsulation Hardware Gateways

Normalize to a common encapsulation in the Fabric and/or between Data Centers
Terminate and map multiple types of encapsulation
VXLAN, NVGRE, MPLS, OTV, LISP
Terminate and re-distribute information between overlay control protocols
Controllers, BGP, LISP
VXLAN, LISP and OTV A common encapsulation
LISP
OTV
VXLAN
LISP/VXLAN/OT LISP Original IP Packet
FCS
Outer MAC Header Outer IP Header Outer UDP Header
V Header VXLAN/OTV Original Ethernet Frame
Header Evolution: Metadata and Overlay Headers
Segmentation (VRFs, VPNs, Instances, Segments)
L2 and L3 Payloads
Policy (End-Point-Groups, Scalable Group Tags)
Service Chaining (Network Services Header)
Underlay integration (load balancing, traffic engineering)
OAM Information
VXLAN and GBP extensions
Ethernet in IP with a shim for scalable segmentation and policy metadata
VXLAN-GBP
VXLAN
FCS
Outer MAC Header Outer IP Header Outer UDP Header VXLAN Header Original Layer 2 Frame
GBP = Group Based Policy
LISP, OTV and VXLAN Normalization with Generic
Protocol Extension (gpe)
draft-ietf-nvo3-vxlan-gpe
Ethernet or IP Payload: Defined in the Protocol Type

Common encapsulation for LISP and VXLAN
L2 and L3 Payloads in both LISP and VXLAN
LISP, OTV and VXLAN GPE Plus Network Service
Header
draft-ietf-sfc-nsh
Protocol Type =
0xNSH
Protocol Type =
IP
Overlay Signalling Evolution
Service Edge Device Signalling
Layer 2 Service Host Overlays Data Plane Learning
Layer 3 Service Network Overlays Control Plane Learning
Overlay Signalling
Service Discovery
Edge devices in an overlay need to
discover each other
Address Advertising and Tunnel
Mapping
Edge devices must exchange host
reachability information
Map end-point to location
Tunnel Management
Overlay
Maintain and manage connections Control
between edge devices Data Plane Signalling
Types Plane
Overlay Signalling
Data Plane Learning
Based on gleaning information from data plane events
Example: Source Learning on bridges
Provides the following:
Address advertisement/mapping (very effectively)
Some tunnel management is possible
Does not provide Service Auto-discovery
Requires a flood facility for data plane events to propagate:
Multicast tree
Unicast replication group at the head-end
Flood facility can be manually configured on every device (e.g. join a mcast group or configure a
list of unicast destinations)
Usually is supplemented with a control protocol for Service Discovery (specially if using unicast
replication)
Overlay Signalling
Control Plane
Provides:
Service Discovery
Address Advertising/Mapping
Tunnel Management
Extensions for multi-homing and advanced services can be provided
Protocol or Controller: Push or Pull:
Routing Protocol amongst Edge Devices Push all information to all Edge
BGP, IS-IS, LISP Devices
Central database on a Controller BGP, IS-IS, Controllers
Distributed Virtual Switches (OVS, N1Kv/VSM) Pull and cache on demand @ ED
LISP, DNS, Controllers
BGP EVPN Control Plane for VXLAN
Host and Subnet Route Distribution
Route-Reflectors deployed for scaling purposes
RR RR
iBGP Adjacencies
V V V V V
Host Route Distribution decoupled from the Underlay protocol

Use MP-BGP on the leaf nodes to distribute internal host/subnet
routes and external reachability information
BGP EVPN Control Plane MAC IP VNI Next- Encap Seq
Host Advertisement Hop
1 1 5000 IP L1 VXLAN 0
MAC
L1
NLRI:
Host MAC1, IP1
NVE IP L1/MAC L1 RR RR
VNI 5000
Ext.Community:
Encapsulation: VXLAN, NVGRE
Sequence 0
V V V V V
VNI 5000
Host 1
VLAN 10
1. Host Attaches
2. Attachment NVE advertises hosts MAC (+IP) through BGP RR
3. Choice of encapsulation is also advertised
BGP EVPN Control Plane MAC IP VNI Next- Encap Seq
Hop
Host Moves 1 1 5000 L3
IP L1 VXLAN 01
MAC L1
L3
NLRI:
Host MAC1, IP1
NVE IP L3/MAC L3 RR RR
VNI 5000
Ext.Community:
Encapsulation: VXLAN, NVGRE
Sequence 1
V V V V V
VNI 5000
Host 1
VLAN 10
1. Host Moves to NVE3

2. NVE3 detects Host1 and advertises H1 with seq#1
3. NVE1 sees more recent route and withdraws its advertisement
LISP Control Plane
Host and Subnet Route Registration
All hosts and subnets registered with the Mapping Servers
MS/ MS/
MR MR
V V V V V
Host Route Registration decoupled from the Underlay protocol

Use LISP on the leaf nodes to resolve internal host/subnet routes
and external reachability information
LISP Pull based control plane
LISP :: Mapping Resolution Level of Indirection
LISP Level of Indirection is analogous to a DNS lookup

DNS resolves IP addresses for URL Answering the WHO IS question
[ who is lisp.cisco.com ] ?
DNS
DNS
host Server Name-to-IP
URL Resolution
[153.16.5.29, 2610:D0:110C:1::3 ]
LISP resolves locators for queried identities Answering the WHERE IS question
[ where is 2610:D0:110C:1::3 ] ? LISP

LISP Identity-to-locator
LISP
Mapping Mapping Resolution
router
System
[ locator is 128.107.81.169, 128.107.81.170 ]
LISP Control Plane MAC IP VNI Next- Encap Seq
Host Registration Hop
1 1 5000 IP L1 VXLAN 0
MAC
L1
Map Register
Host IP1 - Parameter: MAC1
NVE IP L1/MAC L1 MS MS
L3VNI 5000, L2VNI 500
V V V V V
VNI 5000
Host 1
VLAN 10
1. Host Attaches
2. Attachment NVE registers hosts IP (+MAC) in LISP
LISP Control Plane IP MAC VNI Locator
Host Resolution 1 1 5000 IP L1

MAC L1
Map Request:
Where is host IP1?
MS/ MS/
MR MR
Map Cache:
IP1 (MAC1) @ NVE1
V V V V V
VNI 5000
Map Response:
IP1 is @ NVE 1
Host 1
VLAN 10
1. Host 2 wants to talk to host 1, the NVE issues a map-request

2. The Map Resolver responds
3. The response is cached at the requesting NVE
LISP Control Plane
MAC IP VNI Next-
Host Moves Hop
1 1 5000 L3
IP L1
L3
MAC L1
Map Register
MS/ MS/Host IP1 - Parameter: MAC1
MR MR
NVE IP L3 / MAC L3
Map-Notify L3VNI 5000, L2VNI 500
V V V V V
VNI 5000
Host 1
VLAN 10
1. Host Moves to NVE3

2. NVE3 detects Host1 and registers H1
3. NVE1 is notified of the move so it can instruct old cachers to refresh
LISP Control Plane IP MAC VNI Locator
Updating the Map Cache 1 1 5000 IP L1

MAC L1
Map Request:
Solicit Map Request: Where is host IP1?
MS/ MS/
For IP1 MR MR
Map Cache:
IP1 (MAC1) @ NVE 1
3
V V V V V
VNI 5000
Map Response:
IP1 is @ NVE 3
Host 1
VLAN 10
1. Host 2 wants to talk to host 1, the NVE issues a map-request

2. The Map Resolver responds
3. The response is cached at the requesting NVE
Control and Management Planes
Controller Controller
Provisioning
Overlay CP Provisioning
Overlay CP
OpFlex, OVSDB.
OpenFlow OVSDB BGP/LISP NefConf-YANG
Centralised - Database Distributed Network Protocol

Tight integration with Loose integration with
provisioning/management provisioning/management
Limited scale
Global Scale
Overlay Reference Architecture
Orchestration Orchestration
Policy Policy
Overlay & Underlay

Provision & Manage SW Overlay Control Management
BGP-EVPN
Overlay CP Overlay CP Provision Control Plane
Switch Switch Data Plane

Surrogate Forwarders
Autonomous Forwarders
Overlays with Virtual Topology System
Orchestration Orchestration
Policy Policy
Underlay Management
VTS Management
Provision
BGP-EVPN Overlay Control Plane

Overlay CP (RR) BGP VTC CP
Switch Switch VXLAN

Data Plane
Hardware Forwarders Virtual Network Forwarders (VNF)
Overlays Evolve to Meet Network
Challenges
DC-Fabric: Integrated Physical + Virtual overlays
Physical + Virtual:
Hybrid overlay
Overlay Normalization
VXLAN
VXLAN/FP fabrics support a mix of
software and HW end-points on a
hybrid overlay: No gateways
VXLAN VXLAN
ACI Fabrics can Normalize host NVGRE VXLAN
overlay encapsulation:
Terminate the encapsulation from the host
overlay
VM VM VM VM
Translate to a normalized encapsulation
in the fabric
OS OS Physical OS OS
Virtual
Segmentation & Policy End-to-end
Branch/
Campus
Segmentation at many levels
Must be given continuity
WAN WAN/DCI Across the different network places
Segmentation
Across organizations and administrative
boundaries
All relevant technologies include the
Fabric DC Fabric
required segmentation & policy semantics
Segmentation
Tenant Segmentation The network maps the segments and
Front-end segment Front-end segment
A
p
A
policy groups together to provide a
scalable and interoperable e2e solution
p
p p
web O
S
web O
S
segment 2 segment 2
Application
A A
p p
p
app
p
O
S app O
S
Segmentation segment 3 segment 3

db db
Failure Domain Scope
vApp Intra-DC Inter-DC Intra-DC vApp
Network Network Network Network Network
Services Services Services Services Services
Front-end segment Front-end segment
Front-end segment Front-end segmentA
A
A
WAN/DCI
A p p
O p O
p
p p O web
O
p web S
web p S
web S S segment 2
A
segment 2A
p
segment 2
DC Fabric DC Fabric A
segment 2A
p
O p O
p
O app p O app p
S
app p
S
S app p
S
segment 3 segment 3
segment 3 segment 3
db db
db db
Segmentation Service
IP + Mobility Service
Layer 2 Service
Core Principles of Network Resiliency/Scale applied to Overlay Services

Clearly delineated Fault Boundaries and service domains
Control Plane Hierarchy and Federation within and across domains
Data Plane Boundaries
Administrative Domain Delineation and Federation
Data Center Interconnect & WAN Integration
Branch/Cl
oset
North-South
(Client-Server)
WAN / Campus
DCI
East-West
(Server-Server)
Interconnecting Multiple Data Centers
LAN Extensions and IP mobility
Ethernet extensions between independent fabrics
IP traffic is forwarded via the optimal path (no hair-pinning)
L3 Domain
IP Mobility (LISP) N7K/ASR

N7K/ASR
VXLAN L2/L3
Gateway
LAN Extensions (OTV/EVPN)
VXLAN L2/L3
Gateway
VXLAN L2/L3
Gateway
VXLAN L2/L3
Gateway
untagged
VxLAN
VLAN
Data Center 1 Data Center 2
L3 Fabric L3 Fabric
VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3
Gateway Gateway Gateway Gateway Gateway Gateway Gateway Gateway
VLAN 30 VLAN 20 VLAN 30 VLAN 20

VNI 5000 VNI 5000
LAN Extensions for DCI
VXLAN
Domain Boundary:

Failure and Event Containment
Clear Administrative Delineation
VXLAN OTV/EVPN
L3 Domain VXLAN
N7K/ASR N7K/ASR
VXLAN L2/L3 VXLAN L2/L3 VXLAN L2/L3 VXLAN L2/L3
Gateway Gateway Gateway Gateway
DC1 DC2
L3 Fabric L3 Fabric
VLAN 30 VLAN 20 VNI 5000 VLAN 30 VLAN 20

VNI 5000
LISP IP Mobility for Optimized Routing
LISP Mobility: LISP Signalling:
Relay mobility state between sites
LISP registrations and notifications
LISP encapsulation from client sites
No host routing in the IP core
LISP Map System
Direct Path Forwarding L3 Domain
Without Host Routing LISP Signalling
N7K/ASR N7K/ASR
VXLAN L2/L3 VXLAN L2/L3 VXLAN L2/L3 VXLAN L2/L3
Gateway Gateway Gateway Gateway
LISP Host Mobility
Host routes Host routes

DC1 DC2
L3 Fabric L3 Fabric
Fabric Mobile Host

Detection
VNI 5000 Moving Hosts
Over the Top Multicast
Underlay is multicast free
V1 Overlay Edge
VA Re-encapsulating router
S Multicast source
VA VB VC VD L Multicast listener
V1 V2 V3 V4 V5
S L L
Overlay Control Plane creates multicast replication lists
Head-end Replication
Optimized Replication: intermediate replication and re-encapsulation points
Mobility of multicast sources is possible with Pull Control Planes
Traffic Engineering and Service Insertion
Controller path provisioning + metadata assisted forwarding
V1 Overlay Edge & Classify
V3 Re-encapsulating router
S Source D Destination
VA VB VC VD Svc Service Node
V1 V2 V3 V4 V5
S Svc D
The controller distributes the desired Service or TE path to the overlay edges
The Engineered path is expressed in terms of Path ID and Path Index
Forwarding tables are populated accordingly
Path ID and Path Index are encoded in the data plane NSH header
Role of the Underlay
Underlying Fabrics
How The Fabric Forwards Traffic
Types of Network Fabric
IP Network
Leverage traditional routing protocols
Manage point-to-point links
Realise multi-pathed fabric
Fabric Characteristics Standards based
High Capacity (10/40/100 GE)
Unified Fabric Network
Line-rate and Low Latency Simplified provisioning and management of
Multi-pathed and Resilient (16 way ECMP) multi-pathed fabric
Multicast, Load Balancing and multi-topology
Simplified/manageable (single touch provision) optimisations
Programmable (1PK, Scripting: Python, POAP) Supports multiple types of traffic: IP, Ethernet,
FCoE
Overlay aware (inspect encapsulated traffic)
Fabric Relevance to a Hybrid Overlay
ECMP
Multicast Services IP Mobility Services

Fast Re-route L2 Services
L2/L3 Overlay aware instrumentation
Fabric
Distributed Overlay HW GWYs
Routing & TEPs
Services
VM VM Site Demarcation
OS OS DCI
WAN Integration
Physical Virtual
Encapsulation and Effective Throughput
encap
10GE 40GE
1500bytes/packet (10Gbps) 1542 bytes/packet (10.1 Gbps)
64bytes/packet (10Gbps) 106 bytes/packet (10.3 Gbps)
Encapsulation adds bits to the traffic being sent

When receiving traffic at full line rate, the encapsulated traffic will exceed the line-
rate BW of the egress interface
Packet drops
Diminished effective throughput
The uplink BW should be greater than the downlink BW to avoid congestion by
encapsulation
This is naturally done in the network
MTU Issues: Overlay PMTUD
NV-edge Flow of Traffic NV-edge
MTU
MTU backpressure
backpressure
Encapsulated traffic may exceed max MTU of the path

When traffic is encapsulated with the Dont Fragment (DF) bit set:
If MTU is exceeded: IGMP unreachable message (datagram-too-big) is sent back to the
encapsulating NV-edge
Encapsulating NV-edge will lower the tunnel MTU accordingly
Subsequent packets from the source will trigger an ICMP unreachable message from the
NV-edge back to the server (if the traffic from the source has the DF bit set)
If the DF bit is not set, the device sensing the MTU is exceeded should attempt to
fragment the traffic
Multi-pathing and Entropy
NV-edge NV-edge
Tunnel Polarisation: All encapsulated flows tend to look like a single flow between a pair of edge
devices
Encapsulated traffic always hashes to a single path
Adding entropy to the encapsulation header can depolarise the tunnels

Use all available paths
UDP headers: Variable UDP source port

GRE headers: Variable key field
MPLS headers: Variable LSP label
Instrumentation and Overlay Awareness
NV-edge NV-edge
Infrastructure awareness of encapsulated traffic:

Outer/Encapsulation header
Overlay shim header
Internal/Payload header
Payload
Overlay aware Switching & Routing infrastructure:

ACLs, QoS, Netflow
Network Analysis Module (NAM) inspects encapsulated traffic
Data Plane and Control Plane Normalization
Inter-AS
Boundary
Encap A Encap B
Multi-protocol overlay gateway

Terminate and map multiple types of encapsulation
VXLAN, NVGRE, MPLS, OTV, LISP
Terminate and re-distribute information between overlay control protocols
Controllers, BGP, LISP
Management and Orchestration
Data Center Fabric Management
Compute, Storage API

Network Network Management Authority
& Network
Orchestration
Management Cisco Network Controller
DCC/DCI Managed Programmable DCI
Open APIs
Spine Managed Programmable Fabric
Leaf Managed Programmable Hybrid Overlays
Virtual Access Fully integrated Distributed Virtual Switch

VM VM
Hosts OS OS
Virtual Physical
Overlay & Underlay Management
Overlay manager
Provision VXLAN on Virtual and
Physical end-points
NMS/EMS for underlay Underlay Overlay

API
management Manager Manager
PoAP, Topology Discovery and

Inventory, Telemetry, Image
Management, etc.
e.g. DCNM, NFM Physical Virtual
Loosely coupled
API for information exchange
EVPN
Combine Underlay/Overlay management under
single pane of glass VM
OS
Interface with Orchestrators
Orchestrator
Orchestrator events and parameters
exchanged with overlay manager
through orchestrator API
Examples:
Underlay Overlay
OpenStack, Manager
API Manager
UCS director
Physical Virtual
EVPN
VM
OS
Virtual Topology Automation
VLAN, SVI, VRF, BGP
2
Orchestrator brings up a new or Net Mgr
moved host Orchestrator
(VTS)
The event is passed to the

Domain Network Manager 1
3
The Network Manager programs TOR TOR

the right VXLAN profile on the
appropriate access switches
Physical and/or virtual switches
1
VM VM
OS OS
EP to EPG: Contracts + Forwarding
Policy
Policy and Virtual Topology Automation
3
2
Orchestrator brings up a new or moved
Net Mgr
host Orchestrator
(VTS)
Host arrival event is passed to the VLAN, VNID, SVI, VRF, ACL
Network Domain Manager 4

1
Domain Manager queries Policy
Repository TOR TOR
The Domain Manager translates the

policy into concrete network constructs
& programs the appropriate switches
Physical and/or virtual switches
1
VM VM
OS OS
Federated/Normalized Overlays Vision
Inter-DC and Intra-DC LISP/BGP Protocol + Any encapsulation
Virtual and Physical Hosts

Layer 2 and Layer 3
Internet Scale
Federated Network
& Policy DB
IP Network
Private Network & Private Network &
Policy DB Policy DB
DC-west DC-east
POD POD POD POD
App App App App
OS OS OS OS
Normalized encapsulation
VXLAN encapsulation NVGRE (or other) Encapsulation

Normalization GWY Normalization GWY
Q&A
Complete Your Online Session Evaluation
Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 Amazon gift card.
Complete your session surveys
through the Cisco Live mobile
app or from the Session Catalog
on CiscoLive.com/us.
Dont forget: Cisco Live sessions will be available

for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016

11:30 am - 12:30pm, In the Oceanside A room
What to expect from this innovation talk

Insights on market trends and forecasts
Preview of key technologies and capabilities
Innovative demonstrations of the latest and greatest products
Better understanding of how Cisco can help you succeed
Register to attend the session live now or

watch the broadcast on cisco.com
Thank you
Data Center / Virtualization Cisco Education Offerings
Course Description Cisco Certification
Introducing Cisco Data Center Networking (DCICN); Learn basic data center technologies and skills to build a CCNA Data Center
Introducing Cisco Data Center Technologies (DCICT) data center infrastructure.
Implementing Cisco Data Center Unified Fabric (DCUFI); Obtain professional level skills to design, configure, CCNP Data Center
Implementing Cisco Data Center Unified Computing (DCUCI) implement, troubleshoot data center network infrastructure.
Designing Cisco Data Center Unified Computing (DCUDC)
Designing Cisco Data Center Unified Fabric (DCUFD)
Troubleshooting Cisco Data Center Unified Computing
(DCUCT)
Troubleshooting Cisco Data Center Unified Fabric (DCUFT)
Product Training Portfolio: DCNMM, DCAC9K, DCINX9K, Gain hands-on skills using Cisco solutions to configure,
DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K deploy, manage and troubleshoot unified computing, policy-
driven and virtualized data center network infrastructure.
Designing the FlexPod Solution (FPDESIGN); Learn how to design, implement and administer FlexPod Cisco and NetApp Certified
Implementing and Administering the FlexPod Solution solutions FlexPod Specialist
(FPIMPADM)
For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com

BRKDCT 2328

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

BRKDCT 2328

Încărcat de

Drepturi de autor:

Formate disponibile

Evolution of Network Overlays in

Data Center Clouds

Victor Moreno, Distinguished Engineer

Robust Underlay/Fabric Flexible Overlay Virtual Network

Overlay Control Plane

Service = Virtual Network (VN)

Underlay Control Plane

Service Edge Device Signalling

Layer 2 Service Host Overlays Data Plane Learning

Layer 3 Service Network Overlays Control Plane Learning

Layer 2 Overlays Layer 3 Overlays

Hybrid L2/L3 Overlays offer the best of both domains

But network overlays involve edge resiliency

Data Plane Learning Control Protocol

DC- DC- DC- DC-

Pre-set flood facility No predetermined flood tree

Flooded L2 Overlays MAC Routing

Inter-DC (DCI) VPLS OTV & EVPN

Intra-DC (Fabric) Fabric Path EVPN/VXLAN

Fabric Path VXLAN VPLS

MAC in MAC encapsulation MAC in IP encapsulation MAC in MPLS encapsulation

BUM Traffic Flooded over Multicast Tree

Scale of the edge devices On-demand Pull

Broadcast & Link-local multicast traffic to be Combined L2/L3

1. Underlay Control Plane: MPLS calculates all possible

Virtual Physical Virtual Physical

Traditional L2 - centralised L2/L3 boundary L2/L3 fabric (or overlay)

address and a gateway MAC

IP address of the SVIs

1. H1 sends an ARP request for H2

1. If L1 has a valid route to H2, L1 may ARP

destined to G_MAC, then a MAC re- DIP 10.1.1.20

SIP 10.1.1.10 Aggregation

destined to H2_MAC, then overlay DMAC H2_MAC

forwarding can be done without TTL DMAC G_MAC

decrement based on either H2_MAC SMAC H1_MAC

Traditional Forwarding Mode:

1. Underlay Control Plane: IP calculates all possible paths

1. Underlay Control Plane: IP calculates all possible paths

Service Edge Device

Layer 2 Service Host Overlays

Physical Physical Virtual Virtual Virtual Physical

Router/switch end-points Virtual end-points only Physical and Virtual

Application: Cloud Services Variants: VXLAN, NVGRE, STT

AppleTalk Token Ring

Multiple TEPs in independent VRFs

Intelligence in the Control Plane

Pervasive Multi-encap Gateways for VM VM VM VM

optimal traffic patterns OS OS Physical OS OS

POD POD POD POD

VXLAN encapsulation (MPLS or VXLAN) NVGRE (or other) Encapsulation

Multi-encapsulation Hardware Gateways

LISP/VXLAN/OT LISP Original IP Packet

GBP = Group Based Policy

Ethernet or IP Payload: Defined in the Protocol Type

Service Edge Device Signalling

Layer 2 Service Host Overlays Data Plane Learning

Layer 3 Service Network Overlays Control Plane Learning

Host Route Distribution decoupled from the Underlay protocol

1. Host Moves to NVE3

Host Route Registration decoupled from the Underlay protocol

LISP Level of Indirection is analogous to a DNS lookup

[ where is 2610:D0:110C:1::3 ] ? LISP