Enhancing IP&MPLS Based Carrier Services To Address The Data Center Interconnect Market - BRKSPG-2024

Unified MPLS: Advanced Scaling for Core
and Edge Networks

BRKSPG-2405
Rajiv Asati
Distinguished Engineer
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 1
Abstract
Service Providers (SPs) are striving towards becoming 'Experience Providers' while offering many residential
and/or commercial services.
Many SPs have to build an agile Next Gen Networks (NGN) that can optimally deliver the 'Any Play' promise.
However, as the Networks continue to get are getting bigger, fatter and richer, some of the conventional wisdom
of designing IP/MPLS networks is no longer sufficient.
This session introduces a 'Cisco Validated Design' for building Next-Gen Networks' Core and Edge. It briefly
discusses the technologies integral to such a design and focus on their implementation using IOS-XR platforms
(CRS-1/3 and ASR 9000). The session looks at the scaling designs and properties of IP, MPLS, the IGP and BGP
as well as the protection mechanisms IP/LDP FRR and MPLS-TE FRR.
This session is intended to cover -

- Unicast routing + MPLS design
- Fast Restoration
- Topology Dependency
- Test Results
- Case Study
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
Introduction
Solution Overview
Unicast Routing + MPLS Design
Fast Restoration
Topology Dependency
Test Results
Case Study
Conclusion
Agenda
Introduction
Solution Overview
Fast Restoration
Topology Dependency
Results
Case Study
Conclusion
BRKSPG-2405 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Introduction
Trend
Networks becoming larger
Quad-play (Video, Voice, Data & Mobility)
Merger & Acquisition
Growth
Exponential bandwidth consumption
Business Services
Mobile
MPLS in the Access
Seamless MPLS
MPLS-TP
BGP ASN consolidation
Single ASN offering to customers
Introduction
NGN Requirements
Large Network
2000+ routers, say
Multi-Play Services Anywhere in network
Service Instantiation happens anywhere
End-to-End Visibility
v4/v6 Uni/Multicast based Services
Fast Convergence or Restoration
Closer to Zero loss, the better.
Scale & Performance
Agenda
Introduction
Solution Overview
Fast Restoration
Topology Dependency
Results
Case Study
Conclusion
Solution Overview
Unicast Routing + MPLS - Divide & Conquer

1. Isolate IGP domains
2. Connect IGP domains using BGP
Fast Restoration Leverage FRR
1. IP FRR (IGP LFA & BGP PIC)
2. MPLS FRR (LDP FRR & TE FRR)
Topological Consideration Choose it right
1. PoP Design
2. ECMP vs. Link-Bundling
Services Scale
Agenda
Introduction
Solution Overview
Fast Restoration
Topology Dependency
Results
Case Study
Conclusion
Routing + MPLS Design
Must Provide.
PE-to-PE Routes (and Label Switched Paths)
PE needs /32 routes to other PEs
PE placement shouldnt matter
Single BGP ASN
LSP
Backbone
Access Region1
Aggregation Region 2 Aggregation Access
.
.
.
.
PE21
PE21
PE11 PE31
Conventional Wisdom Says
Advertise infrastructure (e.g. PE) routes in IGP
Advertise infrastructure (e.g. PE) labels in LDP
Segment IGP domains (i.e. ISIS L1/L2 or OSPF Areas)
BGP for Services

End-to-End IGP & LDP for Infrastructure
Access Aggregation Region1 Backbone Region 2 Aggregation Access

.
.
.
ISIS Level 1 ISIS Level 2 ISIS Level 1

Or Or Or
R
OSPF Area Y OSPF Area 0 OSPF Area X
.
PE21
PE21
PE11 PE31
Conventional Wisdom Not Good
Enough
Large IGP database size a concern
For fast(er) convergence
Large IGP domain a concern
For Network Stability.
Large LDP database a concern
Divide & Conquer Game Plan
Disconnect & Isolate IGP domains
No more end-to-end IGP view
Leverage BGP for infrastructure (i.e. PE) routes
Also for infrastructure (i.e. PE) labels
BGP for Services
BGP for Infrastructure
Isolated IGP & LDP Isolated IGP & LDP Isolated IGP & LDP

.
.
.

Or Or Or
R
.
PE21
PE21
PE11 PE31
Divide & Conquer End Result
Example - PE31 Reachability
Control Plane Flow RIB/FIB Table View
Data Plane Flow PE11 to PE31 Traffic View
PE31 :: PE31 ::
Next-Hop = P1; BGP; Next-Hop = P2; BGP PE31 ::
Label = L100; BGP Label = L101; BGP Next-Hop = P31; IGP
P1 :: P2:: Label = L110; LDP
Next-Hop = P11; IGP Next-Hop = P100; IGP
Label = L200; LDP Label = L201; LDP
.
.
.

P2
Or P1 Or Or
R
OSPF Area Y OSPF
P100
Area 0 OSPF Area X
.
P11 PE21
PE21 L110 IP
PE11
L201 L101 IP PE31
IP
L200 L100 IP
Divide & Conquer Summary
1. IGP is restricted to carry only internal routes
Non-zero or L1 area carries only routes for that area
Backbone carries only backbone routes *
2. PE redistributes its loopback into IGP as well as iBGP+Label
3. PE peers with its local ABRs using iBGP
ABRs act as Route-reflectors
ABRs reflect _only_ Infrastructure (i.e. PE) routes
4. ABR, as RR, changes the BGP Next-hop to itself
On every BGP advertised routes
5. PEs separately peer for Services (VPN, say)
* ISIS L1->L2 (or L1->L1) Redistribution

Cannot Be Avoided Yet, but OSPF Non-
Presentation_ID Zero<->Zero
2012 Cisco and/or its affiliates. All rights reserved. Area Redistribution Can Be. Cisco Public
Divide & Conquer
1.
1 IGP is restricted to carry only the internal routes
Non-zero or L1 area carries only routes for that area
Backbone carries only backbone routes *
* Unlike OSPF, ISIS Backbone Would Carry Both
L1 and L2 Routes Since L1->L2 (or L1->L1)
Redistribution Cannot Be Avoided Yet
Isolated IGP Isolated IGP Isolated IGP

.
.
.

Or Or Or
R
.
ABR ABR PE21
PE21
PE11 PE31
Divide & Conquer
1.
2 PE redistributes its loopback into IGP as well as
iBGP+Label
Loopback Int Redistributed

into IGP and BGP+Label

.
.
.

Or Or Or
R
.
ABR ABR PE21
PE21
PE11 PE31
Divide & Conquer
1.
3 PE peers with its local ABRs using iBGP+label
ABRs act as Route-reflectors
ABRs reflect _only_ Infrastructure (i.e. PE) routes
RRs also in the backbone
iBGP+Label
Peering
.
.
.

Or Or Or
R
.
ABR ABR PE21
PE21
PE11 PE31
Divide & Conquer
1.
4 ABR, as RR, changes the BGP Next-hop to itself
On each BGP advertised routes
BGP Prefix PE31: ABR Sets BGP NH to Itself ABR Sets BGP NH to Itself
Next-Hop = P1; Label=L100 BGP Prefix PE31:
Next-Hop = PE31; Label=Null
.
BGP Prefix PE31:

.
Next-Hop = P2; Label=L101
.

Or P1 Or P2
Or
R
.
ABR ABR PE21
PE21
PE11 PE31
Divide & Conquer
1.
5 PEs separately peer using iBGP for Services
Dedicated RRs for IPv4/6, VPNv4/6, L2VPN, etc.
More Details on BGP Scale for Services Later
Access Aggregation RRs Region1 Backbone Region 2 Aggregation Access

. RRs RRs
.
.
ISIS Level 2
. Or .
R
OSPF Area 0
.
ABR ABR PE21
PE21
PE11 PE31
Divide & Conquer End Result
Example - L3VPN Services
PE11 sends L3VPN traffic for an L3VPN prefix A to PE31
L3VPN A
Next-Hop = PE31; BGP
Label = L30 ; BGP
PE31 :: PE31 ::
next-hop ==P1;
Next-Hop P1;BGP;
BGP; Next-Hop = P2; BGP PE31 :: L3VPN A::
label ==L100;
Label L100;BGP
BGP Label = L101; BGP Next-Hop = P31; IGP next-Hop = CE31; IGP
P1 :: P2:: Label = L110; LDP Label = Unlabel
Next-hop == P11;
Next-Hop P11; IGP
IGP Next-Hop = P100; IGP
label ==L200;
Label L200;LDP
LDP Label = L201; LDP
.
.
.

Or P1 Or P2
Or
OSPF Area Y OSPF
P100
Area 0 P31 OSPF Area X R
L100 L30 . IP
P11
PE21 L201 L101 L30 IP L110 L30 IP
PE11 PE31
L30 IP
IP L200 L100 L30 IP
Take-Away
Higher Network scale is attainable

1000s of routers
BGP and MPLS Label Stacking are key
Agenda
Introduction
Solution Overview
Fast Restoration
Topology Dependency
Results
Case Study
Conclusion
Fast Restoration
Business Services demanding faster restoration

Against link or node failures
Service Differentiator for many operators
Faster Restoration is driving towards 0 loss
~50ms restoration may be good enough for many
Requirements influence Complexity and Cost
Fast Restoration is optimal with Local Protection
pre-compute and pre-install alternate path
no need for remote nodes to know about the failure
Fast Restoration
Fast Restoration of Services i.e. BGP Prefixes

BGP Prefix Independent Convergence (PIC)
Fast Restoration of BGP next-hops i.e. IGP Prefixes

IP FRR (LFA) with LDP FRR (or RSVP-TE FRR)
Fast Convergence (FC) of IP routing protocols is key and still required
Fast Restoration vs. Fast
Convergence
Detection
(link or node aliveness, routing updates
received) State
Walkthroug
propagation
h routing
Compute (routing
DBs
primary updates send)
path &
label
Download
to HW FIB
Switch to
newer path
Fast Restoration vs. Fast
Convergence
Offline Calculation Detection
Pre-
Compute (link or node aliveness, routing updates
Repair received) State
path Switch to Walkthroug
Download propagation
Repair h routing
to HW FIB Compute (routing
Path DBs
primary updates send)
path &
label
Download
to HW FIB
Switch to
newer path
Edge FC POP FC Core FC
Fast Convergence Edge

FRR
POP
FRR
Core
FRR
IGP Prefixes
Remember that FRR is intended for temporary
restoration
Fast Convergence (FC) is key for IP routing protocols
Faster the routing convergence, faster the permanent
restoration
<1sec restoration is possible
Routing convergence happens at the process level,

hence, depends on the platform processor
Restoration time can not be guaranteed
Fast Convergence Edge

FRR
POP
FRR
Core
FRR
IGP Prefixes
MUST Detect Link/node down event as fast as possible
for
FRR
BFD, Layer2 protocol keep-alives, Alarms, IGP fast hellos, Proactive
Protection
Generate the link state eventLSP/LSA generation is optimized
Propagate the changes in the network as soon as possible
Flooding and passing is optimized
Recalculate the paths (run SPF) as soon as possibleSupport
of incremental SPF and optimized for full SPF
Install the new routes in the routing/forwarding table with Prefix
MUST
for
FC
Prioritization
CRITICAL: IPTV SSM sources
HIGH: Most Important PEs
MEDIUM: All other PEs
LOW: All other prefixes
Fast Convergence Edge POP Core

FRR FRR Reference
FRR
IGP Tuning for FC

OSPF Tuning IS-IS Tuning
OSPF Event Propagation IS-IS hello interval/ Hello Multiplier
timers pacing flood value isis hello-interval { seconds | minimal }
isis hello-multiplier value ------- Valuerange 3
timers pacing retransmission value 20
default values are 33 msec/66 msec IS-IS LSP-Generation Exponential Backoff

OSPF Subsecond Hellos Configuration: lsp-gen-interval lsp-max lsp-start lsp-hold
ipospf dead-interval minimal hello- lsp-max(sec) lsp-hold(msec) lsp-start(msec)
multiplier value IS-IS Event Propagation
Valuerange 320 lsp-interval value
Default rate - one LSP every 33 ms
OSPF LSA Generation Exponential Backoff
Fast LSP Flooding
timers throttle lsa all lsa-start lsa-
hold lsa-max fast-flood lsp-number (Previously ip fast-
convergence)
timers lsa arrival timer
IS-IS SPF Exponential Backoff
OSPF SPF ExponentialBackoff spf-interval spf-max spf-start spf-hold
<spf-max>- (sec) <spf-start> - (msec) <spf-hold> - (msec)
Timers throttle spfspf-start spf-
hold spf-max prc-interval prc-max prc-start prc-hold
<prc-max>- (sec) <prc-start> - (msec) <prc-hold> - (msec)
All LSA/SPF values are in ms
Note: MinLSArrival Must Be <= lsa-Hold
Fast Restoration
IGP Prefixes
MPLS FRR and IP FRR are viable options
Both pre-compute and pre-install alternate path
IP FRR (LFA) is simpler than RSVP-TE based MPLS FRR
Easy to configure and manage
Does not require network-wide support
Has topological dependencies
IP FRR (LFA), with LDP LSP, provides simpler MPLS FRR
Easy to configure and manage
Does not require network-wide support
Removes most of topological dependencies
Use IP FRR & LDP FRR (RSVP-TE FRR only if one have to)
RSVP-TE for bandwidth engineering as usual
FRR Fast Reroute

LFA Loop Free Alternates
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. LSP Label Switched Path Cisco Public
Fast Restoration Edge FC POP FC Core FC
Edge POP Core
IGP Prefixes FRR FRR FRR
IP/LDP FRR: Apply it as an intra PoP and inter PoP FRR

solution
RSVP-TE FRR: Apply it as an inter PoP FRR solution, if IP/LDP
FRR doesnt give enough coverage
Intra PoP Inter PoP

PoP PoP
P
PE PoP
PoP
P
PoP
Fast Restoration Reference
IGP IP FRR
IP FRR (Loop Free Alternates) provides a pre-computed
backup (aka repair path) per destination prefix
IP FRR (LFA) can be deployed in two ways :
Per-Link LFA Protects all the destinations reachable via the
protected link
Per-Prefix LFA Protects a destination against the next-hop
link or node failure
IP FRR (LFA) well applies to most SP topologies
http://tools.ietf.org/html/draft-ietf-rtgwg-lfa-applicability-00
Note: SPF calculations for LFAs are performed in

background and pre-empted in case of any convergence
events
Fast Restoration IGP
IP FRR : Per-Link LFA
Protecting Node Next-hop Node
S F
D
Route D
Primary Next Hop: F
Backup Next Hop: R1 Primary link
Backup link
R1
A backup path for all prefixes reachable via next-hop node

(F) over the protected link (S-F)
1 SPF per protected link
No node protection possible
Sub-optimal forwarding during FRR
Fast Restoration IGP Edge FC POP FC Core FC
Edge POP Core
IP FRR : Per-Link LFA FRR FRR FRR
RP/0/0/CPU0:ospf-3-2(config)#router ospf 1
RP/0/0/CPU0:ospf-3-2(config-ospf)#area 0
RP/0/0/CPU0:ospf-3-2(config-ospf-ar)#int pos 0/3/0/0
RP/0/0/CPU0:ospf-3-2(config-ospf-ar-if)#fast-reroute per-link enable
Route
Route DD
NH:
NH: F
F,
LFA:R1
no 10
LFA: S F
D
10
10
10
Route
Route D
D
NH:
NH:FS
10 10 R3
R1
R2
Availability of the backup NH is dependent on the topology and

link metric assignments
All depends on metric assignment
IP FRR : Per-Prefix LFA
Protecting Node Next-hop Node
S F
D
Route D
Primary Next Hop: F
Backup Next Hop: R1 Backup path1 (link protection)
Backup path2 (node protection)
R1
A backup path for a prefix (e.g. D) reachable via next-hop

node (F)
1 SPF per neighbor
No node protection possible
Sub-optimal forwarding during FRR
(Configuration)
By default, LFA computation is disabled
LFA needs to be enabled only on protecting router
S
!
router isis
fast-reroute per-prefix {level-1 | level-2} {all |
IOS route-map <route-map-name>}
!
router ospf 1
fast-reroute per-prefix enable prefix-priority low
!
IOS-XR
router isis <instance-id>
interface <type> <instance>
address-family ipv4 [unicast]
fast-reroute per-prefix level <1|2>
10.0.0.0/8, NH = D, cost= 10
20.0.0.0/8, NH = D, cost= 7 10.0.0.0/8
2 B 5 6
C
A
1
2
1
4 5
D E
20.0.0.0/
8
F 6
IGP pre-computes a backup path per IGP prefix

FIB pre-installs the backup path in dataplane
10.0.0.0/8, NH = C,
IP FRR : Per-Prefix LFAcost=11
20.0.0.0/8, NH = A, cost=9
10.0.0.0/8, NH = D, cost=
10
20.0.0.0/8, NH = D, cost= 10.0.0.0/8
7
2 B 5 6
C
A
1
2
10.0.0.0/8, NH = A, cost=14 1
20.0.0.0/8, NH = direct, 4 5
D
cost=6 E
20.0.0.0/
8
F 6
10.0.0.0/8, NH = D, cost=
10
20.0.0.0/8, NH = D, cost= 10.0.0.0/8
7
10.0.0.0/8, NH = D, cost=10
LFA: B
20.0.0.0/8, NH = D, cost=7 2 B 5 6
LFA: F
C
A
1
2
1
4 5
D E
20.0.0.0/
8
F 6

LFA with LDP
Protecting
packet labelB
Node
packet labelA A Link
B
P/p
Failure
Primary Path
C
Repair Path
The link between A and B failed.

A sends packets to C instead by swapping labelA with labelC distributed by
C.
LDP requirement: Downstream Unsolicited; Liberal Retention
The backup path for destination P/p must contain the label bound by the
backup neighbor
This is why, whether the IGP computes per-prefix or per-link, the RIB and FIB
representation is always per-prefix
this allows
Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved.
to store the per-path dependent backup label Cisco Public
Remote LFA (aka PQ)
Any node which meets the P Backbone
and Q properties
P: the set of nodes reachable R6 R7
from R2
without traversing [R2-R4]
Q: the set of nodes which can R4 R
5
reach R4
without traversing [R2-R4]
Best PQ node R2 R3
The closest from R2: R5

Establish a directed LDP R1
session with the selected PQ

Access Region
node Presentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Remote LFA (aka PQ)
Backbone
R2s LIB
R4s label for FEC R6 = 408
R1s label for FEC R5 = 103 R6 R7
R5s label for FEC R6 = 502

R2s FIB for destination R6 R4 R
5
Primary: out-label = 408, oif = R4 408

502
Backup: out-label = 502
oif = [push 103, oif = R1] R2 103 R3
R1
Access Region
RSVP-TE FRR Edge

FRR
POP
FRR
Core
FRR
RSVP-TE FRR link protection (and prefix independent): <50ms

Easy to operate with auto-tunnel
RSVP-TE FRR node protection (and prefix independent):
<100ms (depends on time to detect the node failure)
RSVP-TE FRR path protection (and prefix independent):
Time depends of time to signal the path error to the head end
(not a local mechanism)
Challenging to operate (due to due to its end-end / 1:1 protection)
Appropriated to specific scenario
Note: RSVP-TE Provides FRR Mechanism as well as:

Bandwidth Management
Traffic Engineering
Is not Topology Dependent like IP LFA
Edge POP Core
RSVP-TE
interface Tunnel0 FRR Link Protection FRR FRR FRR
tunnel destination Router D

explicit-path R2-R3-R4
notunnel mpls traffic-eng autoroute
announce
Router A Router B Router D Router E

x
interface POS0/0
mpls traffic-eng backup-path
Tunnel0
interface Tunnel0
tunnel destination RouterRouter E C
.. etc ...
tunnel mpls traffic-eng fast-
reroutePresentation_ID 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Edge POP Core
RSVP-TE FRR Node Protection FRR FRR FRR
What if Router D failed?

Link protection would not help as the backup tunnel terminates on Router D
(which is the NHop of the protected link)
Protect tunnel to the next hop PAST the protected link (NNhop)
Protected Link
Router A Router B Router D Router E Router F
NHop NNHop
Fast ReRoute
Backup Tunnel
Router C Fast ReRoute

Backup Tunnel
Fast Restoration
IP FRR (LFA) vs. RSVP-TE FRR
RSVP-TE/MPLS
IP FRR
FRR
Constraints Based
with Bandwidth
1 Repair Path Least Cost
Guarantee and
Path Control
2 SRLG Capable Capable
3 Link Protection Capable Capable
4 Node Protection Capable Capable
5 Path Protection Not Capable Capable
6 Control Plane Requirement None with LFA RSVP-TE
Load Distribution over Multiple
7 Capable Not Capable
Repair Paths
8 Provisioning Complexity Minimal, If Any Significant
9 Topology Dependency Yes No
Edge POP Core

FRR FRR FRR
Fast Restoration
BGP PIC
(Prefix Independent
Convergence)
What Is PIC or BGP FRR?
PIC provides a fast convergence functionality
upon failure to cutover to any backup path
within sub-seconds independent of the number
of prefixes
BGP Fast Reroute (BGP FRR)enables BGP
to use alternate paths within sub-seconds after
a failure of the primary or active paths
PIC or FRR dependent routing protocols (e.g.
BGP) install backup paths
Without backup paths
Convergence is driven from the routing
protocols updating the RIB and FIB one prefix
at a time - Convergence times directly
proportional to the number of affected
prefixes
With backup paths
Paths in RIB/FIB available for immediate use
Predictable and constant convergence time
independent of number of prefixes
PIC Edge vs. PIC Core
1
2
PE2 3
CE1
CE2 PE3
Site2 Site1
PE1
1. PIC Core When IGP Path Changes

2. PIC Edge When Remote PE Node (or Its Reachability)
Fails
3. PIC Edge When PE-CE Link Fails
PIC Core CLI on 7600 - cef table output-chain build favor convergence-speed
BGP PIC Edge Edge FC POP FC Core FC
PE-CE Link Protection

router bgp 100
Edge
FRR
POP
FRR
Core
FRR
address-family ipv4 vrf V1
bgpadvertise-best-external
PE2
MPLS-VPN CE1
PE3
PE1
router bgp 100
CE2 address-family ipv4 vrf V1 Normal Path
bgp additional-paths install Backup Path
PE1 and PE2 pre-compute bgp backup paths using

bgp best-external approach
When primary link PE1 - CE1 fails:
PE1 holds on to the bgp local labels and re-routes CE1s traffic to PE2
using labels advertised by PE2
Uses fixed timer to clean up stale local labels
PE3 is expected to converge and start using PE2s label to send traffic to CE1
BGP PIC Edge Edge FC POP FC Core FC
PE-Node Protection Edge

FRR
POP
FRR
Core
FRR
PE2
MPLS-VPN CE1
PE3
PE1
CE2 Normal Path

Backup Path
PE1, PE2 and PE3 precomputes bgp backup

When node PE1 fails:
IGP notification on PE3 invalidates active path
Switches to backup path
PE3 is expected to converge to start using PE2s label to
send traffic toCE1
Convergence With and Without PIC
BGP PIC Core and PIC Edge
PIC Core PIC Edge

msec
1000000
Core
100000
100000
250k PIC
10000 10000
250k no PIC
PIC 500k PIC
LoC (ms)
1000 1000
no PIC 500k no PIC
100 100
10
10
1
1
12 0
15 0
17 0
20 0
22 0
25 0
27 0
30 0
32 0
35 0
00
0
10 0
1
0
00
00
00
50000
100000
150000
200000
250000
300000
350000
400000
450000
500000
00
50
00
50
00
50
00
50
00
50
00
25
50
75
Prefix
Prefix
Without PIC : Convergence is a function of number of affected

prefixes during failure
With PIC : Convergence is predictable and remains constant
independent of the number of prefixes
Fast Restoration Design
Take-Away
Leverage IP FRR (LFA) with MPLS / LDP
wherever possible
LFA is simpler, local (requires no
interoperability)
Leverage TE FRR, if we must have to.
Bandwidth Engineering, for example.
Leverage BGP PIC for faster BGP convergence
PIC is local (requires no interoperability)
Agenda
Introduction
Solution Overview
Fast Restoration
Topology Consideration
Results
Case Study
Conclusion
Topological Consideration
What topology is chosen makes a big difference

convergence, traffic engineering, capacity planning, routing table, stability..
Topological Options may vary
Flat vs. Hierarchical
Hub & Spoke vs. Ring (Square)
Also, the evergreen question about ECMP vs. LAG
Flat PoP Topology Reference
LFA Applicability
Hierarchical PoP Topology Reference
LFA Applicability
ECMP vs. Link-Bundle
Factors E- ECMP
LinkBun
dle
1 Member Links Speed Must Be Same Yes No
2 Member Links on Any LC Yes Yes
3 Routing Adjacency One Many
4 Routing Table Impact No Yes
5 Max Number of Member Links 64 16 (32*)
6 Line-Rate Multicast (Members on Any LCs) Yes? Yes
7 Port Failure Localized to the Router Yes No
8 BFD on Each Member Link Yes** Yes
9 Video Monitoring Better Accounting No Yes
10 Non-Stop Routing (NSR), Forwarding

Presentation_ID (NSF)
2012 Yes All rights reserved.
Cisco and/or its affiliates. Yes Cisco Public
Take-Away
Triangle topology (i.e. Hub & Spoke) for PE
connectivity is advantageous
Naturally benefits from IP FRR
Linkbundling gaining more traction
Agenda
Introduction
Solution Overview
Fast Restoration
Test Results
Case Study
Conclusion
Results
The solution discussed here is a part of a complete end-to-

end architecture for delivery of residential, business, and
RAN backhaul services
It is thoroughly validated for each service in the areas of:
- Functionality, Scalability, Performance / SLA, QoS, High Availability,
Network Management, OAM
The solution (results) is well documented

Design & Implementation Guide (DIG) available through your
account team
NGN Testbed
PoP A PoP B
Hub & Spoke Aggregation Ring Aggregation Topology
Topology
Internet
Video Headend/DC
SEF Infrastructure
10GE
1GE
PoP C
ASR-9000 7600
Business MSE (Ethernet + TDM)
CRS-1 ASR-1000
MWR-2941 / ISR 3400E / 4948
NGN Testbed Platforms
Role Platform Version

Aggregation ASR-9000 IOS-XR 4.0.1
Node
Core Node CRS-1/3 IOS-XR 4.0.1
Access Node ME-3400E 12.2(55)SE
Access Node 4500, 4948 15.0(2)SG
Access Node MWR-2941-A CSR 3.3
Service Edge ASR-1000, 15.1(1)S
Node Cisco 7600
Presentation_ID
Active Network 3.7.2
2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Results Summary
Test Area Results
1 Topology Scalability PoPs 100 (3 Real + 97 Simulated)

Infrastructure BGP Routes 100K;
Infrastructure ISIS Routes 12K;
2 Service Scalability Residential 120K Triple Play Subscribers;
Business L2VPN 16K E-Line, 4K E-LAN (20K MACs);
Business L3VPN 4K VPNs (1M Routes);
IP RAN/ TDM 4K AToM PWE3;
3 Service (High) Availability Link & Node Failure and Recovery:
<50 msec (Hub & Spoke Topology)
<500 msec (Ring Topology)
* A Few Exceptions During Node Recovery with

Presentation_ID High Scale
Results (Just an Example)
Service Convergence During Link
Failure RAN Backhaul Service
msec
800 750
700
600 500
500
400 RAN Backhaul Service
300 200
200
100 50
0
NNI Failure NNI Failure UNI Failure UNI Failure
(H&S) (ring) (Ethernet) (uWave)
Agenda
Introduction
Solution Overview
Fast Restoration
Results
Case Study
Conclusion
Case Study
SPs are fast embracing Cisco NGN reference
The next two slides illustrate the actual deployed networks -
CSR - Cell Site Router
Case Study #1 SR Service Router
CR Core Router
BR Backbone Router
APAC Mobile Operator / SP
3x40GEs per SR Pair Regional Data Center
SR SR
378Gbps per SR Pair
Backbone
CR CR CR CR
mx40GE mx40GE
Mini-Core
PE PE PE PE
9 Gbps per U-PE nx10GE nx10GE

<10 CSR (Radios) per U-PE Aggregation
U-PE U-PE U-PE U-PE
300 Mbps per CSR (Radio) Star
Access
CSR CSR
CSR CSR
Case Study #2
US Cable Operator / SP
Backbone
Redundant SDC SDC1
May Not Be SDC
Present Distribution
2
7600/ASR9k Aggregation
Service Routers
Tier 2 Hub
7600/ASR9k
Tier 1 Hub Hub Agg
ASR9k
Tier 1 Hub
Legend ASR9k
1 GE Link
Hub Router
10 GE Ring Link
7600/ASR9k
10 GE Point to Point Link
Video EQAM
CMTS
Presentation_ID CPEs
Agenda
Introduction
Solution Overview
Fast Restoration
Results
Case Study
Conclusion
Conclusion
Learned design options for large networks

How to scale Routing (+MPLS) !
What Fast Restoration technique is suitable! Where!
Which Topology makes sense !
Services Consideration !
Got the proof points

Deployed case studies
Additional Slides
LFA Roadmap IPv4
MPLS TE-FRR 7600 ASR1000 ASR9k CRS-1

1-hop link (IOS) (IOS-XE) (IOS-XR) (IOS-XR)
Per Link LFA FRR Not Available Not Available 4.0.1 3.5.0
OSPF LFA FRR (per

15.1(3)S 3.4S 4.2.0 4.2.0
prefix)
ISIS LFA FRR (per
15.1(2)S 3.4S 4.0.1 4.0.1
prefix)
EIGRP FRR (per prefix) 15.2(4)S* 3.7S*
OSPF Remote LFA 15.2(2)S 3.6S 4.3.1* 4.3.1*
ISIS Remote LFA 15.2(2)S 3.6S 4.3.1* 4.3.1*
BGP PIC Core for

12.2(33)SRC 2.5S 3.7.0 3.4
IP/MPLS
BGP PIC Edge 12.2(33)SRE 2.5S 4.0.0 4.0.0

*Future
LFA Roadmap IPv6
MPLS TE-FRR 7600 ASR1000 ASR9k CRS-1

1-hop link (IOS) (IOS-XE) (IOS-XR) (IOS-XR)
Per Link LFA FRR Not Available Not Available 4.3.1* 4.3.1*
OSPF LFA FRR (per

Radar Radar 4.3.1* 4.3.1*
prefix)
ISIS LFA FRR (per prefix) Radar Radar 4.3.1* 4.3.1*
EIGRP FRR (per prefix) Radar Radar Radar Radar
OSPF Remote LFA Radar Radar Radar Radar
ISIS Remote LFA Radar Radar Radar Radar
BGP PIC Core 3.5S 3.5S 3.7.0 3.4
BGP PIC Edge 3.5S 3.5S 4.0.0 4.0.0
*Future
BGP Next-Hop Tracking Edge FC POP FC Core FC
Edge POP Core

FRR FRR FRR
Makes the next-hop failure detection event-driven

instead of timer-driven
Next-hop tracking (NHT) feature allows to track BGP
next-hops in the RIB
If the RIB entry changes, then the client such as BGP
is notified
Allows for new path selection for BGP routes as soon
as the notification is received
On/off knob as well as configuration option on how
long to wait before starting new path selection
Behavior Without NHT Edge FC POP FC Core FC
Edge POP Core

FRR FRR FRR
Traffic Loss for Up to 60 Secs Due to BGP Scanner

Interval
Wait 180 Seconds?? No!!!
RR1 RR2
PE1# show ip route 192.168.1.3
% Subnet Not in Table
PE3
P1 P3
TicTic60sec
CE1 PE1 Site2
Site1
10.1.1.0/24 P4 10.1.2.0/24
P2 PE4
pe1#sh ip bgp vpnv4 vrf vpna 10.1.2.0/24

BGP routing table entry for 100:1:10.1.2.0/24, version 42
Paths: (1 available, best #1, table vpna)
Advertised to update-groups:
1
Local
192.168.1.3 (metric 145) from 192.168.1.2 (192.168.1.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:100:1
Originator: 192.168.1.3, Cluster list: 192.168.1.2,
mpls labels in/out nolabel/28
Behavior with NHT-Enabled Edge FC POP FC Core FC
Edge POP Core

FRR FRR FRR
Potential Time Saving Is Up to 60 Secs

The Time Period Determines How Long BGP Will Wait
router bgp 100 Before Running the Best Path Algorithm After
address-family ipv4 unicast Notification Is Received.
bgp nexthop trigger enable RR1 RR2
bgp nexthop trigger delay 5
PE3
P1 P3
CE1 PE1 Site2

Site1
10.1.1.0/24 10.1.2.0/24
P2 P4 PE4
wg2pe1#sh ip bgp vpnv4 all 10.1.2.0

BGP routing table entry for 100:1:10.1.2.0/24, version 51
Paths: (1 available, best #1, table vpna)
Flag: 0x820
Advertised to update-groups:
1
Local
192.168.1.4 (metric 193) from 192.168.1.2 (192.168.1.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:100:1
Originator: 192.168.1.4, Cluster list: 192.168.1.2,
mpls labels in/out nolabel/32

Enhancing IP&MPLS Based Carrier Services To Address The Data Center Interconnect Market - BRKSPG-2024

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Enhancing IP&MPLS Based Carrier Services To Address The Data Center Interconnect Market - BRKSPG-2024

Încărcat de

Drepturi de autor:

Formate disponibile

Unified MPLS: Advanced Scaling for Core

and Edge Networks

This session is intended to cover -

Unicast Routing + MPLS - Divide & Conquer

BGP for Services

Access Aggregation Region1 Backbone Region 2 Aggregation Access

ISIS Level 1 ISIS Level 2 ISIS Level 1

Access Aggregation Region1 Backbone Region 2 Aggregation Access

ISIS Level 1 ISIS Level 2 ISIS Level 1

ISIS Level 1 ISIS Level 2 ISIS Level 1

* ISIS L1->L2 (or L1->L1) Redistribution

Isolated IGP Isolated IGP Isolated IGP

Access Aggregation Region1 Backbone Region 2 Aggregation Access

ISIS Level 1 ISIS Level 2 ISIS Level 1

Loopback Int Redistributed

Access Aggregation Region1 Backbone Region 2 Aggregation Access

ISIS Level 1 ISIS Level 2 ISIS Level 1

ISIS Level 1 ISIS Level 2 ISIS Level 1

BGP Prefix PE31:

ISIS Level 1 ISIS Level 2 ISIS Level 1

More Details on BGP Scale for Services Later

Access Aggregation RRs Region1 Backbone Region 2 Aggregation Access

ISIS Level 1 ISIS Level 2 ISIS Level 1

Higher Network scale is attainable

Business Services demanding faster restoration

Fast Restoration of Services i.e. BGP Prefixes

Fast Restoration of BGP next-hops i.e. IGP Prefixes

Fast Convergence (FC) of IP routing protocols is key and still required

Fast Convergence Edge

Routing convergence happens at the process level,

Fast Convergence Edge

Fast Convergence Edge POP Core

IGP Tuning for FC

default values are 33 msec/66 msec IS-IS LSP-Generation Exponential Backoff

Note: MinLSArrival Must Be <= lsa-Hold

FRR Fast Reroute

Edge POP Core

IGP Prefixes FRR FRR FRR

IP/LDP FRR: Apply it as an intra PoP and inter PoP FRR

Intra PoP Inter PoP

Note: SPF calculations for LFAs are performed in

A backup path for all prefixes reachable via next-hop node

Edge POP Core

IP FRR : Per-Link LFA FRR FRR FRR

Availability of the backup NH is dependent on the topology and

A backup path for a prefix (e.g. D) reachable via next-hop

IGP pre-computes a backup path per IGP prefix

IGP pre-computes a backup path per IGP prefix

The link between A and B failed.

The closest from R2: R5

session with the selected PQ

R5s label for FEC R6 = 502

Primary: out-label = 408, oif = R4 408

RSVP-TE FRR Edge

RSVP-TE FRR link protection (and prefix independent): <50ms

Note: RSVP-TE Provides FRR Mechanism as well as:

Edge POP Core

tunnel destination Router D

Router A Router B Router D Router E

Edge POP Core

RSVP-TE FRR Node Protection FRR FRR FRR

What if Router D failed?

Router A Router B Router D Router E Router F