ASR9000 Troubleshooting Architecture

ASR9000 Troubleshooting
architectures
BRKSPG-2904
Xander Thuijs CCIE#6775 Principal Engineer

Highend Routing and Optical Group
Agenda
Introduction
Architecture of SW and HW forwarding
Troubleshooting Packet Forwarding
Fabric
NPU
Local Packet Transport Services/LPTS
Software forwarding/handling
Troubleshooting L3 forwarding including Mcast

Satellite
QoS Architecture and verification
Usability, XR strategy and SMUs
BRKSPG-2904
2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Acknowledgements
With contributions from
Santosh Sharma (Satellite)

Aleksandar Vidakovic (FIB)
Sadananda Phadke (Multicast)
Eddie Chami (Usability)
YOU
Thanks for all the feedback on the support forums and last years CL!
BRKSPG-2904
Cisco Public
Introduction
Hardware and Software Forwarding architectures
Getting started
With packet forwarding troubleshooting
One common architecture between all platforms
This applies between the different LC types also

Trident (NP3)
Typhoon (NP4)
Tomahawk (NP5)
IOS XRv
XR virtualization
nV Satellite
9000v,901,903
BRKSPG-2904
9001
9001-S
Cisco Public
9904
6
9006
9010
9912
9922
Forwarding paths
In principal there are 2 forwarding handlers
Hardware based forwarding from Linecard to Linecard over the fabric
Software based
This is not truly forwarding as XR has no true software forwarding path
This is for local packet consumption, eg for me packets
These are either handled by
LC CPU
RP CPU
Inside the CPU SW forwarding there are different switching techniques

SPP (software packet path) (somewhat similar to IOS fast switching).
NETIO (aka process switching)
BRKSPG-2904
Cisco Public
Software forwarding on classic IOS

Bit of History on CEF and Fast switching
CPU
4. OSPF doesnt return in time?

5. HW timer fires
Calls function to spit CPU HOG
Hw timer is defined with scheduler max time
Software forwarding
Calls a receive interrupt (RX interrupt) to the CPU
to handle the packet
Interfaces
PA
We need to turn around this packet quick during

the RX interupt.
PA
PA
Time in the ISR (interrupt service routine) is

limited.
process
IOS main (void)

scheduler
1
jump
2
return
How? With all features and info? CEF! Prebuilt L2

headers, routing, features all in one entry!
BGP
OSPF
jump
CPU utilization for five seconds: 68% / 30%

28% used
Jump procs
Total CPU time used

BRKSPG-2904
Time spent in ISR

(fast switching)
Cisco Public
Interupt
handler
HW timers CPU Complex

IOS classic
Packets not switched in ISR

Are handed over to IP Input (proc switching)
Hardware Forwarding
Transit
Ingress PHY receives frame
Ingress NPU handles input features (MAC is part of ingress NPU)

Handed over to Fabric interface ASIC for ingress schedule over fabric
If typhoon, pass over LC fabric when scheduled to
Central fabric
If RSP2/RSP440 this is on the RSP
If 9922/9910 this is the separate fabric cards
Egress LC fabric and FIA hand over packet to egress NPU for egress feature
processing
Egress NPU hands over to PHY and transmit
BRKSPG-2904
Cisco Public
Software forwarding
The punt path
If ingress NPU detects this packet is for me.
How does it determine it is for me?
The iFIB (internal FIB) portion of the LPTS (local packet transport services)
LPTS consists of 3 key portions

Filtering of what can be punted and categorization of the flow
Holes are dynamically pierced in LPTS and categorized into flows
Director of where it needs to go to

Is it for the LC or RP CPU (or DRP if you have CRS like devices)
Policing of the flow categorizations.

This policing is done per flow and per NPU.
For instance BGP-established is an LPTS flow. We can have multiple established BGP sessions that leverage this
policer.
After the CPU receives the packet after an RX interrupt the packet is handled by
SPP, and potentially the
NETIO chain
BRKSPG-2904
Cisco Public
10
Punt Packet Flow Overview

LPTS
iFIB
Policing
MEM
CPU
SDD
3x 10G
3x10GE
SFP +
Punt
FPGA
Typhoon
3x 10G
3x10GE
SFP +
Typhoon
CPU
FIA
3x 10G
Typhoon
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
Switch Fabric
ASIC
3x10GE
SFP +
Switch
Fabric
USB
Disk0/1
FIA
I/O FPGA
Typhoon
NVRAM
Boot Flash
3x 10G
3x10GE
SFP +
Typhoon
3x10GE
SFP +
BRKSPG-2904
CPU Complex
FIA
3x 10G
Typhoon
Cisco Public
11
Trace-points
Troubleshooting Packet Forwarding: Fabric
Packet Flow Overview

Two-stage IOS-XR packet forwarding
Uniform packet flow: All packet go through central switch fabric
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
Typhoon
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
Switch
Fabric
Switch Fabric
ASIC
3x 10G
3x10GE
SFP +
100GE
MAC/PHY
Switch Fabric
ASIC
3x10GE
SFP +
Ingress 100G
Typhoon
FIA
Egress 100G
Typhoon
FIA
Ingress
Typhoon 100G
100GE
MAC/PHY
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
3x10GE
SFP +
BRKSPG-2904
Switch
Fabric
Typhoon
Cisco Public
13
FIA
Egress
Typhoon 100G
The Magic Of The Switch Fabric

Fabric Arbitration and Redundancy
0 packet loss during RSP failover and OIR
Physically separated from LC. Resides on RSP or dedicated card (9912, 9922)
Logically separated from LC and RSP
All fabric ASICs run in active mode regardless of RSP Redundancy status
Extra fabric bandwidth and instant fabric switch over
If the FAB has been previously initiated then even with RP in rommon FABRIC IS
ACTIVE!
Access to fabric controlled using central arbitration.
One Arbitration ASIC (Arbiter) per RSP

Both Arbiters work in parallel both answer to requests for fab access
FIAs follow active Arbiter, and switch to backup if needed
Arbiter switchover controlled by low level hardware signaling
BRKSPG-2904
Cisco Public
14
Fabric Arbitration
RSP0
Crossbar
Fabric
ASIC
1: Fabric Request
5: credit return
Crossbar
Fabric
ASIC
Arbitration
Fabric Interface
and VOQ
2: Arbitration
Crossbar
Fabric
ASIC
3: Fabric Grant
Crossbar
Fabric
ASIC
4: load-balanced
transmission across
fabric links
Arbitration
RSP1
BRKSPG-2904
Cisco Public
15
Fabric Interface
and VOQ
Fabric Load Sharing Unicast

Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Arbitration
Fabric Interface
and VOQ
RSP0
Fabric Interface
and VOQ
Crossbar
Fabric
ASIC
VOQ (shaped at ~14G)

VQI (queue per priority)
Crossbar
Fabric
ASIC
Arbitration
RSP1
Unicast traffic sent across first available fabric link to destination (maximizes efficiency)
Each frame (or superframe) contains sequencing information
All destination fabric interface ASIC have re-sequencing logic
Additional re-sequencing latency is measured in nanoseconds

BRKSPG-2904
Cisco Public
16
Fabric Load Sharing Multicast

Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Arbitration
Fabric Interface
and VOQ
RSP0
Crossbar
Fabric
ASIC
Arbitration
RSP1
Multicast traffic hashed based on (S,G) info to maintain flow integrity

Very large set of multicast destinations preclude re-sequencing
Multicast traffic is non arbitrated sent across a different fabric plane
BRKSPG-2904
Cisco Public
C1
B2
A3
B1
A2
A1
Flows exit in-order
Crossbar
Fabric
ASIC
Fabric Interface
and VOQ
17
Fabric Super-framing Mechanism

Multiple unicast frames from/to same destinations aggregated into one super frame
Super frame is created if there are frames waiting in the queue, up to 32 frames or
when min threshold met, can be aggregated into one super frame
Super frame only apply to unicast, not multicast
Super-framing significantly improves total fabric throughput
Packet 2
Packet 3
Packet 2
Packet 1
Max
Super-frame
Min
Super-frame
Packet 1
No super-framing
Packet 1
Min reached
Packet 1
Max reached
Jumbo
0 (Empty)
Note that fabric counters are showing super frames not individual packets!!
(show controller fabric fia stats loc 0/X/CPU0)
BRKSPG-2904
Cisco Public
18
Troubleshooting Fabric Forwarding

Check fabric counters on
Ingress LC FIA
Ingress LC crossbar (not applicable to Trident and SIP-700)
Egress LC crossbar (not applicable to Trident and SIP-700)
Egress LC FIA
Check fabric drops

Ingress LC FIA
Egress LC FIA
BRKSPG-2904
Cisco Public
19
Fabric packet counters

RP/0/RSP0/CPU0:A9K-2#sh contr fabric fia stat loc 0/1/CPU0 | utility egrep "\] +[1-9]|^C|FIA"
Tue Jan 28 09:33:40.032 EST
********** FIA-0 **********
Category: count-0
From Unicast Xbar[0]
12684
From Unicast Xbar[1]
12938
To Unicast Xbar[0]
12870
To Unicast Xbar[1]
12752
To Line Interface[0]
25622
From Line Interface[0]
25622
********** FIA-1 **********
Category: count-1
<>
********** FIA-2 **********
Category: count-2
<>
********** FIA-3 **********
Category: count-3
Presuming this is not an 9912 or
<>
9922,
how many RSPs does this system
have?
BRKSPG-2904
Cisco Public
20
Fabric Drop Counters

sh controllers fabric fia drops [ingress|egress]
There are four priority levels and four physical XBAR links. FIA egress drop stats
are per priority, while FIA ingress drop stats are per XBAR link.
The FIA egress drop stats, Tail, Hard, WRED, (offsets 0-3) represent fabric priority
stats and correspond to:
0 - high priority level 1

1 - high priority level 2
2 - low priority
3 - not used (asr9k)
The FIA ingress drop stats offsets (0-3) represent XBAR link stats and correspond
to:
0-1 XBAR links to RSP0 (Trident+RSP2)

2-3 XBAR links to RSP1 (Trident+RSP2)
On Typhoon cards the FIA links with 2 links to the local fabric.
The local fabric connects with 8x55G links to the RSP fabric
BRKSPG-2904
Cisco Public
21
Fabric drop counters - ingress

RP/0/RSP0/CPU0:A9K-2#sh controllers fabric fia drops ingress location 0/1/CPU0
Tue Jan 28 09:40:35.255 EST
https://supportforums.cisco.com/document/12135016/asr9000xrunderstanding-and-troubleshooting-fabric-issues-a9k
********** FIA-0 **********
Category: in_drop-0
From Spaui Drop-0
accpt tbl-0
ctl len-0
short pkt-0
max pkt len-0
min pkt len-0
From Spaui Drop-1
accpt tbl-1
ctl len-1
short pkt-1
max pkt len-1
min pkt len-1
Tail drp
Vqi drp
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Header parsing drp

pw to ni drp
ni from pw drp
sp0 crc err
sp0 bad align
sp0 bad code
sp0 align fail
sp0 prot err
sp1 crc err
sp1 bad align
sp1 bad code
sp1 align fail
sp1 prot err
0
0
0
8
0
2
3
0
3
0
2
3
0
Ingress drops per link

********** FIA-0 **********
<>
BRKSPG-2904
Cisco Public
22
Fabric drop counters - egress

RP/0/RSP0/CPU0:A9K-2#sh contr fabric fia drops egress location 0/1/CPU0
Tue Jan 28 09:51:03.746 EST
********** FIA-0 **********
Category: eg_drop-0
From Xbar Uc Crc-0
From Xbar Uc Crc-1
From Xbar Uc Crc-2
From Xbar Uc Crc-3
From Xbar Uc Drp-0
From Xbar Uc Drp-1
From Xbar Uc Drp-2
From Xbar Uc Drp-3
From Xbar Mc Crc-0
From Xbar Mc Crc-1
From Xbar Mc Crc-2
From Xbar Mc Crc-3
From Xbar Mc Drp-0
From Xbar Mc Drp-1
From Xbar Mc Drp-2
From Xbar Mc Drp-3
Priority
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
**********
BRKSPG-2904 FIA-1 **********
Uc dq pkt-len-crc/RO-seq/len error
Mc rf crc
Mc vl0 src0 buffer full
Egress drops per priority

Cisco Public
23
drp
drp
drp
drp
drp
drp
drp
drp
drp
drp
0
0
0
0
0
0
0
0
0
0
Fabric counters on Typhoon LC (3-stage fabric)

RP/0/RSP0/CPU0:A9K-2#sh controllers fabric crossbar statistics instance 0 location 0/1/CPU0
Tue Jan 28 10:00:38.306 EST
Port statistics for xbar:0 port:0
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read
: 12821
Egress Packet Count Since Last Read
: 9590
Low priority stats (multicast)
===========================
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read
Egress Packet Count Since Last Read
: 12782
: 9778

===========================
<>
BRKSPG-2904
Cisco Public
24
Troubleshooting Packet Forwarding: NP
NPU Packet Processing - Ingress

5 Stages:
Parse
L2/L3 header
packet parsing in
TCAM
Builds keys for
ingress ACL,
QoS and
forwarding
lookups (uCode)
BRKSPG-2904
Search
Resolve
Modify
Performs QoS
and ACL
lookups in
TCAM tables
Performs L2
and L3 lookups
in RLDRAM
Processes Search
results:
ACL filtering
Ingress QoS
classification and
policing
Forwarding (egress
SFP determined)
Performs L2 MAC
learning
Adds internal
system headers
Egress Control
Header (ECH)
Switch Fabric
Header (SFH)
Cisco Public
26
Queueing
Scheduling
Queuing,
Shaping and
Scheduling
functions
All packets go
through this
stage
Troubleshooting NP Forwarding
1. Identify interface in question with problem.
2. Identify the mapping from interface to NPU.
3. Examine NP counters.
4. Look for rate counters that match lost traffic rate.
If none of the counters match the expect traffic, check drops at interface
controller
5. Lookup the counter description.

6. If required capture the packet hitting the counter (Typhoon only).
7. If packets are forwarded to the fabric, run fabric troubleshooting steps.
8. Identify egress NP and repeat steps 3 to 6.
BRKSPG-2904
Cisco Public
27
Interface to NP mapping
RP/0/RSP0/CPU0:A9K-BNG#show controller np ports all loc 0/0/cpU0
Node: 0/0/CPU0:
---------------------------------------------------------------NP
-0
1
2
3
Bridge
----------
BRKSPG-2904
Fia
--0
1
2
3
Ports
--------------------------------------------------GigabitEthernet0/0/0/0 - GigabitEthernet0/0/0/9
GigabitEthernet0/0/0/10 - GigabitEthernet0/0/0/19
TenGigE0/0/1/0
TenGigE0/0/1/1
Cisco Public
28
Examine NP Counters
RP/0/RSP0/CPU0:A9K-2#show controller np counters np0 loc 0/0/CPU0
Node: 0/0/CPU0:
---------------------------------------------------------------Show global stats counters for NP0, revision v2
Read 57 non-zero NP counters:
Offset Counter
FrameValue Rate (pps)
------------------------------------------------------------------------------16 MDF_TX_LC_CPU
22755787
6
17 MDF_TX_WIRE
1614696
0
21 MDF_TX_FABRIC
1530106
0
29 PARSE_FAB_RECEIVE_CNT
1555034
0
33 PARSE_INTR_RECEIVE_CNT
22026578
6
37 PARSE_INJ_RECEIVE_CNT
335774
0
41 PARSE_ENET_RECEIVE_CNT
2115361
1
45 PARSE_TM_LOOP_RECEIVE_CNT
17539300
5
Packets received
MDF=Modify
TX transmit
WIRE to the wire
= egress
from the fabric
If delta between received from Fab to TX-wire is almost 0, then everything forwarded
and not punted. If not, we dropped packets, could be ACL, QOS, or for other
reasons (eg PUNT) or large portion of injected traffic.
BRKSPG-2904
Cisco Public
29
NP Counter Description
RP/0/RSP0/CPU0:A9K-2#sh controllers np descriptions location 0/0/CPU0
Counter
Drop Feature Description
------------------------------------------------------------------------------<...>
PARSE_DROP_IPV4_LENGTH_ERROR
Drop L3
IPv4 packets dropped on receipt
from Ethernet due to IP packet length field inconsistent with L2 frame length
<...>
Look up controller np counters via google OR they can be seen in
show controller np description
BRKSPG-2904
Cisco Public
30
Interface Controller Stats

RP/0/RSP0/CPU0:A9K-2#sh controllers Gi0/0/1/8 stats | e = 0
Statistics for interface GigabitEthernet0/0/1/8 (cached values):
Ingress:
Input total bytes
Input good bytes
= 985886125
= 985886125
Input total packets

= 8343919
Input pkts 64 bytes
= 5536113
Input pkts 65-127 bytes = 2266459
Input good pkts
Input unicast pkts
Input multicast pkts
Input broadcast pkts
BRKSPG-2904
= 8343919
= 6957281
= 1386636
=2
Cisco Public
31
Looks at the PHY. Phy would only drop L1 errors.
MAC inside NPU will drop all identified other errors.
input drop other is accumulate from all NP drops!
Show in show interface | include drops
Filter coming up to define what needs to be accounted for in

the drop counters
This pictured card layout is not accurate to 1G interfaces of

course, illustrational only :)
Note on Aggregate NP Counters

Some counters have an index to a port.
For instance, there is an aggregate count per NPU showing the misses from
vlan to subinterface mapping:
UIDB_TCAM_MISS_AGG_DROP
There is also a specific counter from which port index these drops came from:
UIDB_TCAM_MISS_DROP_1
This means that the second port (starting count from zero) on that NPU
experienced that drop.
So if your show controller np ports tells us that ports X Y and Z are connected to
this NPU, and the drop index is _1, then port Y is the culprit.
BRKSPG-2904
Cisco Public
32
Capturing lost packets in the NPU

CLI:
monitor np counter <COUNTER_NAME> <NPU> count <N>
You can monitor any counter in the NPU on Typhoon generation line cards
Captured packets are always dropped
Exists automatically after capturing <N> packets or when timeout is reached
NPU is reset upon exit (~50ms forwarding stop)
This will be enhanced later
Packets subject to punt cannot be captured by this methodology

Use with care!
BRKSPG-2904
Cisco Public
33
Capturing lost packets in the NPU - Example

RP/0/RSP0/CPU0:A9K-2#monitor np counter PRS_HEALTH_MON np0 count 1 location 0/1/CPU0
Tue Jan 28 10:10:18.824 EST
Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~50ms traffic outage. Links will stay Up.
Proceed y/n [y] >
Monitor PRS_HEALTH_MON on NP0 ... (Ctrl-C to quit)
Tue Jan 28 10:10:22 2014 -- NP0 packet
From OAM RTC: 64 byte packet, bytes[0-3] invalid!
0000: 40 00 00 48 80 00 00 00 00 00 00 00 00 00 10 00 @..H............
0010: 00 00 00 00 00 00 00 00 04 55 aa 55 aa 55 aa 55 .........U*U*U*U
0020: aa 55 aa 55 aa 55 aa 55 aa 55 aa 55 aa 55 aa 55 *U*U*U*U*U*U*U*U
<>
0190: 00 00 00 00 00 00 00 00
........
(count 1 of 1)
Cleanup: Confirm NP reset now (~50ms traffic outage).
Ready? [y] >
RP/0/RSP0/CPU0:A9K-2#
BRKSPG-2904
Cisco Public
34
STOP
Local Packet Transport Services
What is LPTS?
Local packet transport services handling 4 key functions:
Securing the control plane with dynamic ACLs to filter

Categorize the flow
Policing the packets that get punted.
Providing direction as to where a for me flow should go to
FILTER
DIRECT
POLICE
CATEGORIZE
RP
Packet from established neighbor
high rate/critical priority
Bgp-known
Packet from configured neighbor
MPP denied packet to control
Plane, eg telnet from not
medium rate/medium priority
bgp-cfg
known station
LC
BRKSPG-2904
Cisco Public
36
LPTS characteristics
Classification and Policing inside NPU
An LPTS flow is policed at a certain rate
If you have eg multiple BGP neighbors on this NPU they share the same police
rate
For instance:
All our established neighbors are categorized as bgp-known
The bgp-known flow is policed at 10,000 pps
Police rates can be adjusted on a per LC basis, or globally applicable to all LCs.
Each NPU has that configured police rate (rate not shared between all npus on LC)
BRKSPG-2904
Cisco Public
37
LPTS What happens under the hood
CPU
CPU
CLI
TCAM
3x 10G
3x10GE
SFP +
Router bgp 100
Typhoon
Neighbor 1.2.3.4
FIA
3x 10G
3x10GE
SFP +
Typhoon
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
FIA
3x 10G
3x10GE
SFP +
Typhoon
TCP
socket calls pts
3x 10G
3x10GE
SFP +
Socket
Switch Fabric
ASIC
Typhoon
Established TCP
Creates TCP
3x 10G
3x10GE
SFP +
BGP
LPTS
3x 10G
3x10GE
SFP +
Typhoon
3x10GE
SFP +
BRKSPG-2904
CPU Complex
FIA
3x 10G
Typhoon
Cisco Public
38
IOS XR Control Plane

Local Packet Transport Service
LPTS
packets in
App 1
Control Plane Traffic

User Traffic
good packets
for-us packets
LPTS Internal FIB
(IFIB)
FIB
DCoPP
Dynamic Control
Plane Policing
bad packets
LPTS enables applications to reside on any or all RPs, DRPs, or LCs

Active/Standby, Distributed Applications, Local processing
IFIB forwarding is based on matching control plane flows

DCoPP is built in firewall for control plane traffic.
LPTS is transparent and automatic
BRKSPG-2904
App 2
RP
LC
transit
packets out
RP
Cisco Public
39
Local
Stacks
LC
IOS XR LPTS in action

LPTS is an automatic, built in firewall for control
plane traffic.
Every Control and Management packet from the line
card is rate limited in hardware to protect RP and LC
CPU from attacks
Router bgp
neighbor 202.4.48.99
ttl_security
! mpls ldp
LC 1 IFIB TCAM HW Entries

Remote
port
Rate
ICMP
ANY
ANY
1000
Priority
low
any
179
any
any
100
medium
any
179
202.4.48.99
any
1000
medium
202.4.48.1
179
202.4.48.99
2223
10000
medium
200.200.0.2
13232
200.200.0.1
646
100
medium
ttl
255
Socket
port
Any
LPTS
Local
BGP
LDP
SSH
LC 2 IFIB TCAM HW Entries

TCP Handshake
BRKSPG-2904
Cisco Public
40
Verifying LPTS policer values

RP/0/RP0/CPU0:CRS1-4#show lpts pifib hardware police location 0/7/CPU0
------------------------------------------------------------Node 0/7/CPU0:
------------------------------------------------------------Burst = 100ms for all flow types
------------------------------------------------------------FlowType
Policer Type
Cur. Rate Def. Rate Accepted
---------------------- ------- ------- ---------- ---------- ---------unconfigured-default
100
Static 500
500
0
Fragment
106
Global 0
1000
0
OSPF-mc-known
107
Static 20000
20000
0
OSPF-mc-default
111
Static 5000
5000
0
OSPF-uc-known
161
Static 5000
5000
0
OSPF-uc-default
162
Static 1000
1000
0
BGP-known
113
Static 25000
25000
18263
BGP-cfg-peer
114
Static 10000
10000
6
BGP-default
115
Global 0
10000
0
PIM-mcast
116
Static 23000
23000
19186
PIM-ucast
117
Static 10000
10000
0
IGMP
118
Static 3500
3500
9441
ICMP-local
119
Static 2500
2500
1020
ICMP-app
120
Static 2500
2500
0
na
164
Static 2500
2500
72
LDP-TCP-cfg-peer
152
Static 10000
10000
0
LDP-TCP-default
154
Static 10000
10000
0
cut
41
BRKSPG-2904
Cisco Public
lpts pifib hardware police

flow fragment rate 0
flow bgp default rate 0
Dropped
---------0
0
0
0
0
0
0
0
2
0
0
0
0
0
0
0
0
LPTS usage example

the NTP DDOS
Verification of filtering
Example, config used:
ntp
server 3.0.0.1
!
LPTS entries:
RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware entry brief location 0/0/cPU0 | i 123
Wed Apr 2 07:54:53.996 EDT
32 IPV4 default
UDP any
LU(30) any,123 3.0.0.1,any
<< yes we like that
204 IPV4 default
UDP any
LU(30) any,123 any,any
<< crap this we need to close of!
BRKSPG-2904
Cisco Public
43
Verification of policing
RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware entry loc 0/0/cPU0 | be 3.0.0.1
Wed Apr 2 07:55:55.137 EDT

Source IP
: 3.0.0.1
Is Fragment
: 0
Interface
: any
M/L/T/F
: 0/IPv4_LISTENER/0/NTP-known <<<
DestNode
: 48
DestAddr
: 48
>> 48 in binary is 0000-11-0000
SID
: 7
^^ RP on 9010!
L4 Protocol
: UDP
M - Fabric Multicast; L - Listener Tag; T - Min TTL; F Type;
Source port
: Port:any
DestNode - Destination Node;
Destination Port : 123
DestAddr - Destination Fabric Address;
HPo - Policer; Ct - Stats Counter;
Ct
: 0x613110
Lp - Lookup priority; Sp - Storage Priority;
Accepted/Dropped : 0/0
HAr - Hardware Average rate limit; HBu - Hardware Burst;
Lp/Sp
: 1/255
Rsp - Relative sorting position;
Rtp - Relative TCAM position;
# of TCAM entries : 1
HPo/HAr/HBu/Cir
: 14876884/200pps/200ms/200pps
State
: Entry in TCAM
Rsp/Rtp
: 26/40
BRKSPG-2904
Cisco Public
44
Flow
Understanding the outputs

What we looked at is the flow categorization
The policer of the flow:
RP/0/RSP0/CPU0:A9K-BNG#show lpts pifib hardware police location 0/0/cPU0
Wed Apr 2 07:57:29.351 EDT
FLOW NAME
curr.rate
| i NTP
accepted
dropped
default rate
NTPdefault
126
NTP-known
Static
180
200
Static
200
200
applicable TOS values
200
01234567
Configuration
lpts pifib hardware police
flow ntp default rate 1
This adjusts the policer rate to 1pps. (value of 0 is to be fixed up) Explanation of burst!
BRKSPG-2904
Cisco Public
45
01234567
Troubleshooting Punt path (SW forwarding)
Inside the CPU
Memory
CPU
ICMP BGP
CDP
0
1
2
1
2
FTP
HDLC
0
1
3
4
5
raw UDP
Each grey arrow is an IPC inter process call.
TCP
0
1
6
7
8
Because XR has protected or virtual mem space, each process

cant look in the other guys.
9
10
Unless shared mem is used which can be seen by all, this is

limited in size however
11
12
BFD
Netflow
NetIO
CpuCntrl Queues
From high/medium/critical etc
13
14
15
16
SPP
BRKSPG-2904
Cisco Public
47
SPP packet captures
RP/0/RSP0/CPU0:A9K-BNG#packet-trace spp platform

protocol arp start-capture count 5 location 0/0/cpu0
Wed Mar 12 16:28:30.176 EDT
Sending command: trace filter set 40 1 0x20
Trace filter set for protocol: ARP
Sending command: trace start 5
Started capture for 5 packets
RP/0/RSP0/CPU0#run attach 0/0/CPU0

attach: Starting session 1 to node 0/0/CPU0
Wrote ASCII trace to /tmp/spp_packet_trace_ascii

Sending command: trace print
Packet serial 3
client/inject:
length 110 phys_int_index -1 next_ctx 0xdeadbeef time
16:28:30.512
00: 00 65 7a 00 00 00 00 70 72 00 00 02 00 5e 00 00
10: 80 00 00 00 00 00 0f 8c 40 c1 0c c8 50 00 00 00
20: 00 00 0d 34 3f ff f2 90 20 04 fe 03 01 04 00 05
30: 00 00 00 00 5e 00 00 00 00 00 00 00 00 04 00 02
40: 40 00 10 34 ff ff ff ff ff ff 66 66 44 44 22 22
# spp_ui
spp-ui>
spp-ui> trace filter node client/punt
Node "client/punt" set for trace filtering. Index: 11
spp-ui> trace filter set 52 4 0xD4000001

Modified filter for offset 52 successfully
spp-ui> trace filter set 56 4 0xD4000002
Modified filter for offset 56 successfully
Decoder https://scripts.cisco.com/ui/use/xr_spp_ui_to_pcap
BRKSPG-2904
Cisco Public
48
Troubleshooting Packet Forwarding: L3
L3 Control Plane Architecture

LDP
Static
RSVP-TE
BGP
ISIS
LSD
OSPF
EIGRP
RIB
RSP CPU
RSP
LC
ARP/NDP
HW FIB
SW FIB
Adjacency
LC NP
AIB
AIB: Adjacency Information Base
RIB: Routing Information Base
FIB: Forwarding Information Base
LSD: Label Switch Database
LC CPU
BRKSPG-2904
Cisco Public
50
L3 Control Plane Architecture 2 Stage Lookup

Ingress LC
Egress LC
ARP/NDP
SW FIB
SW FIB
AIB
LC CPU
Recursions
resolved
Packet
HW FIB
LC CPU
Packet
HW FIB
Adjacency
LC NP
LC NP
Ingress Lookup:
Find egress NP +
egress interface ID
BRKSPG-2904
Egress Lookup:
Find output interface
+ do L2 rewrite
Cisco Public
51
Packet
Adjacency
stored only for
local and virtual
interfaces
L3 NP FIB Architecture
Adjacency
pointer(s) 1xLDI
Up to 32 ways
Non-Recursive
Prefix Leaf
NR NR
LDI LDI
NR
LDI
Up to 8 or 32 ways
Recursive
Prefix Leaf
R R R R
LDI LDI LDI LDI
NR-LDI: Non-Recursive Load Distribution Information

R-LDI: Recursive Load Distribution Information
BRKSPG-2904
Cisco Public
OIF
Adj
52
Adj
LAG (64
members)
OIF
Protected TE
Adj
LAG
OIF
Backup TE
Adj
OIF
How does ECMP path or LAG member selection work?

Or even VQI selection, but what the heck is that Stay focused (covered in QOS)
Every packet arriving on an NPU will under go a HASH computation. What

fields are used is dependent on encap (see overview shortly)
L2
L3
L4
payload
CRC32
32 bits
HASH
8 bits selected
8 bits selected
(3 drawn) 256 buckets
Path (ECMP)
path2
path1
BRKSPG-2904
path2
Member (LAG)
path2
path1
Buckets distributed over available members/paths
path1
Cisco Public
53
ECMP Load balancing
IPv6 uses first 64 bits in 4.0

releases, full 128 in 42
releases
A: IPv4 Unicast or IPv4 to MPLS (3)

No or unknown Layer 4 protocol: IP SA, DA and Router ID
UDP or TCP: IP SA, DA, Src Port, Dst Port and Router ID
B: IPv4 Multicast
For (S,G): Source IP, Group IP, next-hop of RPF

For (*,G): RP address, Group IP address, next-hop of RPF
C: MPLS to MPLS or MPLS to IPv4
# of labels <= 4 : same as IPv4 unicast (if inner is IP based, EoMPLS, etherheader will follow: 4th label+RID)
# of labels > 4 : 4th label and Router ID
(3) L3 bundle uses 5 tuple as 1 (eg IP enabled routed bundle interface)
(3) MPLS enabled bundle follows C
(1) L2 access bundle uses access S/D-MAC + RID, OR L3 if configured (under l2vpn)
(2) L2 access AC to PW over mpls enabled core facing bundle uses PW label (not FAT-PW label even if configured)
-
BRKSPG-2904
FAT PW label only useful for P/core routers

Cisco Public
54
Load-balancing scenarios
EoMPLS protocol stack
MPLS/IP protocol stack
45 for ipv4
BRKSPG-2904
Cisco Public
55
MPLS vs IP Based loadbalancing

When a labeled packet arrives on the interface.
The ASR9000 advances a pointer for at max 4 labels.
If the number of labels <=4 and the next nibble seen right after that label is
4: default to IPv4 based balancing
6: default to IPv6 based balancing
This means that if you have a P router that has no knowledge about the MPLS service of the packet,
that nibble can either mean the IP version (in MPLS/IP) or it can be the DMAC (in EoMPLS).
RULE: If you have EoMPLS services AND macs are starting with a 4 or 6. You HAVE to use ControlWord
L2
MPLS
MPLS
45 (ipv4)
0000 (CW)
41-22-33 (mac)
4111.0000.
Control Word inserts additional zeros after the inner label showing the P nodes to go for label based
balancing.
In EoMPLS, the inner label is VC label. So LB per VC then. More granular spread for EoMPLS can be achieved with
FAT PW (label based on FLOW inserted by the PE device who owns the service).
Take note of the knob to change the code: PW label code 0x11 (17 dec, as per draft specification). (IANA assignment is 0x17)
BRKSPG-2904
Cisco Public
56
Loadbalancing ECMP vs UCMP and polarization

Support for Equal cost and Unequal cost
32 ways for IGP paths
32 ways (Typhoon) for BGP (recursive paths) 8-way Trident
64 members per LAG
Make sure you reduce recursiveness of routes as much as possible (static route
misconfigurations)
All loadbalancing uses the same hash computation but looks at different bits from that hash.
Use the hash shift knob to prevent polarization.
Adj nodes compute the same hash, with little variety if the RID is close
This can result in north bound or south bound routing.
Hash shift makes the nodes look at complete different bits and provide more spread.
Trial and error (4 way shift trident, 32 way typhoon, values of >5 on trident result in modulo)
BRKSPG-2904
Cisco Public
57
Hash shift, what does it do?

L2
L3
L4
payload
Hash shift 8
Y HASH
HASH
8 bits selected
8 bits selected
(3 drawn) 256 buckets
Path (ECMP)
path2
path1
BRKSPG-2904
path2
Member (LAG)
path2
path1
Buckets distributed over available members/paths
path1
Cisco Public
58
FIB Architecture diffs: IOS XR vs IOS

Adjacency vs RIB prefix preference
Local vs remote ARP learning
https://supportforums.cisco.com/document/12098096/cscse46790-cef-prefers-arp-adjacency-over-rib-next-hop
BRKSPG-2904
Cisco Public
59
Adjacency vs Prefix preference

1
Local adjacency exists

adjacency overrides the /32 RIB entry
FIB creates /32 entry via Te0/0/0/0
router static
address-family ipv4 unicast
10.1.1.1/32 TenGigE0/1/1/0
TenGigE0/1/1/1 10.2.2.1
RIB creates /32 prefix via Te0/1/1/0

ARP/NDP
SW FIB
SW FIB
AIB
LC CPU
TenGigE0/1/1/0
10.2.2.2/24
LC CPU
HW FIB
HW FIB
TenGigE0/0/0/0
10.1.1.2/24
Adjacency
TenGigE0/1/1/1
BRKSPG-2904
LC NP
Cisco Public
LC NP
60
TenGigE0/0/0/1

1
Local
exists
(arp learnt 10.1.1.1 on te 0/0/0/0
Localadjacency
adjacency
exists
router static
10.1.1.1/32 TenGigE0/1/1/0
TenGigE0/1/1/1 10.2.2.1
adjacency overrides the /32 RIB entry

FIB creates 10.1.1.1/32
via Te0/0/0/0
/32 entry viaentry
Te0/0/0/0
RIB creates /32 prefix via Te0/1/1/0

ARP/NDP
SW FIB
SW FIB
AIB
LC CPU
TenGigE0/1/1/0
10.2.2.2/24
LC CPU
HW FIB
HW FIB
TenGigE0/0/0/0
10.1.1.2/24
Adjacency
Packet
BRKSPG-2904
TenGigE0/1/1/1
LC NP
Switch
Fabric
Packet w/ DestIP
10.1.1.1
61
Cisco Public
LC NP
TenGigE0/0/0/1
Packet

All IOS XR releases up to and including 5.1.1:
Adjacency prefix overrides a RIB prefix
IOS XR Release 5.1.2:

New CLI introduced to prevent the override (global configuration mode)
cef adjacency route override rib disable
IOS XR Release 5.2.0:

Default behavior change!
Adjacency prefix will not override a RIB prefix
BRKSPG-2904
Cisco Public
62
Local vs remote ARP learning

IOS XR allows ARP learning when ARP reply/request is outside of the local
subnet
Starting with IOS XR Release 4.3.4:

New CLI introduced to block out-of-subnet ARP learning:
interface g0/0/0/0
arp learning local
BRKSPG-2904
Cisco Public
63
Troubleshooting L3 Forwarding
Identify the input interface, slot and NP
Walk the SW+HW control plane to confirm the expected forwarding decision
RIB
SW FIB on ingress LC
Ingress HW FIB on ingress LC
Egress HW FIB on egress LC
If packets are not forwarded as expected by ingress or egress NP, apply NP

troubleshooting
If packets are not crossing the fabric, apply fabric troubleshooting
BRKSPG-2904
Cisco Public
64
L3 IPv4 Forwarding Show Commands

RIB
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh route 10.1.2.0/24
Routing entry for 10.1.2.0/24

Known via "static", distance 1, metric 0
Installed Jan 29 05:54:26.182 for 00:00:04
Routing Descriptor Blocks
40.0.3.1, via GigabitEthernet0/7/0/2
Route metric is 0
No advertising protos.
BRKSPG-2904
Cisco Public
65

SW FIB on Ingress LC
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 location 0/4/CPU0
10.1.2.0/24, version 256779, internal 0x4004001 (ptr 0xa3d59b84) [1], 0x0 (0xa3610aa0), 0x440 (0xa4fb4d50)
Updated Jan 29 05:54:26.191
remote adjacency to GigabitEthernet0/7/0/2
Prefix Len 24, traffic index 0, precedence routine (0), priority 3
via 40.0.3.1, GigabitEthernet0/7/0/2, 8 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0xa51a43b4 0xa5a5a5b8]
next hop 40.0.3.1
remote adjacency
local label 17012
labels imposed {ImplNull}
BRKSPG-2904
Cisco Public
66

Adjacency and SW FIB on Egress LC
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh adjacency ipv4 g0/7/0/2 detail location 0/7/CPU0
Interface
Address
Version Refcount Protocol
<. . .>
Gi0/7/0/2
40.0.3.1
1562
2( 0) ipv4
001b53ff9a99001b53ff9c320800
mtu: 1500, flags 1 0 0
0 packets, 0 bytes
10.1.2.0/24, version 256779, internal 0x4004001 (ptr 0xa4d3ee6c) [1], 0x0 (0xa35bdc80), 0x440 (0xa5b1bd50)
Updated Jan 29 05:54:26.192
local adjacency 40.0.3.1
path-idx 0 [0xa59683c8 0xa5a30268]
next hop 40.0.3.1
local adjacency
local label 17012
BRKSPG-2904
Cisco Public
67

Adjacency and SW FIB on Egress LC
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh adjacency ipv4 g0/7/0/2 detail location 0/7/CPU0
Interface
Address
Version Refcount Protocol
<. . .>
Gi0/7/0/2
40.0.3.1
1562
2( 0) ipv4
001b53ff9a99001b53ff9c320800
mtu: 1500, flags 1 0 0
0 packets, 0 bytes
Updated Jan 29 05:54:26.192
path-idx 0 [0xa59683c8 0xa5a30268]
next hop 40.0.3.1
local adjacency
local label 17012
Entry missing or not what you have expected?

sh adjacency trace all location
sh cef trace location
BRKSPG-2904
Cisco Public
68

HW FIB Show Command Structure
#
Block
Description
SW FIB Entry
Same as in the SW FIB command output
HW Leaf
Leaf Context: Type of the Leaf entry

Leaf HW result: features associated (e.g.: URPF, BGP policy
accounting, etc.)
NR-LDI
NR-LDI context description

NR-LDI result (one section per path)
TX HW Entry for path #1
Included only if Adjacency exists on location (i.e. if location is egress

LC)
RX HW Entry for path #1
Always included
Displays the egress NP and egress interface ID
4 & 5 are repeated for every available path

BRKSPG-2904
Cisco Public
69

Ingress HW FIB
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 hardware ingress location 0/4/CPU0
< SW FIB Entry >
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0
Leaf H/W Result:
<>
fast_switch_entry
Egress NP identification
rx_adj_field: 0x5300 (LE)
egress_ifhandle: 0x12000140 (LE)
<>
Egress interface identification
nrLDI eng ctx:
flags: 0x541, proto: 2, npaths: 0, nbuckets: 1 ldi_tbl_idx: 0x0, ecd_ref_cft: 0
RX H/W Result on NP:0 [Adj ptr:0x18 (BE)]:

<>
rx_adj_field: 0x5300
UTurn_egress_ifhandle: 0x12000140
<>
RX H/W Result on NP:1 [Adj ptr:0x18 (BE)]:
<>
Little Endian vs Big Endian: 0x5300 LE == 0x53 BE

BRKSPG-2904
Cisco Public
70

Ingress HW FIB Confirm egress NP and Interface ID on egress LC
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#show uidb index location 0/7/CPU0 gig0/7/0/2
------------------------------------------------------------------------------------------------------Location Interface-handle Interface-name
Interface-Type Ingress-index Egress-index
------------------------------------------------------------------------------------------------------0/7/CPU0 0x12000140
GigabitEthernet0_7_0_2 Main interface
5
5
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh controllers pm interface G0/7/0/2 location 0/7/CPU0

Ifname(1): GigabitEthernet0_7_0_2, ifh: 0x12000140 :
iftype
0xf
egress_uidb_index 0x5
ingress_uidb_index 0x5
port_num
0x2
subslot_num
0x0
Matches the egress_ifhandle from show cef hardware
phy_port_num
0x2
channel_id
0x3
channel_map
0x0
lag_id
0x0
virtual_port_id 0x0
Matches the rx_adj_field from show cef hardware
switch_fabric_port 0x53
<>
BRKSPG-2904
Cisco Public
71

Egress HW FIB
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 10.1.2.0/24 hardware egress location 0/7/CPU0
Updated Jan 29 05:54:26.189
path-idx 0 [0xa59683c8 0xa5a30268]
next hop 40.0.3.1
local adjacency
local label 17012
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0
Leaf H/W Result:
<>
fast_switch_entry
rx_adj_field: 0x5300 (LE)
egress_ifhandle: 0x12000140 (LE)
<>
TX H/W Result for NP:0 (index: 0x82c (BE)):
<>
Little Endian vs Big Endian:
uidb_index
: 0x500 (LE)
l3_mtu
: 0xdc05 (LE)
0xdc05 LE == 0x5dc BE == 1500
prefix_adj_cnt_index: 0x0
dest_mac
: 0x001b.53ff.9a99 u1.reserved : 0123456789ab
<>
BRKSPG-2904
Cisco Public
72

Egress HW FIB what would it show in case of more complex adjacencies?
RP/0/RSP0/CPU0:EAST-CORE-ASR9K-1#sh cef 220.0.0.52/32 hardware egress location 0/7/CPU0
220.0.0.52/32, version 256780, internal 0x4004001 (ptr 0xa4d1f4bc) [1], 0x0 (0xa644e520), 0x440 (0xa5b1bd80)
Updated Jan 29 06:30:54.170
local adjacency point2point
via 41.52.0.2, tunnel-ip52, 5 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0xa59cc25c 0x0]
next hop 41.52.0.2
local adjacency
local label 16005
labels imposed {None}
<>
TX H/W Result for NP:0 (index: 0x1421 (BE)):
<>
uidb_index
: 0x6100 (LE)
l3_mtu
: 0xc405 (LE)
prefix_adj_cnt_index: 0x27670300
GRE Adj:
ip_sa
: 110.0.0.41
ip_da
: 110.0.0.52
tos
:0
ttl
: 0xff
df
:1
tos_reflect : 1
rsvd flag bits : 0
encap_checksum : 0xe471
What type of adjacency is this?
BRKSPG-2904
Cisco Public
73
Multicast SSM/SM differences
Multicast forwarding considerations

Difference in forwarding between SSM (source specific multicast) and SM
(sparse mode)
What results are when SSM is misconfigured
Understanding the drop and punt counters/reasons
Characteristics of SM forwarding with no receivers

SSM range and MAP commands
What they do and what the forwarding impact is
BRKSPG-2904
Cisco Public
75
PIM-SM Sender Registration (IGMPv2)
RP
Source
(S, G) Register
(S, G) Joins
Shared Tree
Source Tree
(unicast)
(S, G) State created only

along the Source Tree.
Receiver
BRKSPG-2904
Cisco Public
76
Source Specific Multicast Example

Source
Default ssm:
50.0.0.10
232.x.x.x
NPU
Host learns of source, group/port

Last-hop learns of source, group/port
Last-hop send PIM (S,G) Join
Non-Default ssm:
50.0.0.10
239.x.x.x
RP
A
CPU
PIM (S, G) Join

E
IGMPv3 (S, G) Join
Receiver 1
BRKSPG-2904
Cisco Public
77
Out-of-band
source directory,
example: web server
SSM-range to define SSM groups on first hop router

SSM-MAP to map v2 joins to v3 sources
Roadblock to Deploying PIM-SSM You Need to be

Aware of
Dependent on IGMPv3
Microsoft supports IGMPv3 in Windows XP natively
Many IPTV STBs are adding support.
However, not every host supports IGMPv3 today.

That is where the ssm group mapping comes into play!
Workaround
Source Mapping
Router maps IGMPv2 Joins in SSM range to well-known sources via DNS or static
configuration
The ssm MAP links a multicast group v2 join to a multicast source
BRKSPG-2904
Cisco Public
78
Multicast forwarding in ASR9000

Multicast forwarding is based on FGIDs
PHY
NP0
PHY
NP1
PHY
NP2
PHY
NP3
4 MFIB
5
B0
CP
U
PI
M
2
MRIB
FIA
B1
CrossbarCrossbar
Switch
Fabric Fabric
ASIC
ASIC
Fabric
The FGID/MGID gives instructions to the forwarding asics to which NPUs

(mgid) and LCs (fgid) the mcast needs to be replicated
1.
Incoming IGMP and PIM packets are punted to RP CPU
2.
Protocols (PIM/IGMP) send their Route/OLIST Information to MRIB process to

build multicast route/olist table
3.
MRIB sends the multicast state information to MFIB process on all LCs
4.
MFIB program HW forwarding tables in NP, Bridge FPGA and Fabric interface
ASIC
5.
Software switched multicast packet or data packet for protocol signaling is sent
to local line card CPU
Cisco Public
79
CPU
(fabric group IDs) and MGIDs (mcast group IDs).
BRKSPG-2904
IGMP
Multicast Packet Replication (1)

Switch Fabric and Egress LC Replication
2 FIA Replication replicate single copy
to Bridge which has IGMP join, based
on MGID table in FIA
1 Fabric Replication replicate

single copy to LCs which receive
IGMP join, based on FGID table
in switch fabric
3 Bridge Replication similar as FIA

replication, single copy to NP
MFIB
CPU
MGID
IGMP joins
PHY
PHY
IGMP joins
Multicast
Source
NP1
PHY
NP2
PHY
NP3
FIA
B0
FIA
FGID/
FPOE
Switch
Fabric
B1
BRKSPG-2904
LC3
Cisco Public
B03
NP1
PHY
B1
NP2
PHY
NP3
PHY
NP0
PHY
B0
NP1
PHY
B1
NP2
PHY
NP3
PHY
CPU
MGID Multicast Group ID

MFIB Multicast Forwarding Information Base
PHY
LC2
FIA
FGID Fabric Group ID
IGMP joins
NP04
MGID
CPU
NP0
NP Replication replicate copy

per receiver based on multicast FIB
table
MGIDs and FGIDs

MGID - Multicast Group Identifier
Unique ID assigned to a multicast group
Used by Octopus/Bridge to determine replication requirements per multicast group
FGID - Fabric Group Identifier

Slotmask used by switch fabric to determine replication to line card/RSP slots
Assigned to each group by multicast PD control plane
BRKSPG-2904
Cisco Public
FGID (Slotmask)
FGIDs: 10 Slot Chassis
FGIDs: 6 Slot Chassis

Phy
Slot
Number
LC 0
LC 1
LC 2
LC 3
RSP 1
RSP 0
LC 4
LC 5
LC 6
LC 7
Logical
Slot
Logical
Slot
LC 3
LC 2
LC 1
LC 0
RSP 1
RSP 0
Slot
Slot
Logical
LC7
Slot Mask
Physical
9
Binary
1000000000
Hex
0x0200
LC6
0100000000
0x0100
LC5
0010000000
0x0080
LC4
0001000000
0x0040
RSP0
0000100000
0x0020
RSP1
0000010000
0x0010
LC3
0000001000
0x0008
LC2
0000000100
0x0004
0000000010
LC1
LC0
BRKSPG-2904
0 Cisco Systems,0000000001
2006
Inc. All rights reserved.
0x0002
Cisco0x0001
Confidential
Cisco Public
Slot Mask
Logical
Physical
Binary
Hex
LC3
0000100000
0x0020
LC2
0000010000
0x0010
LC1
0000001000
0x0008
LC0
0000000100
0x0004
RSP1
0000000010
0x0002
FGID Calculation
0
0000000001
Examples
0x0001
RSP0
Target Linecards
FGID Value (10 Slot Chassis)
LC6
0x0100
LC1 + LC5
0x0002 | 0x0080 = 0x0082
LC0 + LC3 + LC7
0x0001 | 0x0008 | 0x0200 = 0x0209

EDCS:xxxx
MGID Tables
MGID Bitmasks
MGID
FIA
MGID
Bit 1
Bit 0
Bridge1
NP3
BRKSPG-2904
MGID
Bridge0
NP2
Cisco Public
NP1
83
Bit 1
Bit 1
NP0
Bit 0
Bit 0
Source specific Multicast configurations

SSM-range
By default hardware assumes that the SSM multicast groups are starting with 232.x.x.x
The SSM range gives the HW the instruction that there are other groups also
SSM enabled.
SSM-MAP
Provides the mapping for non v3 capable receivers to map the mcast group to a specific
source (whatever source address that is).
DNS mapping: allow a name instead of an address for the dynamic mapping of groups
to sources
BRKSPG-2904
Cisco Public
84
L3 Multicast PIM SSM (SSM map)
Not all receivers have IGMP-V3 capability

Use SSM mapping feature to translate V2 as V3
join
Without SSM mapping, V2 joins to 239.x.x.x get
dropped.
RP/0/RSP0/CPU0:ASR9K-1#show logging | in 239.
RP/0/RSP0/CPU0:Apr 23 10:33:45.688 : igmp[1154]: Received v2 Report on
GigabitEthernet0_1_0_1.1501 from 40.0.191.1 for 239.1.1.10
RP/0/RSP0/CPU0:Apr 23 10:33:45.688 : igmp[1154]: Discarding report as group
239.1.1.10 is in SSM range
SSM map, adds source component to V2 join and

creates (S,G) towards source
RP/0/RSP0/CPU0:ASR9K-1#show mrib route 239.1.1.10
(51.2.1.2,239.1.1.10) RPF nbr: 51.2.1.2 Flags:
Up: 00:13:54
Incoming Interface List
GigabitEthernet0/0/0/4.300 Flags: A, Up: 00:13:54
Outgoing Interface List
GigabitEthernet0/1/0/1.1501 Flags: F NS LI, Up: 00:13:54
RP/0/RSP0/CPU0:ASR9K-1#
BRKSPG-2904
Cisco Public
igmp v2 Receiver
0/1/0/1.1501
ASR9K-1
(51.2.1.2, 239.1.1.10)
Source Gig0/0/0/4.300
ipv4 access-list PIM-SSM-groups Define additional

SSM Groups
10 permit ipv4 239.0.0.0 0.255.255.255 any
RP/0/RSP0/CPU0:ASR9K-1#show run multicast-routing

multicast-routing
address-family ipv4
ssm range PIM-SSM-groups add 239.x.x.x as

SSM group
RP/0/RSP0/CPU0:ASR9K-1#show running-config router igmp
router igmp
interface GigabitEthernet0/1/0/1.1501
static-group 239.1.1.10 51.2.1.2

!
SSM MAP
L3 Multicast PIM SSM

Goal of this section is to explain the reasoning for enabling ssm-range
on first hop router if SSM group is different from default group
(232.x.x.x)
This is specifically important if first hop router has V2 and V3 sources
If not configured mcast groups outside default range are perceived to be v2 and forwarded to the
Route Processor and might get dropped (if these are actually SSM mcast adds) or sent to the
Rendez-Vous point
Below sample config, extends SSM group range

ipv4 access-list PIM-SSM-groups
10 permit ipv4 <new group> <reverse mask> any
!
multicast-routing
address-family ipv4
ssm range PIM-SSM-groups
!
!
BRKSPG-2904
Cisco Public
ASR9k behavior on first hop router (closest to source)
SSM range command is needed on first hop router unless:
Each SSM multicast stream has at least one receiver:
In the presence of receivers, last hop router builds (S,G) towards source.
Due to (S,G) from last-hop router, traffic get forwarded correctly.
PIM-SM RP is not configured:
This scenario is applicable when first hop router is enabled for both PIM-SM and
PIM-SSM.
If PIM-SM RP is not configured and if there are no receivers, packets gets dropped
in NP; Example output below:
P/0/RSP0/CPU0:ASR9K-1#show controllers np counters np0 location 0/1/CPU0
81743
781
332 RSV_DROP_IPM4_ING_RTE_DROP
80396
777
Seen when there is no RendezVous Point configured and there is no registered v3 receiver
BRKSPG-2904
Cisco Public
ASR9k behavior on first hop router
The PIM-SM RP is configured but SSM range is not

RP/0/RSP0/CPU0:ASR9K-1#show controllers np counters np0 location 0/1/C$
Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------21 MDF_TX_FABRIC
75156
548 <==
439848
548
RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#
show mrib route shows packets get punted to RP.

RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#show mrib route
(*,232.0.0.0/8) Flags: D P
Up: 02:56:41
Default SSM group. No need for SSM range config
(52.0.0.2,239.1.1.2) RPF nbr: 52.0.0.2 Flags: Group is not part of default SSM group
Up: 00:00:15
GigabitEthernet0/1/0/15 Flags: A, Up: 00:00:15
RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#
Seen when RP is configured, we are sending the traffic to the fabric; forwarding decision is made before we know the
receivers in the NP. The fabric receives erroneous traffic in this case which will get dropped.
BRKSPG-2904
Cisco Public
ASR9k behavior on first hop router (Continued
The PIM-SM RP is configured but SSM range is not
In the absence of SSM range config, packets get punted to RP with FGID Null.
With FGID null, packets get dropped in Crossbar: [Port NP FIA Fabric Crossbar]
RP/0/RSP0/CPU0:ASR9K-1#show mrib route detail
(51.2.1.2,239.1.1.10) Ver: 0x659d RPF nbr: 51.2.1.2 Flags:,
PD: Slotmask: 0x0
FGID zero
MGID: 16918
Up: 00:25:27
GigabitEthernet0/0/0/4.300 Flags: A NS, Up: 00:01:46 Source
RP/0/RSP0/CPU0:ASR9K-1#show controllers fabric crossbar statistics instance 0 loc 0/rsp0/cpu0
===========================
NULL FPOE Drop Count
: 71740 = Packets get dropped in RSP cross-bar
==============================
PIM periodically, sends null registers to PIM-RP to check if there are new receivers but data packets
are not really sent to RP. (few packets leaked from hw to CPU for this purpose)
RP/0/RSP0/CPU0:EAST-PE-ASR9K-1#show pim traffic
PIM Traffic Counters
Elapsed time since counters cleared: 01:03:25
Received
Null Register
0
Register Stop
69
BRKSPG-2904
Cisco Public
Sent
67
0
Sent to RP
Received from RP
ASR9k behavior on first hop router
The ssm range command is configured and no receivers:
Packets get dropped in ingress NP as there is no most specific (S,G), Example

RP/0/RSP0/CPU0:ASR9K-1#show controllers np counters np0 location 0/1/CPU0
44156
651
333 RSV_DROP_IPM4_ING_RTE_DFLAG_DROP
43833
647
RP/0/RSP0/CPU0:ASR9K-1#show mfib hardware route statistics non-zero location 0/1/cpu0
LC Type: Typhoon A9K-MOD80-TR
-------------------------------------------------------------------------ID:
Ingress Drop
ED:
Egress Drop
Source: * Group: 239.0.0.0 Mask:8
------------------------------------------------------------------------NP
R(packets:bytes)/F(packets:bytes)/P(packets)/ID(packets)/ED(packets)
------------------------------------------------------------------------0
0:0 / 0:0 / 0 / 19782 / 0
------------------------------------------------------------------------No OLIST interfaces found for this route
This is the situation where you want to be, dropped in HW, no punting. If there is a receiver
(either
v2 via ssm-map
or a direct v3 then hw Cisco
gets
programmed to forward)
BRKSPG-2904
Public
Best-Practice
Configure SSM-Range on asr9ks if SSM sources are directly connected

Config sample if SSM group is 239.x.x.
0/1/0/1.1501 IGMPv2 join
(51.2.1.2, 239.1.1.10)
Source Gig0/0/0/4.300
RP/0/RSP0/CPU0:ASR9K-1#show mrib route 239.1.1.10 51.2.1.2

IP Multicast Routing Information Base
(51.2.1.2,239.1.1.10) RPF nbr: 51.2.1.2 Flags:
Up: 00:02:23
GigabitEthernet0/0/0/4.300 Flags: A NS, Up: 00:02:23
Outgoing Interface List
GigabitEthernet0/1/0/9 Flags: F NS, Up: 00:02:23 <= V3 join
GigabitEthernet0/1/0/1.1501 Flags: F NS, U V2 join (SSM map)
BRKSPG-2904
Cisco Public
RP/0/RSP0/CPU0:ASR9K-1#show run multicast-routing
multicast-routing
address-family ipv4
ssm range SSM-groups
interface all enable
RP/0/RSP0/CPU0:ASR9K-1#show run ipv4 access-list
SSM-group
ipv4 access-list SSM-groups
10 permit ipv4 239.0.0.0 0.255.255.255 any
20 permit ipv4 232.0.0.0 0.255.255.255 any
RP/0/RSP0/CPU0:ASR9K-1#show run router pim

router pim
address-family ipv4
rp-address 110.0.0.24 Only if you need PIM-SM
RP/0/RSP0/CPU0:ASR9K-1#show run router igmp
Router igmp
interface GigabitEthernet0/1/0/1.1501
static-group 239.1.1.10 51.2.1.2
Satellite
Satellite Basics
Satellite Access port
Satellite Fabric Port
Host Fabric Port
ASR9k
(Host)
Satellite
Ingress: ASR9K uses the SAT-VLAN id and the
incoming uplink to determine the satellite port. Rest of
the packet processing is identical to packets received at
local ports
Ingress: Satellite box adds a SAT-VLAN to the packet

and sends it to ASR9K host. SAT-VLAN id is 1:1
mapped to the incoming port on Satellite
Ingress
MAC DA
MAC SA
VLANs(opt)
Payload/FCS
MAC DA
MAC SA SAT-VLAN
VLANs(opt)
Payload/FCS
Egress
Egress: Satellite box maps the incoming SAT-VLAN to
its front port, strips the SAT-VLAN and forwards the
packet.
BRKSPG-2904
Cisco Public
Egress: After all interface VLAN Ops are performed,

ASR9K adds the SAT-VLAN id of the outgoing port
93
Satellite Bringup
Discovery Protocol
Operates at Layer 2.
Provides the bootstrap mechanism by which a Satellite and Host begin communication.
Enables Host to become aware of a reachable Satellite device, and exchange sufficient
information to set up a full Control session.
The Host device initiates discovery. Satellite devices are factory-shipped to listen for
incoming Discovery probe packets.
Control Protocol
Makes use of the connectivity set up by the discovery protocol to provide a reliable and
extensible mechanism for the Host device and each satellite device to exchange the
required configuration, state, etc.
BRKSPG-2904
Cisco Public
94
Discovery Protocol
Discovery Protocol states
Stopped The link is down. Unshut the satellite Fabric link or the member links incase
Bundle is used as ICL.
Probing for satellites / Discovering The Discovery protocol has sent probe packets
to the satellite. It is waiting for a response from the satellite.
Configuring satellite The Discovery protocol has sent configuring packets to the
satellite. It is waiting for a response from the satellite.
Ready Discovery protocol has completed. Control protocol bring-up can now start.
sh nv satellite status satellite <> gives the state information

RP/0/RSP0/CPU0:nv-cluster-escalation#sh nv satellite status satellite 200
Wed Apr 23 22:27:20.927 UTC
Satellite 200
------------Status: Connected (New image transferred)
BRKSPG-2904
Cisco Public
95
Satellite new topologies in XR51

CFM
VLAN-A
Host A
Satellite
Host A
Satellite
VLAN-B
Satellite
Host B
CFM
Satellite
fabric extension
ICCP
Host B
Dual home ring

Satellite
Satellite
(rings to same host planned for

later)
Host
Cascading
BRKSPG-2904
Cisco Public
96
Discovery Protocol Troubleshooting Contd.

Satellite Stuck in Discovering state and configured with 5.1.1 new topologies
If the satellites are configured with 5.1.1 new topologies(Ring/Chain, L2 Fab, Dual head)
and the satellite is not running with 5.1.1 compatible image, it is expected that satellite
will not be discovered in the new topologies.
Below is the mapping of Sat image and host image:
SVA --- 4.2.1

SVB --- 4.2.3
SVC --- 4.3.0
SVD --- 4.3.1
SVE --- 5.1.0
SVF --- 4.3.2
SVG --- 5.1.1
RP/0/RSP0/CPU0:A9K-BNG#show nv sat stat

IPv4 address: 192.168.0.100 (VRF: default)
RP/0/RSP0/CPU0:A9K-BNG#telnet 192.168.0.100
Trying 192.168.0.100(192.168.0.100)...
Connected to 192.168.0.100.
LC:Satellite#show ver
Version 15.1(3)SVA, RELEASE SOFTWARE (fc1)
Please use following steps to upgrade the image on satellite incase the image on
satellite is not latest: (install nv satellite)
Connect the satellite in hub and spoke mode on the host.

Transfer and activate the 9000v/901 image to the satellite.
Check the output of show nv sat status to check the satellite displays latest version
Move the satellites back to the new 511 topologies as required
BRKSPG-2904
Cisco Public
97

Satellite Stuck in Discovering state(for all topologies)
Satellite state shown as Discovery halted; Conflict: no Identification received yet
Status: Probing for Satellites; Conflict: no Identification received yet state means that
the Host is sending DPM probes to satellite and waiting for a response. The satellite is
either not responding to the discovery probes from the host or the host drops the
discovery replies from the satellite.
BRKSPG-2904
Cisco Public
98
Discovery Protocol Troubleshooting
Contd.
If Host is not sending the discovery probes or if the host is dropping the discovery
replies:
Check show int <ICL> output on host, input/output counter would show SDAC discovery
input/output packets.
Check the NP drop counters of the ICL port.
If IN_SATELLITE_DISCOVERY_DISCARD NP counter is increasing:
Check if icl bit is programmed in uidb for icl interface
show uidb data location 0/0/CPU0 tenGigE 0/0/1/0 ingress
Satellite IC interface
0x1 <<<
If the ICL bit is not set in UIDB, we need to check the vlan ea db
sh ethernet infra internal ea trunks tenGigE 0/14/0/3 location 0/14/cPU0 --> main interfaces
sh ethernet infra internal ea subs tenGigE 0/14/0/3.1 location 0/14/cPU0 --> for icl which is a sub interface
(incase of L2 Fab)
is_in_icl_mode: 1
If the bit is not set in vlan ea then we need to check the vlan ma
sh ethernet infra internal ether-ma trunks tenGigE 0/14/0/3 location 0/14/cPU0 (use subs for sub interfaces)
is_in_icl_mode: 1
BRKSPG-2904
Cisco Public
99

If Host is sending the discovery probes and not receiving the discovery replies
Check the ICL counter on the satellite(sh interface Tengige 1/45-48), if there are input
counters but no output counters then the satellite is dropping the discovery packets. Check the
BCM counters on the ICL link to check for drops:
login to bcm shell by the below command
LC:Satellite#test bcm shell ( for 9000v)
show c xe0 where xe0 -xe3 is tengige 1/45 to tengige 1/48
LC:Satellite#test platform bcm shell for (901)
Show c ge2 or ge3 ,where ge2 is gig 0/10 and ge3 is gi 0/11
BCM.0> show c xe0
RUC.xe0 : 87 +30 5/s
RDBGC0.xe0 : 2 +1 -------------> RDBGC0 counter- this counter represents drops
RDBGC2.xe0 : 15 +4
BRKSPG-2904
Cisco Public
100

511 L2 Fabric Satellite stuck in Discovery State:
Only p2p l2 fab circuits are supported for l2 fab connections.
If you have bridge domains/vpls/trunk ports in the l2fab cloud, convert them to p2p xconnects
or VCs
Trace the discovery probe packets from the host over the l2 cloud with each hop in the l2
cloud. Check the node where the packets are getting dropped and follow the usual triage
process for packet drops.
BRKSPG-2904
Cisco Public
101
Discovery Protocol Troubleshooting contd.

Satellite stuck in configuring state for Dual Head/Ring Topology:
Check if MPLS LDP is up between the 2 hosts

Check if ICCP is up between the 2 hosts (show iccp group)
Check if ORBIT is up between the 2 hosts (show nv satellite protocol redundancy)
IF LDP or ICCP is not up, follow the usual triaging steps for ASR9K.
If ICCP is up and ORBIT is down then show tech satellite will help to debug this issue further.
Host A
E-ICCP
Satellite
Host B
BRKSPG-2904
Cisco Public
102
Control Protocol Troubleshooting

Control Protocol States
Connecting: The Control protocol is trying to connect to the TCP Socket. show tcp on both
the host and Satellite should give more info on this.
Authenticating: The Control Protocol has sent an authentication message and is waiting for a
response from the satellite.
Checking ver: The control protocol has sent a version request to the satellite and is waiting for
a response.
Connected: The control protocol is up
Satellite stuck in Connecting state:
Check if ping works from the host to the satellite IP address.

In case of Dual host/Ring topology, check if you have same ipv4 address or ipv4 unnumbered
loopback address configured for the ICL on both hosts. The loopback address for the ICLs on
both hosts should be different.
Check if TCP session is UP. show tcp brief should show the state as ESTAB
BRKSPG-2904
Cisco Public
103
Satellite image upgrade troubleshooting

Image upgrade consists of following steps:
Image transfer: The satellite binaries are copied from host to satellite.
Install nv satellite <sat-id> transfer can be used to initiate a transfer
Image transfer generally takes 3-5 mins for 9000v and 4-6 mins for 901
Following logs appear on the console once image transfer is done:
RP/0/RSP0/CPU0:dor1101eiuc202#install nv satellite 9002 transfer progress
Mon Dec 2 16:32:08.652 CET Install Op 6: transfer: 90021 configured satellite has been specified for
transfer.
1 satellite has successfully initiated transfer.
Press Ctrl+C at any time to stop displaying the progress bar.
| Working...
RP/0/RSP0/CPU0:Dec 2 16:36:05.676 : icpe_satmgr[1164]: %PKT_INFRA-ICPE_GCO-6TRANSFER_DONE : Image transfer completed on Satellite 9002 Completed.
Image activation: The satellite is booted with the transferred binary.

For Host Upgrade from pre 511 image to 511 image, activation for 9000v is a 2 step process and would
cause the 9000v to reload twice(once with an intermediate image and second time with the 5.1.1 image)
BRKSPG-2904
Cisco Public
104
Satellite image upgrade troubleshooting contd.

image transfer to the 9000v or 901 satellite fails
Check if the router has MPP configured. If yes, then add an exception for tftp under the ICL
Links
control-plane
management-plane
inband
interface TenGigE0/1/0/1
allow TFTP
Check if the router has tftp homedir configured on the ASR9000 host.
tftp vrf default ipv4 server homedir disk0:
If the tftp transfer requests from satellite comes on the default vrf [through manual IP configuration] and tftp
home directory configured on the host is disk0: then image transfer request will fail as the tftp_fs will try to
read the disk0:/ path.
Please remove the tftp homedir and retry the transfer.
BRKSPG-2904
Cisco Public
105
Satellite image upgrade troubleshooting contd.

Check whether the tftp service is configured or not using following cli.
Show cinetd services
RP/0/RSP0/CPU0:R3#show cinetd services
Vrf Name Family Service Proto Port ACL
default v4
telnet tcp 23
10
default
v4
tftp
udp
69
unlimited
max_cnt curr_cnt wait Program Client

0
nowait telnet sysdb
wait
tftpd icpe-cpm /pkg/fpd-nv/
Option
<<<<<<<<<<<<<<<< ICPE
started the service

Try process restart icpe_cpm incase tftp service is not running for icpe-cpm
ASR901 Specific image transfers issues/Checks

If the ASR901 already have a standalone non nV image, image transfer may fail due to lack of
space on the 901. Please delete the non-nV image from the 901 flash and execute "squeeze
flash:" to free up space.
For ASR901 the image transfer may take sufficiently more time sometimes than normal, if the
flash does not have enough space, the squeeze will automatically kick in when image transfer
is attempted, this operation may take close to 30 mins.
BRKSPG-2904
Cisco Public
106
Ping/Packet drop troubleshooting with satellite

Check the nv status(show nv satellite status). The satellite should be in connected
state.
Check the arp table of the destination IP Address. If ARP is not resolved:
Check if cross-links are formed correctly on the satellite and interfaces are up on satellite.
LC:Satellite#sh satellite crosslink tenGigabitEthernet 1/45
Interconnect Link: TenGigabitEthernet1 /45 xos_if: 5
icl_id: 1 host_id: 1 cp_vlan: 0
Link State: Up SDAC State: Discovered Crosslink State: Up
--------------------------------------------------------------Access Port ICLID Link State ForcedDown ? TxDisabled ?
--------------------------------------------------------------Gi1/1 1 Up No No
Gi1/2 1 Up No No
Gi1/3 1 Up No No
Gi1/4 1 Up No No
If crosslink map is missed or not expected, please turn on debug sdac control all feature-channel 3/
feature-channel 4(for bundle ICL) for further debugging
If interfaces are showing up on the satellite but down on the host, collect show tech satellite from host
and debug sdac control all feature-channel 1 from the satellite for further debugging
BRKSPG-2904
Cisco Public
107
Ping/Packet drop troubleshooting with satellite contd.

If ARP is resolved correctly then check for packet drops on host as well as satellite:
Send ping packets from the host with packet size 1400. Check for ICL counters to confirm if
the packet went out of the host.
If ping packets are getting dropped on the host. Check the NP drop counter, which is
increasing.
Check ICL stats on Satellite to see if the ping packets are getting received on satellite
If ARP is not resolved on the host check if the ARP is resolved on the other end
connected to satellite. If ARP is resolved on the other end, send ping packets with
size 1400 and trace the packet path same as above on the opposite direction.
If ARP is not resolved on either end, then initiate ping from the host and host will send
ARP requests, check the qos tm counters to see if packets go out of host.
Configure static arp on the host and initiate ping, now the host will send normal ping
packets(not arp packets), check the counters on the ICL link and the satellite as
above.
BRKSPG-2904
Cisco Public
108
Troubleshooting QoS
ASR9k QoS Architecture

End-to-End priority propagation Guarantee bandwidth, low latency for high priority
trafficset per each
One Queue
at any congestion point
NP on the LC
Ingress side of LC
PHY
NP0
PHY
NP1
PHY
NP2
PHY
NP31
Egress side of LC
CPU
Ingress (sub-)interface
QoS Queues
FIA
FIA
Switch
Fabric
Virtual
Output
Queues
Configure with
Ingress MQC 4-layer hierarchy
Two strict high priority + Normal priority
BRKSPG-2904
NP0
CPU
Implicit Configuration
Two strict high priority +
Normal priority
Cisco Public
110
Egress FIA
Queues
PHY
NP1
PHY
NP2
PHY
NP3
PHY
Egress (sub-)interface
QoS Queues
Configure with Egress MQC

4-layer hierarchy
Two strict high priority + Normal priority

Fabric Bandwidth Access Overview
3 strict priority scheduling/queueing
Back pressure and virtual output queue
RSP0
Crossbar
Fabric
ASIC
1: Fabric Request
Crossbar
Fabric
ASIC
Ingress LC
Arbiter
FIA
2: Arbitration
Crossbar
Fabric
ASIC
3: Fabric Grant
Crossbar
Fabric
ASIC
4: load-balanced
transmission across
fabric links
Arbiter
RSP1
BRKSPG-2904
Cisco Public
111
Multicast and Unicast separation (separated

queues and fabric plane)
5: credit return
Egress LC
FIA
Arbitration & Fabric QoS

Arbitration is being performed by a central high speed arbitration ASIC on the
RSP
At any time a single arbiter is responsible for arbitration (active/active APS like
protection)
The Arbitration algorithm is QoS aware and will ensure that P1 classes have
preference over P2 classes, both of which have preference over non-priority
classes
Fabric QoS is applied at Virtual Output Queue (VoQ) level

Arbitration is performed relative to a given the egress VoQ
BRKSPG-2904
Cisco Public
112
Virtual Queue Identifier (VQI) and

Virtual Output Queue (VOQ)
On Trident LCs, VQI is assigned per NP basis.
NP maps to a single 10GE port or a group of ten 1GE ports
On Typhoon LCs, NP is designed for multiple 10G, 40G, and 100G ports.
Each 10G port is 1:1 mapped to one VQI

Each 40G port is mapped to 8 VQI
Each 100G port is mapped to 16 VQI
VOQs used to load balance across internal connections
One VOQ comprises 4 VQI (Typhoon) or 3 VQI (Trident)
vqi
voq
vqi
BRKSPG-2904
Cisco Public
113

Backpressure and VoQ Mechanism
One VoQ set (4 queues)

per each NP in the system
VOQ congestion on egress NP

backpressure propagated to ingress FIA
Packet is en-queued in the dedicated VOQ
No impact of the packet going to different egress VOQ
No head-of-line-block issue
Backpressure: egress NP egress FIA
fabric Arbiter ingress FIA VoQ
Ingress side of LC1

10Gbps
PHY
5Gbps
NP0
PHY
NP1
PHY
NP2
PHY
NP3
5Gbps
BRKSPG-2904
Egress side of LC2
CPU
CPU
FIA
FIA
Switch
Fabric
Cisco Public
114
NP0
PHY
NP1
PHY
NP2
PHY
NP3
PHY
Default Interface Queues

For every physical port in the system, the following queues get created:
Typhoon
Ingress (NP->Fabric)
Trident
Egress (NP->Line)
Ingress (NP->Fabric)
Egress (NP->Line)
High priority 1 queue

(routing control protocols)
High priority 1 queue (routing

control protocols and critical
traffic like BFD)
High priority queue (routing

control protocols)
High priority queue (routing

control protocols and critical
traffic like BFD)
Medium priority 2 queue

(Unused without policy-map)

Medium priority queue

Medium priority queue (Unused

without policy-map)


Low priority queue (Used by

all other traffic)
Low priority queue (Used by all

other traffic)
Low priority queue (Used by

all other traffic)
Low priority queue (Used by all

other traffic)
Default queue size is 100ms of the line rate

BRKSPG-2904
Cisco Public
115
Default Interface Queues
interface Gig0/0/1/8
RP/0/RSP0/CPU0:A9K#show qoshal default-queue subslot 1 port 8 location 0/0/CPU0

Thu Apr 3 06:00:08.931 UTC
Interface Default Queues : Subslot 1, Port 8
===============================================================
Port 72 NP 1 TM Port 20
Ingress: QID 0x20200 Entity: 1/0/2/4/64/0 Priority: Priority 1 Qdepth: 0
StatIDs: commit/fast_commit/drop: 0x690a00/0x678/0x690a01
Statistics(Pkts/Bytes):
current number of packets
Tx_To_TM 0/0 Fast TX: 58648/12629943
Total Xmt 58648/12629943 Dropped 0/0
in the queue
Ingress: QID 0x20201 Entity: 1/0/2/4/64/1 Priority: Priority 2 Qdepth: 0
<...>
<...>
Egress: QID 0x20220 Entity: 1/0/2/4/68/0 Priority: Priority 1 Qdepth: 0
StatIDs: commit/fast_commit/drop: 0x690aa0/0x67b/0x690aa1
Statistics(Pkts/Bytes):
Tx_To_TM 0/0 Fast TX: 58702/18300724
TX statistics
Total Xmt 58702/18300724 Dropped 0/0
Egress: QID 0x20221 Entity: 1/0/2/4/68/1 Priority: Priority 2 Qdepth: 0
<...>
<...>
BRKSPG-2904
Cisco Public
116
MQC to System QOS mapping

ASR 9000 supports traffic differentiation at all relevant points within the system
P1/P2/LP differentiation or P1/LP differentiation support throughout the system
Classification into these priorities is based on input MQC classification on the

ingress linecard into P1, P2, Other
Once a packet is classified into a P1 class on ingress it will get mapped to PQ1 queue
along the system qos path
Once a packet is classified into a P2 class on ingress it will get mapped to PQ2 queue
along the system qos path, unless no MP is implemented. In this case HP would be
used for P2.
Once a packet is classified into a non-PQ1/2 class on ingress it will get mapped to LP
queue along the system qos path
Note: The marking is implicit once you assign a packet into a given queue on
ingress; its sets the fabric header priority bits onto the packet.
no specific set action is required
BRKSPG-2904
Cisco Public
117
Feature order on ASR 9000 NP (simplified)

TCAM
Ingress linecard
From wire
IFIB action
I/F
classification
ACL
classification
QOS
classification
Fwd lookup
QoS action
L2 rewrite
ACL action
IFIB lookup
To fabric
From fabric
egress linecard
ACL action
QoS action
BRKSPG-2904
QOS
classification
ACL
classification
To wire
Cisco Public
118
L2 rewrite
Fwd lookup
Feature order on ASR 9000 NP: QoS Action Order

Ingress linecard
WRED classifies on marked/remarked
values (doesnt switch class-maps!)
ACL
QOS
I/F
From wire
Fwd lookup
IFIB action
classification
classification
classification
QoS action
L2 rewrite
ACL action
To fabric
IFIB lookup
QoS Action From fabric

egress linecard
ACL action
QOS
classification
Police
QoS action
BRKSPG-2904
ACL
classification
Mark
To wire
Cisco Public
119
L2Queue/sha
rewrite
pe/WRED
Fwd lookup
Injected packets
In general are injected to-wire (same as Pak Priority in IOS)
Means that all features are bypassed.
Including QOS
Few exceptions
ICMP
Netflow
BRKSPG-2904
Cisco Public
120
ASR 9000 QOS Implicit Trust

For Bridged packets on ingress outermost COS would be treated as trusted.
For Routed packets on ingress DSCP/Precedence/outermost EXP would be
treated as trusted based on packet type.
Default QOS will be gleaned from ingress interface before QOS marking is
applied on the ingress policymap.
By default ASR 9000 would never modify DSCP/IP precedence of a packet
without a policy-map configured.
Default QOS information would be used for impositioned fields only
BRKSPG-2904
Cisco Public
121
Typhoon QoS Overview

Super-set of existing Trident linecard QoS functionality
Dedicated TM for queuing
Fabric/internal QoS mechanism
Flexible 4-level H-qos ingress and egress
Higher scale
Higher queue and policer scale
More granular bandwidth control for both policing and queuing
Higher buffer size
Additional new feature capability

Conform-aware policer (a/k/a Coupled Policer)
4 strict priority: P1, P2, P3 (egress only) and normal priority
P3 is egress only;
Ingress TM for <=30G configs only
No input shaping on high-NP loading configs (36x10G, 8x10 MPA, 40G MPA)
By default all queue-limits are set to 100ms of service rate

BRKSPG-2904
Cisco Public
122
Increased Priority Queues

Trident Max of 8 Child Queues per parent , with 1 Priority 1, 1 Priority 2, and 6
Normal-priority queues (including class-default)
Typhoon Max 8 Child Queues per Parent Choices based on user config in
policy.
1 Priority 1, 2 Priority 2 and 5 Normal-priority
1 Priority 1, 1 Priority 2, 1 Priority 3, 5 Normal-Priority (Egress only)
1 Priority 1, 1 Priority 2, and 6 Normal-priority
BRKSPG-2904
Cisco Public
123
Typhoon QoS Overview

Super-set of existing Trident linecard QoS functionality
Dedicated TM for queuing
Fabric/internal QoS mechanism
Flexible 4-level H-qos ingress and egress
Higher scale
Higher queue and policer scale
More granular bandwidth control for both policing and queuing
Higher buffer size
Additional new feature capability

Conform-aware policer (a/k/a Coupled Policer)
4 strict priority: P1, P2, P3 (egress only) and normal priority
P3 is egress only; need special consideration in case of parent shaper
Ingress TM for <=30G configs only

No input shaping on high-NP loading configs (36x10G, 8x10 MPA, 40G MPA)
By default all queue-limits are set to 100ms of service rate

BRKSPG-2904
Cisco Public
124
Typhoon vs Trident Priority Queues

Trident Max of 8 Child Queues per parent , with 1 Priority 1, 1 Priority 2, and 6
Normal-priority queues (including class-default)
Typhoon Max 8 Child Queues per Parent Choices based on user config in
policy.
1 Priority 1, 2 Priority 2 and 5 Normal-priority
1 Priority 1, 1 Priority 2, 1 Priority 3, 5 Normal-Priority (Egress only)
1 Priority 1, 1 Priority 2, and 6 Normal-priority
BRKSPG-2904
Cisco Public
125
ASR9K QoS Classification Criteria

Very flexible L2/L3 field classification on L2 interfaces
Inner/outer cos
Inner/Outer vlan *
DEI*
Outer EXP
Dscp/Tos
TTL, TCP flags, source/destination L4 ports
Protocol
Source/Destination IPv4
Source/Destination MAC address*
Discard-class
Qos-group
match all/match any
Note:
Not all fields are supported on L3 interfaces*
Some fields dont make sense on ingress (e.g. discard-class, qos-group)
MPLS classification is based on EXP only (note in 530 we will be able to apply QOS matching (tentative) and
ACL matching on MPLS labeled packets)
BRKSPG-2904
Cisco Public
126
ASR9K QoS - Classification Formats

Per Policy-map a given classification format is chosen by SW, i.e a given policy-map can
only classify based on a single format
Fields
supported
BRKSPG-2904
Format 0
Format 1
Format 2
Format 3
IPV4 source address

(Specific/Range)
IPV4 Destination address
(Specific/Range)
IPV4 protocol
IP DSCP / TOS / Precedence
IPV4 TTL
IPV4 Source port
(Specific/Range)
IPV4 Destination port
(Specific/Range)
TCP Flags
QOS-group (output policy only)
Discard-class (output-policy
only)
Outer
VLAN/COS/DEI
Inner VLAN/COS
IPV4 Source
address
(Specific/Range)
IP DSCP / TOS /
Precedence
QOS-group (output
policy only)
Discard-class
(output policy only)
Outer
VLAN/COS/DEI
Inner VLAN/COS
IPV4 Destination
address
(Specific/Range)
IP DSCP / TOS /
Precedence
QOS-group (output
policy only)
Discard-class
Outer
VLAN/COS/DEI
Inner VLAN/COS
MAC Destination
address
MAC source
address
QOS-group (output
policy only)
Discard-class
Cisco Public
127
ASR9K QoS - Packet marking details

settable packet fields:
dscp/precedence
EXP imposition
EXP topmost
cos inner/outer
qos-group
discard-class
ASR9K supports maximum of 2 fields per class-map. The same 2 fields can be
placed in any combination below
- 2 sets per police-conform/exceed/violate
- 2 sets without policing.

Note: In MPLS context only EXP marking is supported
Remember that mpls encapped packets cant match on L3 criteria (same for ACL)
(note in 530 we will be able to apply QOS matching (tentative) and security ACL on MPLS labeled packets)
BRKSPG-2904
Cisco Public
128
ASR9K QoS - Policing details

RFC 2698 supported (2r3c) and 1r2c
Ingress & egress policing supported
General Rule: Policing required on priority queues.
Priority level 2 classes can also accept shaping instead of policing.
Granularity of 8Kbps supported (typhoon, 64k on trident)

2-level nested policy maps supported
Note: policers at parent and child work independently
64k policers per NP (shared for ingress/egress) on extended linecards

Policer actions supported:
Policy-map parent
Class class-default
Police rate 10 Mbps peak-rate 20 mbps
conform-action set dscp af12
conform-action set cos 2
exceed-action set dscp af13
exceed-action set cos 3
transmit
drop
set (implicitly behaves like set and transmit)

each color can have two set actions:
BRKSPG-2904
Cisco Public
129
Conform Aware Policer

Normal Hierarchical Policer
Conform Aware Policer
At parent level, if its over the CIR,

packet will be dropped randomly.
There is no awareness which packet
to be dropped
Parent CIR > aggregated child CIR

Parent police only support 1R2C,
child police support all: 1R2C,
2R3C, or 1R3C
policy-map child
class class1
police rate 20 mbps peak-rate 50 mbps
class class2
police rate 30 mbps peak-rate 60 mbps
If drops happen at parent level, it

will drop child out-of-profile packet,
but guarantee the child in-profile
packet
policy-map parent
class class-default
service-policy child
police rate 60 mbps
child-conform-aware
BRKSPG-2904
Cisco Public
130
Common Policer problems

Note that all L2 headers are included, added to the payload and that packet size
is depleting the token bucket (applies to shaping also). Only IFG is not
accounted for (crc is)
Incorrect burst size configuration, allow for some excess burst to catch up.
Mistake between 2 or 3 rate policers (exceed action drop)
Tridents policer cant go negative, Typhoon can borrow
This means that policer behavior is slightly different between the 2 hardware
BRKSPG-2904
Cisco Public
131
ASR 9000 QoS - Queue scheduling

Use shape for a shaped PIR for a graceful enforcement of a maximum bandwidth
shaping at all configurable levels
Min. granularity: 64kbps (L3, L4, 256kbps for L2)
priority levels: priority level 1/2/3, minBw/CIR and Bw remaining
Use bandwidth (minBw) for a CIR guarantee relative to the parent hierarchy level
Min. RATE: 64kbps (8k granularity)
Use bandwidth remaining ratio/percent for the redistribution of excess bandwidth

that is available after PQ classes have been scheduled
configurable ratio values 1-1020
Two parameter scheduler support at class level and subscriber group level (L4, L2):
Shape & BwR (ratio / percent)
Shape & MinBw (absolute / percent)
Not supported: BwR & MinBw on the same class
BRKSPG-2904
Cisco Public
132
ASR 9000 QoS - congestion management/buffering

details
WRED based on: DSCP, IPP, EXP, COS, discard-class
default queue-limit -to prevent buffer exhaustion- is 100ms of service rate
(service rate is the sum of guaranteed bw/bwr assigned to a class)
WRED configuration unit options are: bytes, kbytes, mbytes, us, ms, packets
These values will be rounded up to a set of pre-defined profiles ranging from 8 kB to
262144 kB
The actual implementation uses 512 byte buffer particles
Novelty: ASR 9000 supports WRED on shaped PQ2 classes.

Can be used for differentiation of two kinds of priority within the PQ2 class
BRKSPG-2904
Cisco Public
133
Absolute vs Percentage
All relevant policy actions support both, absolute and percentage based
configuration:
shape
bandwidth
Police
bandwidth remaining*
For tri-rate Copper SFPs (10/100/1000) percentage based QOS will be adjusted
automatically based on the selected rate
*Note: Bandwidth remaining supports ratio/percent, not absolute bandwidth

BRKSPG-2904
Cisco Public
134
Show/debug QOS commands

show running-config
show running-config policy-map <policyname>
Policy map configuration
show running-config class-map <classmap>
Class map configuration
show running-config interface <interface>
Interface running configuration
show policy-map interface <interface> [input | output]
Policy-map statistics on a particular non-bundle interface
show policy-map interface <bundle-interface> [input|output]

member
Policy-map statistics on a member of bundle interface
show qos interface <interface> <input|output> [member

<interface>]
Displays hardware and software configured values of each

class for a service-policy on an interface
show qos-ea interface <interface> <input|ouput> [member

<interface>] [detail]
Displays the detailed information of hardware and software

configured paramters in each class of a service-policy on an
interface
show qos summary <police|policy|queue> [interface <interface>]

[output|iNPt] [member <interface>]
Lists the summary of all queues or policers or interfaces for a policy
show qoshal tm-config

<all|counters|fcu|general|priority|shape|topology|wfq|wred> np
<np> tm <tm>
Displays generic NP TM config
show qoshal <wfq|wred|wred-scale|shape|police|police-node> np

<np> tm <tm> level <level> profile <profile> <num-of-profiles>
[hw|sw]
BRKSPG-2904
Cisco Public
Displays various profiles configured in sw and hw and the values of

each profile
135
Show/debug QOS commands - contd

show qoshal default-queue subslot <n> port <m> location
<location>
Displays the default-queue information
show qoshal resource summary [np <np>]
Displays the summary of all the resources used in hardware and

software for QoS such number of policy instances, queues, profiles
show qoshal fcu <limits|status|profile>
Displays all Traffic Manager (TM) Flow control related info
show qoshal ha chkpt <all|<chkpt-tbl-name> {all|<recid>|info}
Display HA related info for PRM QoS HAL
show qos-ea ha state
Displays the HA State of process QoS EA whether it can accept the

service-policies
show qos-ea ha chkpt <all|<chkpt-tbl-name> {all|<recid>|info}
Display HA Chkpt related info for all the chkpt tables for QoS EA
show qos-ea trace {all|errors|events|internal}
Displays the trace of errors or events or internal events of QoS EA

process
show prm server trace hal
Displays all the trace info of PRM QoS HAL thread
debug qos-ea all
Debug commands for qos ea process
debug qoshal <level|module|events> <word>
Debug commands for PRM qos HAL
debug prm server hal <all|error|events>
Debug commands for PRM qos HAL API
BRKSPG-2904
Cisco Public
136
Troubleshooting: What is programmed in HW?

RP/0/RSP0/CPU0:WEST-PE-ASR9K-2#sh qos interface g0/0/0/0 output
Fri Jan 10 15:05:40.347 EST
Interface: GigabitEthernet0_0_0_0 output
policy-map p-map
Bandwidth configured: 500000 kbps Bandwidth programed: 500000 kbps
class class-default
ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps
service-policy dummy
Port Shaper programed in HW: 500000 kbps
shape average 500 mbps
Policy: p-map Total number of classes: 2
!
---------------------------------------------------------------------end-policy-map
Level: 0 Policy: p-map Class: class-default
!
QueueID: N/A
Shape Profile: 1 CIR: 64 kbps CBS: 10240 bytes PIR: 499968 kbps PBS: 6291456 bytes
WFQ Profile: 4 Committed Weight: 1 Excess Weight: 1
Bandwidth: 0 kbps, BW sum for Level 0: 0 kbps, Excess Ratio: 1
policy-map dummy
---------------------------------------------------------------------class class-default
Level: 1 Policy: dummy Class: class-default
bandwidth percent 100
Parent Policy: p-map Class: class-default
!
QueueID: 642 (Priority Normal)
end-policy-map
Queue Limit: 8388 kbytes Profile: 2 Scale Profile: 2
!
---------------------------------------------------------------------BRKSPG-2904
Cisco Public
137
Troubleshooting: What is programmed in HW?

RP/0/RSP0/CPU0:WEST-PE-ASR9K-2#sh qos interface g0/0/0/0 output
Fri Jan 10 15:05:40.347 EST
Interface: GigabitEthernet0_0_0_0 output
policy-map p-map
Bandwidth configured: 500000 kbps Bandwidth programed: 500000 kbps
class class-default
ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps
Port Shaper programed in HW: 500000 kbps
p-map
Rate Total
is rounded
the nearest
8k or 64k value
Policy:
number to
of classes:
2
!
--------------------------------------------------------------------- Shape sets PIR
end-policy-map
Level: 0PBS
Policy:isp-map
Class:
default
rateclass-default
of 100msec of configured shape rate
!
QueueID: N/A
Parent
BW
is
zero
or
64k,
only
applicable
in
oversubscription
at
sum
of parent levels
Shape Profile: 1 CIR: 64 kbps CBS: 10240 bytes PIR: 499968 kbps PBS: 6291456 bytes
policy-map dummy
---------------------------------------------------------------------class class-default
Level: 1 Policy: dummy Class: class-default
Parent Policy: p-map Class: class-default
!
QueueID: 642 (Priority Normal)
end-policy-map
Queue Limit: 8388 kbytes Profile: 2 Scale Profile: 2
!
---------------------------------------------------------------------BRKSPG-2904
Cisco Public
138
Troubleshooting: Policy-Map Statistics

RP/0/RSP0/CPU0:A9K#sh policy-map interface g0/0/1/8 output
Thu Apr 3 06:47:56.728 UTC
GigabitEthernet0/0/1/8 output: p-map-1
Class class-default
Classification statistics
(packets/bytes) (rate - kbps)
Matched
:
2/116
0
Transmitted
:
4/232
0
Total Dropped
:
0/0
0
Policy dummy Class class-default
Classification statistics
(packets/bytes) (rate - kbps)
Matched
:
2/116
0
Transmitted
:
4/232
0
Total Dropped
:
0/0
0
Queueing statistics
Queue ID
: 131626
High watermark
: N/A
Inst-queue-len (packets)
:0
Avg-queue-len
: N/A
Taildropped(packets/bytes)
: 0/0
Queue(conform)
:
4/232
0
Queue(exceed)
:
0/0
0
RED random drops(packets/bytes)
: 0/0
BRKSPG-2904
Cisco Public
139
policy-map p-map
class class-default
!
end-policy-map
!
policy-map dummy
class class-default
!
end-policy-map
!
Troubleshooting Back-pressure Issues

Look for FIA drops
RP/0/RSP1/CPU0:ios#show drops
Tue Jan 14 20:44:25.360 EST
Node: 0/0/CPU0:
<>
FIA 0 Drops:
---------------------------------------------------------------Ingress Drops
287078960
Egress Drops
1
Total Drops
287078961
Ingress Generic Hard Drop-2
287078960
Egress Mcast RxFab Hdr-1
1
----------------------------------------------------------------
Check if any VQI is dropping packet

RP/0/RSP1/CPU0:ios#show controller fabric fia q-depth location 0/0/CPU0
FIA 0
VoQ
| ddr | pri | pkt_cnt
------+-----+-----+--------23
| 0
| 2
| 118
Total Pkt queue depth count = 118
BRKSPG-2904
Packets in the queue. Not good.

Cisco Public
140
Troubleshooting Back-pressure Issues

Look for FIA drops
RP/0/RSP1/CPU0:ios#show controllers pm loc 0/5/CPU0 | I ^Ifname|switch_fabric_port

Ifname(1): TenGigE0_5_0_0, ifh: 0xe000100 :
switch_fabric_port 0x17 VQI 23 is for interface ten0/5/0/0
Look for egress NP TM Drops:

RP/0/RSP1/CPU0:ios#show controllers NP tm counters all location 0/5/CPU0
Node: 0/5/CPU0:
==== TM Counters (NP 3 TM 1) ====
TM Counters: commit_xmt_paks: 1509333316
excess_xmt_paks: 67641555690
Total Transmitted paks: 69150889006
wred_drop paks: 2441836834 timeout_drop 0 intf_drop 0
==== TM Counters (NP 3 TM 2) ====
TM Counters: commit_xmt_paks: 0
excess_xmt_paks: 0
Total Transmitted paks: 0
wred_drop paks: 0 timeout_drop 0 intf_drop 0
BRKSPG-2904
Cisco Public
141
Shaping with PIR/PBS and CIR

Shaper peaks to linerate for pbs time
Should allow some burst to get to PIR faster
CIR is ignored, will result in queue(exceed) counts, but they dont mean drops!
linerate
PBS
PIR
CIR
RP/0/RSP0/CPU0:A9K-BNG#show policy-map int g 0/0/0/0 | i Queue
Queueing statistics
Queue ID
Queue(conform)
Queue(exceed)
BRKSPG-2904
Cisco Public
142
:
:
: 136
0/0
0/0
0
0
QOS summary
All Ethernet linecards support Queuing, Marking and Policing.
Some high speed linecards do not support ingress Queuing (but support policing
and marking).
Because their ingress TM (Traffic Manager) is disabled
To guarantee priority end to end, make sure high priority traffic is marked on
ingress (This will not burn a queue)
https://supportforums.cisco.com/docs/DOC-15592
BRKSPG-2904
Cisco Public
143
Usability, XR strategy and SMUs
Feedback on XR and actions

The XR drag coefficient, too many SMUs, no maintenance releases
SW quality improvements
Install and manageability improvements in XR4.3
Better install handling

Better smu handling
Disk mirror separation
Auto FPD reducing number of reloads between upgrades
SMU reduction effort and SMU management
Better quality
Better identification of what should be smud
Cisco Software Manager (in PRIME now too). V2 just released
Better DDTS info
SW quality with EMR (extended Maintenance releases)

XR 4.3.4
XR 5.1.3
BRKSPG-2904
Cisco Public
145
Smart Reload
Minimizing Upgrade Time
4.2.1
Benefits
Faster image upgrades
Faster reload SMU installation and activation
Significantly reduced traffic loss due to
unplanned events (e.g. power outages)
Reduces maintenance window duration
Supports any-to-any image upgrade
Improved TFTP image download time
Improved traffic convergence time by
optimizing prefix download and programming
time.
Early availability of IOS-XR prompt, Line
Card services and resources
BRKSPG-2904
Cisco Public
146
34 Minutes
4.3.0
21 Minutes
4.3.1
19 Minutes
5.1.1
11 Minutes
XR Software Manager (CSM)

Highlights
Automatic analysis and optimization of router software
Periodically consult cisco.com for SMU info and updates
Secure, scalable, platforms: Win, Mac, VM
GUI; written in Java; runs as standalone tool
FREE and Public to ALL IOS XR customers
Machine to Machine
BRKSPG-2904
Cisco Public
147
XR Software Manager (CSM)

Benefits
Time saved: No more searching/querying Cisco.com for SMUs
Cost savings by avoiding incorrect and missing SMUs

New SMU Alerts: Similar to iPhone software upgrades
Analyze and recommend your device specific SMUs
Machine to Machine
BRKSPG-2904
Cisco Public
148
XR Software Manager
Next Enhancements
SMU ETA visibility
Auto SW download(BSD)
PDF format conformance report
SMU Size Calculation

Service Pack Support
Machine to Machine
BRKSPG-2904
Cisco Public
149
Introducing ISSU SMUs!

Patching got even better!
SMU Software Maintenance Unit
ISSU SMU
Differentiation
Industry unique ability to patch without a full

upgrade
Non Reload SMUs patching on the fly
Reload SMUs Need a router reload
Previously 50 50 split between Non reload
and reload SMUs
Up to 60% of
reload SMUs can
be converted to
ISSU SMUs
Reduce packet
loss to <60 sec**
from 20-30
Use ISSU infra to convert reload SMUs to

ISSU SMUs
Non Reload :50%; ISSU SMU: 30%; Reload
SMU: 20%
BRKSPG-2904
Cisco Public
**Packet loss time will vary with LC and scale on LC
Reload SMU reduction (4.3.x)

How Do we do this ?.
NP fast reset feature to avoid LC reload for ucode code upgrade.
Every effort is made to build ISSU SMU instead of reload SMU.
Reload SMUs will follow stringent rules before the decision is made.
How do we Measure ?
Current 4.3.2 reload SMUs stands at 0%
Goal: Reload SMUs cannot be more then 20%
BRKSPG-2904
Cisco Public
151
XR Service Packs
Minimizing System Reloads
Service Packs on XR provide packaging of SMUs, and
reduce the number of reboots for software updates
between releases
Benefits
Lowers OpEx & Reduce TCO

Simplify Operations by tracking less SMUs
Reduce downtime & cost of system
upgrades
Highlights
Bundle multiple SMUs into a single package

Combine multiple reload SMUs to reduce
reboots
Critical SMUs are still available external to
Service Packs
BRKSPG-2904
Cisco Public
152
IPV6-Etherconsole:
Customers dont want to use serial (RS232) console.
RS232 to Ethernet conversion using External Dongle (e.g. Raspberry Pi) is not a viable
solution.
Avoid the need to use the serial (RS232) console by ensuring that the router will always
be accessible over IPV6 via Management Ethernet interfaces.
Hardware Solution:
Dedicated physical port in next-Gen RSP front Panel
Goal is to achieve true emulation of console port
Software Solution:
Use front Panel Management Ethernet port for console emulation

Requires ROMMON variable IPV6-ETHERCONSOLE
All tasks that can be done in Rommon CLI can be done in XR CLI
Not true Console emulation.
Router recovery, Password recovery, Rommon access, TFTP boot cannot be done with ether-console and still
requires console connection
BRKSPG-2904
Cisco Public
153
FlexCli: config apply-groups

router isis green
interface GigabitEthernet0/0/0/0
lsp-interval 20
hello-interval 40
metric 10
!
lsp-interval 20
hello-interval 40
metric 10
!
lsp-interval 20
hello-interval 40
metric 10
!
lsp-interval 20
hello-interval 40
metric 10
!
BRKSPG-2904
group G-ISIS-INTERFACE
router isis .*
interface Gig.*
lsp-interval 20
hello-interval 40
metric 10
!
config
router isis green
apply-group G-ISIS-INTERFACE
Apply-group command matches router isis green

config in running-config and applies configs
to specific interfaces
Cisco Public
154
Accelerated Upgrade
Python based Auto upgrade tool
Architecture support Plugins to expand and
customize to any platform
Ordered list of plugins

Pre upgrade plugins
Upgrade plugins
Post upgrade plugins
Can set order of dependency between plugins

On an upgrade failure box will be auto reverted
& restored
Availability: NOW
BRKSPG-2904
Cisco Public
155
Accelerated Upgrade cont..

Where & how can I use AU?
Operators that perform 10 upgrades a night can
now do 15 or 20
Chances of upgrade failures due to human or
system error reduced
Great tool for customers & labs
Feature Highlights
Supported on all XR platforms
Can perform install upgrade or turboboot
Its open sourced and official, anywan can modify
script as needed
Customer can share AU with Cisco or community
Availability: NOW
156
BRKSPG-2904
Cisco Public
Where do I learn more?
Great video by Kashish Golani

https://www.youtube.com/watch?v=2DgXPi0Ink4
Open source AU is now on SourceForge
http://sourceforge.net/projects/acceleratedupgrade/
TechZone article
Questions? <accelerated-upgrade-support@cisco.com>
Roadmap highlights
Integrate AU with CSM
Add support for NG XR
Simple script to support reading config file
Support setting ROMMON variable from file for
turboboot
Availability: NOW
157
BRKSPG-2904
Cisco Public
Accelerated Upgrade
Syntax
DEMO
Pre-Upgrade Check
./accelerated_upgrade -l cisco -p cisco -r

ftp://terastream:cisco@172.20.168.195://h
ome/terastream/echami -f pkg.txt -d
172.28.98.3
Upgrade
Plugins
Eddie Chami in Dubai

Router & FTP in SJ
Plugins
172.28.98.3
= ASR9001
172.20.168.195 = FTP Server

Post-Upgrade Check
Plugins pkg.txt package list added

(mgbl, mpls being added)
158
BRKSPG-2904
Cisco Public
Easy Upgrades
Released: 5.1.1
Flex Upgrades on RSP2 system for ease of upgrades and

downgrades
Enables install manager to use Disk0 and Disk1
Goal! Simplifying upgrades
Benefits
Reduce Maintenance window time

Reduce reloads during upgrades
Provides 3.8G of usable space to install process
Easy of backing out of an upgrade
Feature Highlights
Run 5.1.1 from Disk0

Fewer Commands
Avoid User Errors
See 5.1.1 install documentation for more info

BRKSPG-2904
Simplified Upgrade
On RSP2
Cisco Public
Whats next?
Continued focus on Support forums
Read the blog for useful announcements
Q&A
Tech docs
Increased attention to usability
TCP transfer improvements (so necessary )

Eg Bundle-Ether vs BE
RPL functionality (such as test facility, variables)
BGP show enhancements, tweaks and knobs
TE show and debug improvements
Long standings asks from IOS such as
Logging synchronous (auto ctrl-R after syslog message)
interface-range
BRKSPG-2904
Cisco Public
160
Participate in the My Favorite Speaker Contest

Promote Your Favorite Speaker and You Could be a Winner
Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
Send a tweet and include
Your favorite speakers Twitter handle
Two hashtags: #CLUS #MyFavoriteSpeaker
You can submit an entry for more than one of your favorite speakers
Dont forget to follow @CiscoLive and @CiscoPress
View the official rules at http://bit.ly/CLUSwin
BRKSPG-2904
Cisco Public
162
Complete Your Online Session Evaluation

Give us your feedback and you
could win fabulous prizes. Winners
announced daily.
Complete your session evaluation
through the Cisco Live mobile app
or visit one of the interactive kiosks
located throughout the convention
center.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
BRKSPG-2904
Cisco Public
163
Continue Your Education

Demos in the Cisco Campus
Walk-in Self-Paced Labs
Table Topics
Meet the Engineer 1:1 meetings
BRKSPG-2904
Cisco Public
164

ASR9000 Troubleshooting Architecture

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

ASR9000 Troubleshooting Architecture

Încărcat de

Drepturi de autor:

Formate disponibile

ASR9000 Troubleshooting

Xander Thuijs CCIE#6775 Principal Engineer

Troubleshooting Packet Forwarding

Troubleshooting L3 forwarding including Mcast

2014 Cisco and/or its affiliates. All rights reserved.

Santosh Sharma (Satellite)

2014 Cisco and/or its affiliates. All rights reserved.

This applies between the different LC types also

2014 Cisco and/or its affiliates. All rights reserved.

Inside the CPU SW forwarding there are different switching techniques

2014 Cisco and/or its affiliates. All rights reserved.

Software forwarding on classic IOS

4. OSPF doesnt return in time?

We need to turn around this packet quick during

Time in the ISR (interrupt service routine) is

IOS main (void)

How? With all features and info? CEF! Prebuilt L2

CPU utilization for five seconds: 68% / 30%

Total CPU time used

Time spent in ISR

2014 Cisco and/or its affiliates. All rights reserved.

HW timers CPU Complex

Packets not switched in ISR

Ingress NPU handles input features (MAC is part of ingress NPU)

2014 Cisco and/or its affiliates. All rights reserved.

LPTS consists of 3 key portions

Director of where it needs to go to

Policing of the flow categorizations.

2014 Cisco and/or its affiliates. All rights reserved.

Punt Packet Flow Overview

2014 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Packet Forwarding: Fabric

Packet Flow Overview

2014 Cisco and/or its affiliates. All rights reserved.

The Magic Of The Switch Fabric

Access to fabric controlled using central arbitration.

One Arbitration ASIC (Arbiter) per RSP

2014 Cisco and/or its affiliates. All rights reserved.

2014 Cisco and/or its affiliates. All rights reserved.

Fabric Load Sharing Unicast

VOQ (shaped at ~14G)

Each frame (or superframe) contains sequencing information

All destination fabric interface ASIC have re-sequencing logic

Additional re-sequencing latency is measured in nanoseconds

2014 Cisco and/or its affiliates. All rights reserved.

Fabric Load Sharing Multicast

Multicast traffic hashed based on (S,G) info to maintain flow integrity

2014 Cisco and/or its affiliates. All rights reserved.

Flows exit in-order

Fabric Super-framing Mechanism

2014 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Fabric Forwarding

Check fabric drops

2014 Cisco and/or its affiliates. All rights reserved.

Fabric packet counters

2014 Cisco and/or its affiliates. All rights reserved.

Fabric Drop Counters

0 - high priority level 1

0-1 XBAR links to RSP0 (Trident+RSP2)

2014 Cisco and/or its affiliates. All rights reserved.

Fabric drop counters - ingress

Header parsing drp

Ingress drops per link