ASR9000 Multicast troubleshooting Document Cisco Systems ‘ASR9000 Deployment team ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1Table of Contents 1 1.1 Fabric Replicatios 1.2. FlAand Bridge replication:.. 3 Packet flow and debugging commands at high level 4 Testbed topology: jing the mrib, mfib and IGMP info. 5.1 Explanation of the mb and mrib flags 5.2 MGID and FGID programmin, 5.3. Checking multicast program 6 Veri ig in MFIB: 1g hardware mroute/mfib Table 7 Tracing the packet through Fabric and ASICS: su... 7.1 Ingress NP counters: 7.2. Ingress FIA counter 7.3. Egress LC FIA countei 7.4 Egress NP counters. 8 Debugging multicast Process Memory: 9 Debugging Multicast and related processes:..vcs.s 10 Additional Commands: 11 Multicast drops counters in NP: hardware/softwar 12 Checking multicast resource Allocation 2__ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1 Introduction: Purpose of this document is to outline multicast forwarding packet path as well to provide deep dive on troubleshooting procedures which can be handy to troubleshoot multicast, related issues. ASR9Kis highly distributed system, which utilizes simplified but very i ggAs described below, replication is performed at various points to ensure bandwidth is utilized effectively, Unlike 7600, there is no concept of ingress replication in ASR9K. Hence ingress LC doesn’t take part in replication of multicast traffic. Figure 1: Following diagram outlines various replication points. Multicast Packet Replication Switch Fabric and Egress LC Replication Bridge Replication > similar @)NPR ae FArepieaton, single co LCs which receive IGMP he toNP oon jen, based on FID table inch fare B) FIA Replication > replicate single copy to Bridge which has IGMP join, based on MGIO table in FIA ation > replicate copy per receiver based on multicast FIB table 1.1 Fabric Replicat Fabric is responsible for replicating packet to egress LCs interested in receiving multicast traffic. When ingress LC receives multicast packet, NP performs a lookup on (S,G) and forwards the packet to fabric along with fabric header that has MGID and FGID. Fabric performs the replication based on the FGID value in fabric header of the packet. ASR9000 Multicast troubleshooting document, Cisco Systems Inc 3 FGIDs: 10 Slot Chassi FGIDs: 6 Slot Chassis | Sele | ¢l cla12 [sen [rar [tim [| co 5 —_owoTOE000 | x0tz0 cz 4 onmo010000ox0010 Lot 3 ono00t000 0008 : co 2 onmon06100 —ox0004 a RSPr 1 —cono0001 ——_ox0002 tcs 7 oo10000000 ——_ox0080 sro 0 —_coooonn0r —_ox0001 ce © 01000000 ox0ou0 E=rayes E> lem riace I oc Target Linecards | FGID Value (10 Slot Chassis) sri 4 owos010000 —on0o10 108 20100 ta 300001000 00008 re Ss ae LCT lors {10002 | Gx0060 = Ox0062 tet 1 owonnooo10___oxoon2 TOO+LeS+ LET | OxO00% | 0x0008 | Ox0200 = 0x0200 to, «100000001 ___ox000t + FGID (Fabric Group Identifier) is globally significant and is 12 bits wide. + Slotmask used by switch fabric to determine replication to line card/RSP slots + FGID Values for Megatron (ASR 9022) chassis are assigned as follows Fao [o [a [2 [a [a [s [@ [7 [o _[9 _]a0 [aa Slot RPO [RPI Lco [Ler | ucz | wes ica [acs | uce [uc [ice [ice Slot Na [na [ecto [vert [uei2 [ucis_[icis [cis [ucts [eer [cis | ucis * On the 9022 chassis, due to overlapping slots mapped to same bit, fabric will replicate packets to 2 slots. For example, if receiver is on slot 6, fabric will replicate the packets to slot 6 and 16. FIA and Bridge replication: FIA and Bridge uses MGID (multicast Group ID) information in fabric headers for replicating packets to bridge and NP's. MGID is locally significant to LC. This means content of the same MGID in LC2 can be different from LC3. 4 __ASR9000 Multicast troubleshooting document, Cisco Systems Inc MGID's are programmed both in FIA and bridge. MGID in FIA is used to replicate the packet to Bridge asic and MGID in Bridge asic is used to perform replication to interested NPU's. NOTE: Only tridents LCs have Bridge asic. In typhoon LC's, NP is directly connected to FIA. MGID Tables MGID Bitmasks FIA MGID[Bit 1] Bit 0 Bridae1 fuaintenitaroli irmriaqnen foaptaritern]NP3 NP2 NP1 NPO 1.3 NP Re NP replicates the packets to one or more directly connected interfaces from which it receives IGMP joins. * If given NP has more than one outgoing ports, packet will be replicated in TM loopback in the NP. In the second pass all the egress features will be applied. ASR9000 Multicast troubleshooting document, Cisco Systems Inc 5 Multicast Packet Replication Inside NP To egress, From Fabre initial Lookup ~ Original packet is lookup in the MFIB/L2FIB returning the number of copies to be replicated on the given NPU, each copy per output logical interface = Packot Is sent to a TM (traffic manager). TM replicates the packets and send the copy to the processing engine = Second Lookup “Each copy is processed and sent out of interface. Egress feature like QoS, ‘ACLs applied at this pass2 Multicast software Architecture: As described below LC will punt PIM, IGMP control packets to RP CPU. The MRIB process uses route, outgoing interface list information from PIM, IGMP to build mroute table. MRIB is also responsible to send this information to each of the LC CPU's. IfLC receives a packet to a new (S,G), NPU will punt the packet to LC CPU. LC CPU will punt packet to RSP CPU if transit entry for new (S,G) doesn’t exist in LC CPU (entry is also called pre-fib) 6 _ASR9000 Multicast troubleshooting document, Cisco Systems Inc fl = ae 1. Incoming IGMP and PIM packets are punted to RP CPU 2. Protocols (PIM/IGMP) send their Route/OLIST Information to MRIB process to build multicast route/aliet table 3. MRIB sends the multicast state information to MFIB process on all LCs 4. MFIB program HW forwarding tables in NP, Bridge FPGA and Fabric interface ASIC 5. Software switched multicast packet or data packet for protocol signaling is sent to local line card CPU Multicast control packets destined to LC/RP CPU will undergo CoPP (LPTS : Local PacketTransport system) to protect Ki'/LC Ci'U from excessive contro! packets. Following diagram shows policing rate for PIM/IGMP packets. ‘ASR9000 Multicast troubleshooting document, Cisco Systems Inc 7 'RP/0/RSPO/CPUO:asr9k-34show Ipts pifib hardware police loc 0/4/CPUO | in “IGP|PIM” Node 0/4/cPve: ooms for all flow types Policer Type Cur. Rate Def. Rate Accepted PIM-mast-default 109 Static 2000 2000 ° ° PIM-mcast -known 176 Static 2000 2000 ° ° PIM-ucast 1190 Static 1500 1500 ° ° IaH a1 Static 3000 3000 369376 e RP/0/RSPO/CPUO:asrok-3# RP/0/RSPO/CPUO:asr9k-3ishow Ipts pifib hardware entry statistics location 0/5/cPUO 2 Ime TH? Te0/5/0/1LU(32) 251/0 224,0.0.4,any any any 18 ava * PIN Te0/s/0/1 ——_Lv(30) 01/0 224,0.0.13,any ea 236" va * 10 any Lu(20) 6967/0 any any any,any 237 Twat pm any Lu(30) 9/0 any.any any,any det eva * TOP Optimized ——LU(30) 0/0 22410.0.22,any 3 Packet flow and debugging commands at high level: Below diagram shows commands, which can be executed to trace the packet from ingress NPU to egress NPU.i A 8 _ASR9000 Multicast troubleshooting document, Cisco Systems Inc Following diagram outlines set of show commands, which can be executed to trace the packet and to debug if multicast stream is programmed correctly at software and hardware level. L3 Multicast Show CLIs (Sebel ein tie ede Sideenvecie iran arts os 4 Testbed topology:Following testbed topology used for demonstrating troubleshooting section as well to explain packet path, ASR9000 Multicast troubleshooting document, Cisco Systems Inc 9 anno a902 sovasto2) ‘igoraos-102 smerace igabivethernete/4/0/te.201 5 Verifying the mrib, mfib and IGMP info This section walks you through verification of programming of control plane in RP using mrib (Multicast Routing Information base), which is used by MFIB to program the hardware. NOTE: Role of the routers in the above setup: asr9k-3 is the last hop router for the group 225.0.0.1 and 225.0.0.2 Asr9k-2 is the first hop router for the group 225.0.0.1 and 225.0.0.2 + Step 1: Check if there are receivers for the groups on asr9k-3 © Following output shows that 2 receivers are joining group 225.0.0.1 and one receiver is interested in group 225.0.0.2 P70/RSPO/CPUOTASROK-3#shOw Igmp group | include 225. 225.0.0.1 Gigabitetherneto/4/o/3.102 00:28:52 00:01:34 42.0.1. 225.0.0.1 Gigabitetherneto/4/0/10.101 2de7h 33.0.2.)aes eae eee RP/0/RSPO/CPUO:ASROK-3# RP/0/RSPO/CPUO:ASROK-34show igmp groups 225.0.0.1 IG"? Connected Group Membership Group Address Interface Uptime Expires Last Reporter 225.0.0.1 Gigabitetherneto/4/o/3.102 00:29:10 00:01:15 42.0.1.1 225.0.0.1 Gigabitetherneto/4/o/10.101__2de7h__@0:01:11_33.0.2.2 ASR9000 Multicast troubleshooting document, Cisco Systems Inc (ROK 38 'SROK-38Show igmp groups 225.0.0.1 detail Interface: Gigabitetherneto/4/0/3.102 Group: 25.0.0. Uptime: 00:29:16 Router mode: EXCLUDE (Expires: 00:02:09) Host mode: INCLUDE Last reporter: 42.0.1. Source list is enpty Interface: Gigabitetherneto/4/0/10.101 Group: 225.0.0.1 uptime: 2do7h Router mode: EXCLUDE (Expires: 00:02:05) Host mode: INCLUDE Last reporter: 23.0.2.2 Source list is empty RP/0/RSPO/CPUO:ASROK-3# * Step 2: Verify mrib table is programmed properly in control-plane (RSP). The MRIB table is similar to mroute table in IOS. Output from last and first hop router is below: W7OTRSPA/CPUD:asr9k-24show weib route 225.0.0.1 Ip multicast Routing Information Base Entry flags: L - Oomain-Local source, € - External source to the Oonain, C + Directly-connected check, s- signal, IA - inherit accept, Ie - Inherit From, D = Drop, NA - NOT Address, Me = NOT Encap, ND = NOT Decap, MT - HOT Threshold Crossed, m1 - HOT interface handle CD - Conditional Decap, MPLS - MPLS Decap, MF - MPLS Encap, EX - Extranet MoFE - MoFRR Enabled, MoFS ~ MOFRR State, MoFP ~ HOFRR Primary MoFB - MOFRR Backup Interface flags: F = Forward, A - Accept, IC - Internal copy, UNS = Negate Signal, DP - Don't Preserve, SP ~ Signal present, Iz _ Internal Interest, XD” Internal Disinterect, LY ~ Local Interest, LO = Local Disinterest, OT - Decapsulation Interface EI ~ Encapsulation Interface, MI'~ MOT Interface, LVIF - MPLS Encap, &X ~ Extranet, A2~ secondary Accept (7,225.0.0.1) RPF nbr: 110.0.0.24 Flags: Up: 2d22h. Incoming Interface List Decapstunnelo Flags: A, Up outgoing Interface List Gigabitetherneto/4/o/2 Flags: F NS, Up: 2422h 2d22h (149.0.0.2,225.0.9.1) PF nbet 147.0.0.9 Flags: | Up: adosh Incoming Interface List ‘Gigabitetherneto/4/0/10.100 Flag outgoing Interface List Gigabitetherneto/a/o/2 Flags: F ns, up: 2422h & ux fl A /0/8SP1/CPU0: 359K 2% 1A, up: adeah 4s not + Detail output: ([Re7@7RSPO7CPUG: asr9k-3#show mrib route 225.0.0.1 detail ] a aan: Ui ER(7,225.0.0-1) Ver: @x429a RPF nbr: 40.0.75.1 Flags: C, Po! Slotmask: Oxt0 NGrD: 19919 up: 2d2ah. Incoming Interface List TenGiglo/5/0/1 Flags: A NS, Up: 2d21h outgoing Interface List Gigabitetherneto/4/0/3.102 Flags: FNS LI, Up: 1 Gigabitetherneto/4/0/10.101 Flags: F NS LT, Up: 2d2ih 00 (142.0,0.2,225.0.0-1) Ver: 0x7163 RF nbr: 40.0.75.1 Flags:, 'p0: Slotmask: Oxad € used for Fabric Replication ex40 == 0.0010 4 WoID: 19918 © Used by egress Les FIA and Bridge ASIC to replicate the packet ‘Up: 3d0on Incoming Interface List ‘encige0/5/0/1 Flags: A, Up: ad00h € Interface towards source (RPF to source) outgoing Interface List Gigabitetherneto/4/0/3.102 Flags: F NS, Up: 14:20:00 € interface towards Gigabitetherneto/4/0/10.101 Flag R9/0/%SP0/cPU0: 2 0 0 (slot FNS, Up: 2d2h € interface towards sr9k-38 + For the group 225.0.0.2 RP7O/RSPO/CPUOFaSrOK-3#HON wri route 225-0.0.2 detail --+ snip (£,225.0.0.2) Ver: Ox7c9b RPF nbr: 40.0.75.1 Flags: Cy PD: Slotmask: oxao MGID: 19920 up: 2d23h. Incoming Interface List TenGiglo/5/0/1 Flags: A NS, Up: 2d23h outgoing Interface List Gigabitetherneto/4/o/10.101 Flag: FNS LI, Up: 2d23h (142.0,0.2,225.0.0.2) Ver: Ox2Fba RF nbr: 40.0.75.1 Flags:, PO: Slotmask: 0x0 © Same slot mask as 225.0.0.1. Because egress LC is sane. MeIO: 19921, € different McxD. Packets replicated to only one NP. up: 2423h) Incoming Interface List TenGigeo/5/0/1 Flags: A, Up: 2423h ‘outgoing Interface List Gigabitetherneto/4/o/10.101 Flags: Fs, up: 2d23h 1_ASR9000 Multicast troubleshooting document, Cisco Systems Inc5.1 _ Explanation of the mfib and mrib flags Description of the most common flags you see in mfib/mrib commands. + Flag A: © Accept flag is set by control plane to indicate traffic from this interface can be accepted. * FlagF: © Forward - set by the control plane to indicate that data is allowed to be forwarded out this interface * Flag ‘© Negate Signal* - set by the control plane to indicate whether or not a signal should be sent to the control plane when traffic is received on this interface. + Flag Lh: ‘© Local Source. Set on an SG if Router is the PIM RP for that Group and its receiving PIM Registers + Flag Li: © Local Interest. Set when joins are directly connected to this router. See LI flag is not set on asr9k-2 router, which is directly connected to source but not to receivers. © IGMP is interested in this route. Received a Report indicating interest for the group (and source) on that interface 5.2. MGID and FGID programmin, The “show mrib ...” command shows MGID and FGID values used by FIA and Fabric for multicast replication. Let us verify if values programmed are correct. FGID: In the above show command output (show arib route 225.0.0.1 detail) FGID value programmed is 0x40, which is for slot 4 as per below decode. 0 0 0 1 0 0 0 0 0 0 Slot7 | Slot6 [Slot5 | Slot4 [RsP1 [RSP O | slot3 |Slot2 [Slot1 | Slot Hence fabric will replicate traffic to LC 4, which is correct because both the receivers are on Slot 4 (0/4/0/10.101 and 0/4/0/3.102) MGID (Multicast Group ID) MGID programmed for the stream 225.0.0.1 is 19918, Content of this ID is can be different on each LC. Lets verify whether MGID values programmed correctly. Since our fabric is replicating packets to LC 4, we need to check content of MGID in LC 4. P70/8SPO/CPUB:asrOK-3Hshow controllers np ports all location @/4/CPUS Node: 0/4/¢PvO: Ne eridge Fia Ports ‘ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1 3Gigabitetherneto/4/o/30 - Gigabitctherneto/4/0/20 Gigabitetherneto/4/o/10 - ° ° © Gigabitetherneto/a/o/o - P/0/RSPO/CPUO: P/0/RSPO/CPUO: asrok-3# ‘asrok-3#show controllers Device NGID-Bits FIA 10 aridge-0} aridgeno a Bridget uu RP/0/RSPO/CPUO: asrok-3# RP/0/RSPO/CPUO: asr9k-3#show controllers Device MGID-Bits FIA 10 Bridge-o ° nerBva eridge-1 1 RP/0/RSPO/CPUO: asrok-3# MFrBVa € Replicated to Bridge-1 MeTaUA & tnt raplicated hare MFIBVa € Replicated to NP 2 and 3 [NP 3|NP 2] Gigabitetherneto/4/0/39 Gigabitctherneto/4/0/29 Gigabitetherneto/4/0/19 €16"P Join on Gigabitetherneto/4/o/9 €16"P Join on gidprgm mgidindex 19918 location 0/4/cPue Client-Last-hodified [eridge-1 | [wo 1 | mo 0] gidprgm mgidindex 19921 location 0/4/cPue Client-Last-modified NFIBVa € Replicated to Bridge-1 MFIBVa € Replicated to NP 2 [NP-3| NP-2 ] Another way to check MGID and FGID programming: You also see that content of the MGID for 225.0.0.1 and 225.0.0.2 are different. Below command gives great deal of information about replication taking place, MGID, FGID info as well as number of ports in each NP interested in the traffic. 'RP/0/RSPO/CPUB:asr9k-S¥show mfib hardware route olist 22: Le Type: Tragent AaK-a0Ge-E 2 location 0/4/¢Pue Legend: Route Information B: —BACL check Directly connected RPF Interface signal PL: Punt to LC CPU Punt to RP Punt if forwarded Drop all RF interface Fabric slotmask hulticast group 10 Multicast leaf index Not Installed Interface Information Intf: Interface OLE Type Ur UIDB index Table 10 AG a flag Hw BALL flag sourct Route Information Mask: 64 RPF Int: Te0/5/0/1 8S OC PL PR PF DR RI 1 _ASR9000 Multicast troubleshooting document, Cisco Systems Inc PRRFFRE F exe9oo100 @x49 19921 3801 € FGID and MGID values Interface Information oru hp Intt T Ice2 Gi0/4/0/10.101 REG 8S AF OF OLIST counts © © 1 © € Shows 2 port from NP 2 interested in ne/e/Rspe/cPue: aerok-2#chow mfib hardware route oliet 225. SHIP Source: 142.0.0.2 Group: 225.0. Route Information @.1 location @/4/epue i Mask: 64 RPF Int: Te0/5/0/1 8 S OC PL PR PF OR AT F FOF F F F F exe000100 Interface Information hp Int T 2 Gia/4/o/i0.101 REG aS =F «OF | & NP and Outgoing port info 3 Gi0/4/0/3.102 REG 109 1 ~F F — € NP and Outgoing port info LIST counts € shows 1 port from NP 2 and 3 interested in P/0/RSPO/CPUO: asrok-3# 5.3 Checking multicast programming in MFIB: MFIB is a key component in multicast architecture. MFIB receives updates from MRIB and updates the platform control plane. MFIB in LC will store routes only relevant to the LC using selective route downloads. First we build software mfib table based on mrib routing table. This is used incase packets needs to be software switched. Verifying mfib programming: RP/0/RSPO/CPUO:asr9K-3ashow mFib route 225.0-0.1 IP multicast Formarding Information Base Entry flags: ¢ - Directly-Connected Check, $ - Signal, 0 - Drop, IA - Inherit Accept, IF - Inherit From, MA - MOT Address, ME - MOT Encap, MD” MOT Decap, HT - MOT Threshold Crosséd, Mi - MoT interface handle, cb ~ Conditional Decap, ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1. 5 DT = WOT Decap True, EX ~ Extranet NoFE - MoFRR Enabled, MOFS - NOFRR state Interface flags: F - Forward, A - Accept, IC - Internal Copy, NS - legate Signal, NP - nn't Preserve,

snip - (142.0.0.2,225.0.0.1), Flags: 3d09h Last Used: never ‘SW Forwarding Counts: 0/0/@ SH Replication Counts: 0/0/0 SH Failure Counts: 0/0/0/0/0 TenGigé0/5/0/1 Flags: A, Up:3409h Gigabitetherneto/4/o/3.102 Flags: MS EG, Up:23:19:39 € Traffic will be forwarded Gigabitetherneto/4/o/10.101 Flags: NS EG, Up:3d06h RP/0/RSPO/CPUO: asr9k-3# Now check if packets are getting forwarded in hardware using mfib commands: RP /0/RSPO/CPUD:asr9k-3ashow fib route statistics 225.0.0.1 142.0.0.2 detail IP multicast Forwarding Information Base Entry flags: ¢ - Directly-Connected check, $ - Signal, D - Drop, Ik - Inherit Accept, IF - Inherit From, MA - MOT Address, ME = MOT Encap, MO - MOT Decap, NT - MOT Threshold Crossed, Ma - MOT interface handle, cD ~ Conditional Decap, OT - MOT Decap True Interface flags: F - Forward, A - Accept, IC - Internal Copy, NS - legate Signal, OP - Don't Preserve, SP - Signal Present, EG - Egress, EI - Encapsulation interface, MI - MOT Interface ‘Su/Hi Forwarding/Replication Counts: Packets in/Packets out/aytes out SW Failure Counts: RPF / TTL / Empty Olist / Encap RL / other 1_ASR9000 Multicast troubleshooting document, Cisco Systems Inc Hii Drop Counts: Ingress 7 Egress Hu Forwarding Rates: bps In/pps In/bps out/pps out }-0.2,225.0.0-1), Flags: 257 ‘SH Forwarding Counts: 0/0/20 SW Replication counts: w/o/e SW Failure Counts: 0/0/0/0 Hu Forwarding Counts: W/A /W/A /W/A Hu Replication Counts: N/A /M/A /W/A Hu Drop counts: N/A /W/A Hu Forwarding Rates: N/A /N/A /N/A /u/A TenGigeo/5/0/1 Flags: A, Up:00:12: Gigabitetherneto/4/o/3.102 Flags: WS, Up:00:12:57 Gigabitetherneto/4/o/10.101 Flags: NS, Up:00:12:57 RP/0/RSPO/CPUO: asr9k-3# P/0/RSPO/CPUO:asrSk-3#show mfib route statist 225.0.0.1 142.0.0.2 detail loc @/4/cPu0(aa. -0.2,225.0.0-1), Flags: 3do9h Last Used: never ‘SH Forwarding Counts: 0/0/20 ‘SH Replication Counts: 0/0/0 SW Failure Counts: 0/0/0/0 Hui Forwarding Counts: 0/169493467/7796699482 Hu Replication Counts: 0/736054404/33858506724 Hu Drop counts: 0/0 Hu Forwarding Rates: N/A /N/A /725395/1970 TenGigeo/5/a/1 Flags: A, Up:3desh Gigabitetherneto/a/o/3.102 Flags: WS EG, Up:23 Gigabitetherneto/4/o/10.101 Flags: NS EG, Up:3de6h P/0/RSPO/CPUO: asrok-3# 28 © Verifying hardware mroute/mfib Table Following command output shows the whether packets forwarded in hardware for _a given (S,G ‘ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1 7 P/0/RSPO/CPUO:asrok-3show mFib hardware route stati 225.0.0.1 142.0.0.2 loc 0/5/CPUO Le Type: Typhoon ASK-NOD160-SE np 10 R: Received Forwarded P: punted to cpu 1D: Ingress Drop ED: egress Drop 182.0.0.2 Group: 225.0.0.1 Mask:64 rytes)/P(packet s)/I0(packets)/ED(packets) ‘€ THis Ne is receiving traffic from wire No OLIST interfaces found for this route P/0/RSPO/CPUO:asr9k-3# P/@/RSPO/CPUO:asr9k-3¥show mfib hardware route stati 225.0.0.1 142.0.0.2 loc 0/4/CPUO Le Type: Trident A9K-40Ge-E ae & © peceivelForwarded Ingress Drop Punted to CPU Egress Drop 0: 10) Source: 142.0.0.2 Group: 225.0.0.1 Mask:64 NP R(packets:bytes)/F(packets rytes)/P(packet s)/I0(packets)/EO(packets) 1.0 /0/ 0 € This NP 4s sending traffic out on wire 10/0 /0 € This NP is sending traffic out on wire 2 Gio/a/e/10.101 434208:19973568, 3 Gio/a/o/3.102 443309:20392214 -going interface on the NP2 'going interface on the NP3 P/0/RSPO/cPUO:asr9k-3# 1 _ASR9000 Multicast troubleshooting document, Cisco Systems Inc 7 Tracing the packet through Fabric and ASICs: At very high-level path taken by a packet is as follows. Only Trident LC has bridge asic. PHY > ingress NP > Bridge > FIA > Fabric > FIA > Bridge > Egress NP PHY Lets check the counters in each asic using show commands to trace the packet getting forwarded. 7.1_Ingress NP counters: WP/0/RSPO/CPUD: asrok-3eshow controllers np counters npo Location @75/@PUS Node: 0/5/cPUO: Show global stats counters for Pe, revision v2 Read 23 non-zero WP counters: offset counter Framevalue Rate (pps) 21 MDF_TX_FABRIC 23279 4001 29° PARSE_FAB_RECEIVE_CNT 7 1 33. PARSE_INTR_RECEIVE_CNT 43 7 41 PARSE_ENET_RECEIVE_CNT 23278 4001 P/0/RSPO/CPUO: asr9k-3# 7.2__Ingress FIA countersseeeeceee prgig tetenentee category: count-0 From Unicast xbar[0] 6 From unicast xbar[1] 4 From unicast xbar[2] @ From unicast xbar[3] ° From Multicast xbar[o] e From Multicast xbar(1]} ° From Multicast xbar(2] ° From Multicast xbar[3] ° To unicast xbar[o] 3 To Unicast xbar[1] 3 To unicast xbar[2] ° To unicast xbar{3] ° To multicast xbar[o] 17829 & To multicast xbar[2] 17820 € To multicast xbar[2] 8 To multicast xbar[3] ° To Line Interface[0] ° To Line interface(2) 10 From Line Interface[o] ° From Line Interface(1] 35668 Ingress drop: @ Egress drop e ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1. 9 Total drop? e seveneeee praca tenensnees RP/0/RSPO/CPUO:asrOk-3# 7.3__Egress LC FIA counters: B70/#SPO/CPUO:asr9k-Sashow Controllers Fabric fia stats Location 0/4/CPUS sesneeseee FIA-g seeeeeeees Category: count-0 From Line Interface[o] 6 From Line Interface[1] 2 To Line Interface(o] 6 To tine tnterface(t] 63807 ‘To unicast xbar[o] To unicast To Unicast To unicast To multicast To multicast To multicast To multicast From unicast From unicast From unicast From unicast From Multicast From Multicast From Multicast From Multicast xbar[2] xbar[2] xbar{3] xbar0] xbar{2] xbar{2] xbar{3] xbar [0] xbar{] xbar{2] xbar[3] Xbar[o] xbar[1] xbar[2] xbar[3] 331753 331752 € From Fabric € From Fabric Ingress drop: Egress drop Total drop: 7.4_Egress NP counters P70/8SPO/CPUO:asr9k-34show controllers np ports all location 0/4/CPUS Node: 0/4/cPvO:eee ee Gigabitetherneto/4/o/30 - Gigabitetherneto/4/0/39 oon ae 1 © © Gigabitetherneto/4/0/20 - Gigabitetherneto/4/0/29 2 1 © Gigabitetherneto/a/o/10 - Gigabitetherneto/4/0/19 3 1 0 _Gigabitetherneto/4/o/o - cigabitetherneto/a/o/9 RP/0/RSPO/CPUO:asrok-3# RP/0/RSPO/CPUO:asr9k-34show controllers np counters np2 location 0/4/cPUO Node: 0/4/cPU0: show global stats counters for NP2, revision v3 Read 16 non-zero NP counters: Offset” Counter Franevalue Rate (pps) 24 PARSE_LooPaack RECEIVE. cwT 3674751 2000 30 MOOTFY_ENET_TRANSATT ci 11349784 4000 38 MODIFY _NCAST FLO, LOOPBACK_cnT 3674751 2000 p/0/RSPO/CPUO:asr9k-38show controllers np counters np3 location 0/4/cPuO Node: 0/4/CPU0: 2__ASR9000 Multicast troubleshooting document, Cisco Systems Inc show global stats counters for NP3, revision v3, Read 19 non-zero NP counters: offset counter Framevalue Rate (pps) 23 PARSE_FABRIC_RECEIVE_CNT 3604472 2000 24 PARSE LOOPBACK_RECEIVE_CNT 47 ° 30 MODIFY_ENET_TRANSMIT_CHT s604a25 2000 np /0/RSPO/CPUO: as ok-3# 8 Debugging multicast Process Memory: This section outlines some basic commands, which can help to troubleshoot whether system is running with low memory, memory leak and or process crashes. + Collect the output of “show process memory location <>” and check if memory in dynamic column increasing. If so, this might be sign of one or more process holding up memory and not releasing it. Please take few snap shot of the output. Output shows memory utilization for only multicast process. 1 70/#SPO/CPUB:asrSK-S¥shOn processes wenory location 073/cPUs | In ipva_wfwd 1D Text Data Stack Dynamic Process 237 475136 192512 «126976 12337352 ipva_mfwd_partner RP/0/RSPO/CPUO:asr9k-3# 9 Debugging Multicast and related processes: ‘+ Following outputs were taken from one of the Service request for the explanation purpose. This output shows one of the multicast process blocked * Check if multicast process is blocked: R70/8SPO/CPUO:on-bari-vord#sh processes blocked Location o/rspo/cpud nid Pid Tid ane State Tinelnstate Blocked-on 6554916397 1 ksh Reply 6572:57:32:0613 16396 devc-conaux 6555457362 devb-umass Reply 0:00:07:0644 45096 io-usb A F 96 40996 2 —_umass-enum Reply 6572:57:36:0954 1 kernela , es 52 69677 2 attachd Reply 6572 49196 eth_server 52 69677 3 attachd Reply 6572 24595 mquéue 65582 57390 2 devb-umass Reply :00:37:0655 45096 io-usb 87 69080 G net Reply 49196 eth_server, 51 69684 2 attach server Reply 6572:57:33:0215 24595 mqueue 65594 69690 2 devb-umass Reply @ 0158 45096 io-usb 3349917781 ttp_server Reply 29R6:32:74:0177 74505 mquoNe pai 258252 2 Ipts_fm Reply 0485 245891 Ipts_pa 65741 39502797 1 exec Reply 1:13:10 258244 _deve-vty 353° 1835278 6 ipv4_mfwd_partner hutex 3238 69654-56184 #2 attach_server 65844 6035072 1 exec Reply a kernel 65045 61145397 1 ‘more Repl} 0065 24593 pipe ‘ASR9000 Multicast troubleshooting document, Cisco Systems Inc 2 1 GSH46 61145398 1 show processes Reply P/0/RSPO/CPUO:0n-bari-vorié T Kernel + Above output shows that ipv4_mfwd_partner process blocked on attach. server. Lets check status of the attach server process * Check state of the attach server process and ipv4_mfwd_partner 'R970/8SPO/CPUOTOn-barL-vOriash processes attach server Thu May 10 15:42:26.745 EOT dob Id: 51 €2== J0b 10 PID: 69684 €=. Executable path: /asr9k-0s-4.0.1/bin/attach_server Instance #: 1 version 10: 00.00.0000 Respawn: oN Respawn count: 1 Max. spawns per minute: 12 Last started: Wed Aug 10 15:02:46 2011 Process state: Run Package state: Normal core: MAINMEH Max. core: Level: 39 MaintrodeProc: on Placenent: tione startup_path: /pkg/init.d/attach_server. init Process cpu time: 0.012 user, 0.000 kernel, 0.012 total PID JID TID CPU Stack pri state Timeznstate _HR:MN:SS:MSEC S110 20K 10 Receive 6576:30:04: Si 2 -1-20K «55 Reply —__6576:39:31:0128 _@:00:20:0000 attach_server RP/0/RSPO/CPUD:0n-bari-vorisshow processes ipv Thu May 10 15:44:38.446 EOT Job Id: 353 PID: 1835278 Version 1D: 00.00.0000 Respawn: ON Respawn count: 1 Max. 2 Fri Aug 19 21:3 Run Normal MaTiOEM ° 199 None 9 2011 327 @:00:00:0012 attach_server /install /iosxr-mcast-4.0.1/startup/ipva_nfud. startup NaveProcess cpu time: 222629.394 user, 60507.334 kernel, 263336.728 total 31D TID CPU Stack pri state TimeInState | HR:MN:SS:MSEC NAME 3531 @ 76K 10 Receive a:ee:e0:e0a0 48:58:31: ipva_mfedl_parn 353 6 1 76K 10 Mutex 20: ipva_mfud_partner 2__ASR9000 Multicast troubleshooting document, Cisco Systems Inc 50 31:0794 28:39:32:0556 Collect output of “show follow process " if the process is in blocked state. 10 Additional Commands: Some additional useful command: Show mfib interface loca * Show pim topology + Show drops € Take this output 2-3 times with some gap + Show controller fabric fia q-depth loc € 2-3 times + Show controller pm location + Show controller fabric fia stat loc € 2-3 times * Show log + Show tech-support multicast hardware loc + Show controllers np tm counter loc € 2-3 times + Show tech-support fabric + Show mfib hardware loc + Show mfib hardware Itrace * Show controllers np struct 22 all location [egress] + Show mfib hardware route summary loc <> + Show mfib hardware trace location <> 11 Multicast drops counters in NP: + IPV4MC_DO_ALL_EXCD «Punt policier rate limited packet drops + IPV4MC_DO_ALL_BUT_FWD_EXCD Punt policier rate limited packet drops + MODIFY_RPF_FAIL_DROP_CNT Ingress on RPF interface + RESOLVE_IPM4_ING_RTE_DROP_CNT * No multicast route found on ingress + RESOLVE_IPM4_EGR_RTE_DROP_CNT * No multicast route found on ingress + RESOLVE_IPM4_NO_OLIST_DROP_CNT + No multicast output interfaces found on egress route + RESOLVE_IPM4. FILTER DROP_CNT Ingress interface = egress interface + RESOLVE_IPM4.TTL_DROP_CNT * TTLexceeded on packet + PUNT_IFIB (L3 PUNT) + PIM/IGMP control plane packets punted to RP + IPV4MC_DO_ALL