TR 40 Asr 9 Kbook

Cisco IOS XR Troubleshooting Guide for
the Cisco ASR 9000 Aggregation Services

Router
Cisco IOS XR Software, Release 4.0
April, 2011
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Text Part Number: OL-23591-02
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCBs public
domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1005R)
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display
output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in
illustrative content is unintentional and coincidental.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2010 Cisco Systems, Inc. All rights reserved.
CONTENTS
Preface
xi
Changes to This Document
xi
Obtaining Documentation and Submitting a Service Request
CHAPTER
General Troubleshooting Procedures
xii
1-1
Prerequisite Documentation for Troubleshooting
1-1
Verifying and Troubleshooting CLI Access 1-2

General CLI Access Information 1-2
User Access Privileges 1-2
Cisco-support Task ID 1-3
CLI Access Through a Console Port 1-3
CLI Access Through a Terminal Server 1-3
CLI Access Through the Management Ethernet Interface
1-4
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Verifying the Software Version 1-8
Validating the Installation 1-10
Validating and Troubleshooting Cisco IOS XR Software Configuration
Local and Global Configurations 1-16
Collecting Configuration Information 1-19
Verifying the Running Configuration 1-20
Using the show configuration failed Command 1-24
Verifying the System
1-41
Basic Cisco IOS XR Verification and Troubleshooting Commands

man Command 1-46
describe Command 1-49
show platform Command 1-49
top Command 1-50
show context Command 1-50
show users Command 1-52
show history Command 1-52
show configuration Command 1-53
Using Trace Commands
1-16
1-26
Troubleshooting the Backplane Ethernet Control System
Displaying ASIC Errors
1-7
1-46
1-54
1-56
OL-23591-02
iii
Contents
MIB Location
1-57
Gathering Information Before You Call Cisco TAC 1-58

Gathering Information about Crashes and Core Dumps 1-58
Capturing Logs 1-58
Using Debug Commands 1-59
Using Diagnostic Commands 1-59
Commands Used to Display Process and Thread Details 1-59
CHAPTER
Verifying and Troubleshooting Interface Status
2-61
Verifying and Troubleshooting Gigabit Ethernet Interfaces
2-61
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
CHAPTER
Troubleshooting Interface Connectivity
2-68
3-75
Troubleshooting Ping and ARP Connectivity
3-75
Troubleshooting Bidirectional Forwarding Detection 3-81

Using show and debug Commands 3-82
BFD Sessions in Down State 3-83
BFD Sessions Flap 3-83
BFD Sessions Down on Neighboring Router 3-85
BFD Sessions Are Not Created on the LC 3-85
Troubleshooting Ethernet CFM 3-85
MEPs Are Not Created 3-88
MIPs Are Not Created 3-88
No CCMs are Received at the MEP or Peer MEPs Are Not Seen 3-89
Peer MEP Defects and Mismatches Are Seen 3-90
Remote Defect Indication Received 3-91
Peer MEP Times Out But No Alarm Or Action Occurs 3-92
No Debugs or Counters for Higher-Level Packets at a MEP or MIP 3-92
Dropped CFM PDUs 3-92
CFM ping Or traceroute Returns a not found Error 3-93
AIS Messages Are Not Sent 3-93
CHAPTER
Troubleshooting Packet Forwarding

Understanding IPv4 CEF
Troubleshooting IPv4 CEF
4-95
4-95
4-96
Troubleshooting Adjacency Information
4-101
Troubleshooting Transient Traffic Drop
4-106
iv
OL-23591-02
Contents
CHAPTER
Troubleshooting Packet Drop in the Fabric
4-109
Troubleshooting Control Plane Information
4-109
Troubleshooting Bundles and Load Balancing
5-115
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing
Verifying Routing Table Entries for Parallel Links 5-115
Verifying the CEF Database and Measuring Flows 5-117
5-115
Troubleshooting Problems with Link Bundles 5-118

Bundle Does Not Come Up 5-118
Bundle Member Not Distributing 5-119
Bundle Not Using MAC-Address From Backplane 5-119
Layer 3 Data Traffic Not Flowing 5-120
Ping Failed over Bundle 5-120
Layer 3 Packets Not Synching Over Bundle 5-121
Layer 2 Traffic Not Flowing 5-121
Bundle Statistics 5-122
Troubleshooting Layer 2 Bundles and Load Balancing 5-122
Verifying the Bundle Status, IGP Route, and CEF Database 5-122
Viewing the Expected Paths and Measuring the Flows 5-123
Troubleshooting Layer 3 Bundles and Load Balancing
CHAPTER
Troubleshooting Layer 3 Connectivity

Using show and debug Commands
Traffic Loss
5-124
6-125
6-125
6-128
Packets Are Punted and Switched in Software

Traceroute Fails
6-129
6-130
Adding Routes Fails
6-131
Continuous Tracebacks
6-133
fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts
CEF Entries Out of Sync
fib_mgr Crashes
6-134
6-135
6-136
Tracebacks Appearing
6-136
Traffic Loss Because of Changing encap on a Subinterface

Traffic Loss during RSP Failover
6-137
6-138
Troubleshooting Virtual Router Redundancy Protocol

VRRP Fails to Reach Active State 6-140
6-138
OL-23591-02
Contents
Tracked Interface Failing, Router State Not Changed 6-140

VRRP State Flapping 6-140
More Than One VRRP Router Active 6-141
VRRP Active Router Not Forwarding Traffic 6-141
Traffic Loss or Unexpected VRRP State After Interface shut/no shut
Additional Information On Routing Configuration Commands
CHAPTER
Troubleshooting Router Switch Fabric and Data Path

Understanding Switch Fabric Architecture
6-142
7-143
7-143
Getting Started with Fabric Troubleshooting
7-145
Troubleshooting Packet Drops 7-146

Displaying Traffic Status in Line Cards and RSP Cards
Locating Packet Drops by Examining Counters 7-148
Locating Drops of Punted Packets 7-155
Packet Drop from LC to LC 7-157
Packet Drop Between RSP and LC 7-158
Packet Drop After Certain Actions 7-160
Packet Drop After a Redundancy Switchover 7-161
Packet Drop with Unknown Reason 7-163
Troubleshooting RSP and LC Crashes
Active RSP Is Crashing 7-165
Standby RSP Is Crashing 7-166
LC Is Crashing 7-167
6-142
7-147
7-165
Troubleshooting Complete Loss of Traffic 7-168

No Traffic from LC to LC 7-169
No Traffic Between RSP and LC 7-170
Gathering Fabric Information Before Calling TAC
CHAPTER
Troubleshooting MPLS Services
7-172
8-173
Verifying MPLS PIE Activation and MPLS Configuration

Troubleshooting Connectivity Over MPLS
IP Packets Not Forwarded to LSP
8-174
8-174
8-175
IP Packets Not Forwarded to MPLS TE Tunnel
8-176
MPLS Packets Not Forwarded to MPLS TE Tunnel

MPLS TE Tunnels Do Not Come Up
8-176
8-176
FRR-Protected Tunnel Goes Down After Triggering FRR

MPLS TE FRR Database Not Built
8-173
8-177
8-178
vi
OL-23591-02
Contents
MPLS FRR Switch Time Debugging
CHAPTER
8-178
Troubleshooting L2VPN and Ethernet Services
9-181
Troubleshooting VLAN Traffic and L2 TCAM Classification 9-181

Understanding Problems with VLAN Traffic and L2 TCAM Classification
Verifying the Configuration Is Correct 9-182
Verifying Interfaces, Subinterfaces, and Packet Forwarding 9-183
Troubleshooting Multipoint Layer 2 Services 9-190
Basic Bridging: Example 9-190
Verifying MAC Address Updates 9-192
Troubleshooting Multipoint Layer 2 Bridging Services (VPLS)
Troubleshooting Bridge Domains That Use BGP-AD 9-201
9-182
9-195
Troubleshooting Point-to-Point Layer 2 Services 9-206

Example of Point-To-Point Layer 2 Deployment 9-206
AC Is Down 9-211
Pseudowire Is Down 9-212
VPWS Not Forwarding Traffic from AC to Pseudowire 9-212
Pseudowire Up but Ping Fails 9-213
Traffic Loss 9-213
Traffic Loss During RSP Fail Over 9-213
Preferred Path Not Working 9-214
Troubleshooting Specific Outage Scenarios In Layer 2 Services 9-214
L2VPN Discovery Not Working 9-217
AC Is Down 9-218
Pseudowire Is Down 9-219
VPLS Not Forwarding Flooding Traffic 9-220
VPLS Not Forwarding Flooding Traffic from AC to Pseudowire 9-224
VPLS Not Forwarding Flooding Traffic from Pseudowire to AC 9-224
VPLS Not Forwarding Unicast Traffic from AC to AC 9-225
VPLS Not Forwarding Unicast Traffic from AC to Pseudowire 9-225
VPLS Not Forwarding Flooding Traffic from Pseudowire to AC 9-225
Pseudowire Up but Ping Fails 9-226
Traffic Loss 9-226
Pseudowire Flap Causing Traffic Loss 9-226
Traffic Loss During RSP Fail Over 9-227
Preferred Path Not Working 9-227
Troubleshooting Dynamic Host Configuration Protocol Snooping
9-227
OL-23591-02
vii
Contents
Show Commands 9-228

Trace Commands 9-228
Syslog Commands 9-228
Tech-support Commands 9-229
Action Commands 9-229
L2VPN Commands 9-229
L2Snoop Commands 9-229
Interface Controller Commands 9-230
Troubleshooting Multiple Spanning Tree 9-230
MSTP Incorrectly or Inconsistently Formed 9-230
MSTP Correctly Formed, but Traffic Flooding 9-231
Packet Forwarding Does Not Match MSTP State 9-231
MSTAG Access Network Does Not Recognize MSTAG Node as Root
Traffic Not Switching Through MSTAG Node(s) 9-232
Additional ReferencesCommand Reference and Configuration Guides
CHAPTER
10
Troubleshooting Quality of Service and Access Control Lists

Packets in Wrong Queue
10-233
10-235
10-235
10-236
Packets Incorrectly Marked
10-236
Packets Incorrectly Policed
10-237
Shaping Incorrect
9-232
10-234
Service-Policy Configuration Is Rejected

Packets are Incorrectly Classified
9-231
10-237
Weighted Random Early Detection Incorrect

Bandwidth Not Guaranteed
10-237
10-238
Bandwidth Ratio Not Working
10-238
Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands

Unable to Modify or Delete policy-map or class-map
Unable to Modify or Delete class-map ACL
Unable to Delete service-policy
10-239
10-240
10-240
10-240
After QoS EA Restarts, show policy-map interface Fails

After QoS EA Restarts, service-policy config Fails
show policy-map interface Output Error
10-241
10-241
Bundle Members Not Configured with service-policy

Troubleshooting Access Control Lists
10-240
10-241
10-241
viii
OL-23591-02
Contents

ACL Messages Not Appearing 10-243
Fragmented Packets Being Accepted 10-243
Egress Counter Incorrect or Not Working 10-244
ACL Interface Bind Rejected 10-244
Single ACE Using Many TCAMs 10-244
ACL Using Varying TCAM Space 10-245
ACL Logs Not Working for Ethernet Services 10-245
Ethernet Services ACL Bind on Interface Rejected 10-245
Changing ACL Exhausts TCAM 10-245
Cannot Delete ACL 10-246
DF Bit Not Supported 10-246
Max ACL Limit Reached 10-246
Unsupported Combinations in ACL 10-246
No Statistics Counters 10-246
TCAMs Out of Resources 10-246
CHAPTER
11
Troubleshooting Multicast Services
11-247
Troubleshooting IGMP Snooping (Layer 2 Multicast) 11-247

Using show Commands 11-247
Using the debug, trace, and show tech-support Commands 11-249
Troubleshooting Missing Routes and Forwarding Errors 11-250
Troubleshooting Native Multicast Routing (Layer 3) 11-256
Multicast PIE Installation Fails 11-262
Multicast CLI Unavailable Although PIE Is Installed 11-263
This command not authorized Error Message 11-263
Dynamic IGMP Failure 11-263
Traffic Fails on Some Interfaces 11-267
Traffic Fails on Some InterfacesMGID 11-268
Throughput Loss at Receiver Interfaces 11-268
Reverse Path Forwarding IP Address Problems 11-268
INDEX
OL-23591-02
ix
Contents
OL-23591-02
Preface
This guide describes how to troubleshooting a router using the Cisco IOS XR software.
This preface contains the following sections:
Changes to This Document, page xi
Obtaining Documentation and Submitting a Service Request, page xii

Table 1 lists the technical changes made to this document since it was first printed.
Table 1
Revision
Date
OL-23591-02
April, 2011 Added details for the following topics:
Change Summary
Chapter 1, General Troubleshooting ProceduresAdded

information on prerequisite documentation for troubleshooting,
Cisco-support task ID, show tech-support command, displaying
ASIC errors, gathering logs and system information. Modified the
information on diagnostics.
Divided interface troubleshooting into two separate

chaptersChapter 2, Verifying and Troubleshooting Interface
Statusand Chapter 3, Troubleshooting Interface Connectivity.
Chapter 3, Troubleshooting Interface ConnectivityAdded

information on connectivity fault management (CFM).
Chapter 6, Troubleshooting Layer 3 ConnectivityAdded

information on CEF and interface accounting.
Chapter 7, Troubleshooting Router Switch Fabric and Data

PathAdded information on NP counters.
Chapter 8, Troubleshooting MPLS ServicesCorrected syntax of

several commands.
continued ...
OL-23591-02
xi
Preface
Table 1
Changes to This Document (continued)
Revision
Date
OL-23591-02
April, 2011 ... continued
Change Summary
Chapter 9, Troubleshooting L2VPN and Ethernet

ServicesEnhanced information about VLAN verification,
included sample VLAN and PW configurations, added a section on
Verifying MAC Address Updates, enhanced information about
multipoint Layer 2 services (VPLS), moved information on DHCP
snooping to this chapter, enhanced information on MST access
gateways (MSTAGs). Reorganized the chapter for ease of use.
Chapter 10, Troubleshooting Quality of Service and Access

Control ListsAdded information on queue conform and queue
exceed counters displayed by the policy-map command.
Chapter 11, Troubleshooting Multicast ServicesReorganized

this chapter to highlight IGMP snooping (Layer 2 MC) and native
MC (Layer 3), and added information to each of these sections.
OL-23591-01
November
2010
(Initial release of this document as a multichapter book.) The content was

reorganized for usability and updated to reflect Release 3.9 and 4.0
features.
OL-20794-01
December
2009
Initial release of this document as a single module.
Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gathering additional
information, see the monthly Whats New in Cisco Product Documentation, which also lists all new and
revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the Whats New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed
and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free
service and Cisco currently supports RSS version 2.0.
xii
OL-23591-02
CH A P T E R

This chapter describes general troubleshooting techniques you can use to troubleshoot the
Cisco ASR 9000 Aggregation Series Router. It includes the following sections:
Prerequisite Documentation for Troubleshooting, page 1-1
Verifying and Troubleshooting CLI Access, page 1-2
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package, page 1-7
Validating and Troubleshooting Cisco IOS XR Software Configuration, page 1-16
Verifying the System, page 1-26
Troubleshooting the Backplane Ethernet Control System, page 1-41
Basic Cisco IOS XR Verification and Troubleshooting Commands, page 1-46
Displaying ASIC Errors, page 1-54
Using Trace Commands, page 1-56
MIB Location, page 1-57
Gathering Information Before You Call Cisco TAC, page 1-58
Prerequisite Documentation for Troubleshooting

As a starting point for troubleshooting, we strongly recommend that you have a system of maintaining
and accessing detailed information about your network and ASR 9000 router. This should include:
Current documentation about the system, including chassis numbers, serial numbers, installed cards,
and location of chassis details.
Diagrams illustrating the connectivity of the router control plane Ethernet network.
Detailed documentation about the network, including the following:

Up-to-date internetwork map that outlines the physical location of all the devices on the network
and how they are connected, as well as a logical map of interfaces, network addresses, network
numbers, subnetworks, and so on
List of all network protocols implemented in your network; and for each of the protocols
implemented, a list of the network numbers, subnetworks, zones, areas, and so on that are
associated with them
All points of contact to external networks
Routing protocol for each external network connection
OL-23591-02
1-1
Chapter 1
Verifying and Troubleshooting CLI Access
Established baseline for your network, that is, the normal network behavior and performance at
different times of the day so that you can compare any problems with a baseline
Name of the device that is the spanning tree root bridge for the system control plane Ethernet
network
Captured output of all commands

Ensure that the system has been booted. If the system has not booted, see Cisco IOS XR Getting Started
Guide for the Cisco ASR 9000 Aggregation Services Router for information on booting a router running
Cisco IOS XR software. The following CLI access troubleshooting information is provided:
General CLI Access Information, page 1-2
User Access Privileges, page 1-2
Cisco-support Task ID, page 1-3
CLI Access Through a Console Port, page 1-3
CLI Access Through a Terminal Server, page 1-3
CLI Access Through the Management Ethernet Interface, page 1-4
General CLI Access Information

The following CLI access information applies to a console port, terminal server, and management
Ethernet interface connections.
Once the terminal emulation software is started and you press Enter, a router prompt should appear. If
no prompt appears, verify the physical connection to the console port and press Enter again. If the
prompt still does not appear, contact Cisco Technical Support. See the Obtaining Documentation and
Submitting a Service Request section on page xii for Cisco Technical Support contact information.
If a prompt appears, indicating that the CLI is accessible, but your login username and password are
invalid, you are prevented from accessing the router. Verify that you have the correct username and
password. If you have the correct username and password, but are locked out of the router, you may need
to perform password recovery to access the system again. See Cisco ASR 9000 Aggregation Services
Router ROM Monitor Guide for password recovery procedures.
User Access Privileges

When you log on to the router, use a username that is associated with a valid user group that has the
authorization to execute the required commands. If you suspect user group assignment is preventing you
from using a command, contact your AAA administrator for assistance.
See Cisco ASR 9000 Aggregation Services Router System Security Command Reference and
Cisco ASR 9000 Aggregation Services Router System Security Configuration Guide for information on
users, usernames, and user groups.
1-2
OL-23591-02
Chapter 1

Cisco-support Task ID
Many of the troubleshooting commands can be performed only by users who are assigned to a user group
that includes the cisco-support task ID. Users without the cisco-support task ID receive a This
command is not authorized response if they attempt to use those commands. The cisco-support
commands are normally reserved for use by Cisco Technical Support personnel, because there is some
risk that they may cause performance or other issues.
Caution
These Cisco support commands are normally reserved for use by Cisco Technical Support personnel
only. There is some risk that they may cause performance or other issues that impact products without
proper usage, and we highly recommend that you contact Cisco Technical Support prior to using any of
these commands. See the Obtaining Documentation and Submitting a Service Request section on
page xii for information on contacting Cisco TAC.
CLI Access Through a Console Port

The first time a router is started, you must use a direct connection to the console port to connect to the
router and enter the initial configuration. See Cisco ASR 9000 Aggregation Services Router Router
Getting Started Guide for information on connecting to the router through a console port. When you use
a direct connection to the Console port, CLI commands are entered at a terminal or at a computer running
terminal emulation software. A direct Console port connection is useful for entering initial
configurations and performing some debugging tasks.
CLI Access Through a Terminal Server

A terminal server connection provides a way to access the Console port from a remote location. A
terminal server connection is used when you need to perform tasks that require Console port access from
a remote location.
Connecting to a router through a terminal server is similar to directly connecting through the Console
port. For both connection types, the physical connection takes place through the Console port. The
difference is that the terminal server connects directly to the Console port, and you must use a Telnet
session to establish communications through the terminal server to the router.
If you are unable to access the CLI through a terminal server, perform the following procedure.
Step 1
Disable flow control (XON/XOFF) on the Terminal Server.
Step 2
Disable local echo mode on the Terminal Server.
Step 3
Verify the router name configured using the hostname command.
Step 4
Check whether the port address is configured correctly.
Step 5
Verify whether the address (interface) used for the reverse Telnet is up/up. The output of the show
interfaces brief command provides this information. Cisco recommends you to use loopbacks because
they are always up.
Step 6
Ensure that you have the correct type of cabling. For example, you must not use a crossover cable to
extend the length.
Step 7
Establish a Telnet connection to the IP address port to test direct connectivity. You must Telnet from both
an external device and the terminal server. For example, telnet 172.21.1.1 2003.
OL-23591-02
1-3
Chapter 1
Step 8
Ensure that you have the transport input telnet command under the line for the target device. The target
device is the device that is connected to the terminal server.
Step 9
Use a PC/dumb terminal to connect directly to the console of the target router. The target router is the
device connected to the terminal server. This step helps you identify the presence of a port issue.
Step 10
If you are disconnected, check timeouts. You can remove or adjust timeouts.
Note
If you encounter authentication failures, remember that the terminal server performs the first
authentication (if configured), while the device to which you try to connect performs the second
authentication (if configured). Verify whether AAA is configured correctly on both the terminal server
and the connecting device.
Step 11
Contact Cisco Technical Support. See the Obtaining Documentation and Submitting a Service Request
section on page xii for Cisco Technical Support contact information.
CLI Access Through the Management Ethernet Interface

The Management Ethernet interface allows you to manage the router using a network connection. Before
you can use the Management Ethernet interface, the interface must be configured. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on
configuring the interface.
Once configured, the network connection takes place between client software on a workstation computer
and a server process within the router. The type of client software you use depends on the server process
you use. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for
information on the client and server services supported by the Cisco IOS XR software.
If you are unable to access the CLI through a management Ethernet interface, perform the following
procedure.
SUMMARY STEPS
1.
show interface MgmtEth interface-instance
2.
show arp MgmtEth interface-instance
3.
show ipv4 interface type instance
4.
ping
5.
Contact Cisco Technical Support if the problem is not resolved
1-4
OL-23591-02
Chapter 1

DETAILED STEPS
Step 1
Command or Action
Purpose
show interfaces MgmtEth interface-instance
Displays statistics for all interfaces configured on the

router.
Example:
Check the following:
RP/0/RSP0/CPU0:router#
0/RSP0/CPU0/0
show interfaces MgmtEth
MgmtEth interface is up
Line protocol (state of the Layer 2 line protocol) is up
Number of input and output errors
If an interface is administratively down, use the no

shutdown command to enable the interface.
If an interface is down (operationally down), input or output
errors are not within an acceptable range, the management
Ethernet interface is not enabled when the no shutdown
command is used, or the line protocol is down, see
Chapter 2, Verifying and Troubleshooting Interface
Status, for detailed information on troubleshooting
interfaces.
If the interface is up and the input and output errors are
within an acceptable range, proceed to Step 2.
Step 2
show arp MgmtEth interface-instance
Displays the Address Resolution Protocol (ARP) table for

the management Ethernet interface.
Example:
Ensure that the expected ARP entries exist for the

management Ethernet interface.
0/RSP0/CPU0/0
show arp MgmtEth
If the expected ARP entries exist, proceed to Step 3.

If the expected ARP entries do not exist, verify the physical
layer Ethernet interface connectivity. Use the show arp
trace command to display the ARP entries in the buffer. See
the Chapter 2, Verifying and Troubleshooting Interface
Status, for more information on troubleshooting interfaces.
Step 3
show ipv4 interface type instance
Displays the usability status of interfaces configured for

IPv4.
Example:
If the interface is in the expected state, proceed to Step 4.
MgmtEth 0/RSP0/CPU0/0
show ipv4 interface
If the status of the interface is not as expected, see

Chapter 2, Verifying and Troubleshooting Interface
Status, for more information on troubleshooting interfaces.
OL-23591-02
1-5
Chapter 1
Step 4
Command or Action
Purpose
ping
Checks host reachability and network connectivity on the IP

network.
Note
Example:
ping
Enter a specific IP address or follow the prompts to

send the ping message to the target address.
If no problems are detected, proceed to Step 5.

Step 5
Contact Cisco Technical Support.
If the problem is not resolved, contact Cisco Technical

Support. For Cisco Technical Support contact information,
see the Obtaining Documentation and Submitting a
Service Request section on page xii.
Examples
The output from the show interfaces MgmtEth command displays the status of the management
Ethernet interface. In the following example, the management Ethernet interface is up, and there are 20
input errors and 8 output errors.
RP/0/RSP0/CPU0:router# show interface MgmtEth 0/RSP0/CPU0/0
Tue Sep 14 14:21:07.496 DST
MgmtEth0/RSP0/CPU0/0 is up, line protocol is up
Interface state transitions: 1
Hardware is Management Ethernet, address is 001b.53ff.4a62 (bia 001b.53ff.4a62)
Description: Connected to Lab LAN
Internet address is 172.29.52.137/24
MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)
reliability 73/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Half-duplex, 100Mb/s, THD, link type is autonegotiation
output flow control is off, input flow control is off
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 2000 bits/sec, 3 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
373082 packets input, 51028824 bytes, 239105 total input drops
62028 drops for unrecognized upper-level protocol
Received 2601 broadcast packets, 194653 multicast packets
10 runts, 0 giants, 0 throttles, 0 parity
20 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
45232 packets output, 3042775 bytes, 0 total output drops
Output 24 broadcast packets, 0 multicast packets
8 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
The output from the show arp MgmtEth 0/RSP0/CPU0/0 command displays the ARP table for the
management Ethernet interface. Use the output from this command to verify that there are dynamic ARP
addresses in the table and that ARP is functioning over the interface. The output shows that ARP is
functioning over the management Ethernet interface 0/RSP0/CPU0/0.
RP/0/RSP0/CPU0:router# show arp MgmtEth 0/RSP0/CPU0/0
Tue Sep 14 14:24:03.962 DST
------------------------------------------------------------------------------0/RSP0/CPU0
------------------------------------------------------------------------------Address
Age
Hardware Addr
State
Type Interface
1-6
OL-23591-02
Chapter 1

172.29.52.1
172.29.52.13
172.29.52.21
172.29.52.27
172.29.52.28
172.29.52.32
172.29.52.36
172.29.52.46
172.29.52.47
172.29.52.60
172.29.52.69
172.29.52.70
172.29.52.71
172.29.52.75
172.29.52.76
172.29.52.81
172.29.52.83
172.29.52.127
172.29.52.134
172.29.52.135
172.29.52.136
172.29.52.137
172.29.52.138
172.29.52.161
172.29.52.171
172.29.52.172
172.29.52.173
172.29.52.180
172.29.52.217
172.29.52.226
172.29.52.243
01:44:00
01:16:59
01:40:25
02:18:16
02:05:29
01:42:16
02:39:34
01:36:50
01:36:39
01:35:20
00:00:00
01:23:38
02:00:47
01:44:59
01:41:10
00:15:35
00:21:05
01:43:38
01:15:53
01:01:46
00:43:39
01:32:12
00:16:12
00:17:47
01:57:04
01:26:50
01:16:21
01:28:32
01:48:25
0000.0c07.ac01
0010.79e9.6038
0022.0d5a.a6c4
0012.7fd6.ba08
0012.7fd6.ba09
0022.0d26.3bc5
0026.527c.5341
0012.7fd6.b9aa
0012.7fd6.b9ab
0003.a099.8000
001b.7852.4bd1
0011.93ef.e8e6
0011.93ef.e8fe
5a59.0000.0202
0011.93ef.e8ea
001a.6c40.d89c
001a.6c40.d89c
0013.c4cb.a200
001f.6c26.7fc0
001f.6c25.c480
0022.5560.8840
001b.53ff.4a62
001b.53ff.4a62
0019.aaa3.3d48
001c.5838.5b28
001c.5838.5b29
0015.c75f.09f8
0015.c75f.0800
0019.aaa3.b5ff
0010.f60e.8400
001e.79c1.e0c1
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Interface
Interface
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
MgmtEth0/RSP0/CPU0/0
The ping command checks to see if the neighbor is reachable.

RP/0/RSP0/CPU0:router# ping 172.16.52.28 count 10
Tue Sep 14 14:36:52.441 DST
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 172.16.52.28, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/1/2 ms
Validating and Troubleshooting Installation of the Cisco IOS XR

Software Package
The Cisco IOS XR software is divided into software packages allowing you to select which features run
on your router. Each package contains the components to perform a specific set of router functions, such
as routing, security, or Modular Services Card (MSC) support. Bundles are groups of packages that can
be downloaded as a set. For example, the Unicast Routing Core Bundle provides six packages for use on
every router.
This section provides information on how to validate and troubleshoot the Cisco IOS XR software
package installation. The following sections are provided:
Verifying the Software Version, page 1-8
Validating the Installation, page 1-10
OL-23591-02
1-7
Chapter 1
Verifying the Software Version

To verify the Cisco IOS XR software version, perform the following procedure.
SUMMARY STEPS
1.
show version
2.
show install
DETAILED STEPS
Step 1
Command or Action
Purpose
show version
Displays a variety of system information, including

hardware and software version, router uptime, boot settings
(configuration register), and active software.
Example:
show version
Determine if all expected packages are installed and the

current software versions are the expected versions.
If the expected packages are not installed or are not the
expected version, install the correct package. See
Cisco ASR 9000 Aggregation Series Router Getting Started
Guide for information on installing and upgrading
Cisco IOS XR software packages.
Step 2
Displays a list of all installed and active packages on each

node.
show install
Example:
show install
Determine if the expected packages are installed on each

node.
If the software or active package versions are not as
expected for a node, the package is not compatible with the
node for which it is being activated, or the package being
activated is not compatible with the current active software
set, install the correct software or package on the node. See
Cisco ASR 9000 Aggregation Series Router Getting Started
Guide for information on installing and upgrading
Cisco IOS XR software packages.
The following example shows that the Cisco IOS XR software and active packages are version 4.0.0.
RP/0/RSP0/CPU0:router# show version
Cisco IOS XR Software, Version 4.0.0
Copyright (c) 2010 by cisco Systems, Inc.
ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],
router uptime is 1 day, 18 hours, 34 minutes
System image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"
cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory.
MPC8641D processor at 1333MHz, Revision 2.2
2 Management Ethernet
1-8
OL-23591-02
Chapter 1

12 DWDM controller(s)
12 TenGigE
40 GigabitEthernet
2 SONET/SDH
2 Packet over SONET/SDH
219k bytes of non-volatile configuration memory.
975M bytes of compact flash card.
33994M bytes of hard disk.
1605616k bytes of disk0: (Sector size 512 bytes).
Configuration register on node 0/RSP0/CPU0 is 0x0
Boot device on node 0/RSP0/CPU0 is disk0:
Package active on node 0/RSP0/CPU0:
asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0
Built on Wed Sep 8 16:17:30 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0
asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0
asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0
asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0
--More-.
.
.
The following example shows that the Cisco IOS XR software and active packages are version 4.0.0. If
there is an expected package missing or an active package is not an expected package, install and activate
the missing package or upgrade the unexpected package to the appropriate package. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for details on installing,
activating, and upgrading software packages.
RP/0/RSP0/CPU0:router# show install
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-doc-p-4.0.0
disk0:asr9k-k9sec-p-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mgbl-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/1/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
OL-23591-02
1-9
Chapter 1
Boot Device: mem:
Active Packages:
Boot Device: mem:
Active Packages:
Boot Device: mem:
Active Packages:
Validating the Installation

Validate the Cisco IOS XR software package installation to ensure the packages were installed correctly.
The following commands are used to validate the currently installed software packages:
install verify Command, page 1-10
show install active Command, page 1-12
show install committed Command, page 1-14
install verify Command

Use the install verify command to verify the consistency of a previously installed software set with the
package file from which it originated.
This command can be used as a debugging tool to verify the validity of the files that constitute the
packages to determine if there are any corrupted files. The command is also used to check that the install
infrastructure is up and running and to determine if all files are expected. If there are corrupted files, see
deactivating and removing software packages and adding and activating software packages.
1-10
OL-23591-02
Chapter 1

Note
The install verify command can take up to two minutes per package to process.
Note
The install verify command ignores secure domain router (SDR) boundaries and performs the operation
in global scope.
The following example shows the output of the install verify command. The output is used to verify the
consistency of a previously installed software set with the package file from which it originated.
RP/0/RSP0/CPU0:router(admin)# install verify
Sat Sep 25 08:18:14.077 DST
Install operation 3 '(admin) install verify packages' started by user_A
'dwolman-r' via CLI at 08:18:14 DST Sat Sep 25 2010.
The install operation will continue asynchronously.
RP/0/RSP0/CPU0:router(admin)#Info:
This operation can take up to 2 minutes per package
being verified.
Info:
Please be patient.
Info:
0/0/CPU0 [LC] [SDR: Owner]
Info:
meta-data: [SUCCESS] Verification Successful.
Info:
/install/asr9k-optics-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/asr9k-scfclient-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-video-adv-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/iosxr-mcast-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-routing-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-infra-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-diags-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.
Info:
Info:
meta-data: [SUCCESS] Verification Successful.
Info:
/install/asr9k-optics-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/asr9k-scfclient-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-video-adv-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/iosxr-mcast-4.0.0: [SUCCESS] Verification
OL-23591-02
1-11
Chapter 1
Info:
Successful.
Info:
/install/iosxr-routing-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-infra-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-diags-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.
Info:
.
.
.
Info:
Verification Summary:
Info:
0/0/CPU0: SUCCESSFUL. No anomalies found.
Info:
Info:
Info:
Info:
Info:
Info:
Info:
0/RSP0/CPU0: SUCCESSFUL. No anomalies found.
Info:
The system needs no repair.
Install operation 3 completed successfully at 08:19:48 DST Sat Sep 25 2010.
show install active Command

Use the show install active command to display active software packages. Verify that the command
output matches the output of the show install committed command. If the output does not match, when
you reload the router, the software displayed in the show install committed command output is the
software that will be loaded. For example, the following output shows two different software package
versions, one is the active version and the other is the committed version, so when the router reloads, the
3.9.1 version will be loaded even though 4.0.0 is the currently active version on 0/RSP0/CPU0.
RP/0/RSP0/CPU0:router(admin)# show install active location 0/RSP0/cpu0
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm << 4.0.0 is active, not committed
Active Packages:
RP/0/RSP0/CPU0:router(admin)# show install committed location 0/RSP0/cpu0
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-3.9.1/mbiasr9k-rp.vm<< 3.9.1 is committed
1-12
OL-23591-02
Chapter 1

Committed Packages:
If the expected active software packages are not displayed, install the packages (if required) and activate
the packages. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for
information on installing and activating Cisco IOS XR software packages. The following example
output shows the active packages for all cards in a router. The output displays the disk on which each
package is located.
RP/0/RSP0/CPU0:router# show install active
Boot Device: disk0:
Active Packages:
Boot Device: mem:
Active Packages:
Boot Device: mem:
Active Packages:
Boot Device: mem:
Active Packages:
Boot Device: mem:
OL-23591-02
1-13
Chapter 1

Active Packages:
Boot Device: mem:
Active Packages:
Boot Device: mem:
Active Packages:
Boot Device: mem:
Active Packages:
The output shows the name of the disk on which the packages are located. In the above example, the
active packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0
is active. Additional packages shown are optional packages that have been activated after the initial
loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.
show install committed Command

Use the show install committed command to display committed software packages. The committed
software packages are the software packages that will be booted on a router reload.
Committed packages are the packages that are persistent across router reloads. If you install and activate
a package, it remains active until the next router reload. If you commit a package set, all packages in that
set remain active across router reloads until the package set is replaced with another committed package
set. The show install committed command is useful to ensure software is installed and committed after
a router reload. If the expected software is not installed and committed, see
Cisco ASR 9000 Aggregation Services Router Getting Started Guide for information on installing and
committing Cisco IOS XR software packages.
The following command output shows the committed software packages on all cards in the router. The
output displays the disk on which each package is located.
RP/0/RSP0/CPU0:router# show install committed
1-14
OL-23591-02
Chapter 1

Boot Device: disk0:

Committed Packages:
Boot Device: mem:
Committed Packages:
Boot Device: mem:
Committed Packages:
Boot Device: mem:
Committed Packages:
Boot Device: mem:
Committed Packages:
Boot Device: mem:
Committed Packages:
Boot Device: mem:
OL-23591-02
1-15
Chapter 1

Committed Packages:
Boot Device: mem:
Committed Packages:
The output shows the name of the disk on which the packages are located. In the above example, the
committed packages for each node are on disk0, and for all nodes, the composite package
asr9k-os-mbi-4.0.0 is committed. Additional packages shown are optional packages that have been
committed after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing
Core Bundle.
Validating and Troubleshooting Cisco IOS XR Software

Configuration
Validating the Cisco IOS XR software configuration includes collecting configuration information on
the router to determine configuration changes and verifying the current running configuration. When a
configuration fails during a commit, the failed configuration can be viewed to help determine why the
configuration was not committed.
The following sections are provided:
Local and Global Configurations, page 1-16
Collecting Configuration Information, page 1-19
Verifying the Running Configuration, page 1-20
Using the show configuration failed Command, page 1-24
Local and Global Configurations

To troubleshoot configurations, you need to determine whether the problem is in the local configuration
or the shared (global) configuration.
The local configuration is specific to the individual LC or RP to which it belongs. Every LC and RP
has a data store containing the local data for that node, including configuration and operational data
for the local interfaces. An example of a local configuration is the port designations on a particular
LC.
The shared (global) configuration applies to the entire router, and is shared with all of the LCs and
RPs. An example of a shared configuration is the routing protocol parameters.
To view the local configuration, use the show running-config interface * command. The output
displays all the configured interfaces on the node.
1-16
OL-23591-02
Chapter 1

RP/0/RSP0/CPU0:router# show running-config interface *

interface Bundle-Ether16
description Connect to router-S Port-Ch 16
mtu 9216
bundle maximum-active links 1
!
interface Bundle-Ether16.160 l2transport
description Connect to router-S Port-Ch 16 Service Instance 160
encapsulation dot1q 160
!
.
.
.
!
interface Loopback0
ipv4 address 10.144.144.144 255.255.255.255
!
interface tunnel-ip10
!
interface tunnel-te44190
description Primary GE Tunnel from router-S to router-T
ipv4 unnumbered Loopback0
priority 0 0
autoroute announce
signalled-bandwidth 100000
destination 10.19.19.19
fast-reroute
record-route
path-option 1 explicit name Primary_GE_Path_to_router-T ospf 100 area 0
!
.
.
.
interface MgmtEth0/RSP0/CPU0/0
description Connected to LAN
ipv4 address 172.29.52.137 255.255.255.0
!
interface MgmtEth0/RSP0/CPU0/1
shutdown
!
interface GigabitEthernet0/0/0/0
shutdown
!
shutdown
!
shutdown
!
.
.
.
interface TenGigE0/7/0/1
shutdown
!
shutdown
!
shutdown
!
interface POS0/2/0/0
description Connected to PE_router-2 POS 0/2/0/0
OL-23591-02
1-17
Chapter 1
!
interface POS0/2/0/1
description Connected to PE_router-3 POS 0/2/0/1
!
controller SONET0/2/0/0
clock source internal
!
controller SONET0/2/0/1
clock source internal
!
Use the show sysdb trace commands to display the contents of the system database after a configuration
change. The trace information includes a history of any changes to the running configuration. You can
specify either a local node or the shared plane.
The following example output shows the contents of the local database, that is, for a specific location
(node):
RP/0/RSP0/CPU0:router# show sysdb trace verification location 0/5/cpu0 reverse
.
.
.
Timestamp
nid
reqid
jid
tid reg_hndl connid
action
path
432 wrapping entries (8192 possible, 158 filtered, 590 total)
Sep 23 04:35:39.969
0/RSP0/CPU0 8168
354
1
94
4483
reply
'--'
Sep 23 04:35:39.960
0/RSP0/CPU0 8168
354
1
94
4505
called
'cfg/if/act/tunnel-ip10/v'
Sep 23 04:35:39.960
0/RSP0/CPU0 8168
354
1
94
4505
reply: accept
'--'
Sep 23 04:35:39.685
0/RSP0/CPU0 8168
354
1
94
4505
called
'cfg/if/act/tunnel-ip10/v'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/mtu/tunnel-ip'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/im/bw'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/keepalive'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/dfbit_disable'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/ttl'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/tos'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/mode'
.
.
.
apply
Apply
verify
Verify
The following example output shows the contents of the shared database, that is, the configuration data
that is shared with all LC and RP in the router:
RP/0/RSP0/CPU0:router# show sysdb trace verification shared-plane reverse

Config Shared Server
====================
Timestamp
nid
reqid
jid
tid reg_hndl connid
path
action
1-18
OL-23591-02
Chapter 1

Sep 23 19:34:40.202
unregister
Sep 23 19:34:40.197
unregister
Sep 23 19:34:40.196
unregister
Sep 23 19:14:45.076
register
Sep 23 19:14:41.679
register
Sep 23 19:14:41.593
register
Sep 23 19:12:36.472
unregister
Sep 23 19:12:36.471
unregister
Sep 23 19:12:36.470
unregister
Sep 23 19:07:56.914
register
.
.
.
0/3/CPU0
0
241
8
384
1430
'from-'
0/3/CPU0
0
241
15
385
1434
'from-'
0/3/CPU0
0
163
1
386
1440
'from-'
0/3/CPU0
0
163
1
386
1440
'cfg/gl/ipv4/cef/hardware/forwarding/update/synchronous'
0/3/CPU0
0
241
15
385
1434
'cfg/gl/dbgtrace/node/831/'
0/3/CPU0
0
241
8
384
1430
'cfg/gl/dbgtrace/node/831/'
0/3/CPU0
0
241
8
381
1375
'from-'
0/3/CPU0
0
241
15
382
1378
'from-'
0/3/CPU0
0
163
1
383
1383
'from-'
0/3/CPU0
0
163
1
383
1383
'cfg/gl/ipv4/cef/hardware/forwarding/update/synchronous'
The show processes location node-id | include sysdb command displays all active database processes
for a specified node.
RP/0/RSP0/CPU0:router# show process location 0/1/CPU0 | include sysdb
Thu Nov 4 14:06:30.191 DST
279
1
0
56K 10 Sigwaitinfo 739:28:22:0145
0:00:00:0057 sysdb_svr_local
279
2
1
56K 10 Receive
0:00:00:0779
279
3
1
56K 10 Receive
0:03:34:0474
279
4
1
56K 10 Receive
0:05:03:0006
277
1
0
64K 10 Sigwaitinfo 739:28:21:0305
0:00:00:0046 sysdb_mc
277
2
0
64K 10 Receive
739:28:21:0274
0:00:00:0003 sysdb_mc
277
3
1
64K 10 Receive
166:59:14:0698
0:00:00:0038 sysdb_mc
277
4
1
64K 10 Receive
0:01:49:0941
0:00:00:0106 sysdb_mc
277
6
1
64K 10 Receive
739:15:22:0734
0:00:00:0058 sysdb_mc
See Chapter 1, General Troubleshooting Procedures for additional information on troubleshooting

processes.
Collecting Configuration Information

Collecting configuration information allows you to determine if changes to the system have occurred. It
also allows you to determine if these changes could impact the system. The following commands allow
you to determine if there was an unknown commit, if there was a commit that overwrote a previous
configuration, or there are configuration changes that should be removed from the running configuration.
show configuration commit changes {[since] commit-id | last number-of-commits} [diff]the

command output displays changes made to the running configuration by previous configuration
commits.
RP/0/RSP0/CPU0:router# show configuration commit changes since 1000000319
Wed May 17 09:30:27.877 UTC
Building configuration...
no logging console
no domain ipv4 host ce1
no domain ipv4 host ce2
domain ipv4 host ce6 172.29.52.73
domain ipv4 host ce7 172.29.52.78
OL-23591-02
1-19
Chapter 1
no domain ipv4 host pe1

no domain ipv4 host pe2
domain ipv4 host pe6 172.29.52.128
domain ipv4 host pe7 172.29.52.182
no negotiation
!
end
show configuration commit list [number-of-commits] [detail]the command output displays a list
of the commit IDs (up to 100) available for rollback.
RP/0/RSP0/CPU0:router# show configuration commit list
Wed May 17 09:31:21.727 UTC
SNo. Label/ID
User
Line
~~~~ ~~~~~~~~
~~~~
~~~~
1
1000000324 userA
vty0
2
1000000323 userA
vty0
3
1000000322 userB
vty0
4
1000000321 userC
vty2
5
1000000320 userA
vty2
6
1000000319 userB
vty2
7
1000000318 userB
vty2
8
1000000317 userB
vty2
9
1000000316 userC
vty2
10
1000000315 userC
vty2
11
1000000314 userA
vty2
12
1000000313 userA
vty2
13
1000000312 userD
con0_RSP0_C
Client
~~~~~~
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
Time Stamp
~~~~~~~~~~
16:50:33 UTC Wed May 10 2006
16:49:51 UTC Wed May 10 2006
16:48:05 UTC Wed May 10 2006
19:11:26 UTC Wed May 03 2006
19:10:45 UTC Wed May 03 2006
18:03:01 UTC Wed May 03 2006
18:02:43 UTC Wed May 03 2006
18:02:38 UTC Wed May 03 2006
17:59:16 UTC Wed May 03 2006
17:46:38 UTC Wed May 03 2006
15:40:04 UTC Wed May 03 2006
13:05:09 UTC Wed May 03 2006
13:49:31 UTC Mon May 01 2006
commit confirmed minutes (executed from config mode)This command commits the
configuration on a trial basis for a minimum of 30 seconds and a maximum of 300 seconds (5
minutes). During the trial configuration period, enter commit to confirm the configuration. If
commit is not entered, then the system will revert to the previous configuration when the trial time
period expires.
Verifying the Running Configuration

To verify the running configuration, perform the following procedure.
SUMMARY STEPS
1.
configure
2.
show running-config
3.
describe hostname hostname
4.
end
5.
show sysdb trace verification shared-plane | include path
6.
show sysdb trace verification location node-id
7.
show cfgmgr trace
8.
show configuration history commit
9.
show configuration commit changes {last | since | commit-id}
1-20
OL-23591-02
Chapter 1

10. show config failed startup

11. cfs check
DETAILED STEPS
Step 1
Command or Action
Purpose
configure
Enters global configuration mode.
Example:
Step 2
configure
Displays the contents of the running configuration.
show running-config
Verify that the running configuration is as expected.

Example:
RP/0/RSP0/CPU0:router(config)# show
running-config
Step 3
Determines the path.
describe hostname hostname
Example:
RP/0/RSP0/CPU0:router(config)# describe
hostname router_A
Step 4
Saves configuration changes.
end
Example:
RP/0/RSP0/CPU0:router(config)# end
When you issue the end command, the system prompts

you to commit changes:
Uncommitted changes found, commit them before
exiting(yes/no/cancel)?
[cancel]:
Entering yes saves configuration changes to the
running configuration file, exits the configuration

session, and returns the router to EXEC mode.
Entering no exits the configuration session and
returns the router to EXEC mode without

committing the configuration changes.
Entering cancel leaves the router in the current
configuration session without exiting or

committing the configuration changes.
OL-23591-02
1-21
Chapter 1
Step 5
Command or Action
Purpose
show sysdb trace verification shared-plane |

include path
Displays details of recent verification sysDB transactions

and changes on the shared plane allowing you to verify
whether the configuration was verified correctly.
Example:
Specifying the path filters the data to display only the

sysDB path for the router.
RP/0/RSP0/CPU0:router# show sysdb trace

verification shared-plane | include
gl/a/hostname
Step 6
show sysdb trace verification location node-id
Displays details of recent verification sysDB transactions

and changes on local plane configurations.
Example:
Verify that changes to the SysDB were verified and

accepted.
RP/0/RSP0/CPU0:router# show sysdb trace

verification location 0/3/CPU0
Step 7
Verify that changes to the SysDB were verified and

accepted.
Displays cfgmgr trace information.
show cfgmgr trace
Example:
Step 8
show cfgmgr trace
show configuration history commit
Displays a list of historical changes to the configuration.

Verify that the timeline of changes is as expected.
Example:
history commit
Step 9
show configuration
show configuration commit changes {last | since

| commit-id}
Displays detailed committed configuration history

information.
Verify that the history information is as expected.
Example:
RP/0/RSP0/CPU0:router# show configuration
commit changes last 15
Step 10
show configuration failed startup
Displays information on any configurations that failed

during startup.
Example:
failed startup
Step 11
show configuration
Checks the current configuration to see if there are any

missing configurations.
cfs check
Example:
cfs check
Examples
The following example shows the output of the show running-config command:
RP/0/RSP0/CPU0:router# show running-config
!! IOS XR Configuration 4.0.1.10I
!! Last configuration change at Thu Sep 23 04:35:38 2010 by user_A
!
hostname router
.
1-22
OL-23591-02
Chapter 1

.
.
logging suppress duplicates
telnet vrf default ipv4 server max-servers 100
domain name cisco.com
domain lookup disable
taskgroup default
!
.
.
.
interface preconfigure GigabitEthernet0/3/0/7
shutdown
!
shutdown
!
shutdown
!
shutdown
!
.
.
.
The output is used to determine if the configuration is as expected.

In the following example, the path to SysDB where the configuration is stored in the database is
displayed.
RP/0/RSP0/CPU0:router(config)# describe hostname router
Package:
iosxr-infra
iosxr-infra V4.0.0 IOS-XR Infra Package Definition
Vendor : Cisco Systems
Desc
: IOS-XR Infra Package Definition
Build : Built on Wed Sep 8 16:07:48 DST 2010
Source : By sc-g-01 in /auto/ioxbuild8/production/4.0.0/asr9k/workspace for pie
Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE,
NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500
Restart information:
Default:
parallel impacted processes restart
Size Compressed/Uncompressed: 38MB/85MB (44%)
Component:
shellutil V[ci-401/7]
Common shell utility applications
User needs ALL of the following taskids:

host-services (READ) or root-lr (READ WRITE)
It will take the following actions:
Create/Set the configuration item:
Path: gl/a/hostname
Value: router
OL-23591-02
1-23
Chapter 1
Using the show configuration failed Command

Use the show configuration failed command to browse a failed configuration. The configuration can be
classified as failed during startup or during a configuration commit.
Startup Failed Configuration, page 1-24
Commit Configuration Failed, page 1-25
Startup Failed Configuration

A configuration can be classified as failed during startup for three reasons:
Syntax errors
Syntax errors are generated by the parser and usually indicate that there is an incompatibility with
the CLI commands. Correct the syntax errors and reapply the configuration. A syntax error can be
an invalid CLI entry or a CLI syntax change. See the Obtaining Documentation and Submitting a
Service Request section on page xii in the Preface for information on obtaining
Cisco IOS XR software CLI documentation.
Semantic errors
Semantic errors are generated by the backend components when the configuration is being restored
by the configuration manager during startup of the router. Semantic errors include logical problems
(invalid logic).
Apply errors
Apply errors are generated when a configuration has been successfully verified and accepted as part
of running configuration but the backend component is not able to update its operational state. The
configuration shows both as the running configuration (since it was correctly verified) and as a failed
configuration because of the backend operational error. To find the component apply owner, use the
describe on the CLI that failed to be applied.
Note
You may browse startup failed configurations for up to the previous four router reloads.
Use the show configuration failed startup command and the load configuration failed startup
command to browse and reapply any failed configuration. The load configuration failed startup
command can be used in configuration mode to load the failed startup configuration into the target
configuration session, then the configuration can be modified and committed. See
committing a configuration.
RP/0/RSP0/CPU0:router# show configuration failed startup
!! CONFIGURATION FAILED DUE TO SYNTAX/AUTHORIZATION ERRORS
telnet vrf default ipv4
server max-servers 5 interface POS0/7/0/3 router static
address-family ipv4 unicast
0.0.0.0/0 172.18.189.1
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS
router bgp 217
!!% Process did not respond to sysmgr !
RP/0/RSP0/CPU0:router# config
1-24
OL-23591-02
Chapter 1

RP/0/RSP0/CPU0:router(config)# load config failed startup noerror

Loading. 263 bytes parsed in 1 sec (259)bytes/sec
RP/0/RSP0/CPU0:mike3(config-bgp)#show configuration
telnet vrf default ipv4 server max-servers 5 router static
0.0.0.0/0 172.18.189.1
!
!
router bgp 217
!
end
The failed configuration is loaded into the target configuration, minus the errors that caused the startup
configuration to fail.
RP/0/RSP0/CPU0:router(config-bgp)# commit
Use the show configuration failed command to display failed items in the last configuration commit,
including reasons for the error.
In any mode, the configuration failures from the most recent commit operation are displayed.
The show configuration failed command can be used in EXEC mode and configuration mode. The
command is used in EXEC mode when the configuration does not load during startup. The command is
used in configuration mode to display information when a commit fails.
The following example shows the show configuration failed command.
RP/0/RSP0/CPU0:router(config)# interface pos 0/6/0/4
RP/0/RSP0/CPU0:router(config-if)# no vrf
RP/0/RSP0/CPU0:router(config-if)# commit
% Failed to commit one or more configuration items during an atomic operation, no changes
have been made. Please use 'show configuration failed' to view the errors
RP/0/RSP0/CPU0:router(config-if)# exit
RP/0/RSP0/CPU0:router(config)# show configuration failed
Wed May 2 13:14:08.426 EST EDT
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS interface POS0/6/0/4 no vrf !!
% The interface's numbered and unnumbered IPv4/IPv6 addresses must be removed prior to
changing or deleting the VRF !
Note
The show configuration failed command in configuration mode only exists as long as the configuration
session is active. Once you exit configuration mode, the command cannot be used to display the failed
configuration.
Commit Configuration Failed

The following example shows an invalid task ID configuration that fails to commit. The show
configuration failed command provides information on why the configuration failed.
RP/0/RSP0/CPU0:router(config)# taskgroup isis
RP/0/RSP0/CPU0:router(config-tg)# commit
% Failed to commit one or more configuration items during an atomic operation, s
RP/0/RSP0/CPU0:router(config-tg)# show configuration failed
OL-23591-02
1-25
Chapter 1
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS

taskgroup isis
!!% Usergroup/Taskgroup names cannot be taskid names
!
If a configuration commit fails, do not exit configuration mode (return to EXEC mode) as you will not
be able to view the failed configuration.
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# taskgroup bgp
RP/0/RSP0/CPU0:router(config-tg)# end
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:y
% Failed to commit one or more configuration items during an atomic operation, s
RP/0/RSP0/CPU0:router(config)# exit
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:n
RP/0/RSP0/CPU0:router# show configuration failed

To verify the general status and state of a router using Cisco IOS XR software, perform the following
procedure.
SUMMARY STEPS
1.
admin
2.
show platform [node-id]
3.
show version
4.
show running-config
5.
show logging
6.
show environment
7.
show context
8.
exit
9.
show context
10. show memory summary detail location all

11. show memory heap summary {job-id | all}
12. top processes
13. show running-config
14. show system verify start
show system verify report

15. show {ipv4 | ipv6} interface brief
1-26
OL-23591-02
Chapter 1

DETAILED STEPS
Step 1
Command or Action
Purpose
admin
Enters administration mode.
Example:
Step 2
admin
Displays information about the status of cards and modules

installed in the router.
show platform [node-id]
Example:
Some cards support a CPU module and service

processor (SP) module. Other cards support only a
single module.
A card module is also called a node. When all nodes are

working properly, the status of each node displayed in
the State column is IOS-XR RUN.
If you run the command without a node-id (show

platform as shown in the example), the output will
include all nodes in the system.
Type the show platform node-id command to display

information for a specific node. Replace node-id with a
node name from the show platform command Node
column.
RP/0/RSP0/CPU0:router(admin)# show platform
Step 3
show version
Displays information about the router, including image

names, uptime, and other system information.
Example:
Verify that the expected software version and images are

installed.
RP/0/RSP0/CPU0:router(admin)# show version
Step 4
show running-config
Example:
RP/0/RSP0/CPU0:router(admin)# show
running-config
Displays all of the nondefault commands currently running,

including hardware module power status, secure domain
router (SDR) configuration, and fabric configuration. The
output also displays the users defined in administration
mode with root-system access.
Verify that the serial numbers for the nodes in the current
running configuration are what you expected. The expected
rack numbers and serial numbers should be listed in the
current system documentation. See the Prerequisite
Documentation for Troubleshooting section on page 1-1.
Also verify that the hardware module power status is as
expected and the SDR and fabric configurations are as
expected.
Step 5
Displays all syslog messages stored in the buffer. The

command output displays the device operation history from
a system perspective.
show logging
Example:
RP/0/RSP0/CPU0:router(admin)# show logging
Analyze the logged events and their order of happening.

Check for anything out of the ordinary such as errors,
tracebacks, or crashes. Also check for any Severity 1 or
Severity 2 errors.
OL-23591-02
1-27
Chapter 1
Step 6
Command or Action
Purpose
show environment
Displays environmental monitor parameters for the system.

Verify that the parameters are as expected.
Example:
RP/0/RSP0/CPU0:router(admin)# show environment
Step 7
show context
Example:
RP/0/RSP0/CPU0:router(admin)# show context
Step 8
Displays core dump context information on fabric cards,

alarm modules, fan controllers, and service processors
(system-owned cards). See the show context Command
section on page 1-50 for more information on the show
context command output.
Exits administration mode.
exit
Example:
RP/0/RSP0/CPU0:router(admin)# exit
Step 9
show context
Example:
Step 10
show memory summary detail location all
Displays information about the memory available on the

router after the system image decompresses and loads.
Example:
Verify that the expected memory is available or installed.

Ensure that all memory regions have adequate free space
available.
detail location all
Step 11
show context
Displays core dump context information on CPUs

responsible for routing and Cisco Express Forwarding
(CEF). See the show context Command section on
page 1-50 for more information on the show context
command output.
show memory summary
show memory heap summary {job-id | all}
Example:
summary all
show memory heap
Displays a summary of the information about the heap

space. The output displays each process and the amount of
memory allocated for each process.
Note
The job-id is the output of the show processes

command.
Verify if there are any processes using a large amount of

memory.
Step 12
Provides a live update of process resource consumption.
top processes
Press M to sort by memory usage.

Example:
RP/0/RSP0/CPU0:router# top processes
Verify that the resource consumption is as expected.
Press q to exit the command.

Step 13
Displays the contents of the currently running

configuration.
show running-config
Example:
show running-config
Verify that the contents of the current running configuration

are what you expected.
1-28
OL-23591-02
Chapter 1

Step 14
Command or Action
Purpose
show system verify start

show system verify report
A two-step command that produces system reports.
Example:
start
report
show system verify

show system verify
show system verify startStarts the system verify

process (creates the initial baseline file)
show system verify reportGenerates a report for the

system verification process (report of the current
status)
The output of the show system verify report command

provides a comparison of the system at the time of the show
system verify start snapshot and the show system verify
report snapshot. The output provides a sanity check of the
system provided the show system verify start system
snapshot was taken when the system was healthy or before
an event.
Verify that the system parameters are as expected.
Step 15
show (ipv4 | ipv6} interface brief
Displays the usability status of interfaces.
Example:
Verify that all expected interfaces are listed, that they have
the correct assigned address, and that they are in the
expected states.
brief
show ipv4 interface
Examples
The output from the show platform command indicates that all expected nodes are in the run state. If
all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the
OK state. The example output shows that all expected nodes are in the run state.
RP/0/RSP0/CPU0:router(admin)# show platform
Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/FT0/SP
FAN TRAY
READY
0/FT1/SP
FAN TRAY
READY
0/1/CPU0
A9K-40GE-B
IOS XR RUN
PWR,NSHUT,MON
0/2/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,MON
0/2/0
SPA-2XOC48POS/RPR
OK
PWR,NSHUT,MON
0/3/CPU0
A9K-2T20GE-B
IN-RESET
PWR,NSHUT,MON
0/4/CPU0
A9K-8T/4-B
IOS XR RUN
PWR,NSHUT,MON
0/6/CPU0
A9K-4T-B
IOS XR RUN
PWR,NSHUT,MON
0/PM0/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM1/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM2/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
The output from the show version command indicates the version of software being run on the nodes
and from which location (disk or network). Check that the expected software version and images are
installed. The example output shows that the Cisco IOS XR software version is 4.0.0 and that the
installed pie versions are also 4.0.0.
RP/0/RSP0/CPU0:router(admin)# show version
Cisco IOS XR Software, Version 4.0.0[Default]
Copyright (c) 2010 by Cisco Systems, Inc.
ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],
OL-23591-02
1-29
Chapter 1
router uptime is 1 day, 15 hours, 53 minutes

System image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"
cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory.
MPC8641D processor at 1333MHz, Revision 2.2
2 Management Ethernet
12 DWDM controller(s)
12 TenGigE
40 GigabitEthernet
2 SONET/SDH
2 Packet over SONET/SDH
219k bytes of non-volatile configuration memory.
975M bytes of compact flash card.
33994M bytes of hard disk.
Configuration register on node 0/RSP0/CPU0 is 0x102
Boot device on node 0/RSP0/CPU0 is disk0:
Package active on node 0/RSP0/CPU0:
asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0
asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0
asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0
asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0
asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0
iosxr-security, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:iosxr-security-4.0.0
--More--
The output from the show running-config command displays the current running configuration, that is,
all of the nondefault commands currently active. Verify that the contents of the current running
configuration are as expected.
Tip
The output of this command in exec mode is different from the output in admin mode. You should run
the command from each of these modes to locate all of the configuration information.
!! Last configuration change at 18:56:31 UTC Tue Feb 28 2006 by user_A
!
hostname router
clock timezone PST 8.
1-30
OL-23591-02
Chapter 1

logging console informational

telnet vrf default ipv4 server max-servers 100
domain name cisco.com
domain lookup disable
taskgroup default
!
taskgroup igpadmin
task write rib
task write isis
task write ospf
!
taskgroup routeadmin
task write bgp
inherit taskgroup igpadmin
!
usergroup default
taskgroup default
!
usergroup igp_admin
taskgroup igpadmin
!
usergroup route_admin
taskgroup routeadmin
!
tacacs-server host 172.29.52.69 port 49
key 7 060506324F41
aaa group server tacacs+ DOC_LAB_TACACS+
server 172.29.52.69
server 172.29.52.68
!
aaa authorization exec LAB_AAA group DOC_LAB_TACACS+ local
aaa authorization exec CONSOLE_AAA group DOC_LAB_TACACS+ none
aaa authorization commands LAB_AAA group DOC_LAB_TACACS+ none
aaa authorization commands CONSOLE_AAA group DOC_LAB_TACACS+ none
aaa authentication login default group DOC_LAB_TACACS+ local
aaa authentication login CONSOLE_AAA group DOC_LAB_TACACS+ local
aaa default-taskgroup default
explicit-path name Primary_GE_Path_to_P19
index 1 next-address strict ipv4 unicast 10.114.4.44
!
line console
accounting exec CONSOLE_AAA
accounting commands CONSOLE_AAA
authorization exec CONSOLE_AAA
authorization commands CONSOLE_AAA
login authentication CONSOLE_AAA
exec-timeout 600 0
session-timeout 600
!
line default
exec-timeout 600 0
session-timeout 600
!
.
.
.
shutdown
OL-23591-02
1-31
Chapter 1
!
shutdown
!
interface preconfigure TenGigE0/3/0/0
shutdown
!
interface preconfigure TenGigE0/3/0/1
shutdown
!
router static
0.0.0.0/0 172.29.52.1 200
!
!
router isis 100
is-type level-2-only
net 49.0001.0000.0000.0044.00
nsf cisco
metric-style wide
mpls traffic-eng level-2-only
mpls traffic-eng router-id Loopback0
!
interface Loopback0
passive
!
!
bfd minimum-interval 50
bfd multiplier 3
bfd fast-detect ipv4
!
!
!
router ospf 100
nsr
router-id 10.144.144.144
bfd minimum-interval 50
bfd multiplier 3
mpls ldp sync
nsf cisco
area 0
mpls ldp sync-igp-shortcuts
mpls traffic-eng
interface Loopback0
passive enable
!
.
.
.
http server
ssh server vrf default
igmp snooping profile default
system-ip-address 10.144.144.144
minimum-version 2
internal-querier
tcn query solicit
ttl-check disable
router-alert-check disable
!
igmp snooping profile mrouter
1-32
OL-23591-02
Chapter 1

router-guard
mrouter
!
router pim
address-family ipv4
mofrr mofrr-acl
rp-address 10.11.11.11
rp-address 10.144.144.144 bidir-acl bidir
spt-threshold infinity
!
!
end
RP/0/RSP0/CPU0:router# admin
Wed Oct 27 14:52:07.000 DST
RP/0/RSP0/CPU0:router(admin)# show running-config
Wed Oct 27 14:52:12.766 DST
!! IOS XR Admin Configuration 4.0.0
username doclabuser-c
group root-system
group cisco-support
secret 5 $1$RJVQ$6w7saUHgk16v5HXRWEp6m/
!
username doclabuser-r
group root-system
secret 5 $1$.uOF$O9N0aRRk.V1qe250IavLw1
!
alias cr copy run disk0a:/usr/base_config_admin
alias sa show alias
alias sc show config commit list
alias sd show diag
alias si show install req
alias sl show led
alias sp show platform
alias sr show run
alias sv show version
alias nda no debug all
end
The output from the show logging command displays the contents of the logging buffer. The output
displays details on syslog historical events. Analyze the logged events and the order in which they
happened. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for
any Severity 1 or Severity 2 errors.
RP/0/RSP0/CPU0:router(admin)# show logging
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
Console logging: level informational, 693 messages logged
Monitor logging: level debugging, 0 messages logged
Trap logging: level informational, 0 messages logged
Buffer logging: level debugging, 4467 messages logged
Log Buffer (307200 bytes):
LC/0/3/CPU0:Sep 13 23:58:03.272 : pfm_node_lc[230]: %PLATFORM-NP-0-NP_INIT_FAILURE :
Set|prm_server[110670]|Network Processor Unit(0x1007000)|Persistent Initialization
Failure.
OL-23591-02
1-33
Chapter 1
LC/0/3/CPU0:Sep 13 23:58:03.276 : pfm_node_lc[230]: %PLATFORM-PFM-0-CARD_RESET_REQ :

pfm_dev_sm_perform_recovery_action, Card reset requested by: Process ID: 110670
(prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device Handle: 0x1007000,
CondID: 1027, Fault Reason: Persistent Initialization Failure.
LC/0/3/CPU0:Sep 13 23:58:03.276 : syslog_dev[85]: pfm_node_lc[230]: Request Graceful
Reboot via Sysmgr: Reason: pfm_dev_sm_perform_recovery_action, Card reset requested by:
Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device
Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure.
LC/0/3/CPU0:Sep 13 23:58:03.277 : sysmgr[87]: %OS-SYSMGR-2-REBOOT : reboot required,
process (pfm_node_lc) reason (pfm_dev_sm_perform_recovery_action, Card reset requested by:
Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device
Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure. )
LC/0/3/CPU0:Sep 13 23:58:03.467 : sysmgr[87]: %OS-SYSMGR-3-ERROR :
sysmgr_shutdown_cleanup_handler: shutdown script execution timed-out! Node will reset
LC/0/3/CPU0:Sep 13 23:58:03.467 : sysmgr[87]: %OS-SYSMGR-7-DEBUG :
sysmgr_shutdown_cleanup_handler: shutdown script execution timed-out! Node will reset
LC/0/3/CPU0:Sep 13 23:58:03.468 : sysmgr[87]: %OS-SYSMGR-3-ERROR :
sysmgr_shutdown_cleanup_handler: shutdown triggered by (pfm_node_lc) did not complete in
45 seconds, shutting down
RP/0/RSP0/CPU0:Sep 13 23:58:16.859 : shelfmgr[299]:
%PLATFORM-SHELFMGR-0-MAX_RESET_BRINGDOWN : Can not boot node 0/3/CPU0 A9K-2T20GE-B due to
multiple resets, putting it IN_RESET state. The probable cause is an unexpected event on
the node or a failure in communication with the node. Please refer to the Cisco ASR 9000
System Error Message Reference Guide for further information if needed.
--More--
The output from the show environment command displays environmental monitor parameters for the
system. Verify that the environment parameters are as expected. Environment parameter anomalies are
logged in the syslog, so if an environment parameter displayed in the show environment command
output is not as expected, check the syslog using the show logging command. The syslog provides details
on any logged problems.
RP/0/RSP0/CPU0:router(admin)# show environment
Wed Sep 15 09:48:27.178 DST
Temperature Information
--------------------------------------------R/S/I
Modules Sensor
(deg C)
host
host
Inlet0
Hotspot0
36.4
46.7
spa0
spa0
InletTemp
Hotspot
35.5
35.5
host
host
Inlet0
Hotspot0
34.5
61.0
host
host
Inlet0
Hotspot0
31.1
32.5
host
host
Inlet0
Hotspot0
31.3
42.0
0/1/*
0/2/*
0/3/*
0/RSP0/*
0/4/*
1-34
OL-23591-02
Chapter 1

host
host
Inlet0
Hotspot0
34.9
45.2
host
host
Inlet0
Hotspot0
38.0
48.9
host
host
Inlet0
Hotspot0
31.9
33.5
host
host
Inlet0
Hotspot0
32.3
35.3
0/6/*
0/FT0/*
0/FT1/*
Voltage Information
--------------------------------------------R/S/I
Modules Sensor
(mV)
Margin
0/1/*
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
IBV
5.0V
VP3P3_CAN
3.3V
2.5V
1.8VB
1.2VB
1.8VA
0.9VB
1.2V_LDO_BRG0
1.2V_LDO_BRG1
1.8VC
1.5VB
1.5VA
1.1V(1.05V_CPU)
0.75VA
0.75VB_0.75VC
1.1VB
1.2V_TCAM0
1.2V_TCAM1
1.0V_Bridge_LDO
1.0VB
0.75VD_and_0.75VE
1.2V_TCAM2
1.2V_TCAM3
1.5VC
1.8VD
1.1VC
ZARLINK_3.3V
ZARLINK_1.8V
1.2V_DB
3.3V_DB
2.5V_DB
1.5V_DB
10592
4925
3289
3302
2516
1812
1193
1806
886
1193
1195
1811
1505
1503
1053
752
754
1103
1003
1000
999
1042
752
1006
1002
1504
1804
1100
3272
1807
1195
3318
2535
1509
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
.
.
.
LED Information
OL-23591-02
1-35
Chapter 1
--------------------------------------------R/S/I
Modules
0/RSP0/*
host
host
host
host
host
LED
Status
Critical-Alarm
Major-Alarm
Minor-Alarm
ACO
Fail
Off
Off
Off
Off
Off
Fan Information
--------------------------------------------Fan speed (rpm):
FAN0
FAN1
FAN10
FAN11
FAN2
FAN3
FAN4
FAN5
FAN6
FAN7
FAN8
FAN9
3510
3510
3480
3540
3540
3510
3390
3510
3510
3540
0/FT0/*
3510
3480
3510
3570
3540
3480
3480
0/FT1/*
3510
3510
3510
3540
3480
3450
3480
Power Supply Information
--------------------------------------------R/S/I
Modules
Capacity
(W)
Status
0/PM0/*
host
PM
3000
Ok
host
PM
3000
Ok
host
PM
3000
Ok
Voltage
(V)
53.7
53.8
54.1
Current
(A)
11.6
14.4
0.0
0/PM1/*
0/PM2/*
R/S/I
Power Draw
(W)
0/PM0/* 622.9
0/PM1/* 774.7
0/PM2/* 0.0
-------------Total: 1397.6
Power Shelves Type: AC

Total Power Capacity:
Usable Power Capacity:
Supply Failure Protected Capacity:
Feed Failure Protected Capacity:
Worst Case Power Used:
9000W
9000W
6000W
3000W
3170W
Slot
---0/1/CPU0
0/2/CPU0
0/RSP0/CPU0
0/RSP1/CPU0
0/4/CPU0
0/6/CPU0
0/FT0/SP
Max Watts
--------350
450
235
235
350
350
600
(default)
1-36
OL-23591-02
Chapter 1

0/FT1/SP
600
Worst Case Power Available:

Supply Protected Capacity Available:
Feed Protected Capacity Available:
5830W
2830W
Not Protected
The output from the show context command displays core dump context information. See the show
context Command section on page 1-50 for more information on the show context command output.
RP/0/RSP0/CPU0:router# show context
node:
node0_1_CPU0
-----------------------------------------------------------------Crashed pid = 61524 (pkg/bin/tcam_mgr)
Crashed tid = 1
Crash time: Wed Apr 05, 2006: 18:27:26
Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
R0
R4
R8
R12
R16
R20
R24
R28
R32
R36
r0
00000000
r4
0000f054
r8
00000000
r12
4823be90
r16
00000048
r20
00000000
r24
00000003
r28
00000006
cnt
00000000
cnd
28004024
Registers info
r1
r2
481ff7b0 4824a55c
r5
r6
00000001 00000006
r9
r10
fc220000 481fffc0
r13
r14
4824a4a0 48230000
r17
r18
00000001 00000019
r21
r22
00000000 00000003
r25
r26
00000000 00000003
r29
r30
0000f054 48254064
lr
msr
fc1c6340 0000d932
xer
00000008
r3
00000000
r7
00000000
r11
00000000
r15
00000000
r19
48256520
r23
00000045
r27
4825dc34
r31
481ff810
pc
fc1d3fa0
DLL Info
DLL path
Text addr. Text size
Data addr. Data size
Version
/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8
0
/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000
0x00000bbc
0
Crash Package Infomation

Package: hfr-mgbl, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
OL-23591-02
1-37
Chapter 1
Package: hfr-mcast, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo

rkspace for c2.95.3-p8
Package: hfr-mpls, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-rout, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-k9sec, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
Package: hfr-lc, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/works
pace for c2.95.3-p8
Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-os-mbi, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/w
orkspace for c2.95.3-p8
node:
node0_6_CPU0
-----------------------------------------------------------------node:
node0_RSP0_CPU0
-----------------------------------------------------------------node:
node0_RSP1_CPU0
------------------------------------------------------------------
The example output shows that the pkg/bin/tcam_mgr process crashed.

The output from the show memory command displays information about the memory available on the
router after the system image decompresses and loads. Verify that the expected memory is available or
installed. Ensure that all memory regions have adequate free space available. The example output shows
that there is 2.003 gigabits of application memory available.
RP/0/RSP0/CPU0:router# show memory summary detail location all
Physical Memory: 4.000G total (2.003G available)
Application Memory : 3.826G (2.003G available)
Image: 48.725M (bootram: 48.725M)
Reserved: 128.000M, IOMem: 1.980G, flashfsys: 0
Shared window mfwdv6: 449.910K
Shared window mfwd_info: 701.910K
Shared window soasync-app: 242.402K
Shared window soasync: 242.402K
Shared window li: 3.359K
Shared window ipv4_fib: 1.003M
Shared window l2fib: 2.425M
Shared window statsd_db: 67.386K
Shared window mgid: 587.390K
Shared window ifc-protomax: 1.290M
Shared window ifc-mpls: 7.981M
Shared window ifc-ipv6: 7.212M
Shared window ifc-ipv4: 11.286M
Shared window infra_statsd: 3.402K
Shared window im_rd: 1.104M
Shared window im_db: 1.204M
Shared window infra_ital: 67.316K
Shared window netio_fwd: 292
Shared window vkg_bmp_adj: 211.371K
Shared window aib: 623.375K
Shared window rspp_ma: 3.351K
Shared window im_rules: 293.308K
Shared window aaa: 67.382K
1-38
OL-23591-02
Chapter 1

Shared window pfm_node: 131.304K

Shared window atc_cache: 35.359K
Shared window spp: 619.312K
Shared window qad: 134.707K
Shared window pcie-server: 43.246K
Total shared window: 37.931M
Allocated Memory: 378.742M
Program Text: 46.175M
Program Data: 37.796M
Program Stack: 16.539M
The show system verify start command starts the system verification process and the show system
verify report generates the output from the system verification process. The output allows you to verify
that the system parameters are as expected.
RP/0/RSP0/CPU0:router# show system verify start
Storing initial router status ...
done.
The example output compares the system from the time the show system verify start command took the
first snapshot to the snapshot taken of the system when the show system verify report command took
the second snapshot and generated the comparison. If there are no changes, [OK] is displayed. If there
are changes between the first and second snapshot, the specific change is noted and marked with
[WARNING] or [FAIL].
RP/0/RSP0/CPU0:router# show system verify report
Getting current router status ...
System Verification Report
==========================
- Verifying Memory Usage
- Verified Memory Usage
- Verifying CPU Usage
- Verified CPU Usage
-
Verifying Blocked Processes

Verified Blocked Processes
Verifying Aborted Processes
Verified Aborted Processes
Verifying Crashed Processes
Verified Crashed Processes
- Verifying LC Status
- Verified LC Status
- Verifying QNET Status
Unable to get current LC status info
- Verified QNET Status
-
Verifying GSP Fabric Status

Verified GSP Fabric Status
Verifying GSP Ethernet Status
Verified GSP Ethernet Status
Verifying POS interface Status

Verified POS interface Status
Verifying TenGigE interface Status
Verified TenGigE interface Status
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
: [FAIL]
: [OK]
: [OK]
: [OK]
: [OK]
- Verifying TCP statistics

- Verified TCP statistics
: [OK]
- Verifying UDP statistics
tcp_udp_raw WARNING messages for router
UDP Packets sent has not increased during this period.
OL-23591-02
1-39
Chapter 1
- Verified UDP statistics

- Verifying RAW statistics
- Verified RAW statistics
-
: [WARNING]
: [OK]
Verifying RIB Status

Verified RIB Status
Verifying CEF Status
Verified CEF Status
Verifying CEF Consistency Status
Verified CEF Consistency Status
Verifying BGP Status
Verified BGP Status
Verifying ISIS Status
Verified ISIS Status
Verifying OSPF Status
Verified OSPF Status
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
- Verifying Syslog Messages

- Verified Syslog Messages
: [OK]
System may not be stable. Please look into WARNING messages.
The show interface brief command displays the usability status of the configured interfaces. Verify that
all expected interfaces are listed. For an interface to be usable, both the interface hardware (Status) and
line protocol must be up. The protocol is Up if the interface can provide two-way communication. The
example output displays IP addresses, status, and protocol status for each interface. The output shows
that all assigned interfaces (interfaces that are configured with IP addresses) have an interface hardware
status and line protocol status of Up.
RP/0/RSP0/CPU0:router# show ipv4 interface brief
Interface
Bundle-Ether16
Bundle-Ether16.160
Bundle-Ether16.161
Bundle-Ether16.162
Bundle-Ether16.163
Loopback0
tunnel-te44190
tunnel-te44192
tunnel-te44194
tunnel-te44196
GigabitEthernet0/1/0/0
GigabitEthernet0/1/0/3.160
IP-Address
unassigned
unassigned
unassigned
10.194.8.44
10.194.12.44
10.144.144.144
10.144.144.144
10.144.144.144
10.144.144.144
10.144.144.144
172.29.52.137
unassigned
unassigned
unassigned
10.147.4.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
Status
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Shutdown
Shutdown
Shutdown
Up
Up
Up
Up
Up
Up
Up
Shutdown
Shutdown
Shutdown
Up
Up
Up
Up
Up
Up
Up
Protocol
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Down
Down
Down
Up
Up
Up
Up
Up
Up
Up
Down
Down
Down
Up
Up
Up
Up
Up
Up
Up
1-40
OL-23591-02
Chapter 1

POS0/2/0/0
POS0/2/0/1
TenGigE0/4/0/0
TenGigE0/4/0/1
TenGigE0/4/0/2
TenGigE0/4/0/3
TenGigE0/4/0/4
TenGigE0/4/0/5
TenGigE0/4/0/6
TenGigE0/4/0/7
TenGigE0/6/0/0
TenGigE0/6/0/1
TenGigE0/6/0/2
TenGigE0/6/0/3
10.146.4.44
unassigned
unassigned
unassigned
10.194.16.44
unassigned
unassigned
unassigned
unassigned
unassigned
10.194.4.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
10.114.4.44
unassigned
unassigned
unassigned
10.145.4.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
10.114.8.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
Up
Shutdown
Shutdown
Shutdown
Up
Shutdown
Shutdown
Shutdown
Up
Up
Up
Up
Up
Up
Up
Up
Shutdown
Shutdown
Up
Shutdown
Shutdown
Shutdown
Up
Shutdown
Shutdown
Up
Up
Up
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Up
Up
Up
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Up
Down
Down
Down
Up
Down
Down
Down
Up
Up
Up
Up
Up
Up
Up
Up
Down
Down
Up
Down
Down
Down
Up
Down
Down
Up
Up
Up
Down
Down
Down
Down
Down
Down
Down
Down
Up
Up
Up
Down
Down
Down
Down
Down
Down
Down
Down
Down
Down
Down

This section describes techniques that you can use to troubleshoot the control plane Ethernet network on
routers using Cisco IOS XR software. The system control plane Ethernet network is used for processes
on different devices to communicate for functions such as system device discovery, image transfers,
heartbeat messages, alarms, and configuration management.
OL-23591-02
1-41
Chapter 1
All devices in a system using Cisco IOS XR software connect to the system control plane Ethernet
network, also called the Ethernet over backplane channel (EOBC). The control plane is provided using
Gigabit Ethernet (GE) links between nodes. The GE links are internal to the chassis and cannot be
removed.
1-42
OL-23591-02
Chapter 1

Figure 1-1 shows the control plane Ethernet network (the dotted line in the drawing).
Figure 1-1
Cisco ASR 9000 Aggregation Services Router Control Ethernet Topology
RSP 0
RSP 1
Fabric
Interface
Chip
System
Timing
Fabric
Chip
Fabric
Interface
Chip
CPU
VOQ
Scheduler
System
Timing
GE
Switch
CPU
Data Plane
Fabric
Chip
VOQ
Scheduler
Control Plane
GE
Switch
Backplane
40x1GE
Line Card
Fabric
Interface
Chip
8x10GE 2:1
Fabric
Oversubscribed Interface
Line Card
Chip
GE PHY
CPU
FPGA
NPU
NPU
10 x
SFP
10 x
SFP
GE PHY
FPGA
NPU
NPU
CPU
FPGA
NPU
NPU
FPGA
NPU
X
F
P
10 x
SFP
X
F
P
X
F
P
X
F
P
X
F
P
X
F
P
X
F
P
FPGA
NPU
NPU
NPU
NPU
10
GE
10
GE
10
GE
10
GE
X
F
P
X
F
P
X
F
P
X
F
P
10 10 10 10 10 10 10 10
GE GE GE GE GE GE GE GE
10 x
SFP
Fabric
Interface
Chip
CPU
FPGA
NPU
4x10GE
Line Card
GE PHY
X
F
P
Backplane
8x10GE 80G
Line Rate Card
2x10GE + 20x1GE
Combo Line Card
Fabric
Interface
Chip
Fabric
Interface
Chip
GE
PHY
FPGA
FPGA
CPU
FPGA
FPGA
CPU
NPU NPU NPU NPU
NPU NPU NPU NPU
GE
SW
NPU NPU
NPU NPU
GE
SW
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
To
NPUs
To
FPGAs
10x 10x
S
F
P
S
F
P
GE
PHY
10
GE
X
F
P
10
GE
X
F
P
To
NPUs
To
FPGAs
247272
10
GE
X
F
P
Fabric
Interface
Chip
OL-23591-02
1-43
Chapter 1
To verify and troubleshoot booting of the system control plane Ethernet network, perform the following
procedure.
SUMMARY STEPS
1.
show platform
2.
show controllers backplane ethernet clients all location node-id
3.
show controllers backplane ethernet clients 18 statistics location node-id
4.
Contact Cisco Technical Support if the problem is not resolved.
DETAILED STEPS
Step 1
Command or Action
Purpose
show platform
Displays information about the status of cards and modules

installed in the router.
Example:
Verify that the expected nodes display IOS XR RUN under

the State column of the command output.
Step 2
show platform
show controllers backplane ethernet clients all

location node-id
Displays information about all local client applications.

Each row contains the client Ethernet server ID and the
client process ID (PID).
Example:
The system allows client processes to send and receive

packets over the control Ethernet. It uses client IDs to
demultiplex packets that arrive at the node.
RP/0/RSP0/CPU0:router# show controllers

backplane ethernet clients all location
0/RSP0/CPU0
Two client IDs in the output are important for

troubleshooting boot problems:
Step 3
show controllers backplane ethernet clients 18

statistics location node-id
Client Ethernet server ID 18used for boot requests
Client Ethernet server ID 22used for heartbeats
Displays a list of client statistics for the specified client ID.

Check the values for:
Example:
backplane ethernet clients 18 statistics
location 0/RSP0/CPU0
Packets input
Packets delivered
If they contain values other than 0, boot requests have been

received and replies have been sent (packets output).
If they contain values of 0, check the system control plane
Ethernet network physical connectivity.
If there are no problems with the physical connectivity,
contact Cisco Technical Support. For Cisco Technical
Support contact information, see the Obtaining
Documentation and Submitting a Service Request section
on page xii.
Step 4
Contact Cisco Technical Support if the problem is not If the problem is not resolved, contact Cisco Technical
resolved.
Service Request section on page xii.
1-44
OL-23591-02
Chapter 1

Examples
The output from the show platform command indicates that all expected nodes are in the run state. If
all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the
OK state. The example output shows that all expected nodes are in the run state.
RP/0/RSP0/CPU0:router# show platform
Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/1/CPU0
A9K-40GE-B
IOS XR RUN
PWR,NSHUT,MON
0/2/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,MON
0/2/0
SPA-2XOC48POS/RPR
OK
PWR,NSHUT,MON
0/3/CPU0
A9K-2T20GE-B
IN-RESET
PWR,NSHUT,MON
0/4/CPU0
A9K-8T/4-B
IOS XR RUN
PWR,NSHUT,MON
0/6/CPU0
A9K-4T-B
IOS XR RUN
PWR,NSHUT,MON
The following example shows the current state of each Ethernet server client.
RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients all location
0/RSP0/CPU0
Intf
Client ethernet
Client
Description
Name
server id
Process Id
-----------------------------------------------------------------------------GE0_RSP0_CPU0
1
110639
QNX network manager
2
221272
Group services
3
0
Reserved for Attach
4
221274
Plugin controller
5
0
Designated SC
6
0
ASR9K H/W diags
7
221279
IP packet handler
8
217149
Redundancy controller
9
0
ASR9K Virtual console
10
110638
ASR9K Virtual terminal
11
49196
Control ethernet echo
12
0
Control eth echo reply
13
221274 Card Configuration Protocol
14
0
Reserved for Attach
15
0
Chassis controller
16
0
Forwarding driver
17
0
MBI hello
18
110640
MBI Boot Server Source
19
0
HSR ES client
20
0 Packets for ethernet server
21
0
For Diag application
22
233589
heartbeat request
23
0
heartbeat reply
24
221275
Async IPC client
25
0
Test application 1
26
0
Test application 2
27
0
Test client out-of-band
The following example shows that there are 18 nodes in the run state, which means that 12 boot requests
have been received by eth_server and 12 replies have been sent:
RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients 18 statistics location
0/RSP1/CPU0
Client ShelfMgr, ES Client Id 18, PID 53338 running on FastEthernet0_RSP0_0
12 packets input, 8676 bytes
12 packets delivered, 8676 bytes
0 packets discarded (0 bytes) in garbage collection
OL-23591-02
1-45
Chapter 1
0 (0 bytes) unicast packets filtered

0 (0 bytes) multicast packets filtered
0 (0 bytes) buffer mgmt policy discards
0 (0 bytes) locking error discards
12 packets output, 8676 bytes, 0 could not be transmitted
Basic Cisco IOS XR Verification and Troubleshooting

Commands
The following commands are used to collect information to aid in verifying the system and
troubleshooting problems:
man Command, page 1-46
describe Command, page 1-49
show platform Command, page 1-49
top Command, page 1-50
show context Command, page 1-50
show users Command, page 1-52
show history Command, page 1-52
show configuration Command, page 1-53
man Command
The man command provides online help for standard Cisco IOS XR CLI commands using manual (man)
pages. The command is used to display the manual pages for a specific command based on the command
name, a feature, or a keyword. Each man page contains the command name, syntax, command mode,
usage, examples, and related commands.
Note
To run the man command, you must have the Cisco IOS XR Documentation Package,
asr9k-doc.pie-4.0.0, .man pages for Cisco IOS XR software on the Cisco ASR 9000 Series Router
chassis, loaded. If you are running a release later than 4.0.0, the package installation envelope (PIE)
name might be different. For the appropriate PIE name and an explanation of PIE installation, see the
Upgrading Cisco IOS XR Software section of the Release Notes document for the IOS XR version you
are running
The following example shows the output from the man command show users command.
RP/0/RSP0/CPU0:router# man command show users
Tue Sep 14 14:39:16.409 DST
Building index table...
Total Number of Command Entries:2726
[OK]
DESCRIPTION
Displays information about the active lines on the router.
To display information about the active lines on the router, use the show users
1-46
OL-23591-02
Chapter 1

command in EXEC mode.

show users
SYNTAX DESCRIPTION
This command has no arguments or keywords.
COMMAND DEFAULT
No default behavior or values
COMMAND MODES
EXEC
COMMAND HISTORY
Release
Modification
Release 3.7.2
This command was introduced.
USAGE GUIDELINES
To use this command, you must be in a user group associated with a task group
that includes the proper task IDs. If you suspect user group assignment is
preventing you from using a command, contact your AAA administrator for
assistance.
Use the show users command to display the line number, connection name, idle
time, hosts, and terminal location. An asterisk (*) indicates the current
terminal session.
===================================================================================
Note:
To display all user groups and task IDs associated with the currently logged-in
user, use the show user command in EXEC mode. See the Authentication,
Authorization, and Accounting Commands on Cisco IOS XR Software module in
Cisco^B^`ASR^B^`9000 Series Aggregation Services Router System Security Command
Reference.
===================================================================================
TASK ID
Task ID
Operations
tty-access
read
EXAMPLES
The following example shows sample output identifying an active vty terminal
session:
OL-23591-02
1-47
Chapter 1
* * * * * * * * * * * * * * START OF LISTING * * * * * * * * * * * * * *
RP/0/RSP0/CPU0:router# show users
Line
con0_RSP0_CPU0
vty0
vty1
User
cisco
cisco
cisco
* * * * * * * * * * * * * *
Service Conns
Idle
hardware
0 18:33:48
telnet
0 00:30:36
telnet
0 00:00:00
Location
10.33.54.132
10.33.54.132
END OF LISTING * * * * * * * * * * * * * * *
Table 1 describes the significant fields shown in the display.

show users Field Descriptions
Field
Description
Line
All current connections. An asterisk (*) indicates the active connection.
User
Username of the user logged into the line.
Service
Physical or remote login service used.
Conns
Number of outgoing connections.
Idle
Interval (in hours:minutes:seconds) since last keystroke.
Location
IP address of remote login host. For local (physical) terminal
connections, this field is blank.
RELATED COMMANDS
Command
Description
show line Displays the parameters of terminal lines.
Displays the parameters of a terminal line.
show user
Displays all user groups and task IDs associated with the currently
logged-in user.
1-48
OL-23591-02
Chapter 1

describe Command
The describe command provides a preview of a command without actually implementing it. This
command lists information about the package, component, and task ID for a specific command. You
must be in the appropriate configuration mode for the specific command. For example, to display the
package, component, and task ID information for the router bgp 1 command, you must be in global
configuration mode.
The following example shows the output from the describe router bgp 1 command.
RP/0/RSP0/CPU0:router(config)#describe router bgp 1
Package:
iosxr-routing
iosxr-routing V4.0.0[DT_IMAGE] IOS-XR Routing Package Definition
Vendor : Cisco Systems
Desc
: IOS-XR Routing Package Definition
Build : Built on Wed Sep 8 16:10:14 DST 2010
Source : By router-021 in /files/4.0.0.DT_IMAGE/asr9k/workspace fo8
Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE,
NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500
Restart information:
Default:
parallel impacted processes restart
Size Compressed/Uncompressed: 8556KB/22MB (37%)
Component:
ipv4-bgp V[ci-401/18]
IPv4 Border Gateway Protocol (BGP)
User needs ALL of the following taskids:

bgp (READ WRITE)
show platform Command

The show platform command displays a high level overview of the entire physical system. Use the show
platform command in administration mode to display a summary of the nodes in the system, including
node type and status.
The following example shows the output from the show platform command in administration mode.
RP/0/RSP0/CPU0:router(admin)#show platform
Tue Sep 14 14:52:52.558 DST
Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/FT0/SP
FAN TRAY
READY
0/FT1/SP
FAN TRAY
READY
0/1/CPU0
A9K-40GE-B
IOS XR RUN
PWR,NSHUT,MON
0/2/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,MON
0/2/0
SPA-2XOC48POS/RPR
OK
PWR,NSHUT,MON
0/3/CPU0
A9K-2T20GE-B
IN-RESET
PWR,NSHUT,MON
0/4/CPU0
A9K-8T/4-B
IOS XR RUN
PWR,NSHUT,MON
0/6/CPU0
A9K-4T-B
IOS XR RUN
PWR,NSHUT,MON
0/PM0/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM1/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM2/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
OL-23591-02
1-49
Chapter 1
top Command
The top command is used to monitor CPU usage on the system through interactive process statistics.
The following example show the output from the top command.
RP/0/RSP0/CPU0:router# top
Computing times...
224 processes; 803 threads;
CPU states: 93.6% idle, 3.6% user, 2.7% kernel
Memory: 4096M total, 3504M avail, page size 4K
JID
91
256
340
294
1
65816
1
60
256
340
TID LAST_CPU PRI STATE HH:MM:SS

1
0
10 Rcv
0:16:29
10
0
10 Rcv
0:19:43
10
0
10 Rcv
0:03:21
1
1
10 Rcv
0:03:24
12
1
10 Rcv
0:00:03
1
0
10 Rply
0:00:00
11
1
10 Run
0:00:19
5
1
10 Rcv
0:00:43
11
1
10 Rcv
0:00:15
14
1
10 Rcv
0:01:06
CPU COMMAND
2.25% spp
2.16% netio
0.13% udp
0.09% sc
0.09% procnto-600-smp-instr
0.07% top
0.05% procnto-600-smp-instr
0.02% eth_server
0.02% netio
0.01% udp
Press q to exit the command.
show context Command

The show context command displays core dump context information for the last ten core dumps. The
command output is used for post-analysis in the debugging of processes (determine if any process
crashes have occurred).
If there are no crashed processes, the show context command displays no output for each node. The
following example shows the output of the show context command with no crashed processes.
node:
node0_1_CPU0
-----------------------------------------------------------------node:
node0_6_CPU0
-----------------------------------------------------------------node:
node0_RSP0_CPU0
-----------------------------------------------------------------node:
node0_RSP1_CPU0
------------------------------------------------------------------
The following example shows the output from the show context command where there is a crashed
process.
node:
node0_1_CPU0
-----------------------------------------------------------------Crashed pid = 61524 (pkg/bin/tcam_mgr)
Crashed tid = 1
Crash time: Wed Apr 05, 2006: 18:27:26
1-50
OL-23591-02
Chapter 1

Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z

Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
R0
R4
R8
R12
R16
R20
R24
R28
R32
R36
r0
00000000
r4
0000f054
r8
00000000
r12
4823be90
r16
00000048
r20
00000000
r24
00000003
r28
00000006
cnt
00000000
cnd
28004024
Registers info
r1
r2
481ff7b0 4824a55c
r5
r6
00000001 00000006
r9
r10
fc220000 481fffc0
r13
r14
4824a4a0 48230000
r17
r18
00000001 00000019
r21
r22
00000000 00000003
r25
r26
00000000 00000003
r29
r30
0000f054 48254064
lr
msr
fc1c6340 0000d932
xer
00000008
r3
00000000
r7
00000000
r11
00000000
r15
00000000
r19
48256520
r23
00000045
r27
4825dc34
r31
481ff810
pc
fc1d3fa0
DLL Info
DLL path
Text addr. Text size
Data addr. Data size
Version
/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8
0
/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000
0x00000bbc
0
Crash Package Infomation

Package: hfr-mgbl, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-mcast, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
Package: hfr-mpls, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-rout, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-k9sec, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
Package: hfr-lc, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/works
pace for c2.95.3-p8
Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
Package: hfr-os-mbi, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/w
OL-23591-02
1-51
Chapter 1
orkspace for c2.95.3-p8

node:
node0_6_CPU0
-----------------------------------------------------------------node:
node0_RSP0_CPU0
-----------------------------------------------------------------node:
node0_RSP1_CPU0
------------------------------------------------------------------
Use the show context command to locate the core dump file path. For example, the core dump file path
shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z. The
command output shows a crash on a node. The process is pkg/bin/tcam_mgr.
Collect the following information and send it to Cisco Technical Support. For Cisco Technical Support
contact information, see the Obtaining Documentation and Submitting a Service Request section on
page xii.
ppc.Z fileThis file contains the binary core dump information. Use the path listed in the command
output to copy the contents of the ppc.Z file. The path shown in the command output is:
harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
ppc.txt fileThis file contains content on the core dump similar to the show context command
output. Use the path listed in the command output to copy the contents of the ppc.txt file. The path
shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.txt
Collect the show version or show install active command output.
show users Command

The show users command displays information on active lines on the router including the line number,
user, service, number of connections, idle time, and remote terminal location. An asterisk (*) indicates
the current terminal session.
The following example shows the output from the show users command.
RP/0/RSP0/CPU0:router# show users
Line
vty0
vty1
User
User_A
User-B
Service
telnet
telnet
Conns
0
0
Idle
00:00:00
00:00:03
Location
161.44.1925
161.44.1929
show history Command

The show history command displays a history of the command entered for the current command mode.
You can enter the show history command to display a history of commands entered in EXEC, ADMIN,
or CONFIG mode.
Examples
RP/0/RSP0/CPU0:router# show history
RP/0/RSP0/CPU0:router(admin)# show history
RP/0/RSP0/CPU0:router(config)# show history
The following example shows the output from the show history command in EXEC mode:
1-52
OL-23591-02
Chapter 1


Thu Oct 28 14:20:50.328 DST
show vrrp interface gigabitEthernet 0/1/0/0
show vrrp interface brief
show vrrp brief
show vrrp detail
show vrrp interface gigabitEthernet 0/1/0/1
show vrrp interface gigabitEthernet 0/1/0/1 statistics all
The detailed history provides a timestamp also:

Thu Oct 28 14:26:06.199 DST
1 Thu Oct 28 14:02:25.310 show
2 Thu Oct 28 14:03:34.854 show
3 Thu Oct 28 14:04:02.042 show
4 Thu Oct 28 14:04:08.167 show
5 Thu Oct 28 14:08:25.180 show
6 Thu Oct 28 14:09:03.402 show
detail
vrrp
vrrp
vrrp
vrrp
vrrp
vrrp
interface
interface
brief
detail
interface
interface
gigabitEthernet 0/1/0/0
brief
gigabitEthernet 0/1/0/1 statistics all
show configuration Command

The show configuration command displays details on uncommitted configuration changes, that is, the
commands you are about to commit. You can enter the show configuration command to display the
changes in EXEC, ADMIN, or CONFIG mode.
Use the show configuration command with the running keyword to display the running (active)
configuration.
Prior to committing the target configuration, use the show configuration command with the merge
keyword from any configuration mode to display the result of merging the target configuration with the
running configuration.
Examples
RP/0/RSP0/CPU0:router# show configuration
RP/0/RSP0/CPU0:router# show configuration running-config
RP/0/RSP0/CPU0:router(admin)# show configuration running
RP/0/RSP0/CPU0:router(config)# show configuration
RP/0/RSP0/CPU0:router(config)# show configuration running
RP/0/RSP0/CPU0:router(config)# show configuration merge
In this example, the show configuration command displays uncommitted changes made during a
configuration session:
RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3
RP/0/RSP0/CPU0:router(config-if)# description faq
RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0
RP/0/RSP0/CPU0:router(config-if)# show configuration
description faq
ipv4 address 10.10.11.20 255.0.0.0
end
OL-23591-02
1-53
Chapter 1
The following example shows sample output from the show configuration command with the optional
merge keyword. The command is entered during a configuration session. The output displays the result
of merging the target and running configuration, without committing the changes.
RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3
RP/0/RSP0/CPU0:router(config-if)# description faq
RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0
RP/0/RSP0/CPU0:router(config-if)# show configuration merge
hostname router
ipv4 address 10.2.3.4 255.0.0.0
exit
description faq
ipv4 address 10.1.1.1 255.0.0.0
shutdown
end

The following example shows how to display ASIC errors for each ASIC in a SIP-700 optical LC. If an
error is displayed, dump the individual ASIC instance number to obtain details on the ASIC error.
RP/0/RSP0/CPU0:router# show asic-errors all location 0/6/CPU0
Thu Oct 21 19:00:54.178 DST
************************************************************
*
Fia ASIC Error Summary
*
************************************************************
Instance
: 0
Number of nodes
: 0
SBE error count
: 0
MBE error count
: 0
Parity error count : 0
CRC error count
: 0
Generic error count : 0
Reset error count
: 0
-------------------************************************************************
*
Mace ASIC Error Summary
*
************************************************************
Instance
: 0
Number of nodes
: 0
SBE error count
: 0
MBE error count
: 0
Parity error count : 0
CRC error count
: 0
Generic error count : 0
Reset error count
: 0
-------------------************************************************************
*
Prm_np ASIC Error Summary
*
************************************************************
Instance
: 0
1-54
OL-23591-02
Chapter 1

Number of nodes
:
SBE error count
:
MBE error count
:
Parity error count :
CRC error count
:
Generic error count :
Reset error count
:
-------------------Instance
:
Number of nodes
:
SBE error count
:
MBE error count
:
CRC error count
:
Reset error count
:
-------------------Instance
:
Number of nodes
:
SBE error count
:
MBE error count
:
CRC error count
:
Reset error count
:
-------------------Instance
:
Number of nodes
:
SBE error count
:
MBE error count
:
CRC error count
:
Reset error count
:
--------------------
3
0
0
0
0
0
0
1
3
0
0
0
0
0
0
2
3
0
0
0
0
0
0
3
3
0
0
0
0
0
0
The following ASIC error types are supported:
FIA (Fabric Interface ASIC)
Mace ASIC
Prm_np ASIC
The following ASIC error classifications are supported:
Single Bit Errors (SBE)Correctable ECC protected single bit errors in external or internal
memory.
Not reported to PM on each occurrence and reported to the platform manager (PM) as Minor when
software threshold rate is exceeded. Report alarm using Alarm Logging, and Debugging Event
Management System (ALDEMS).
Error data:
AddressAddress that encountered the SBE
SyndromeSyndrome if available
Multiple Bit ErrorsUncorrectable multiple bit error in memory.

Reported to PM as Major and ALDEMS for each occurrence.
Error data:
AddressAddress that encountered the SBE.
OL-23591-02
1-55
Chapter 1
DataActual error data.
PARITY ErrorsParity error in all applicable memory.

Reported to PM as Major.
Cyclic redundancy check (CRC) ErrorsCRC errors in EIO other links.

Not reported for each occurrence. When the threshold is reached it is reported as Major to the PM.
GENERIC ErrorsErrors that do not fall under any of the other classifications.
Threshold and alarm reporting is done.
RESET ErrorsLogged for each reset instance of the ASIC.

Reported to PM when threshold is exceeded.
Error data:
Interrupt statusInterrupt status bits due to ASIC reset.
Halt statusHalt status bits.
Reset node keyKey for the error node that causes the reset.
TimeReset time.
The following ASIC error fault severities are supported:
CriticalAffected component is unusable or card is reset if no redundant card exists.
MajorPartially service affecting fault, causing the card to run in degraded mode. For redundant
cards, consider performing a switchover.
MinorNon-service affecting fault.
OKNo fault.

Trace commands provide an always on debug feature. Many major functions in Cisco IOS XR software
have trace functionality to show the last actions it conducted allowing you to analyze function events.
Use the show trace commands to display the trace data for a specific feature or process. Use the ? in the
CLI to determine if a command has the trace keyword. The following example shows that the show arp
command has the trace keyword.
RP/0/RSP0/CPU0:router# show arp ?
A.B.C.D
IP address or hostname of ARP entry
BVI
Bridge-Group Virtual Interface
Bundle-Ether
Aggregated Ethernet interface(s)
GigabitEthernet GigabitEthernet/IEEE 802.3 interface(s)
H.H.H
48-bit hardware address of ARP entry
MgmtEth
Ethernet/IEEE 802.3 interface(s)
TenGigE
TenGigabitEthernet/IEEE 802.3 interface(s)
api-stats
Show ARP API statistics data
client
ARP Client show commands
dagr
Show Direct Attached Gateway Redundancy group information
idb
Show the internal ARP interface data block
location
specify a node name
resolution
Show the ARP resolution history
trace
Show trace data for the ARP component
traffic
ARP traffic statistics
vrf
Specify a VRF
|
Output Modifiers
<cr>
1-56
OL-23591-02
Chapter 1

MIB Location
The following example shows the last 20 events in the address resolution protocol (ARP) table.
RP/0/RSP0/CPU0:router# show arp trace tailf last 20
Apr 19 09:52:29.857 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for
address: 172.18.105.255
Apr 19 09:52:34.501 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
Apr 19 09:52:41.856 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
172.18.105.255
Apr 19 09:52:59.979 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted
from table
request for 172.18.105.255
address: 172.18.105.255
request for 172.18.105.255
172.18.105.255
from table
request for 172.18.105.255
address: 172.18.105.255
172.18.105.255
from table
request for 172.18.105.255
address: 172.18.105.255
172.18.105.255
from table
request for 172.18.105.255
address: 172.18.105.255
172.18.105.255
MIB Location
To locate and download MIBs, use the Cisco MIB Locator found at the following URL and choose a
platform under the Cisco Access Products menu:
http://cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml.
OL-23591-02
1-57
Chapter 1
Gathering Information Before You Call Cisco TAC

We recommend that you have a system of maintaining and accessing detailed information about your
network and ASR 9000 router, including system hardware and software, network diagrams, and captured
output from commands. For additional details, see the Prerequisite Documentation for
Troubleshooting section on page 1-1.
Before calling the Cisco Technical Assistance Center (TAC), you should gather the information
described in the following sections, if possible. This information will be helpful for troubleshooting.
Caution
Timesaver
We strongy recommend that, if possible, you gather the information described in this section before you
reset any cards. If you reset cards before you gather information, the system erases the information and
it will be more difficult to diagnose and repair the problem.
Gathering Information about Crashes and Core Dumps, page 1-58
Capturing Logs, page 1-58
Using Debug Commands, page 1-59
Using Diagnostic Commands, page 1-59
Commands Used to Display Process and Thread Details, page 1-59
Before contacting Cisco Technical Support, review the information provided at the following URL:
http://www.cisco.com/web/services/ts/access/index.html.
For information on contacting Cisco Technical Support, see the Obtaining Documentation and
Submitting a Service Request section on page xii.
Gathering Information about Crashes and Core Dumps

Gather system information with the following commands:
show install active summary
show version
show run
show context
show log
show inventory
show diagnostics
Upload any core dumps that were written to disk0, disk1, or harddisk directories.
Capturing Logs
See the Prerequisite Documentation for Troubleshooting section on page 1-1 in Chapter 1, General
Troubleshooting Procedures, for information on collecting current system information.
Collect system information using the following commands:
1-58
OL-23591-02
Chapter 1

Note
show tech-supportDisplays system information for Cisco Technical Support and includes a
traditional dump of the configuration and show command outputs.
Some tech-support commands require the user to be assigned the cisco-support task ID. For a mapping
of commands to task IDs and allowed operations, see Cisco IOS XR Task ID Reference Guide.
show loggingDisplays the contents of the logging buffers
show system verifyDisplays system verification information
Using Debug Commands

For details on using debug commands, see Cisco IOS XR Using Debug Guide.
Using Diagnostic Commands

The Cisco ASR 9000 Aggregation Series Router diagnostic tests verify control Ethernet and fabric data
paths. If a diagnostic tests fails, it indicates a bad data path. The integrity of the covered data paths is
verified when the diagnostic tests pass.
The diagnostic tests generally test data paths between multiple nodes, therefore error reports need to be
analyzed to narrow down the possible points of failure in a system.
All diagnostic tests run within the 1 second to 1 minute range.
Note
On the Cisco ASR 9000 Aggregation Services Router, only online diagnostics are supported.
To run a specified on-demand diagnostic test or series of tests, use the diagnostic start location
command.
Example:
RP/0/RSP0/CPU0:router(admin)# diagnostic start location 0/RSP1/CPU0 test 1
RP/0/RSP0/CPU0:router(admin)# diagnostic stop location 0/RSP1/CPU0
For details on the diagnostic commands, see Cisco ASR 9000 Series Aggregation Services Router
Interface and Hardware Component Command Reference.
Commands Used to Display Process and Thread Details

For details on processes and threads, see the Understanding Processes and Threads section in
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide.
OL-23591-02
1-59
Chapter 1
1-60
OL-23591-02
CH A P T E R

This chapter describes how to verify that interfaces are up (operational), and how to troubleshoot
problems on interfaces. It includes the following sections:
Verifying and Troubleshooting Gigabit Ethernet Interfaces, page 2-61
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces, page 2-68

To troubleshoot gigabit Ethernet interfaces, perform the following procedure.
SUMMARY STEPS
1.
show interfaces type instance
2.
show controllers interface-type interface-instance stats
3.
show netio idb interface-type interface-instance
4.
show hw-module subslot address counters

or
show hw-module subslot counters framer
5.
If your system has pluggable optical line cards, perform additional troubleshooting steps applicable
to these cards.
6.
OL-23591-02
2-61
Chapter 2
DETAILED STEPS
Step 1
Command or Action
Purpose
show interfaces type instance
Displays statistics for all interfaces configured on the

specified node. Check for interface errors and conflicting
configurations, such as IP addresses defined incorrectly on
interfaces.
Example:
RP/0/RSP0/CPU0:router# show interfaces
Step 2
show controllers interface-type

interface-instance stats
Example:
gigabitEthernet 0/1/0/23 stats
Step 3
show netio idb interface-type

interface-instance
Example:
RP/0/RSP0/CPU0:router# show netio idb
Step 4
show hw-module subslot address counters
or
Displays interface controller status and configuration

statistics for the specified node. Check for input drops.
If input drops are found, contact Cisco Technical Support.
For Cisco Technical Support contact information, see the
Obtaining Documentation and Submitting a Service
Request section on page xii in the Preface.
Displays network input and output information for a
specified node.
Check the software counters for the interface. Under Chains
and Protocol chains in the output check if any drops
occurred at a particular point in the Encap or Decap of the
packet. The drops are displayed in the column on the right,
showing drop packets and bytes for each step in the chain.
Use these commands to check for status and interface drops
on shared port adapters (SPAs).
show hw-module subslot counters framer
Example:
RP/0/RSP0/CPU0:router# show hw-module subslot
0/0/0 counters
or
counters framer
Step 5
If your system has pluggable optical line cards,

perform the additional troubleshooting steps
applicable to these cards. See Verifying and
Troubleshooting Pluggable Optical Line Card
Interfaces section on page 2-68.
Check for interface drops on the pluggable optical line

cards.
Step 6

The following example shows POS 0/0/1/0 with no input drop counters.
RP/0/RSP0/CPU0:router# show interfaces pos 0/0/1/0
POS0/0/1/0 is up, line protocol is up
Hardware is Packet over SONET/SDH
MTU 4474 bytes, BW 155520 Kbit
2-62
OL-23591-02
Chapter 2

Encapsulation HDLC, crc 32, controller loopback not set, keepalive set (10 sec)
RP/0/RSP0/CPU0:router# show controllers gigabitEthernet 0/1/0/2 stats

Wed Nov 3 12:41:54.201 DST
Statistics for interface GigabitEthernet0/1/0/2 (cached values):
Ingress:
Input total bytes
Input good bytes
= 0
= 0
Input
Input
Input
Input
Input
Input
Input
Input
Input
Input
total packets
802.1Q frames
pause frames
pkts 64 bytes
pkts 65-127 bytes
pkts 128-255 bytes
pkts 256-511 bytes
pkts 512-1023 bytes
pkts 1024-1518 bytes
pkts 1519-Max bytes
=
=
=
=
=
=
=
=
=
=
0
0
0
0
0
0
0
0
0
0
Input
Input
Input
Input
good pkts
unicast pkts
multicast pkts
broadcast pkts
=
=
=
=
0
0
0
0
Input
Input
Input
Input
Input
Input
drop
drop
drop
drop
drop
drop
=
=
=
=
=
=
0
0
0
0
0
0
Input
Input
Input
Input
Input
Input
Input
Input
error
error
error
error
error
error
error
error
=
=
=
=
=
=
=
=
0
0
0
0
0
0
0
0
overrun
abort
invalid VLAN
invalid DMAC
invalid encap
other
giant
runt
jabbers
fragments
CRC
collisions
symbol
other
Input MIB giant

Input MIB jabber
Input MIB CRC
Egress:
Output total bytes
Output good bytes
Output total packets
Output 802.1Q frames
= 0
= 0
= 0
= 0
= 0
= 0
= 0
OL-23591-02
2-63
Chapter 2
Output
Output
Output
Output
Output
Output
Output
Output
pause frames
pkts 64 bytes
pkts 65-127 bytes
pkts 128-255 bytes
pkts 256-511 bytes
pkts 512-1023 bytes
pkts 1024-1518 bytes
pkts 1519-Max bytes
=
=
=
=
=
=
=
=
0
0
0
0
0
0
0
0
Output
Output
Output
Output
good pkts
unicast pkts
multicast pkts
broadcast pkts
=
=
=
=
0
0
0
0
Output drop underrun

Output drop abort
Output drop other
= 0
= 0
= 0
Output error other
= 0
RP/0/RSP0/CPU0:router# show netio idb pos 0/0/1/0

POS0/0/1/0 (handle: 0x010800a0, nodeid:0x1) netio idb:
--------------------------------name:
POS0_0_1_0
interface handle:
0x010800a0
interface global index: 2
physical media type:
14
dchain ptr:
<0x482dd660>
echain ptr:
<0x48247d58>
fchain ptr:
<0x482dd774>
driver cookie:
<0x4824cd68>
driver func:
<0x4824cd54>
number of subinterfaces: 0
subblock array size:
0
DSNCNF:
0x00000000
interface stats info:
IN unknown proto pkts: 0
IN unknown proto bytes: 0
IN multicast pkts:
0
OUT multicast pkts:
0
IN broadcast pkts:
0
OUT broadcast pkts:
0
IN drop pkts:
0
OUT drop pkts:
0
IN errors pkts:
0
OUT errors pkts:
0
Chains
-------------------Base decap chain:
hdlc
<14>
<0xfd6a0a74, 0x00000000>
<
0,
0>
Protocol chains:
--------------<Protocol number> (name) Stats
Type Chain_node
<caps num> <function, context> <drop pkts, drop bytes>
<9> (chdlc)
Stats IN: 48466 pkts, 3559516 bytes; OUT: 41378 pkts, 910312 bytes
Encap:
l2_adj_rewrite
<86> <0xfceada88, 0x482c390c> <
0,
0>
queue_fifo
<56> <0xfcedea68, 0x482ddc30> <
0,
0>
2-64
OL-23591-02
Chapter 2

txm_nopull
<60>
Decap:
queue_fifo
<56>
chdlc
<13>
Fixup:
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
<10> (clns)
Stats IN: 0 pkts,
Encap:
clns
<15>
hdlc
<14>
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
Decap:
queue_fifo
<56>
clns
<15>
Fixup:
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
<12> (ipv4)
Stats IN: 0 pkts,
Encap:
ipv4
<26>
hdlc
<14>
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
Decap:
queue_fifo
<56>
ipv4
<26>
Fixup:
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
<0xfce8fa5c, 0x482ddda4>
<
0,
0>
<0xfcedea4c, 0x482ddc30>
<0xfd6a252c, 0x00000000>
<
<
0,
0,
0>
0>
<0xfcead45c, 0x00000000> <

<0xfcedea68, 0x482ddc30> <
<0xfce8fa5c, 0x482ddda4> <
0 bytes; OUT: 0 pkts, 0 bytes
0,
0,
0,
0>
0>
0>
<0xfcfaa030,
<0xfd6a0678,
<0xfceada88,
<0xfcedea68,
<0xfce8fa5c,
0x00000000>
0x00000000>
0x48305a90>
0x482ddc30>
0x482ddda4>
<
<
<
<
<
0,
0,
0,
0,
0,
0>
0>
0>
0>
0>
<0xfcfa9508, 0x00000000>
<
<
0,
0,
0>
0>
<0xfcead45c, 0x00000000> <

<0xfcedea68, 0x482ddc30> <
<0xfce8fa5c, 0x482ddda4> <
0 bytes; OUT: 0 pkts, 0 bytes
0,
0,
0,
0>
0>
0>
<0xfd0f341c,
<0xfd6a0678,
<0xfceada88,
<0xfcedea68,
<0xfce8fa5c,
0x482dd460>
0x00000000>
0x48349b54>
0x482ddc30>
0x482ddda4>
<
<
<
<
<
0,
0,
0,
0,
0,
0>
0>
0>
0>
0>
<0xfd0f3474, 0x00000000>
<
<
0,
0,
0>
0>
<0xfcead45c, 0x00000000>
<0xfcedea68, 0x482ddc30>
<0xfce8fa5c, 0x482ddda4>
<
<
<
0,
0,
0,
0>
0>
0>
Protocol SAFI counts:

-------------------Protocol
--------------ipv4
ipv4
ipv4
ipv6
ipv6
SAFI
---------Unicast
Multicast
Broadcast
Unicast
Multicast
Pkts In
---------0
0
0
0
0
Bytes In
---------0
0
0
0
0
Pkts Out
---------0
0
0
0
0
Bytes Out
---------0
0
0
0
0
The following example shows counters implemented for Multiprotocol Label Switching (MPLS)
packets. The following output under Protocol Chains in the show netio idb command shows the MPLS
packets incrementing:
mpls
<25>
<0xfcc7b2b8, 0x00000000>
<
152,
17328>
RP/0/RSP0/CPU0:router# show netio idb gigabitEthernet 0/2/0/1

GigabitEthernet0/2/0/1 (handle: 0x01280040, nodeid:0x21) netio idb:
--------------------------------name:
GigabitEthernet0_2_0_1
interface handle:
0x01280040
30
dchain ptr:
<0x482e0700>
OL-23591-02
2-65
Chapter 2
echain ptr:
<0x482e1024>
fchain ptr:
<0x482e13ec>
driver cookie:
<0x4829fc6c>
driver func:
<0x4829f040>
7
DSNCNF:
0x00000000
IN multicast pkts:
0
OUT multicast pkts:
0
IN broadcast pkts:
0
OUT broadcast pkts:
0
IN drop pkts:
0
OUT drop pkts:
0
IN errors pkts:
0
OUT errors pkts:
0
Chains
-------------------Base decap chain:
ether
<30>
<0xfd018cd8, 0x482c736c>
<
0,
0>
Protocol chains:
Type Chain_node
<7> (arp)
Encap:
l2_adj_rewrite
<86> <0xfcaa997c, 0x4831a33c> <
0,
0>
pcn_output
<54> <0xfd054bfc, 0x48319f04> <
0,
0>
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
0>
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
0>
Decap:
pcn_input
<55> <0xfd054bfc, 0x4830ba8c> <
0,
0>
q_fq_input
<96> <0xfd05f330, 0x48312c7c> <
0,
0>
arp
<24> <0xfcbfc2cc, 0x00000000> <
0,
0>
Fixup:
l2_adj_rewrite
<86> <0xfcaa945c, 0x00000000> <
0,
0>
pcn_output
<54> <0xfd054bfc, 0x48319f04> <
0,
0>
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
0>
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
0>
<10> (clns)
Encap:
clns
<15> <0xfcbe2c80, 0x00000000> <
0,
0>
ether
<30> <0xfd0189b4, 0x482c736c> <
0,
0>
l2_adj_rewrite
<86> <0xfcaa997c, 0x482d8660> <
0,
0>
pcn_output
<54> <0xfd054bfc, 0x48319f04> <
0,
0>
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
0>
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
0>
Decap:
pcn_input
<55> <0xfd054bfc, 0x4830ba8c> <
0,
0>
q_fq_input
<96> <0xfd05f330, 0x48312c7c> <
0,
0>
clns
<15> <0xfcbe2444, 0x00000000> <
0,
0>
Fixup:
l2_adj_rewrite
<86> <0xfcaa945c, 0x00000000> <
0,
0>
pcn_output
<54> <0xfd054bfc, 0x48319f04> <
0,
0>
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
0>
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
0>
<12> (ipv4)
Encap:
ipv4
<26> <0xfcc03dfc, 0x482e0414> <
0,
0>
ether
<30> <0xfd0189b4, 0x482c736c> <
0,
0>
2-66
OL-23591-02
Chapter 2

l2_adj_rewrite
<86> <0xfcaa997c, 0x4831a294> <
0,
pcn_output
<54> <0xfd054c48, 0x48319f04> <
0,
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
Decap:
pcn_input
<55> <0xfd054c48, 0x4830ba8c> <
0,
q_fq_input
<96> <0xfd05f330, 0x48312c7c> <
0,
ipv4
<26> <0xfcc03e80, 0x00000000> <
0,
Fixup:
l2_adj_rewrite
<86> <0xfcaa945c, 0x00000000> <
0,
pcn_output
<54> <0xfd054c48, 0x48319f04> <
0,
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
<13> (mpls)
Encap:
mpls
<25> <0xfcc7ddbc, 0x00000000> <
0,
ether
<30> <0xfd0189b4, 0x482c736c> <
0,
l2_adj_rewrite
<86> <0xfcaa997c, 0x4831a2e8> <
0,
pcn_output
<54> <0xfd0561f0, 0x48319f04> <
0,
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
Decap:
pcn_input
<55> <0xfd0561f0, 0x4830ba8c> <
0,
q_fq_input
<96> <0xfd05f330, 0x48312c7c> <
0,
mpls
<25> <0xfcc7b2b8, 0x00000000> <
152,
Fixup:
l2_adj_rewrite
<86> <0xfcaa945c, 0x00000000> <
0,
pcn_output
<54> <0xfd0561f0, 0x48319f04> <
0,
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
<22> (ether_sock)
Encap:
ether_sock
<98> <0xfd01a774, 0x482c736c> <
0,
l2_adj_rewrite
<86> <0xfcaa997c, 0x482d85f0> <
0,
pcn_output
<54> <0xfd054bfc, 0x48319f04> <
0,
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
Decap:
pcn_input
<55> <0xfd054bfc, 0x4830ba8c> <
0,
q_fq_input
<96> <0xfd05f330, 0x48312c7c> <
0,
ether_sock
<98> <0xfd01a91c, 0x482c736c> <
0,
Fixup:
l2_adj_rewrite
<86> <0xfcaa945c, 0x00000000> <
0,
pcn_output
<54> <0xfd054bfc, 0x48319f04> <
0,
q_fq
<43> <0xfd05f4b8, 0x48320fec> <
0,
txm_nopull
<60> <0xfcadba38, 0x4824c0fc> <
0,
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
17328>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>
0>

-------------------Protocol
--------------ipv4
ipv4
ipv4
ipv6
ipv6
SAFI
---------Unicast
Multicast
Broadcast
Unicast
Multicast
Pkts In
---------0
0
0
0
0
Bytes In
---------0
0
0
0
0
Pkts Out
---------0
7
0
0
0
Bytes Out
---------0
434
0
0
0
The following example shows the SPA status.

RP/0/RSP0/CPU0:router# show hw-module subslot 0/4/1 counters
Wed Nov 3 09:00:34.258 EDT
OL-23591-02
2-67
Chapter 2
Subslot 0/4/1 counts info:

-----------------------SPA inserted: YES
SPA type: 1xCHOC48 SPA
SPA operational state: READY
SPA insertion time: Mon Nov 1 08:28:12 2010
SPA last time ready: Mon Nov 1 08:30:04 2010
SPA uptime [HH:MM:SS]: 48:30:30
The following Packet-over-SONET/SDH (POS) port example shows the SPA counters. The output
displays any drop counters or error counters incrementing for the interface.
RP/0/RSP0/CPU0:router# show hw-module subslot counters framer
SPA 0/2/0 device framer 0/0 info:
ABC (port 0) framer registers:

DEF Framer counters:
STREAM 0
Rx Bytes
Rx Good Bytes
Rx Good Packets
Tx Byte Cnt Reg
Tx Good Bytes Cnt Reg
Tx Transmitted Packet Cnt Reg
(48-bit)
(48-bit)
(48-bit)
(48-bit)
(48-bit)
(48-bit)
(#0x60e7d078-0x883c):
(#0x60e7d080-0x8840):
(#0x60e7d040-0x8820):
(#0x60e81070-0xa838):
(#0x60e81068-0xa834):
(#0x60e81040-0xa820):
567674
479474
21795
567709
479496
21796
(48-bit)
(48-bit)
(48-bit)
(48-bit)
(48-bit)
(48-bit)
(#0x60dfd078-0x883c):
(#0x60dfd080-0x8840):
(#0x60dfd040-0x8820):
(#0x60e01070-0xa838):
(#0x60e01068-0xa834):
(#0x60e01040-0xa820):
567628
479438
21793
567690
479478
21795
SPA 0/2/0 device framer 1/0 info:
ABC (port 1) framer registers:

DEF Framer counters:
STREAM 0
Rx Bytes
Rx Good Bytes
Rx Good Packets
Tx Byte Cnt Reg
Tx Good Bytes Cnt Reg
Tx Transmitted Packet Cnt Reg
Verifying and Troubleshooting Pluggable Optical Line Card

Interfaces
Troubleshooting the pluggable optical line card interfaces includes verifying that you have an optical
card installed, enabled, and functioning properly. To troubleshoot the configured pluggable optical line
card interfaces, perform the following procedure.
SUMMARY STEPS
1.
show controllers plim asic SPAQFPBridgeCtrl counters instance <0 3> all location node-id
2.
show hw-module subslot brief pluggable-optics
3.
show hw-module subslot address status pluggable-optics
2-68
OL-23591-02
Chapter 2

4.
show hw-module subslot address errors pluggable-optics
5.
show hw-module subslot address registers pluggable-optics
6.
DETAILED STEPS
Step 1
Command or Action
Purpose
show controllers plim asic SPAQFPBridgeCtrl

counters instance <0 3> all location node-id
Displays the drop counters and error counters incrementing

on the interface.
Example:
RP/0/RSP0/CPU0:router# show controllers plim
asic SPAQFPBridgeCtrl counters instance 0 all
location 0/4/CPU0
Step 1
show hw-module subslot brief pluggable-optics
Example:
brief pluggable-optics
Step 2
show hw-module subslot address status

pluggable-optics
Example:
Displays a brief summary of the pluggable optics status for

all line card nodes, including optics type, vendor, and state.
Check that the state is enabled for the node that you are
troubleshooting.
Displays the status of the pluggable optics for the specified
line card node, including faults and environmental data.
Check that the state and the transceiver are enabled. Check
for any warnings or alarms.

0/2/0 status pluggable-optics
Step 3
show hw-module subslot address errors

pluggable-optics
Displays any errors that are present on the node. Note if

there are any errors.
Example:
Verify that Phased Initialization displays Phase Reached: 4.

Verify that Socket Verification displays passed for both
Compatibility and Security.

0/2/0 errors pluggable-optics
Step 4
show hw-module subslot address registers

pluggable-optics
Displays all available information on the optics including

the IDPROM contents.
Example:
0/2/0 registers pluggable-optics
Step 5

Examples
The following example displays the drop counters and error countersincrementing on the SPA in the line
card.
RP/0/RSP0/CPU0:router# show controllers plim asic SPAQFPBridgeCtrl counters instance 0 all
location 0/4/CPU0
Wed Nov 3 08:58:07.062 EDT
OL-23591-02
2-69
Chapter 2
SPAQFPBridgeCtrl 0:
=======================================
------------ SPAQFPBridgeCtrl 0 SPA1 Counters -------RX Bytes: 0
RX Packets: 0
TX Bytes: 0
TX Packets: 0
--------------- SPAQFPBridgeCtrl 0 HT Counters ---------HT RX Bytes: 0
HT RX Packets: 0
HT TX Bytes: 0
HT TX Packets: 0
-------------- SPAQFPBridgeCtrl 0 NP Counters --------RXNP eSPI Bytes: 0
RXNP eSPI Packets: 0
TXNP eSPI Bytes: 0
TXNP eSPI Packets: 0
-------------- SPAQFPBridgeCtrl 0 Diag Counters --------RXNP Online Diag Bytes: 0
RXNP Online Diag Packets: 0
TXNP Online Diag Bytes: 0
TXNP Online Diag Packets: 0
-------------- SPAQFPBridgeCtrl 0 Error Counters --------ERP Read out of sync Error: 0
CPUIF ERP Parity Error: 0
CPUIF ERP Protocol Error: 0
RXRLDC Mem Low Byte ECC: 0
RXRLDC Mem High Byte ECC: 0
TXRLDC Mem Low Byte ECC: 0
TXRLDC Mem High Byte ECC: 0
-------------- SPAQFPBridgeCtrl 0 Drop Counters --------TXNP Errored Low Priority Packets: 0
TXNP Errored High Priority Packets: 0
TXNP Errored Loopback Packets: 0
TXNP Errored HT Packets: 0
TXNP Errored Low Priority Bytes: 0
TXNP Errored High Priority Bytes: 0
TXNP Errored Loopback Bytes: 0
TXNP Errored HT Bytes: 0
--------- SPAQFPBridgeCtrl 0 RXDCM Drops ---------------- SPAQFPBridgeCtrl 0 TXDCM Drops --------
The following example shows typical outputs for these commands.

RP/0/RSP0/CPU0:router# show hw-module subslot brief pluggable-optics
SPA 0/2/0 device pluggable-optics 0/0 info:
POS0/2/0/0:
ID: SFP
Extended ID: 4
Xcvr Type: OC48 SR/STM16 I-16 (44)
Connector: LC
Vendor name: CISCO-FINISAR
Vendor part number: FTRJ1321P1BTL-C4
State: Enabled

POS0/2/0/1:
ID: SFP
2-70
OL-23591-02
Chapter 2

Extended ID: 4
Connector: LC
Vendor name: CISCO-FINISAR
Vendor part number: FTRJ1321P1BTL-C4
State: Enabled
RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 status pluggable-optics

POS0/2/0/0:
State: Enabled
Environmental Information - raw values
Temperature: 48.136 C
Supply voltage: 32970 in units of 100uVolt
Tx bias: 12678 in units of 2uAmp
Tx power: -4 dBm (2848 in units of 0.1 uW)
Rx power: -7 dBm (1554 in units of 0.1 uW)
Transceiver: Enabled
SW TX Fault: None
SW LOS: Active
No active alarms
No active warnings
Version Identifier (VID): V01
Product Identifier (PID): SFP-OC48-SR
Part Number (PN): 10-1961-01
CLEI: WM1T2TTAAA
POS0/2/0/1:
State: Enabled
Environmental Information - raw values
Temperature: 46.172 C
Supply voltage: 32914 in units of 100uVolt
Tx bias: 11166 in units of 2uAmp
Tx power: -5 dBm (2282 in units of 0.1 uW)
Rx power: -5 dBm (2411 in units of 0.1 uW)
Transceiver: Enabled
SW TX Fault: unavailable
SW LOS: unavailable
No active alarms
No active warnings
Version Identifier (VID): V01
Product Identifier (PID): SFP-OC48-SR
Part Number (PN): 10-1961-01
CLEI: WM1T2TTAAA
RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 errors pluggable-optics

POS0/2/0/0:
Phased Initialization
Phase Reached: 4
Phase Exit Code: Success 0
Phase Read Offset: 256
Socket Verification
OL-23591-02
2-71
Chapter 2
Compatibility: Compatibility passed

Security: Security passed
POS0/2/0/1:
Phased Initialization
Phase Reached: 4
Phase Exit Code: Success 0
Phase Read Offset: 256
Socket Verification
Compatibility: Compatibility passed
Security: Security passed
RP/0/RSP0/CPU0:router# show hw-module subslot 0/2/0 registers pluggable-optics

POS0/2/0/0:
ID: SFP
Extended ID: 4
Connector: LC
Encoding: reserved
Bit Rate: 2500 Mbps
Single mode fiber supported length: 2 km
Single mode fiber supported length: 20 m
Upper bit rate limit: not specified
Lower bit rate limit: not specified
Date code (yy/mm/dd): 05/12/13
Vendor name: CISCO
Vendor OUI: 36965
Vendor Part Number (PN): FTRJ1321P1BTL-C4
Vendor Rev: B
Vendor SN (SN): FNS0951J0VN
Options implemented:
LOS Signal
TX Fault Signal
TX Disable Signal
Enhanced options implemented:
Alarm/Warning Flags
Diagnostic monitoring implemented:
Exernally Calibrated
Digital Diagnostic Monitoring
Idprom contents (hex):
0x00:
03 04 07 00 01 00 00 12 00 01 05 05
0x10:
00 00 00 00 43 49 53 43 4F 2D 46 49
0x20:
52 20 20 20 00 00 90 65 46 54 52 4A
0x30:
50 31 42 54 4C 2D 43 34 42 20 20 20
0x40:
00 1A 00 00 46 4E 53 30 39 35 31 4A
0x50:
20 20 20 20 30 35 31 32 31 33 20 20
0x60:
00 00 02 4C D2 86 D7 04 F8 8D 92 6D
0x70:
60 F3 BD 00 00 00 00 00 00 00 00 00
0x80:
FF FF FF FF FF FF FF FF FF FF FF FF
0x90:
Status/Control Register: 10f8
Alarm Status: 0000
Warning Status: 0000
19
4E
31
05
30
58
3C
62
FF
FF
00
49
33
1E
56
80
8B
56
FF
FF
02
53
32
00
4E
01
D2
5D
FF
FF
14
41
31
23
20
D3
2D
CD
FF
FF
THRESHOLDS
Temperature
high alarm high warning

low warning
low alarm
+110.000
+093.000
-30.000
-40.000
2-72
OL-23591-02
Chapter 2

Voltage
V
003.9000
Bias Current
mA
080.0000
Transmit power mW
002.7234
Receive power mW
005.0328
Diagnostics contents (hex):
0x00:
6E 00 D8 00 5D 00 E2
0x10:
9C 40 03 E8 88 B8 07
0x20:
C4 98 00 78 82 B5 01
0x30:
00 00 00 00 00 00 00
0x40:
00 00 00 00 3E 72 B9
0x50:
00 77 FF AB 01 00 00
0x60:
21 EC 81 18 01 0E 00
0x70:
00 00 00 80 00 00 00
0x80:
57 4D 31 54 32 54 54
0x90:
31 2D 30 31 56 30 31
0xA0:
00 00 00 00 00 00 00
0xB0:
00 00 00 00 00 00 00
0xC0:
53 46 50 2D 4F 43 34
0xD0:
20 20 20 20 20 20 20
0xE0:
00 00 00 00 00 00 00
0xF0:
00 00 00 00 00 00 00
003.7000
070.0000
001.7251
003.3461
00
D0
75
00
A5
00
BA
00
41
20
00
00
38
20
00
00
98
6A
00
00
41
01
19
00
41
8A
00
00
2D
20
00
00
58
62
00
00
37
00
DA
00
41
FB
00
00
53
20
00
40
69
04
00
00
B7
00
00
00
31
55
00
00
52
20
00
00
002.9000
004.0000
000.1704
000.0373
78
77
00
00
FE
00
00
00
30
00
00
00
20
20
00
40
90
43
00
00
01
10
00
00
2D
00
00
00
20
20
00
00
88
63
00
00
00
00
00
00
31
00
00
00
20
20
00
00
71
06
00
00
00
00
10
00
39
00
00
AA
20
20
00
00
48
A8
00
00
00
90
F8
01
36
64
00
AA
20
66
00
00
19
4E
31
05
30
58
CB
88
FF
FF
00
49
33
1E
42
80
9B
17
FF
FF
02
53
32
00
56
01
FD
A0
FF
FF
14
41
31
23
20
CE
A3
7B
FF
FF
002.7000
002.0000
000.1143
000.0120

POS0/2/0/1:
ID: SFP
Extended ID: 4
Connector: LC
Encoding: reserved
Bit Rate: 2500 Mbps
Single mode fiber supported length: 2 km
Single mode fiber supported length: 20 m
Upper bit rate limit: not specified
Lower bit rate limit: not specified
Date code (yy/mm/dd): 06/04/25
Vendor name: CISCO
Vendor OUI: 36965
Vendor Part Number (PN): FTRJ1321P1BTL-C4
Vendor Rev: B
Vendor SN (SN): FNS1017R0BV
Options implemented:
LOS Signal
TX Fault Signal
TX Disable Signal
Enhanced options implemented:
Alarm/Warning Flags
Diagnostic monitoring implemented:
Exernally Calibrated
Digital Diagnostic Monitoring
Idprom contents (hex):
0x00:
03 04 07 00 01 00 00 12 00 01 05 05
0x10:
00 00 00 00 43 49 53 43 4F 2D 46 49
0x20:
52 20 20 20 00 00 90 65 46 54 52 4A
0x30:
50 31 42 54 4C 2D 43 34 42 20 20 20
0x40:
00 1A 00 00 46 4E 53 31 30 31 37 52
0x50:
20 20 20 20 30 36 30 34 32 35 20 20
0x60:
00 00 02 91 44 1B BF 32 33 64 55 09
0x70:
39 11 A4 00 00 00 00 00 00 00 00 00
0x80:
0x90:
Status/Control Register: 10f8
Alarm Status: 0000
Warning Status: 0000
OL-23591-02
2-73
Chapter 2
THRESHOLDS
high alarm high
Temperature
C
+110.000
Voltage
V
003.9000
Bias Current
mA
080.0000
Transmit power mW
003.5276
Receive power mW
005.0147
Diagnostics contents (hex):
0x00:
6E 00 D8 00 5D 00 E2
0x10:
9C 40 03 E8 88 B8 07
0x20:
C3 E3 00 9B 82 49 01
0x30:
00 00 00 00 00 00 00
0x40:
00 00 00 00 3E 73 C5
0x50:
00 5B FF F2 01 00 00
0x60:
21 D8 80 B8 00 54 00
0x70:
00 00 00 80 00 00 00
0x80:
57 4D 31 54 32 54 54
0x90:
31 2D 30 31 56 30 31
0xA0:
00 00 00 00 00 00 00
0xB0:
00 00 00 00 00 00 00
0xC0:
53 46 50 2D 4F 43 34
0xD0:
20 20 20 20 20 20 20
0xE0:
00 00 00 00 00 00 00
0xF0:
00 00 00 00 00 00 00
warning
low warning
low alarm
+093.000
-30.000
-40.000
003.7000
002.9000
002.7000
070.0000
004.0000
002.0000
002.2272
000.2022
000.1291
003.3353
000.0407
000.0155
00
D0
97
00
30
00
22
00
41
20
00
00
38
20
00
00
98
89
00
00
40
01
2A
00
41
8A
00
00
2D
20
00
00
58
CC
00
00
46
00
02
00
41
FB
00
00
53
20
00
40
69
05
00
00
23
00
00
00
31
55
00
00
52
20
00
00
78
0B
00
00
7C
00
00
00
30
00
00
00
20
20
00
40
90
57
00
00
01
10
00
00
2D
00
00
00
20
20
00
00
88
00
00
00
00
00
00
00
31
00
00
00
20
20
00
00
71
07
00
00
00
00
10
00
39
00
00
AA
20
20
00
00
48
E6
00
00
00
7C
F8
01
36
64
00
AA
20
66
00
00
2-74
OL-23591-02
CH A P T E R

This chapter explains how to troubleshoot problems with connectivity between interfaces on the
ASR 9000 and interfaces on remote devices. It includes the following topics:
Troubleshooting Ping and ARP Connectivity, page 3-75
Troubleshooting Bidirectional Forwarding Detection, page 3-81
Troubleshooting Ethernet CFM, page 3-85

Follow the steps in this section to troubleshoot ping andAddress Resolution Protocol (ARP) connectivity
problems. The overall approach is to verify that the routing protocol is up, the network topology is
properly configured, and neighbors are up and reachable.
This procedure sends ping messages to the remote end and analyzes the resulting responses. For Ethernet
interfaces, Address Resolution Protocol (ARP) connectivity is a prerequisite for ping
connectivityARP must work first before ICMP echo can work. Therefore, check ARP and ensure it is
working so that ICMP echo and ping can work. (Optical interfaces do not involve ARP.)
You should trace the path of the ping packets to see if they are getting dropped at any point along the
path. Typical steps to locate these drops are:
Run the command show inject stats See if the packets went from the CPU to the NP.
Run the command show interface statsLook at the Tx counters to see if the packet was sent.
Check the remote interface statistics to see if they received the packets.
Check the remote punt and ARP statistics.
Run commands in this list (above) on the return patch to check for drops.
Figure 3-1 shows the general approach to troubleshooting ping and ARP connectivity issues.
Throubleshooting is required because a ping attempt has failed. In this example, the IGP protocol refers
to the protocol currently configured on the network you are troubleshootingOSPF, EIGRP, IS-IS, or
RIP.
OL-23591-02
3-75
Chapter 3
Figure 3-1
Example of Troubleshooting Ping and ARP Connectivity Issues
Yes
Is the
IGP protocol
running?
Is this
an Ethernet
interface?
No
No
Yes
Is the
IGP protocol
up?
No
Try
ping
Yes
Was an
ARP entry
created?
Yes
No
Debug
ARP
Try
ping
Was ping
successful?
No
Debug
ping packets
Exit
279971
Yes
Follow these steps to troubleshoot ping and ARP connectivity issues. See Figure 3-1 to help you locate
the steps that apply to your network scenario.
Step 1
Tip
Step 2
Ping the remote end and check for a response. If there is no response, continue with this procedure to
determine why the ping was unsuccessful and to connect successfully to the remote end.
Use a systematic process to isolate the location of the failure. Ping the local interface first. If that is
successful, ping the directly connected neighbor (single hop). If that is successful, ping the next hop, and
so forth.
Verify that the interface is configured as Layer 3.
3-76
OL-23591-02
Chapter 3

Step 3
Check the routing table to make sure the IP address you are trying to ping has a route in the Routing
Information Base (RIB) table.
RP/0/RSP0/CPU0:router# show route
Step 4
Verify that the IGP protocol is running and the connection to the neighbor is up. The following example
assumes OSPF as the IGP.
RP/0/RSP0/CPU0:router# show protocols ospf
RP/0/RSP0/CPU0:router# show ospf neighbor
Step 5
Verify that packets are coming in and going out on the Ethernet interface.
Be aware of the following ARP behaviors when you are reviewing the display from the show commands
in this step:
A normal ping will send an ARP packet out followed by the actual ICMP echo packets. ARP must
work before ICMP echo can work. If the system is receiving zero packets back, then there was no
ARP reply. Even a single packet back means there was an ARP reply. (The system sends the ARP
packet only if there is no ARP entry. Otherwise, it skips the ARP and proceeds with the ICMP echo.)
By default, the system attempts to ping the remote router five times. If the remote router was recently
connected to the network, the first ping will fail because the system needs time to resolve the ARP
packet.
RP/0/RSP0/CPU0:router# show interfaces location node-id
or
RP/0/RSP0/CPU0:router# show interfaces type
Step 6
Display the ARP information. The IP address that you attempted to ping should be in the output.
RP/0/RSP0/CPU0:router# show arp
Step 7
If this port was previously attached to another device, or some othe major change has taken place on the
remote end, use the clear arp-cache command to build a new entry. Verify that the MAC address in the
ARP table is correct (see the MAC address in the Hardware Addr column in the example in Step 6).
RP/0/RSP0/CPU0:router# clear arp-cache
Step 8
Determine whether an ARP entry exists for the destination IP.

RP/0/RSP0/CPU0:router# show arp location node-id
a.
If an ARP entry does not exist or is incomplete, add a static ARP entry. Ensure that the Tx adjacency
points to COMPLETE.
RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id
Caution
After you finish using the static ARP entry for troubleshooting purposes, you must remove it. If you do
not remove the static ARP entry, it will cause traffic to be misdirected.
b.
If the ARP entry points to COMPLETE, it means that the ARP entry is not being updated.
Troubleshooting should now focus on why the ARP entry is not getting added (this includes steps
such as show arp, show arp idb, show adjacency gig node-id detail location node-id, and show
arp trace).
OL-23591-02
3-77
Chapter 3
Step 9
c.
If the Tx adjacency still points to PUNT, it means ARP is adding the entry in the database, but
fib_mgr fails to mark the adjacency as COMPLETE.
d.
This could be a fib_mgr, ARP, or AIB problem. Delete and reconfigure the static ARP entry with
AIB and CEF debugs on. The debugs show if ARP is adding the entry inside the AIB and if the AIB
is informing fib_mgr.
Send a burst of traffic to help troubleshoot whether traffic is getting through. This can help in scenarios
such as a disconnected cable, intermittent drops at unknown locations, and so forth.
a.
Configure a static ARP entry, then send a large number of ping packets (for example, 100 or 1,000
packets) with a zero timeout. This sends out a burst of traffic from the router. When the ping fails,
the system displays a dot instead of an exclamation point. If the ping is not possible, the system
displays a U instead of a dot.
Example:
RP/0/RSP0/CPU0:router# ping 192.0.2.55 count 100 timeout 0
Wed Sep 29 15:09:29.809 EDT
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 192.0.2.55, timeout is 0 seconds:
......................................................................
..............................
Success rate is 0 percent (0/100)
Step 10
b.
There are parameters to the ping command that can be used to change the time delay for the reply.
Try lowering the delay, changing parameters to send 100 ping requests, and so forth.
c.
There is a mode that allows you to suppress the ARP request and send out only the ICMP echo
packets. See if the pings are failing intermittently or all the time.
If ARP connectivity fails, perform the following steps to find out why.
a.
Remove the static ARP entry.
b.
Local pingPing your own interface (the interface your router uses to send out the pings).
c.
Determine whether the local ping was successful. If the local ping failed (no response), pings out of
that interface will also fail.
d.
If you have an ARP entry, verify that there are outgoing/incoming ICMP echo/reply packets.
Note
Step 11
A ping is represented by a dot, exclam point, or capital U. The RSP allows a specific number of
seconds for the ping to complete. The ARP is hidden inside this event.
If you have to dig deeper into the issue, use the following commands to dump ping packets.
Example
RP/0/RSP0/CPU0:router# show route
C 172.21.116.0/24 is directly connected, 2d19h, MgmtEth0/RSP0/CPU0/0
is directly connected, 2d19h, MgmtEth0/RSP1/CPU0/0
L 172.21.116.10/32 is directly connected, 2d20h, MgmtEth0/RSP0/CPU0/0
L 172.21.116.11/32 is directly connected, 2d19h, MgmtEth0/RSP1/CPU0/0
L 172.21.116.12/32 [0/0] via 172.21.116.12, 2d19h, MgmtEth0/RSP0/CPU0/0
O 192.168.12.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1
[110/2] via 192.168.121.12, 2d19h, GigabitEthernet0/2/0/2
[110/2] via 192.168.121.12, 2d19h, GigabitEthernet0/2/0/2
C 192.168.111.0/24 is directly connected, 2d20h, GigabitEthernet0/2/0/1
L 192.168.111.1/32 is directly connected, 2d20h, GigabitEthernet0/2/0/1
3-78
OL-23591-02
Chapter 3


RP/0/RSP0/CPU0:router# show protocols ospf

Routing Protocol OSPF 100
Router Id: 10.144.144.144
Distance: 110
Non-Stop Forwarding: Enabled
Redistribution:
None
Area 0
MPLS/TE enabled
Loopback0
TenGigE0/4/0/0* Indicates MADJ interface
Neighbors for OSPF 100
Neighbor ID
10.164.164.164
Neighbor is
10.166.166.166
Neighbor is
10.19.19.19
Neighbor is
10.11.11.11
Neighbor is
10.11.11.11
Neighbor is
Pri
State
1
FULL/DR
up for 2d17h
1
FULL/DR
up for 2d17h
1
FULL/BDR
up for 2d18h
1
FULL/BDR
up for 2d18h
1
FULL/BDR
up for 2d18h
Dead Time
00:00:36
Address
10.147.4.64
Interface
00:00:39
10.146.4.66
00:00:33
10.194.4.19
00:00:34
10.114.4.11
00:00:39
10.114.8.11
TenGigE0/4/0/0
Total neighbor count: 5
The following example shows the packet counts for a line card. Note that there are packets being input
and output.
RP/0/RSP0/CPU0:router# show interfaces location 0/4/CPU0
Wed Sep 1 09:22:03.427 DST
TenGigE0/4/0/0 is up, line protocol is up
Hardware is TenGigE, address is 001b.53ff.a780 (bia 001b.53ff.a780)
Layer 1 Transport Mode is LAN
Description: Connected to P11_CRS-4 10GE 0/2/5/0
Encapsulation ARPA,
Full-duplex, 10000Mb/s, LR, link type is force-up
loopback not set,
OL-23591-02
3-79
Chapter 3


Wed Sep 1 10:16:22.472 DST
------------------------------------------------------------------------------0/4/CPU0
------------------------------------------------------------------------------Address
Age
Hardware Addr
State
Type Interface
10.114.8.11
02:40:44
001b.0c63.67ff Dynamic
ARPA TenGigE0/4/0/0
10.114.8.44
001b.53ff.a780 Interface ARPA TenGigE0/4/0/0
------------------------------------------------------------------------------0/1/CPU0
------------------------------------------------------------------------------Address
Age
Hardware Addr
State
Type Interface
10.114.4.11
00:15:22
001b.0c63.67e7 Dynamic
ARPA GigabitEthernet0/1/0/23
10.114.4.44
001b.53ff.87f7 Interface ARPA GigabitEthernet0/1/0/23
10.145.4.38
01:43:50
001e.f77d.5219 Dynamic
10.145.4.44
001b.53ff.87fb Interface ARPA GigabitEthernet0/1/0/27
10.146.4.44
001b.53ff.87e8 Interface ARPA GigabitEthernet0/1/0/8
10.146.4.66
02:56:39
0022.0d26.3bc4 Dynamic
10.147.4.44
001b.53ff.87e2 Interface ARPA GigabitEthernet0/1/0/2
10.147.4.64
00:33:21
0022.0d26.36c4 Dynamic
10.194.4.19
03:16:59
001a.3029.d400 Dynamic
10.194.4.44
001b.53ff.87f2 Interface ARPA GigabitEthernet0/1/0/18
10.194.8.44
001b.53ff.87f0 Interface ARPA Bundle-Ether16.162
10.194.12.44
001b.53ff.87f0 Interface ARPA Bundle-Ether16.163
10.194.16.44
001b.53ff.87ec Interface ARPA GigabitEthernet0/1/0/12
------------------------------------------------------------------------------0/RSP0/CPU0
------------------------------------------------------------------------------Address
Age
Hardware Addr
State
Type Interface
172.29.52.1
01:51:49
001e.f77d.2a19 Dynamic
ARPA MgmtEth0/RSP0/CPU0/0
172.29.52.13
03:29:42
0010.79e9.6038 Dynamic
172.29.52.21
00:50:04
0022.0d5a.a6c4 Dynamic
172.29.52.22
02:44:58
0001.6443.1678 Dynamic
172.29.52.27
02:36:46
0012.7fd6.ba08 Dynamic
172.29.52.28
03:04:37
0012.7fd6.ba09 Dynamic
RP/0/RSP0/CPU0:router# show cef 192.168.1.1/32 hardware egress detail location 0/4/CPU0
192.168.6.73/32, version 0, internal 0x40040001 (ptr 0x9f613944) [1], 0x0 (0x9eb9bdfc),
0x4500 (0xa0156184)
Updated Sep 22 20:20:54.369
remote adjacency to GigabitEthernet0/1/0/23
Prefix Len 32, traffic index 0, precedence routine (0)
gateway array (0x9e935bbc) reference count 249, flags 0xd00, source lsd (2),
[84 type 5 flags 0x101001 (0x9fb06898) ext 0x0 (0x0)]
LW-LDI[type=5, refc=3, ptr=0x9eb9bdfc, sh-ldi=0x9fb06898]
via 10.114.4.11, GigabitEthernet0/1/0/23, 10 dependencies, weight 0, class 0 [flags
0x0]
path-idx 0
next hop 10.114.4.11
remote adjacency
local label 16021
labels imposed {16031}
via 10.114.8.11, TenGigE0/4/0/0, 13 dependencies, weight 0, class 0 [flags 0x0]
3-80
OL-23591-02
Chapter 3

Troubleshooting Bidirectional Forwarding Detection
path-idx 1
next hop 10.114.8.11
local adjacency
local label 16021
.
.
.
TX Adjacency
Raw data for tx adj struct:
Raw result1: 0x03000100 0x01000000 0x7e23001b 0x0c6367ff
Raw result2: 0x95650300 0x00000000 0x00000000 0x00000000
--------------------------------------------Search Ctrl Flags:
-----------------match
: 1
valid
: 1
gre_adj
: 0
null_route
: 0
tx_punt
: 0
tx_drop
: 0
next_hop_down : 0
adj_complete : 0
punt_ifib
: 0
nhop_down
: 0
stop
: 0
match_all_bit: 0
default_action: 1
uidb_index
: 0x0001
l3_mtu
: 9086
dest mac
: 001b.0c63.67ff
prefix_adj_cnt_index: 0x95650300
RX Adjacency
Raw data for rx adj struct:
Raw result1: 0x13000100 0x00001300 0x0c000280 0x00000000
--------------------------------------------Search Ctrl Flags:
-----------------rx_punt
: 0
rx_drop
: 0
rx_adj_SFP
: 1
rp_destined
: 0
rp_drop
: 0
match
: 1
valid
: 1
rx_LAG_adj
: 0
match_all_bit : 0
pri_adj_down : 0
default_action: 1
rx_adj_field : 0x0013
egress_ifh
: 0xc000280
Load distribution: 0 1 (refcount 84)

Hash
0
1
OK
Y
Y
Interface
TenGigE0/4/0/0
Address
remote
10.114.8.11

Bidirectional Forwarding Detection (BFD) is a detection protocol designed to provide fast forwarding
path failure detection times for all media types, encapsulations, topologies, and routing protocols. This
section contains the following subsections:
Using show and debug Commands, page 3-82
BFD Sessions in Down State, page 3-83
BFD Sessions Flap, page 3-83
OL-23591-02
3-81
Chapter 3
BFD Sessions Down on Neighboring Router, page 3-85
BFD Sessions Are Not Created on the LC, page 3-85

SUMMARY STEPS
1.
show bfd [ipv4 | all] [location node-id]
2.
show bfd client [detail]
3.
show bfd [ipv4 | all] session [detail | [interface ifname] | [location node-id] ] [detail]
4.
show bfd counters packet [ interface ifname] location node-id
5.
show bfd trace {adjacency | error | fsm | packet} [interface ifname] [location node-id]
6.
show tech-support routing bfd {file | location | rack}
DETAILED STEPS
Step 1
Command or Action
Purpose
show bfd [ipv4 | all] [location node-id]
View general BFD information on the Route Switch

Processor (RSP), such as the number of sessions. Use the
location keyword to display information for a specific LC.
If not specified, information for all locations displays.
Example:
RP/0/RSP0/CPU0:router# show bfd location
0/4/CPU0
Step 2
show bfd client [detail]
View BFD clients. Use the detail keyword to display more

information.
Example:
RP/0/RSP0/CPU0:router# show bfd client detail
Step 3
show bfd [ipv4 | all] session [detail |

[interface ifname] | [location node-id] ]
[detail]
Example:
RP/0/RSP0/CPU0:router# show bfd session
interface Gig2/1/0/0 detail
Step 4
show bfd counters packet [interface ifname]

location node-id
Example:
RP/0/RSP0/CPU0:router# show bfd counters packet
interface POS 0/3/0/0 location 0/3/cpu0
View BFD session information. Filter results using the

following parameters and keywords:
locationBFD sessions hosted on this location.
interfaceBFD sessions on the specified interface (no

wildcards).
detailDetailed session information: statistics,

number of state transitions.
View packet counters information. Filter results using the

locationPacket counters for BFD sessions hosted on

this location.
interfacePacket counters for BFD sessions on the

specified interface (no wildcards).
invalidInvalid packet counter information.
3-82
OL-23591-02
Chapter 3

Step 5
Command or Action
Purpose
show bfd trace {adjacency | error | fsm |

packet} [interface ifname] [location node-id]
View tracing information from the RSP. Filter results using

the following parameters and keywords:
adjacencyTraces generated when BFD receives an

adjacency update from the Adjacency Information Base
(AIB) Finite State Machine (FSM) display.
errorTraces generated when an error is detected.
fsmTraces generated when there is a state change in

a session.
packetTraces generated when there is a change in a

Tx or Rx packet.
locationTraces for BFD traces on the specified

interface.
Example:
RP/0/RSP0/CPU0:router# show bfd trace fsm
location 0/4/CPU0
Note
Step 6
show tech-support routing bfd {file | location

| rack}
Log the trace to a file to save the results.
View BFD debugging information.
Example:
RP/0/RSP0/CPU0:router# show tech-support
routing bfd location 0/1/CPU0
BFD Sessions in Down State

To troubleshoot BFD sessions in the down state, perform the following steps.
Step 1
Verify IP connectivity. Verify there is no IP packet loss.

RP/0/RSP0/CPU0:router# ping local-remote-address
Step 2
Ensure that the router and remote device are configured with the following parameters:
Number of BFD sessions they can support
Timers to support the police rates
BFD Sessions Flap

To check various BFD parameters, perform the following steps. Also see:
Step 1
BFD Sessions Down on Neighboring Router, page 3-85
BFD Sessions Are Not Created on the LC, page 3-85
Verify IP connectivity.
RP/0/RSP0/CPU0:router# ping local-IP-address
OL-23591-02
3-83
Chapter 3
Step 2
View input and output counters.

RP/0/RSP0/CPU0:router# show interface
Step 3
View session detail information.

RP/0/RSP0/CPU0:router# show bfd session detail
Step 4
View session packet counters.

RP/0/RSP0/CPU0:router# show bfd counter
Step 5
View SPP counters.

RP/0/RSP0/CPU0:router# show spp node
Note
Step 6
SPP means software packet processing, but is more commonly referred to as vector path
processing (VPP).
View resource usage.

RP/0/RSP0/CPU0:router# monitor process
Step 7
View IP connectivity. Verify there is no IP packet loss.

RP/0/RSP0/CPU0:router# ping
local remote address
If the following message appears, the BFD flap is a result of the application flap.
bfd_agent[104]: %BFD-6-SESSION_REMOVED : BFD session to neighbor 192.168.1.1 on interface
Gi0/5/0/0 has been remove
Step 8
Verify that the SPP is not losing packets.

RP/0/RSP0/CPU0:router# show spp node location
Step 9
Check LC CPU and memory usage.

RP/0/RSP0/CPU0:router# monitor processes location
Step 10
Check the local interface counters.

RP/0/RSP0/CPU0:router# show interfaces type interface-name
Step 11
Check any QoS policies applied to the interface.

RP/0/RSP0/CPU0:router# show policy-map interface
Step 12
Repeat Step 1 through Step 11on the remote end.
BFD Sessions Flap Because of Local Echo Failure

BFD sessions flap may be locally triggered because the router detects echo failure.
3-84
OL-23591-02
Chapter 3

Troubleshooting Ethernet CFM
Examine LC CPU utilization:

RP/0/RSP0/CPU0:router# monitor process location
Examine the SPP process on the LC CPU to determine the delay encountered by BFD echo packets:
RP/0/RSP0/CPU0:router# show bfd trace performance reverse location
Rule out BFD echo packet loss: show bfd counters packet location
BFD Sessions Flap Because of SPP Process Restart

If BFD failure detection is configured to be within 1 second, the BFD session would flap if the SPP
process is restarted on the LC.
BFD Sessions Down on Neighboring Router

The neighbor router sends this message to indicate its BFD is going down:
LC/0/6/CPU0:Aug 8 16:42:56.821: bfd_agent[104]: %L2-BFD-6-SESSION_STATE_DOWN: BFD session
to neighbor 192.168.1.1 on interface Gi0/5/0/0 has gone down. Reason: Nbor signalled down
BFD Sessions Are Not Created on the LC

Up to 1024 BFD sessions are allowed per LC. Configuring more than 1024 BFD sessions may result in
random BFD sessions not being created.

Ethernet Connectivity Fault Management (CFM) monitors, detects, and diagnoses remote network faults
end-to-end across the network. It does this using keepalives and MAC-based ping and traceroutes.
Unlike most other Ethernet protocols which are restricted to a single physical link, CFM frames can
transmit across the entire end-to-end Ethernet network.
This section describes how to troubleshoot problems with CFM on the local ASR 9000 router. For more
information on how to use CFM to troubleshoot problems across the network, see the Ethernet CFM
section in Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component
Configuration Guide.
Figure 3-2 shows an example of maintenance domains across a network.
OL-23591-02
3-85
Chapter 3
Figure 3-2
CFM Maintenance Domains Across a Network
Customer Domain Level 7

Service Provider Domain Level 6
Operator Domains
Operator 1
CE 1
PE 1
Operator 2
PE 3
PE 4
PE 2
MEP
MIP
MIP
CE 2
MEP
Level 6
Level 6
MEP
MIP
MIP
MEP
MIP
MIP
MEP
Level 3
207581
Level 4
MEP
This section contains the following topics:
Tip
MEPs Are Not Created, page 3-88
MIPs Are Not Created, page 3-88
No CCMs are Received at the MEP or Peer MEPs Are Not Seen, page 3-89
Peer MEP Defects and Mismatches Are Seen, page 3-90
Remote Defect Indication Received, page 3-91
Peer MEP Times Out But No Alarm Or Action Occurs, page 3-92
No Debugs or Counters for Higher-Level Packets at a MEP or MIP, page 3-92
Dropped CFM PDUs, page 3-92
CFM ping Or traceroute Returns a not found Error, page 3-93
AIS Messages Are Not Sent, page 3-93
For an extensive discussion of CFM usage and CFM command examples, see Cisco ASR 9000 Series
Aggregation Services Router Interface and Hardware Component Configuration Guide.
3-86
OL-23591-02
Chapter 3


The show and debug commands in this section are useful for troubleshooting CFM. Further details on
these commands can be found in Cisco ASR 9000 Series Aggregation Services Router Interface and
Hardware Component Command Reference.
These commands are useful for checking the validity of the configuration commands:
show run ethernet cfmView all CFM global configuration.
show ethernet cfm configuration-errorsDisplays any problems that have been detected in the CFM
configuration.
These commands are useful for verifying CFM functions:
show ethernet cfm local maintenance-pointsDisplays a summary of the MEPs and MIPs that have
been created.
show ethernet cfm local mepsDisplays information about local MEPs, including continuity check
messages (CCMs), details about the types of packets being sent and received, and counters for each
packet type.
show ethernet cfm peer mepsDisplays information about peer MEPs, including details about any
peer MEP defects that have been detected.
show ethernet cfm traceroute-cacheDisplays the contents of the traceroute cache, that is, the result
of recent traceroute operations.
show ethernet cfm interfaces aisDisplays a summary of interfaces where AIS messages are being
sent or received.
show ethernet cfm interfaces statisticsDisplays counters for CFM PDUs that are dropped per
interface.
show ethernet cfm ccm-learning-databaseDisplays the contents of the CCM learning database,
which the system uses when it responds to received traceroute (linktrace) messages.
debug ethernet cfm packetsEnables debugging of sent and received CFM PDUs.
debug ethernet cfm protocol-stateEnables debugging of major CFM protocol state-machine

operations.
If you need to collect information to provide to Cisco, the following commands are also useful, in
addition to those listed above. Note that many of these commands require the cisco-support task ID.
show tech-support ethernet
show ethernet cfm trace
show ethernet cfm interfaces status
show ethernet cfm services
debug ethernet cfm platformDisplays platform-specific debugging information for CFM.
debug ethernet oam platformDisplays platform-specific debugging information for OAM.
show spp nodeDisplays SPP counters.
show spp sid statsCheck the SPP stream ID (SID) statistics to see that CFM traffic is injected and
punted.
show spp clientDisplays information from the RSP about traffic on the SPP. Check to see if there
are any SPP drops.
OL-23591-02
3-87
Chapter 3
Note
To clear the spp counters, run the command clear spp {client | interface | node-counters} location
node-id. This command clears client statistics, interface statistics, and per-node counters, depending on
the keyword you use.
MEPs Are Not Created

If MEPs have been configured but have not been created, follow the troubleshooting steps in this section.
Step 1
Display information about errors that might be preventing configured CFM operations from becoming
active, as well as any warnings that have occurred.
show ethernet cfm configuration-errors
Step 2
Display a list of local maintenance points that have been created. Verify that the list contains the
expected nodes.
show ethernet cfm local maintenance-points
Step 3
Display operational states of local MEPs. Verify that the states are as expected.
show ethernet cfm local meps
MIPs Are Not Created

This section explains what to do if MIP creation has been configured but MIPs have not been created as
expected. Understand the factors that can impact MIP creation, then troubleshoot the specific MIP
creation issues.
Understanding the Factors that Impact MIP Creation

Configuring MIP creation for a service does not guarantee that MIPs will actually be created for all
interfaces in that service. MIPs are only created on interfaces that are correctly configured for Layer 2
switching, that is, interfaces that:
Are configured as Layer 2 interfaces
Have an appropriate encapsulation configured
Have been added to a bridge domain or point-to-point xconnect.
The CFM standard (IEEE 802.1ag-2007) specifies an algorithm that is used to determine whether a MIP
should be created, and at what level. For details of this implementation, see the mip auto-create
command in Cisco ASR 9000 Series Aggregation Services Router Interface and Hardware Component
Command Reference.
Troubleshooting MIP Creation Issues

Follow these steps to troubleshoot MIP creation issues.
3-88
OL-23591-02
Chapter 3

Step 1
Display information about errors that might be preventing configured CFM operations from becoming
active, as well as any warnings that have occurred.
show ethernet cfm configuration-errors
Step 2
Display a list of local maintenance points that have been created. Verify that the list contains the
expected nodes. Check for MEPs configured on the interface, and for MIPs enabled on a service at a
lower level.
show ethernet cfm local maintenance-points
Step 3
If MIP creation is not functioning, verify that the bridge domain or xconnect is configured correctly. To
verify or troubleshoot these bridge domain and xconnect configurations, see Chapter 9,
Troubleshooting L2VPN and Ethernet Services.
No CCMs are Received at the MEP or Peer MEPs Are Not Seen
This section explains how to troubleshoot the following conditions:
Continuity check messages (CCMs) are not seen at one or more maintenance end points (MEPs).
Peer MEPs are not seen.
CFM MEPs exchange CCMs periodically according to parameters configured on the system. These
CCMs are multicast to all other MEPs in the service at the same level. When the local MEP receives a
CCM, it creates an entry in the peer MEP table. If CCMs are not being exchanged correctly, perform the
following steps.
Step 1
Verify that CCM is enabled and there is a supported encapsulation on the interface.
Step 2
View configured MEPs and maintenance intermediate points (MIPs).

RP/0/RSP0/CPU0:router# show ethernet cfm local maintenance-points
Step 3
Display operational states of local and peer MEPs. Verify that CCM is enabled and the states are as
expected.
Step 4
show
show
show
show
ethernet
ethernet
ethernet
ethernet
cfm
cfm
cfm
cfm
local meps
local meps verbose
peer meps
peer meps detail
View packets seen by the CFM PI. Enable all of the options. The output shows if packets are dropped,
forwarded, or processed.
RP/0/RSP0/CPU0:router# debug ethernet cfm packets packet-type ccm
Step 5
View remote MEPs shown by the specific LC CFM instance. If CCMs are not received, the peer does not
display.
Step 6
View CFM SID statistics seen by the SPP. This displays any CFM traffic that is injected and punted.
RP/0/RSP0/CPU0:router# show spp sid stats
Step 7
View SPP drops.

RP/0/RSP0/CPU0:router# show spp client location location
OL-23591-02
3-89
Chapter 3
Step 8
Show bi-state alarms, to check for invalid encapsulation.
Step 9
Check for dropped PDUs as described in the Dropped CFM PDUs section on page 3-92.
Peer MEP Defects and Mismatches Are Seen

This section explains what to do when MEPs are exchanging CCMs, but there are defects or mismatches
in the received CCMs. Perform the following steps to locate the specific problem. Then take appropriate
corrective action.
Step 1
Run the following commands to obtain the output you will need for detailed troubleshooting.
Step 2
Step 3
show
show
show
show
ethernet
ethernet
ethernet
ethernet
cfm
cfm
cfm
cfm
local meps
local meps verbose
peer meps
peer meps detail
Wrong levelCheck the output of the commands in Step 1 to see if CCMs are being received at a lower
level than the level of the local MEP. This indicates a misconfiguration, for example:
The domain level is configured incorrectly on the local device or the peer device.
An intended MEP at the lower level has not been configured, and as a result the CCMs it would
consume are reaching the local MEP.
The forwarding path within the network has been misconfigured, such that CCM packets are being
received from an unintended source.
Cross-connect (wrong MAID)Check the output of the commands in Step 1 to see if CCMs are being
received with an maintenance association identifier (MAID) that does not match the MAID configured
locally for the service. The MAID is formed from the maintenance domain identifier (MDID) and the
short maintenance association name (SMAN). By default, the MDID is set to the name of the domain
and the SMAN is set to the name of the service. A crossconnect error indicates a misconfiguration, for
example:
The domain name or ID is configured incorrectly on the local device or on the peer device.
The service name or ID is configured incorrectly on the local device or on the peer device.
The forwarding path within the network has been misconfigured, such that CCM packets are being
received from an unintended source.
Step 4
Wrong intervalCheck the output of the commands in Step 1 to see if CCMs are being received with a
CCM interval that does not match the locally configured CCM interval. This indicates that the interval
is configured incorrectly on either the local device or the peer device. For a given service, the same CCM
interval must be configured on all devices.
Step 5
Loop (local MAC address received)Check the output of the commands in Step 1 to see if CCMs are
being received with the source MAC equal to the MAC address of the interface for the local MEP. This
indicates that there is a loop in the network such that the local device is receiving its own packets, or that
two devices in the network are configured with the same MAC address.
Step 6
Configuration (local MEP ID received)Check the output of the commands in Step 1 to see if CCMs
are being received from a peer MEP with the same MEP ID as the local MEP. This defect indicates that
two MEPs are configured with the same MEP ID. Across the entire network, each MEP in the service
must be configured with a different MEP ID.
3-90
OL-23591-02
Chapter 3

Step 7
Peer interface downCheck the output of the commands in Step 1 to see if CCMs are being received
that indicate the interface on the peer MEP is down, or that the interface on every peer MEP is STP
blocked. This indicates a problem with the operational state of the network.
Step 8
Missing (crosscheck)Check the output of the commands in Step 1. If crosscheck is configured

specifying this peer MEP, but no CCMs are being received, the peer MEP is missing. This might indicate
a failure in the network.
Step 9
Unexpected (crosscheck)Check the output of the commands in Step 1. If crosscheck is configured and
CCMs are being received from a peer MEP that is not specified, these CCMs are unexpected. This may
indicate a misconfiguration or that CCMs are being received from an unintended source.
Step 10
Remote defect receivedCheck the output of the commands in Step 1. If received CCMs indicate that
the peer MEP has detected a defect, take the action recommended in the Remote Defect Indication
Received section on page 3-91.
Remote Defect Indication Received

This section explains what to do if the local router receives a remote defect indication (RDI) from a peer
router.
Understanding How RDIs are Exchanged

When a MEP detects a defect (as described in the Peer MEP Defects and Mismatches Are Seen section
on page 3-90), it sets the remote defect indication (RDI) in the CCMs it is sending. When another MEP
receives the RDI, it recognizes that the peer MEP sending the CCMs has detected a defect.
In a point-to-point service, most defects will be detected by both MEPs; therefore both MEPs will
send the RDI and both will receive the RDI. However, a unidirectional failure in the network could
cause one of the MEPs to detect a crosscheck missing defect, while the other MEP does not detect
any defect. In this case, the RDI sent by the first MEP serves to notify the second MEP of the
problem.
In a multipoint service, if there is a defect on any MEP or pair of MEPs, all other MEPs in the service
receive the RDIs from the MEP or MEPS that detected the defect.
Locating the Source of RDIs and Resolving Defects

Step 1
Run the following commands to obtain the output you will need for troubleshooting RDIs.
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail
Step 2
Determine the peer MEP from which the RDI is being received.
Step 3
Log into the peer device and follow the steps in the Peer MEP Defects and Mismatches Are Seen
section on page 3-90.
OL-23591-02
3-91
Chapter 3
Peer MEP Times Out But No Alarm Or Action Occurs

This section explains what to do if a peer MEP times out without generating an alarm or automatic
corrective action. You need to verify that crosscheck is configured and functioning correctly.
If the local MEP stops receiving CCMs from a peer MEP, it times out by default after 3.5 times the CCM
interval. This is refered to as a loss of continuity. By default, a loss of continuity does not trigger any
other actions, such as log messages, SNMP traps, AIS, or Ethernet fault detection (EFD). These actions
are only triggered if crosscheck is configured for the service and the peer MEP is specified. In this case,
the loss of continuity causes a crosscheck missing defect, and this in turn triggers the other actions.
Step 1
Check that MEP crosscheck is configured for the service.

Step 2
Check for the crosscheck missing defect.

No Debugs or Counters for Higher-Level Packets at a MEP or MIP

CFM packets at a higher level than the highest MEP or MIP configured on the interface are forwarded
by the network processor and are not handled by the software. Therefore, it is not possible to display
debugs or counters for these packets. In addition, certain packets at the same level as a MIP are
forwarded by the network processor. Again it is not possible to display debugs or counters for these.
Dropped CFM PDUs

If CFM PDUs are not reaching the expected destination or are not being processed as expected, it is
possible that they are being dropped. The PDU drops could be caused by any of these reasons:
Dropped by the network processor
Dropped when being passed to software due to exceeding the supported CFM packet rate of 16,000
packets per second per line card
Note
Note that the CFM packet rate limit (16,000 CFM packets per second per line card) includes all
CFM packet types, including linktrace (traceroute) and loopback (ping) packets, as well as
CCMs and Ethernet SLA probes. Normally, the number of linktrace or loopback packets is low;
however, the use of continuity-check auto-traceroute can cause a high number of linktrace
packets to be sent, if a number of peer MEPs time out in quick succession.
Dropped because the interface or the forwarding node is down
Dropped because the PDU is invalid or not formed properly
Dropped because a higher level MEP was reached
Dropped due to an unknown PDU type
Dropped because the configured maximum MEPs limit (default 100) has been reached for the
service
3-92
OL-23591-02
Chapter 3

To display information for troubleshooting dropped CFM PDUs, perform the following steps. Take
corrective actions based on the outputs of the commands in these steps.
Step 1
Enable packet debugging to determine whether forwarded packets are being received at the MIP.
RP/0/RSP0/CPU0:router# debug ethernet cfm packets [received dropped interface
gigabitEthernet node-id]
Step 2
Display the statistics of the CFM PDUs per interface. Look for any drops ted to packets that are
improperly formed, invalid, wrong level, or unknown type.
RP/0/RSP0/CPU0:router# show ethernet cfm interfaces statistics
Step 3
Display the local MEPs and look for discarded CCMs. Discarded CCMs might indicate the the
configured maximum MEPs limit (default 100 MEPs per service) is reached.
RP/0/RSP0/CPU0:router# show ethernet cfm local meps verbose
Step 4
View peer MEPs seen by every local MEP.

Step 5
Check the STP status on the interfaces with MEPs or MIPs. CFM PDUs originating at MEPs on a STP
block port get forwarded, however, PDUs forwarded on a MIP are subject to the STP port state. This
means that if MIP is on a port that is STP blocked, then CFM PDUs will be dropped at the MIP.
Step 6
View STP state and CFM peer MEP status.

RP/0/RSP0/CPU0:router# show spanning-tree mst mstp
CFM ping Or traceroute Returns a not found Error

This section explains what to do if you perform a CFM ping or traceroute and receive a not found error.
For the ping or traceroute commands, the target is specified by means of a MAC address or a MEP ID.
If the target is specified as a MAC address, the MAC address is copied directly into the message.
However, if a MEP ID is specified, the system looks in the peer MEP table to find the MAC address for
the corresponding peer MEP. If there is no peer MEP for the service with the specified MEP ID, or if
there is more than one peer MEP for the service with the specified MEP ID, this lookup fails and the
system returns a not found error.
View the peer MEPs and check that there is an entry for the MEP ID that was being used as the target
MEP ID in the ping or traceroute command.
AIS Messages Are Not Sent

Alarm indication signal (AIS) can be enabled in configuration, either for MEPs or explicitly on an
interface. The system sends AIS messages when it detects a peer MEP defect, when it receives AIS or
LCK messages, or when the interface is down. AIS messages are sent in one of two ways:
OL-23591-02
3-93
Chapter 3
If there is another MEP on the interface at a higher level, and in the same direction, the AIS messages
are sent internally from the lower level MEP to the next highest level MEP. In this case, no actual
PDUs are transmitted.
Otherwise, if there is a MIP on the interface then AIS PDUs are transmitted at the level of the MIP.
If there is no MIP on the interface, no AIS messages are transmitted.
Use the following steps for troubleshooting.

Step 1
Verify that AIS is enabled in the configuration.

Step 2
Verify that there is a MIP.

RP/0/RSP0/CPU0:router# show ethernet cfm local maintenance-points
Step 3
Display the information published in the interface AIS table, including a record of the AIS
transmissions. Determine whether AIS messages are actually being sent.
RP/0/RSP0/CPU0:router# show ethernet cfm interfaces ais
Step 4
Determine whether the system should be sending AIS messages.

RP/0/RSP0/CPU0:router# show ethernet cfm local meps detail
3-94
OL-23591-02
CH A P T E R

This chapter explains how to troubleshoot router forwarding.
Cisco Express Forwarding (CEF) is the mechanism that enables packet forwarding. CEF information is
examined when data forwarding is not occurring as expected. Troubleshooting CEF involves comparing
the Routing Information Base (RIB) information to the software Forwarding Information Base (FIB),
verifying that the hardware is programmed correctly, verifying that the adjacencies are built correctly,
verifying the control plane is built correctly, and gathering any necessary trace information.
The only prerequisite for CEF is a valid route in the RIB.
This chapter includes the following sections:
Understanding IPv4 CEF, page 4-95
Troubleshooting IPv4 CEF, page 4-96
Troubleshooting Adjacency Information, page 4-101
Troubleshooting Transient Traffic Drop, page 4-106
Troubleshooting Packet Drop in the Fabric, page 4-109
Troubleshooting Control Plane Information, page 4-109
Understanding IPv4 CEF

CEF is an advanced, Layer 3 IP switching technology that optimizes network performance. It also
improves the scalability for networks with large and dynamic traffic patterns, such as the Internet and
networks characterized by intensive Web-based applications.
Information conventionally stored in a route cache is stored in several data structures for CEF switching.
The data structures provide optimized lookup for efficient packet forwarding. The two main components
of CEF operation are forwarding information base (FIB) and adjacency tables:
CEF uses a FIB to make IP destination prefix-based switching decisions. FIB maintains a mirror
image of the forwarding information contained in the IP routing table. When routing or topology
changes occur in the network, the IP routing table is updated, and those changes are reflected in the
FIB. The FIB maintains next hop address information based on the information in the IP routing
table. There is a one-to-one correlation between FIB entries and routing table entries, therefore FIB
contains all known routes and eliminates the need for route cache maintenance that is associated
with switching paths such as fast switching and optimum switching.
Nodes in the network are said to be adjacent if they can reach each other with a single hop across a
link layer. In addition to the FIB, CEF uses adjacency tables to prepend Layer 2 addressing
information. The adjacency table maintains Layer 2 next-hop addresses for all FIB entries.
OL-23591-02
4-95
Chapter 4
Figure 4-1 shows the components that contribute information to the CEF process, including
autosynchronization of the RIB with the FIB.
Note
In this document, the FIB is also referred to as the CEF table.

Figure 4-1
CEF Process
RP
LDP
LC-CPU
RSVP
AIB
LSD
BGP
HW
IFMGR
BCDL
OSPF
GSP
RIB
FIB
Process
ISIS
Sw
FIB
Hw
FIB
Netio
208808
STATIC
ROUTES

To troubleshoot IPv4 CEF information, perform the following procedure.
This procedure checks that neighbors are recognized, packets are flowing along the expected path, and
packets are not being dropped between neighbor interfaces.
SUMMARY STEPS
1.
show route ipv4 prefix
2.
show cef ipv4 prefix mask detail
3.
show cef ipv4 prefix mask detail location node-id (on ingress line card)
4.
show cef ipv4 prefix mask detail location node-id (on egress line card)
5.
show cef ipv4 prefix mask hardware ingress detail location node-id
6.
show cef ipv4 prefix mask hardware egress detail location node-id
7.
show cef ipv4 interface type instance location node-id
8.
show cef ipv4 summary location node-id
9.
show cef ipv4 trace location node-id
10. show cef platform trace ipv4 all location node-id

11. show controllers pse qfp feature forward client ltrace unicast error location node-id
12. Contact Cisco Technical Support if the problem is not resolved
4-96
OL-23591-02
Chapter 4

DETAILED STEPS
Step 1
Command or Action
Purpose
show route ipv4 prefix
Displays the current routes in the Routing Information Base

(RIB).
Example:
Check the prefix and mask, as well as the next hop and
outgoing interface, to ensure that they are what is
expected.
Note the timer value that shows how long the route has
been in the routing table. If the timer value is low the
route may be flapping.
show route 192.168.2.0
A lower timer value is present when a route is installed

in the RIB for a short period of time. A low timer value
may indicate flapping. For example, if a BGP route was
being installed and removed from the RIB table every
sixty seconds, then the route is flapping.
Look for routes that have not been installed in the
routing table for very long. The route will either be
stable or flapping. If the route is flapping, contact
contact Cisco Technical Support. For Cisco Technical
Support contact information, see the Obtaining
Documentation and Submitting a Service Request
section on page xii in the Preface.
Step 2
show cef ipv4 prefix mask detail
Example:
Check that route is learned via the routing protocol you

are expecting, and that the metric is what you expect.
Displays the IPv4 Cisco Express Forwarding (CEF) table

detailed entry information.
Compare the prefix, mask, next hop ip, and outgoing

interface information with the information in the RIB.
The information in the RIB is displayed using the show
route ipv4 prefix mask command as in Step 1.
Check that the adjacency is valid or the expected type

of adjacency. For example, if it is a remote adjacency,
then the adjacency information exists on another node.
Check that the expected hash (load balance) and egress

interfaces are listed.
RP/0/RSP0/CPU0:router# show route ipv4

192.168.2.0 255.255.255.0 detail
OL-23591-02
4-97
Chapter 4
Step 3
Command or Action
Purpose
show cef ipv4 prefix mask detail location

node-id
Displays the IPv4 CEF table for the designated ingress

node.

route ipv4 prefix command as in Step 1.

Example:
RP/0/RSP0/CPU0:router# show cef ipv4
192.168.2.0 255.255.255.0 detail location
0/14/cpu0

Step 4
show cef ipv4 prefix mask detail location

node-id
Displays the IPv4 CEF table for the designated egress node.

route ipv4 prefix mask command.


Example:
192.168.2.0 255.255.255.0 detail location
0/13/cpu0
Step 5
show cef ipv4 prefix mask hardware ingress

detail location node-id
Example:
192.168.2.0 255.255.255.0 hardware ingress
detail location 0/14/cpu0
Step 6
show cef ipv4 prefix mask hardware egress

detail location node-id
Example:
192.168.2.0 255.255.255.0 hardware detail egress
location 0/13/cpu0
Step 7
show cef ipv4 interface type instance location

node-id
Example:
show cef ipv4 interface
tengige 1/3/0/7 location 1/3/cpu0
Displays the IPv4 CEF table and corresponding forwarding

chain for the designated ingress node.
Check that the prefix and mask are valid.
Check the nexthop IP address is as expected
Check that the entry type is set to forward.
Check the adjacency packet counter and byte counter.
Displays the IPv4 CEF table and corresponding forwarding

chain for the designated egress node.
Check the nexthop IP address is as expected
Check that the entry type is set to forward.
Check the adjacency packet counter and byte counter.
Displays IPv4 CEF-related information for an interface.

Verify the interface handle interface is marked is as
expected. The command output also shows how many
references there are to the interface in CEF table and the
IPv4 MTU.
Use this command for the ingress and egress interfaces.
4-98
OL-23591-02
Chapter 4

Step 8
Command or Action
Purpose
show cef ipv4 summary location node-id
Displays a summary of the IPv4 CEF table. Check the VPN

routing and forwarding (VRF) names associated with the
node, the route update drops, and that there are the expected
number of incomplete adjacencies.
Example:
location 0/3/cpu0
show cef ipv4 summary
Note the number of routes CEF has entries for, the number
of load sharing elements, and the number of references to
this node.
Use this command for the ingress and egress line cards and
route processor (RP).
Step 9
show cef ipv4 trace location node-id
Displays IPv4 CEF trace table information.

Check if there is any flap on the prefix.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4 trace
location 0/3/cpu0
Step 10
show cef platform trace ipv4 all location

node-id
Use this command for the RP, and ingress and egress
interfaces for the local line card.
Displays CEF IPv4 hardware status and configuration trace

table information.
Use this command for the ingress and egress interfaces for
the local line card.
Example:
RP/0/RSP0/CPU0:router# show cef platform trace
ipv4 all location 0/3/cpu0
Step 11
show controllers pse qfp feature forward client

ltrace unicast error location node-id
(For SIP-700 line cards only) Displays trace files that

contain information on any engine error (if any) that
occurred in the unicast hardware structure programming.
Example:
RP/0/RSP0/CPU0:router# show contro pse qfp
feature forward client ltrace unicast error
location node-id
Step 12

Service Request section on page xii in the Preface.
Examples
The following examples show routes to two networks, one that is directly connected and one that is
learned. In the first example, the route was installed about 19 days ago, which might be as expected.
However, in the second example, the route was installed only 54 seconds ago, so it appears to be
flapping:
RP/0/RSP0/CPU0:router# show route ipv4 10.114.4.11
Tue Jul 13 09:25:47.754 DST
Routing entry for 10.114.4.0/24
Known via "connected", distance 0, metric 0 (connected)
Installed Jul 12 14:18:06.668 for 19:07:41 <<< This route appears to be stable
Routing Descriptor Blocks
directly connected, via GigabitEthernet0/1/0/23
Route metric is 0
Redist Advertisers:
OL-23591-02
4-99
Chapter 4
ospf 100
RP/0/RSP0/CPU0:router# show route ipv4 10.119.4.19
Tue Jul 13 09:28:38.407 DST
Routing entry for 10.119.4.0/24
Known via "ospf 100", distance 110, metric 2, type intra area
Installed Jul 12 15:00:10.327 for 00:00:54 <<< This route appears to be flapping
Routing Descriptor Blocks
10.114.4.11, from 10.19.19.19, via GigabitEthernet0/1/0/23
Route metric is 2
10.114.8.11, from 10.19.19.19, via TenGigE0/4/0/0
Route metric is 2
No advertising protos.
The following examples show interface details.

RP/0/RSP0/CPU0:router# show cef ipv4 interface TenGigE 0/6/0/1 location 0/4/CPU0
Tue Jul 13 11:39:13.693 DST
UNKNOWN intf 0x00000001 is unknown if_handle 0x00000001 if_type 0x0
idb info 0xa4d610d8 flags 0x301 ext 0xa5fe50cc
Vrf Local Info (0x0)
Interface last modified Jul 12, 2010 14:17:49, modify
Interface is marked as point to point interface
Reference count 1
Next-Hop Count 8
Protocol Reference count 1
Protocol ipv4 not configured or enabled on this card
Primary IPV4 local address NOT PRESENT
RP/0/RSP0/CPU0:router# show cef ipv4 interface TenGigE 0/6/0/1 location 0/6/CPU0
Tue Jul 13 11:39:39.969 DST
TenGigE0/6/0/1 is down if_handle 0x100000c0 if_type 0x1e
idb info 0xa4d61298 flags 0x1 ext 0x0
Vrf Local Info (0x0)
Interface last modified Jul 12, 2010 14:17:48, create
Reference count 1
Next-Hop Count 0
Protocol Reference count 0
Protocol ipv4 not configured or enabled on this card
Primary IPV4 local address NOT PRESENT
The following example shows the CEF summary. Use this display to check the VRF names, route update
drops, and adjacencies:
RP/0/RSP0/CPU0:router# show cef ipv4 summary location 0/1/CPU0
Tue Jul 13 12:50:48.259 DST
Router ID is 10.144.144.144
IP CEF with switching (Table Version 552) for node0_1_CPU0
Load balancing: L4
Tableid 0xe0000000 (0xa4a6ddb0), Vrfid 0x60000000, Vrid 0x20000000, Flags 0x301
Vrfname default, Refcount 251
163 routes, 0 reresolve, 0 unresolved (0 old, 0 new), 13040 bytes
60 load sharing elements, 129968 bytes, 342 references
8 shared load sharing elements, 8564 bytes
52 exclusive load sharing elements, 121404 bytes
0 CEF route update drops, 0 CEF rcc update drops
176 revisions of existing leaves
Resolution Timer: 15s
0 prefixes modified in place
0 deleted stale prefixes
99 prefixes with label imposition, 111 prefixes with label information
23 next hops
0 incomplete next hops
0 PD backwalks on LDIs with backup path
4-100
OL-23591-02
Chapter 4


To troubleshoot adjacency information on Cisco IOS XR software, perform the following procedure.
SUMMARY STEPS
1.
show arp location node-id
2.
show arp traffic location node-id
3.
show adjacency interface-type interface-instance remote detail location node-id
4.
show adjacency interface-type interface-instance remote detail hardware location node-id
5.
show adjacency ipv4 nexthop ipv4-address detail location node-id
6.
show adjacency interface-type interface-instance detail location node-id
7.
show adjacency ipv4 nexthop ipv4-address detail hardware location node-id
8.
show adjacency interface-type interface-instance detail hardware location node-id
9.
show adjacency trace location node-id
10. show adjacency trace client aib-client location node-id

11. show adjacency hardware trace location node-id
12. show cef adjacency tunnel-te tunnel-id hardware {egress | ingress} location node-id
13. Contact Cisco Technical Support if the problem is not resolved
DETAILED STEPS
Step 1
Command or Action
Purpose
Displays the Address Resolution Protocol (ARP) for an

egress line card with a broadcast interface.
Example:
Ensure that you can find the IP address and that correct
MAC address of the neighbor is learned.
RP/0/RSP0/CPU0:router# show arp location

0/12/cpu0
Step 2
show arp traffic location node-id
Displays ARP traffic statistics for an egress line card with a

broadcast interface.
Example:
Check for any errors or IP packet drops.
RP/0/RSP0/CPU0:router# show arp traffic

location 0/12/cpu0
Step 3
show adjacency interface-type

interface-instance remote detail location
node-id
Displays detailed CEF adjacency table information for a

remote ingress line card.
Ensure that the output shows IPv4 adjacency information
and that an adjacency exists.
Example:
RP/0/RSP0/CPU0:router# show adjacency pos
0/13/0/2 remote detail location 0/14/cpu0
OL-23591-02
4-101
Chapter 4
Step 4
Command or Action
Purpose

interface-instance remote detail hardware
location node-id
Displays adjacency information for a remote ingress line

card.
Example:
Check that the table look-up (TLU) pointers match the

TLU pointers in the show cef ipv4 prefix mask
hardware ingress detail location node-id command.
For example:

0/13/0/2 remote detail hardware location
0/14/cpu0
Step 5
show adjacency ipv4 nexthop ipv4-address detail

location node-id
Displays adjacencies on an egress line card with a broadcast

interface that are destined to the specified IPv4 next hop.
Example:
When an egress interface is broadcast, use the show

adjacency ipv4 nexthop command to display the adjacency
information.
RP/0/RSP0/CPU0:router# show adjacency ipv4

nexthop 192.168.2.0 detail location 0/12/cpu0
Compare the mac layer rewrite information that shows the

destination L2 address in the first part followed by the
source L2 address, and the Ethernet value with the output
from the show arp location node-id command.
Step 6

interface-instance detail location node-id
Example:
0/13/0/2 detail location 0/13/cpu0
Displays CEF adjacency table information for an egress line

card with a point to point interface.
There should be two IPv4 entries in the command output.
Ensure both entries exist.
The SRC MAC only entry is used for multicast

switching
The point to point entry is used for unicast switching.
On broadcast interfaces you will have a SRC MAC only and

one for each nexthop IP address. Please note the MTU is for
the IPv4 minus the Layer 2 header. Use the show im chains
command to display MTU details.
Step 7
show adjacency ipv4 nexthop ipv4-address detail

hardware location node-id
Displays the hardware programming associated with the

adjacency. Verify that the packets are being switched in the
hardware.
Example:
RP/0/RSP0/CPU0:router# show adjacency ipv4
nexthop 192.168.2.0 detail hardware location
0/12/cpu0
Step 8

interface-instance detail hardware location
node-id
Example:
0/13/0/2 detail hardware location 0/13/cpu0
Displays the hardware programming information for a

point-to-point interface such as the
Packet-over-SONET/SDH (POS) interface. The rewrite
information is slightly different because there is no MAC
rewrite string as there is in Ethernet.
Verify that the rewrite is appropriate for the encapsulation
on the interface. Compare the CEF hardware output and
verify that the pointer matches the egress adjacency.
4-102
OL-23591-02
Chapter 4

Step 9
Command or Action
Purpose
show adjacency trace location node-id
Displays CEF adjacency trace table information.
Example:
Use this command for the egress interfaces for the local line
card.
RP/0/RSP0/CPU0:router# show adjacency trace

location 0/1/cpu0
Step 10
show adjacency trace client aib-client location

node-id
Displays CEF adjacency trace table information for a

specified adjacency information base (AIB) client.
card.
Example:
RP/0/RSP0/CPU0:router# show adjacency trace
client ipv4_fib_mgr location 0/13/cpu0
Step 11
show adjacency hardware trace location node-id
Displays CEF adjacency hardware trace table information.
Example:
card.
RP/0/RSP0/CPU0:router# show adjacency hardware

trace location 0/13/cpu0
Step 12
show cef adjacency tunnel-te tunnel-id hardware

{egress | ingress} location node-id
Displays the IPv4 tunnel engineering (TE) tunnel

adjacencies. Verify the tunnel adjacencies are as expected.
Example:
show cef adjacency tunnel-te 1 hardware egress
location 0/13/CPU
Step 13

Examples
RP/0/RSP0/CPU0:router# show adjacency pos 0/2/0/1 remote detail hardware location 0/0/CPU0
Wed Nov 3 13:16:32.119 DST
Interface
Address
Version Refcount Protocol
PO0/2/0/1
(remote)
15
1(
0) fint_n2n
040001c0
flags 1 0 2
0 packets, 0 bytes
RP/0/RSP0/CPU0:router# show cef 10.3.3.3 hardware ingress location 0/2/CPU0

Wed Nov 3 13:19:23.263 DST
10.3.3.3/32, version 0, internal 0x40040001 (ptr 0xa667ad70) [1], 0x0 (0xa5728bc4), 0x4500
(0xa754df28)
Updated Oct 12 18:26:50.344
remote adjacency to GigabitEthernet0/1/0/23
Prefix Len 32, traffic index 0, precedence routine (0)
via 10.114.4.11, GigabitEthernet0/1/0/23, 10 dependencies, weight 0, class 0 [flags
0x0]
path-idx 0
next hop 10.114.4.11
remote adjacency
local label 16018
via 10.114.8.11, TenGigE0/4/0/0, 12 dependencies, weight 0, class 0 [flags 0x0]
OL-23591-02
4-103
Chapter 4
path-idx 1
next hop 10.114.8.11
remote adjacency
local label 16018
TBM Node Data:
Node (0x00000002):0
Node (0x89527010):1
Node (0x8944c2f0):2
Node (0x88f94530):3
Node (0x88f94540):4
Node (0x88f94550):5
0x8952700d
0x8944c2dd
0x88f9453d
0x88f9454d
0x88f94555
0x00002020
0x00000004
0x88f92d50
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x08888888
0x10000000
0x80000000
0x01000000
0x88fccc60
0xf7ff0000
0x88888888
0x00000000
0x00000000
0x00000000
0x88f96320
Hardware Leaf Data (0x88f94550):0x00002020 0x00000000 0x88fccc60 0x88f96320

IP Leaf Data:
as:0
prefix_len:32
for_us:0x0
dft_route:0x0
real_intf:0x1
free1: 0x0
hw_use_only: 0x0
lspa_ptr: 0x0
oce_chain_p: 0x88f96320
extre_fib_data_ptr: 0x88fccc60
Hardware Extended Leaf Data:
fib_leaf_extension_length: 0
interface_receive: 0x0
traffic_index_valid: 0x0
qos_prec_valid: 0x0
qos_group_valid: 0x0
valid_source: 0x0
traffic_index: 0x0
nat_addr: 0x0
reserved: 0x0
qos_precedence: 0x0
qos_group: 0x0
peer_as_number: 0
path_list_ptr: 0x0
connected_intf_id: 0x0
ipsub_session_uidb: 0xffffffff
Path_list:
urpf loose flag: 0x0
List of interfaces:
OCE Loadbalance Data for ptr 0x88f96320:
num_entries:2
level:0x1
pad_1:0x0
l3_lbe_ptr:0x8942d140
LBE Array for 0x8942d140
Entry 0: oce_chain_p 0x88f975b0
Entry 0: bgp_ipv4_next_hop_addr: 0x0
Entry 1: oce_chain_p 0x88f96e40
OCE Label Object Data for ptr 0x88f975b0:
flags: 0x0
number of labels: 1
protocol: 0
number bk labels: 0
out labels: 0x3e92
next_hw_oce_ptr: 0x88f97850
counter_ptr: 0x893e9720
Stats for ptr 0x893e9720:
byte count: 0
packet count: 0
OCE RX Adj Data for 0x88f97850:
base: 37(CPP HW RX ADJ MPLS)
adj_flags: 0x0
pd_16: 0x1005
pd_32: 0x2f
output_uidb: 0x1fea
counters_ptr: 0x893dc8a0
byte count: 0
packet count: 0
OCE Label Object Data for ptr 0x88f96e40:
flags: 0x0
number of labels: 1
protocol: 0
number bk labels: 0
out labels: 0x3e92
4-104
OL-23591-02
Chapter 4

next_hw_oce_ptr: 0x88f97840
counter_ptr: 0x893e9750
Stats for ptr 0x893e9750:
byte count: 0
packet count: 0
OCE RX Adj Data for 0x88f97840:
base: 37(CPP HW RX ADJ MPLS)
adj_flags: 0x0
pd_16: 0x6013
pd_32: 0x1
output_uidb: 0x1fd0
counters_ptr: 0x893dc8b0
byte count: 0
packet count: 0
The following example shows that the address information matches. The addresses are indicated in bold.
RP/0/RSP0/CPU0:router# show arp location 0/1/cpu0
Address
10.27.50.157
Age
02:08:34
Hardware Addr
0016.c761.f509
State
Dynamic
Type Interface
ARPA TenGigE0/1/0/2
RP/0/RSP0/CPU0:router# show adjacency ipv4 nexthop 212.27.50.157 detail loccation 0/1/cpu0

Interface
TenGigE0/1/0/2
Address
Version
10.27.50.157
41
0016c761f5090015fa9959890800
mtu: 1500, flags 0 0 0
2894 packets, 156876 bytes
0xffffffff
Refcount
2
Protocol
ipv4
RP/0/RSP0/CPU0:router# show adjacency gigabitEthernet 0/1/0/1 remote detail hardware

location all
Wed Nov 3 13:10:23.519 DST
------------------------------------------------------------------------------0/1/CPU0
------------------------------------------------------------------------------Interface
Address
Gi0/1/0/1
(remote)
6
1(
0) fint_n2n
020000c0
flags 1 0 2
0 packets, 0 bytes
------------------------------------------------------------------------------0/RSP1/CPU0
------------------------------------------------------------------------------Interface
Address
------------------------------------------------------------------------------0/RSP0/CPU0
------------------------------------------------------------------------------Interface
Address
Gi0/1/0/1
(remote)
7
1(
0) fint_n2n
020000c0
flags 1 0 2
RP/0/RSP0/CPU0:router# show cef adjacency tunnel-te 1 hardware egress location 0/3/CPU0

Wed Nov 3 13:37:17.935 DST
Interface not found (tunnel-te1)
Display protocol is ipv4
Interface
Address
BE16.162
Type
Refcount
special 2
Interface: BE16.162 Type: glean
Interface Type: 0x19, Base Flags: 0x4400 (0x9e4e9bb0)
Nhinfo PT: 0x9e4e9bb0, Idb PT: 0x9e3591d8, If Handle: 0x80001a0
Dependent adj type: remote (0x9f8af79c)
Dependent adj intf: BE16.162
OL-23591-02
4-105
Chapter 4
Ancestor If Handle: 0x0
BE16.163
special 2
Interface: BE16.163 Type: glean
Interface Type: 0x19, Base Flags: 0x4400 (0x9e4e9d1c)
Nhinfo PT: 0x9e4e9d1c, Idb PT: 0x9e359218, If Handle: 0x80001e0
Dependent adj type: remote (0x9f8b033c)
Dependent adj intf: BE16.163
Ancestor If Handle: 0x0
tt44190
Prefix: 0.0.0.0/32
no next-hop adj
Interface: NULLIFHNDL
local
--More--

Perform this procedure to troubleshoot transient drops in packet forwarding. The approach to
troubleshooting transient drops is as follows:
1.
Determine the interface drops.
2.
Determine the line card type. This is necessary because the next steps depend on whether you are
troubleshooting an Ethernet or SIP-700 line card (LC).
3.
(For Ethernet LC) Determine which NP contains the counters for the interface your are
troubleshooting.
4.
(For Ethernet LC) View the counters on the appropriate NP.
5.
For SIP-700 LC, display the drop statistics on the LC.
1.
show interface interface-type node-id
2.
show platform
3.
show controllers np ports all location node-id (for Ethernet)
4.
show controllers np count np-id location node-id (for Ethernet)
5.
show controllers pse qfp stat drop location node-id (for SIP-700)
SUMMARY STEPS
DETAILED STEPS
Step 1
Command or Action
Purpose
show interface interface-type node-id
Displays the interface drops.
Example:
show interface gigabitEthernet 0/0/0/0
Step 2
show platform
Determines the line card type. This is necessary because the

next steps depend on whether you are troubleshooting an
Ethernet or SIP-700 line card (LC).
4-106
OL-23591-02
Chapter 4

Step 3
Command or Action
Purpose
show controllers np ports all location node-id
(For Ethernet LC) Displays the port mapping between the

interface and the NP. View the output and determine which
NP contains the counters for the interface your are
troubleshooting.
Example:
Step 4
show controllers np count {np-id all} location

node-id | i DROP
(For Ethernet LC) View the counters on the appropriate NP.

The first command displays all counters, whether related to
drops or not. The second command limits the display to
only those counters that include the string DROP.
Example:
For additional information on interpreting NP counters, see

the Displaying Traffic Status in Line Cards and RSP
Cards section on page 7-147.
show controllers np count {np-id | all} ocation

node-id
show controllers np count all location 0/0/CPU0

show controllers np count all location
00/0/CPU0 | i DROP
Step 5
show controllers pse qfp stat drop location

node-id
(For SIP-700 LC) Display the drop statistics on the LC.
Example:
show controllers pse qfp stat drop location
0/6/CPU0
Example
RP/0/RSP0/CPU0:router# show interface gigabitEthernet 0/0/0/0
Tue Oct 26 21:04:12.805 UTC
GigabitEthernet0/0/0/0 is up, line protocol is up
Hardware is GigabitEthernet, address is 001b.53ff.a018 (bia 001b.53ff.a018)
Encapsulation ARPA,
Full-duplex, 1000Mb/s, SXFD, link type is force-up
loopback not set,
Last clearing of "show interface" counters 1w4d
1590651 packets input, 551036131 bytes, 0 total input drops <<< drops by framer or HW
97206 drops for unrecognized upper-level protocol <<< drops
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort <<< drops
1536152 packets output, 1427163508 bytes, 0 total output drops <<< sum of all output
drops, including drops from buffer, qos, or HW.
OL-23591-02
4-107
Chapter 4
Tue Oct 26 20:58:49.575 UTC

Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/0/CPU0
A9K-40GE-L
IOS XR RUN
PWR,NSHUT,NMON <<< Ethernet
0/3/CPU0
A9K-8T/4-E
IOS XR RUN
PWR,NSHUT,MON <<< Ethernet
0/4/CPU0
A9K-8T-E
IOS XR RUN
PWR,NSHUT,MON <<< Ethernet
0/6/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,NMON <<< SIP-700
0/6/0
SPA-2XCHOC12/DS0
OK
PWR,NSHUT,MON <<< SPA
0/6/1
SPA-5X1GE-V2
OK
PWR,NSHUT,MON <<< SPA
RP/0/RSP0/CPU0:router# show controllers np ports all loc 0/0/CPU0

Tue Oct 26 20:57:11.468 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------NP
-0
1
2
3
Bridge
-----0
0
1
1
Fia
--0
0
0
0
Ports
--------------------------------------------------GigabitEthernet0/0/0/30 - GigabitEthernet0/0/0/39
GigabitEthernet0/0/0/20 - GigabitEthernet0/0/0/29
RP/0/RSP0/CPU0:router# show controllers np counters all location 0/0/CPU0

Tue Oct 26 20:54:53.095 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------Show global stats counters for NP0, revision v3
Read 23 non-zero NP counters:
Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------22 PARSE_ENET_RECEIVE_CNT
74772482296
60925
23 PARSE_FABRIC_RECEIVE_CNT
80571
0
26 MODIFY_FABRIC_TRANSMIT_CNT
36431746029
29685
28 PARSE_INGRESS_DROP_CNT
18816500
0
RP/0/RSP0/CPU0:router# show controllers np count all location 0/0/CPU0 | i DROP
Tue Oct 26 20:56:10.714 UTC
38183944221
0
30 RESOLVE_INGRESS_DROP_CNT
157639443
0
31 RESOLVE_EGRESS_DROP_CNT
2559
0
291 DROP_IPV4_NOT_ENABLED
38174791832
0
438 RESOLVE_MAC_NOTIFY_CTRL_DROP_CNT
2559
0
18816500
0
Note
For a description of how to interpret NP counter information, see the Displaying Traffic Status in Line
Cards and RSP Cards section on page 7-147.
RP/0/RSP0/CPU0:router# show controllers pse qfp stat drop location 0/6/CPU0

Tue Oct 26 20:57:49.864 UTC
Global Drop Statistics for QFP 0
---------------------------------------------------------------Global Drop Stats
Packets
Octets
----------------------------------------------------------------
4-108
OL-23591-02
Chapter 4

AttnInvalidSpid
BadAdj
BadBhdr
0
0
0
0
0
0

To check whether packets are being dropped in the fabric, use the following commands.
show controllers fabric fia bridge stats location node-id
show controllers fabric fia drops ingress location node-id
show controllers fabric fia drops egress location node-id
show controllers fabric fia stats location node-id
For detailed fabric troubleshooting procedures, see Chapter 7, Troubleshooting Router Switch Fabric
and Data Path.

To troubleshoot control plane information on Cisco IOS XR software, perform the following procedure.
SUMMARY STEPS
1.
show netio idb interface-type interface-instance location node-id
2.
show uidb index
3.
show uidb data location node-id interface-type interface-instance {ingress | egress}
4.
show imds interface brief
5.
Contact Cisco Technical Support if the problem is not resolved
OL-23591-02
4-109
Chapter 4
DETAILED STEPS
Step 1
Command or Action
Purpose
show netio idb interface-type

interface-instance location node-id
Displays control plane information for the software

switching path. The output provides useful statistics for
determining software forwarding issues.
Example:
Verify the encap and decap paths
RP/0/RSP0/CPU0:router# show netio idb

tengige0/0/0/0 location 0/0/cpu0
Ensure that all the appropriate steps in the chain are

shown for all the features that may be enabled on the
interface.
Note
Fixup is a direct pointer to a routine in the output

path after a CEF rewrite. this is an optimized path if
a CEF rewrite exists and is usable.
Verify that the ifhandle and global uidb value is correct.
Step 2
Displays the micro-interface descriptor block (IDB) index

assigned by the software.
show uidb index
Example:
show uidb index
Check that the interface and the universal interface

descriptor block (UIDB) value are what is expected.
Compare the IDB index to the uidb index value in the show
adjacency ipv4 interface-type interface-instance detail
hardware location node-id command output.
Step 3
show uidb data location node-id interface-type

interface-instance {ingress | egress}
Example:
show uidb data location 0/6/CPU0
gigabitEthernet 0/0/0/2 ingress
Displays, from a software perspective, features that are

enabled on a selected interface.
Check the UIDB value.
Check what flags are enabled for the UIDB.
Check the ifhandle in the UIDB to make sure it is

correct.
Compare the output to the configuration of the interface and

expected features.
Step 4
Displays interface manager distribution server (IMDS)

interface information.
Example:
Note
RP/0/RSP0/CPU0:router# show imds interface

brief
Step 5
This is just a partial output not full output.
Check the state, MTU, encapsulation being used, and the

ifhandle for each interface.
4-110
OL-23591-02
Chapter 4

Examples
The following example displays the control plane information for the software switching path. Check
for any errors or drops.
RP/0/RSP0/CPU0:router# show netio idb tenGigE 0/1/1/0 location 0/1/cpu0
TenGigE0/1/1/0 (handle: 0x01180020, nodeid:0x11) netio idb:
--------------------------------name:
TenGigE0_1_1_0
interface handle:
0x01180020
30
dchain ptr:
<0x482ae8e0>
echain ptr:
<0x482d791c>
fchain ptr:
<0x482d79b8>
driver cookie:
<0x4824ad58>
driver func:
<0x4824ad44>
3
DSNCNF:
0x00000000
IN multicast pkts:
0
OUT multicast pkts:
0
IN broadcast pkts:
0
OUT broadcast pkts:
0
IN drop pkts:
0
OUT drop pkts:
0
IN errors pkts:
0
OUT errors pkts:
0
Chains
-------------------Base decap chain:
ether
<30>
<0xfd7aef88, 0x48302824>
<
0,
0>
Protocol chains:
Type Chain_node
<7> (arp)
Encap:
l2_adj_rewrite
<86> <0xfcec7a88, 0x4834efec> <
0,
0>
queue_fifo
<56> <0xfcedda68, 0x482dbee4> <
0,
0>
txm_nopull
<60> <0xfcea2a5c, 0x482dc11c> <
0,
0>
Decap:
queue_fifo
<56> <0xfcedda4c, 0x482dbee4> <
0,
0>
arp
<24> <0xfd1082cc, 0x00000000> <
0,
0>
Fixup:
l2_adj_rewrite
<86> <0xfcec745c, 0x00000000> <
0,
0>
queue_fifo
<56> <0xfcedda68, 0x482dbee4> <
0,
0>
txm_nopull
<60> <0xfcea2a5c, 0x482dc11c> <
0,
0>
<12> (ipv4)
Encap:
ipv4
<26> <0xfd10f41c, 0x482d7724> <
0,
0>
ether
<30> <0xfd7aeb44, 0x48302824> <
0,
0>
l2_adj_rewrite
<86> <0xfcec7a88, 0x4834f104> <
0,
0>
queue_fifo
<56> <0xfcedda68, 0x482dbee4> <
0,
0>
txm_nopull
<60> <0xfcea2a5c, 0x482dc11c> <
0,
0>
Decap:
queue_fifo
<56> <0xfcedda4c, 0x482dbee4> <
0,
0>
OL-23591-02
4-111
Chapter 4
ipv4
<26>
Fixup:
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
<22> (ether_sock)
Stats IN: 0
Encap:
ether_sock
<98>
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
Decap:
queue_fifo
<56>
ether_sock
<98>
Fixup:
l2_adj_rewrite
<86>
queue_fifo
<56>
txm_nopull
<60>
<0xfd10f474, 0x00000000>
<
0,
0>
<0xfcec745c, 0x00000000> <

0,
<0xfcedda68, 0x482dbee4> <
0,
<0xfcea2a5c, 0x482dc11c> <
0,
pkts, 0 bytes; OUT: 0 pkts, 0 bytes
0>
0>
0>
<0xfd7b1630,
<0xfcec7a88,
<0xfcedda68,
<0xfcea2a5c,
0x48302824>
0x48304c1c>
0x482dbee4>
0x482dc11c>
<
<
<
<
0,
0,
0,
0,
0>
0>
0>
0>
<0xfcedda4c, 0x482dbee4>
<0xfd7b1874, 0x48302824>
<
<
0,
0,
0>
0>
<0xfcec745c, 0x00000000>
<0xfcedda68, 0x482dbee4>
<0xfcea2a5c, 0x482dc11c>
<
<
<
0,
0,
0,
0>
0>
0>

-------------------Protocol
--------------ipv4
ipv4
ipv4
ipv6
ipv6
SAFI
---------Unicast
Multicast
Broadcast
Unicast
Multicast
Pkts In
---------0
0
0
0
0
Bytes In
---------0
0
0
0
0
Pkts Out
---------0
0
0
0
0
Bytes Out
---------0
0
0
0
0
The following example shows that the micro-idb index value is 12.
RP/0/RSP0/CPU0:router# show uidb index tengige1/3/0/6.30 location 1/3/cpu0
-----------------------------------------------------------------------------Location Interface-name
Interface-Type
Ingress-index Egress-index
--------------------------------------------------------------------------1/3/CPU0 TenGigE1_3_0_6.30
Sub-interface
20
12
Comparing the IDB index value of 12 in the show uidb index command to the uidb index value in the
following command output shows that the values are the same.
RP/0/RSP0/CPU0:router# show adjacency ipv4 tengige1/3/0/6.30 detail hardware location
1/3/cpu0
Interface
TenGigE1/3/0/6.30
Address
Version Refcount
(src mac only)
90
1
000000000000001243602d8b8100001e0800
mtu: 1500, flags 1 0 1
453 packets, 42582 bytes
453 hw-only-packets, 42582 hw-only-bytes
Protocol
ipv4
ether egress adjacency

TLU1
: 0x4407
[HW: 0x00401862 0xc4170800 0x8100001e 0x01060700]
num. entries : 1
uidb index
: 12
counter msb
: 0x2
counter lsb
: 0xc417
vlan e or len : 0x800
ether len
: 0x8100 (33024)
vlan info
: 30
next ptr
: 0x10607
4-112
OL-23591-02
Chapter 4

The following example displays, from a software perspective, features that are enabled on a selected
interface. Compare the output to the configuration of the interface and expected features. Verify that the
configured features are correctly enabled.
RP/0/RSP0/CPU0:router# show uidb data location 0/6/cpu0
-------------------------------------------------------------------------Location = 0/6/CPU0
Index = 0
Pse direction = INGRESS
Global general 16 bytes:
-----------------------ROUTER_ID: 45.104.151.108
MINIMUM MASK DESTINATION: 0 / 0
MINIMUM MASK SOURCE: 0 / 0
BYTES OF SNIFF PACKET: 0
SUPPRESS PUNT ACL: 0
MPLS PROPAGATE TTL FLAG: 1
PARITY: 0
FABRIC QOS ENABLE FLAG: 0
-------------------------------------------------------------------------Location = 0/6/CPU0
Index = 0
Pse direction = EGRESS
Global general 16 bytes:
-----------------------ROUTER_ID: 45.104.151.108
MINIMUM MASK DESTINATION: 0 / 0
MINIMUM MASK SOURCE: 0 / 0
BYTES OF SNIFF PACKET: 0
SUPPRESS PUNT ACL: 0
MPLS PROPAGATE TTL FLAG: 1
PARITY: 0
IPV4 PREFIX ACCNTG: 0
-------------------------------------------------------------------------Location = 0/6/CPU0
Ifname/Ifhandle = GigabitEthernet0_6_5_0
Index = 1
Pse direction = INGRESS
General 16 bytes:
----------------IFHANDLE: 0x168002
STATUS: 0
IPV4 ENABLE: 0
IPV6 ENABLE: 0
MPLS ENABLE: 0
STATS POINTER: 0x2c400
SPRAYER QUEUE: 32
IPV4 MULTICAST: 0
IPV6 MULTICAST: 0
USE TABLE ID IPV4: 0
USE TABLE ID IPV6: 0
USE TABLE ID MPLS: 0
TABLE ID: 0
QOS ENABLE: 0
QOS ID: 0
NETFLOW SAMPLING PERIOD: 0
L2 PKT DROP: 0
L2 QOS ENABLE: 0
SRC FWDING: 0
*BUNDLE IFHANDLE: 0
OL-23591-02
4-113
Chapter 4
*TUNNEL IFHANDLE: 0
*L2 ENCAP: 3
* Not programmed in hardware
.
.
.
4-114
OL-23591-02
CH A P T E R

This chapter explains the procedures for troubleshooting link bundles and load balancing on the
Cisco ASR 9000 Aggregation Services Router.
A link bundle is a group of ports that are bundled together and act as a single link. The advantages of
link bundles are:
Multiple links can span several LCs to form a single interface; thus, the failure of a single link does
not cause a loss of connectivity.
Bundled interfaces increase bandwidth availability, because traffic is forwarded over all available
members of the bundle. Therefore, traffic can move onto another link if one of the links within a
bundle fails. This allows you to add or remove bandwidth without interrupting packet flow.
This chapter contains the following sections:
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing, page 5-115
Troubleshooting Problems with Link Bundles, page 5-118
Troubleshooting Layer 2 Bundles and Load Balancing, page 5-122
Troubleshooting Layer 3 Bundles and Load Balancing, page 5-124
Troubleshooting Routing and CEF Issues Related to Bundles and

Load Balancing
Cisco Express Forwarding (CEF) uses the path information in the IP routing table to balance traffic over
link bundles. For this reason, verifying correct load balancing with CEF begins with confirming the
contents of the IP routing table and CEF database.
Verifying Routing Table Entries for Parallel Links, page 5-115
Verifying the CEF Database and Measuring Flows, page 5-117
Verifying Routing Table Entries for Parallel Links

Perform this procedure to verify the contents of the IP routing table.
SUMMARY STEPS
1.
show route destination-address
OL-23591-02
5-115
Chapter 5
Step 1
2.
configure
3.
router ospf process
4.
maximum paths number
5.
end
6.
7.
show ospf process interface brief
8.
show running-config router ospf process
Command or Action
Purpose
Displays the routes to a destination address. Use a

destination address on another host that is reachable
through the parallel links.
Example:
Step 2
RP/0/RSP0/CPU0:router# show route 10.1.2.1
Verify that number of routes in the routing table equals the

number of parallel links. If you have fewer routes than
expected, continue with this procedure.
configure
Enters global configuration mode.
Example:
Step 3
router ospf process
Enters configuration mode for the OSPF process.
Example:
RP/0/RSP0/CPU0:router# router ospf 200
Step 4
maximum paths number
Example:
Configures the maximum number of paths over which to

load balance. By default, OSPF balances up to 4 equal-cost
paths.
RP/0/RSP0/CPU0:router# maximum paths 3
Step 5
Ends the configuration process. Enter yes at the prompt to

commit the changes.
end
Example:
RP/0/RSP0/CPU0:router# end
Step 6
Displays the routes to a destination address.
Example:
Verify that number of routes in the routing table equals the

number of parallel links. If you have fewer routes than
expected, continue with this procedure.
5-116
OL-23591-02
Chapter 5

Step 7
Command or Action
Purpose
show ospf process interface brief
Shows interface information for all routes to the destination

address, which displays the cost metric. OSPF balances
loads over equal-cost routes only, so verify that the
interfaces have equal costs. To load balance over unequal
paths, use Enhanced Interior Gateway Routing Protocol or
Interior Gateway Routing Protocol (EIGRP/IGRP) as the
IGP instead.
Example:
RP/0/RSP0/CPU0:router# show ospf 200 interface
brief
Step 8
show running-config router ospf process
Example:
Displays the running configuration for the OSPF process.

This is another way to determine if the interfaces have
different costs.
RP/0/RSP0/CPU0:router# show
running-configuration router ospf process
Verifying the CEF Database and Measuring Flows

Perform this procedure to verify the contents of the CEF database.
SUMMARY STEPS
1.
show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]
2.
show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type]
[source-port source-port] [destination-port destination-port] [ingress-interface type
interface-path-id] [policy-class value] [detail | location node-id]
3.
show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail |
summary]
DETAILED STEPS
Step 1
Command or Action
Purpose
show cef ipv4 [prefix [mask]] | interface-type

interface-path-id] [detail] [location node-id]
Displays the CEF forwarding table. Verify that it contains

the same interfaces that the routing table has for this
destination.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1
detail
OL-23591-02
5-117
Chapter 5
Troubleshooting Problems with Link Bundles
Step 2
Command or Action
Purpose
show cef [ipv4 | ipv6] exact-route

source-address destination address [protocol
type] [source-port source-port]
[destination-port destination-port]
[ingress-interface type interface-path-id]
[policy-class value] [detail | location
node-id]
Displays the exact route that a specific flow would take,

including the egress interface for a specific source and
destination IP. Use this command for several flows to verify
that they are distributed equally over the parallel interfaces.
Example:
RP/0/RSP0/CPU0:router# show cef exact-route
192.168.254.1 10.1.2.1 protocol ospf
source-port 5500 destination-port 80
ingress-interface gi0/6/5/4
Step 3
show interfaces [type interface-path-id | all |

local | location node-id] [accounting | brief |
detail | summary]
Displays the traffic rates by interface. Use this command to

verify that the simulated traffic takes the expected egress
interface.
Example:
accounting rates

This section explains how to troubleshoot problems with link bundles. It contains the following
subsections:
Bundle Does Not Come Up, page 5-118
Bundle Member Not Distributing, page 5-119
Bundle Not Using MAC-Address From Backplane, page 5-119
Layer 3 Data Traffic Not Flowing, page 5-120
Ping Failed over Bundle, page 5-120
Layer 3 Packets Not Synching Over Bundle, page 5-121
Layer 2 Traffic Not Flowing, page 5-121
Bundle Statistics, page 5-122
Bundle Does Not Come Up

Step 1
Ensure that the member port is not shutdown. Ensure that the MAC burned-in address (BIA) of the
port is valid.
Step 2
If running Link Aggregation Control Protocol (LACP), ensure that LACP packets are able to send and
receive accordingly. If LACP packets are not able to send and receive accordingly, check interface
counters to identify at what stage packets are dropped.
RP/0/RSP0/CPU0:router# show lacp counters
5-118
OL-23591-02
Chapter 5

Step 3
View LACP statistics.

RP/0/RSP0/CPU0:router# show lacp
Step 4
Ensure that the other side of the link is up (bundle and members).
RP/0/RSP0/CPU0:router# show bundle
Bundle Member Not Distributing

Step 1
Ensure that the member is up.

RP/0/RSP0/CPU0:router# show interface node-id
Step 2
Ensure that the remote side is up.
Step 3
Ensure that the LACP parameters are the same on both sides.
a.
If LACP is enabled, check its status.

RP/0/RSP0/CPU0:router# show lacp bundle
b.
Ensure that bundle members have the same characteristics.

RP/0/RSP0/CPU0:router# show running interface interface-name
c.
If the bundle members have different characteristics, make them all the same.
d.
Ensure that LACP packets are transmitted and received.

RP/0/RSP0/CPU0:router# debug bundlemgr local packets port node-id
Workaround
If the bundle with LACP cannot come up, use one side of the bundle in passive mode and the other in
active mode. At least one side must be active.
Bundle Not Using MAC-Address From Backplane

Step 1
Ensure that the backplane MAC is programmed. Note that this command has to be run in the admin
mode.
RP/0/RSP0/CPU0:router(admin)# show diag chassis eeprom-info
Step 2
Display the backplane information.

RP/0/RSP0/CPU0:router# show controllers backplane bpe-trace
OL-23591-02
5-119
Chapter 5
Layer 3 Data Traffic Not Flowing

Regular Interface (No Subinterfaces)
Step 1
View the Address Resolution Protocol (ARP).

Step 2
Verify that the lag table is programmed properly in the hardware.

RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id
Step 3
View the running configuration information.

Step 4
View information about packets forwarded by CEF.

RP/0/RSP0/CPU0:router# show cef
Step 5
RP/0/RSP0/CPU0:router# show cef hardware ingress location node-id
Step 6
RP/0/RSP0/CPU0:router# show cef hardware egress location node-id
Subinterface
Step 1
Troubleshoot Layer 3 IPv4 traffic.
Step 2
Ensure that VLAN traffic coming in matches that on the incoming interface.
Ping Failed over Bundle

Step 1
View the ARP.

Step 2
View the ARP information on the particular LC or RSP.

RP/0/RSP0/CPU0:router# show arp location node-id
Step 3
RP/0/RSP0/CPU0:router# show cef hardware detail location node-id ingress
Step 4
Step 5
Use the hash calculator to determine which bundle member (interface) to test.
Step 6
Remove the interface from the bundle.
Step 7
Assign an IP address to the interface.
Step 8
Ping the interface.
Step 9
Ensure that the ARP is resolved between the router and the node being pinged.
5-120
OL-23591-02
Chapter 5

Step 10
Ensure that the MAC address in the ARP table of the other side corresponds to that on the router.
Step 11
Ensure that the MAC address of the bundle is valid.
Step 12
Ensure that the routing and hardware routing table has an entry to the next hop.
Step 13
Check the interface counters to see if ping packets are transmitted and being received on the router
member port of the bundle.
Step 14
Check the ucode counters to see where packets are dropped on the incoming or outgoing member of the
bundle.
Step 15
Make sure that the table lookup (TLU) entries are allocated and bundle adjacency information is properly
programmed.
show cef adajacency bundle-type bundle-number hardware egress detail location location-id
show cef adajacency bundle-type bundle-number hardware ingress remote detail location
location-id
Workaround
Try a different port.
Layer 3 Packets Not Synching Over Bundle

Step 1
View the interface information:

Step 2
Turn on the debug of that protocol or look at the protocol counters to see if the protocol packets are being
sent and received.
Step 3
If the protocol packets are not being sent or received, check the interface counters to see if interface
indicates packets in and out.
Step 4
If the interface level indicates that packets are coming in and out but not reaching protocol, check the
ucode counters to see if there are any drops.
Layer 2 Traffic Not Flowing

VPLS
Step 1
Verify that the AC is up.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain
Step 2
Verify that the bridge domain is up.

OL-23591-02
5-121
Chapter 5
Step 3
Look for MTU mismatches.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
VPWS
Step 1
View brief information on configured cross-connects.

RP/0/RSP0/CPU0:router# show l2vpn xconnect summary
Step 2
RP/0/RSP0/CPU0:router# show l2vpn xconnect state
Step 3
RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether bundle-id location node-id
Bundle Statistics
Layer 2 statistics are not supported in the show interface accounting command for bundle interfaces in
the current release.

This section describes how to troubleshoot Layer 2 bundles and load balancing. It includes the following
topics:
Verifying the Bundle Status, IGP Route, and CEF Database, page 5-122
Viewing the Expected Paths and Measuring the Flows, page 5-123
Verifying the Bundle Status, IGP Route, and CEF Database

CEF uses the path information in the IP routing table to balance traffic over multiple links. For this
reason, confirming proper CEF load balancing begins with confirming the contents of the IP routing
table. When troubleshooting a bundle, verify that the bundle is up and that the IGP route to the desired
destination includes the bundle interface.
SUMMARY STEPS
1.
show running-config
2.
show bundle {Bundle-Ether | Bundle-POS} interface-path-id
3.
show interface {Bundle-Ether | Bundle-POS} bundle-id
4.
show arm router-id
5.
show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id
6.
7.
show cef ipv4 prefix
5-122
OL-23591-02
Chapter 5

DETAILED STEPS
Step 1
Command or Action
Purpose
show running-config
Displays the running configuration.
Example:
Verify that the configuration related to the bundles are

correct.
Step 2
show bundle {Bundle-Ether | Bundle-POS}

interface-path-id
Displays the bundle status.

Verify that the bundle has the expected number of links. If
not, troubleshoot the bundle first.
Example:
bundle-ether 12
Step 3
show interface {Bundle-Ether | Bundle-POS}

bundle-id
Displays the interface status.

Verify that the interface is assigned to the bundle.
Example:
bundle-ether 12
Step 4
show arm router-id

Example:
RP/0/RSP0/CPU0:router# show arm router-id
Step 5
show controllers bundle {Bundle-Ether |

Bundle-POS} bundle-id location node-id
Example:
RP/0/RSP0/CPU0:router# show controllers bundle
bundle-ether 12 location 0/4/CPU0
Step 6
Example:
Step 7
Displays the routes to a destination address. Use a

destination address on another host that is reachable
through the bundle.
Verify that the route to the desalination address includes the

bundle interface. If not, make sure that the bundle interface
is included in the IGP process configuration.
show cef ipv4 prefix
Displays the CEF forwarding table. Verify that it contains

the same bundle interface that the routing table has for this
subnet prefix.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1
Viewing the Expected Paths and Measuring the Flows

Cisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member
links. This is an interactive tool prompts for the information that the load balancing algorithm uses to
allocate flows to member links.
OL-23591-02
5-123
Chapter 5
SUMMARY STEPS
1.
bundle-hash {Bundle-Ether | Bundle-Pos} interface-path-id
2.
show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail |
summary]
DETAILED STEPS
Step 1
Command or Action
Purpose
bundle-hash {Bundle-Ether | Bundle-Pos}

interface-path-id
Launches the bundle-hash utility. This is an interactive

utility that prompts for the necessary information.
Example:
RP/0/RSP0/CPU0:router# bundle-hash
bundle-ether 12
Step 2
show interfaces [type interface-path-id | all |

local | location node-id] [accounting | brief |
detail | summary]
Displays interface information, which includes the traffic

rates. Use this command for each link in the bundle to verify
that the simulated traffic takes the expected link. Use clear
counters to make it easier to view the traffic allocation.
Example:
RP/0/RSP0/CPU0:router# show interfaces tenGigE
0/5/0/1

This section provides commands for troubleshooting Layer 3 bundles and load balancing.
Step 1
RP/0/RSP0/CPU0:router# show arm router-id
Step 2
Step 3
RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id
Step 4
Step 5
RP/0/RSP0/CPU0:router# show arm router-ids
Step 6
Find out which member is carrying the traffic out.

RP/0/RSP0/CPU0:router# bundle-hash bundle-ether bundle-id
Step 7
View each member of the bundle to see which member is actually carrying the traffic out.
Step 8
Display the exact route, including the egress interface for a specific source and destination IP. Use this
command for several flows to verify that they are distributed equally over the parallel interfaces.
show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type]
[source-port source-port] [destination-port destination-port] [ingress-interface type
interface-path-id] [policy-class value] [detail | location node-id]
5-124
OL-23591-02
CH A P T E R

This section explains how to troubleshoot Layer 3 routing problems. If a ping to a remote site fails, the
cause could be in an interface or in the Layer 3 routing. The overall approach for troubleshooting a failed
ping should be to troubleshoot the interface failures and interface connectivity first, then proceed to
troubleshooting Layer 3 routing if necessary.
Note
For interface troubleshooting, perform the procedures listed in Chapter 2, Verifying and
Troubleshooting Interface Status and Chapter 3, Troubleshooting Interface Connectivity.
This chapter contains the following topics:
Traffic Loss, page 6-128
Packets Are Punted and Switched in Software, page 6-129
Traceroute Fails, page 6-130
Adding Routes Fails, page 6-131
Continuous Tracebacks, page 6-133
fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts, page 6-134
CEF Entries Out of Sync, page 6-135
fib_mgr Crashes, page 6-136
Tracebacks Appearing, page 6-136
Traffic Loss Because of Changing encap on a Subinterface, page 6-137
Traffic Loss during RSP Failover, page 6-138
Troubleshooting Virtual Router Redundancy Protocol, page 6-138
Additional Information On Routing Configuration Commands, page 6-142

SUMMARY STEPS
1.
show cef location node-id
2.
show cef ipv4 {prefix/mask} location node-id
OL-23591-02
6-125
Chapter 6
3.
show bgp summary
4.
show bgp [{ipv4 | all} {unicast | multicast | all}] dampened-paths
5.
show bgp flap-statistics [ip-address[/mask]]
6.
show arp [vrf vrf-name] [ip-address [location node-id] | hardware-address [location node-id] |
traffic [location node-id | interface-name]
7.
show interface accounting [location]
8.
show cef ipv4 [prefix/mask] hardware [ingress | egress] location node-id
9.
show cef platform trace common [all | errors | events | info] [location node-id]
10. show cef vrf [vrfname] [prefix]
DETAILED STEPS
Step 1
Command or Action
Purpose
show cef location node-id
View all IPv4 routes of Cisco Express Forwarding (CEF) on

an LC).
Note
Use this when there are only a few routes.
Example:
RP/0/RSP0/CPU0:router# show cef location
0/2/CPU0
Step 2
show cef ipv4 {prefix/mask} location node-id
View a prefixs route on an LC.
Example:
192.168.1.1/32 location 0/2/CPU0
Step 3
show bgp summary
View Border Gateway Protocol (BGP) neighbors without an

inbound and outbound policy for each active address family.
Note
Step 4
show bgp [{ipv4 | all} {unicast | multicast |

all}] dampened-paths
Use this when there are many routes.
View which routes have dampening enabled.
Example:
RP/0/RSP0/CPU0:router# show bgp dampened-paths
Step 5
show bgp flap-statistics [ip-address[/mask]]
View BGP flap statistics.

Note
Use this for routes that have had dampening

enabled.
Example:
RP/0/RSP0/CPU0:router# show bgp flap-statistics
If you do not specify arguments or keywords, all routes for

the address family are displayed.
If you enter an IP address without mask or prefix length, the
longest matching prefix is displayed.
6-126
OL-23591-02
Chapter 6

Step 6
Command or Action
Purpose
show arp [vrf vrf-name] [ip-address [location

node-id] | hardware-address [location node-id]
| traffic] [location node-id | interface-name]
View Address Resolution Protocol (ARP) records.

For bundle and VLAN-on-Bundle interfaces, enter location
node-id. This tells the system which cache entries to show.
Note
Example:
If vrf is entered, it must appear immediately after

show arp, and you must enter a vrf-name.
Step 7
show interface accounting [location]
View packet accounting on an interface per protocol.
Example:
accounting location 0/4/CPU0
Step 8
show cef ipv4 {prefix/mask} hardware {ingress |

egress} location node-id
View IPv4 prefix/route in the hardware of an LC.

This information helps determine if the destination IP or
prefix action is COMPLETE, PUNT or DROP.
Example:
38.1.1.2/32 hardware egress location 0/4/CPU0
Step 9
show cef platform trace common [all | errors |

events | info] [location node-id]
View common Dynamic Link Library (DLL) code traces.
Example:
RP/0/RSP0/CPU0:router# show cef platform trace
common all errors location 0/4/CPU0
Step 10
show cef vrf [vrfname] [prefix]
Verify that the L3 MTU value, encapsulation string value,

byte count, and packet count are as expected. (See the
example below.)
Example:
RP/0/RSP0/CPU0:router# show cef vrf 0xx
RP/0/RSP0/CPU0:router# show cef vrf vrf1
192.168.1.2 hardware egress location 0/1/CPU0
Example
RP/0/RSP0/CPU0:router# show cef vrf vrf1 192.168.1.2 hardware egress location 0/1/CPU0
192.168.1.2/32, version 0, internal 0x40800001 (ptr 0xaac1c468) [1], 0x0 (0xaab8c7b0), 0x0
(0x0)
Updated Oct 1 21:29:37.684
local adjacency 130.130.1.2
Prefix Len 32, traffic index 0, Adjacency-prefix, precedence routine (0)
via 130.130.1.2, GigabitEthernet0/1/0/0, 3 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0
next hop 130.130.1.2
local adjacency
TBM Node Data:

Node (0x00000100):0 0x8d40047d 0x00000000 0xffffffff 0xffffffff
OL-23591-02
6-127
Chapter 6
Traffic Loss
Node
Node
Node
Node
Node
Node
Node
(0x8d400470):1
(0x8cb92900):2
(0x8d815110):3
(0x8d8151e0):4
(0x8d814ea0):5
(0x8d8cd410):6
(0x8d8dc0c0):7
0x8cb928bd
0x8d81511d
0x8d8151ed
0x8d814ead
0x8d8cd40d
0x8d8dc0a5
0x00002000
0x00000005
0x89885f70
0x00000000
0x00000000
0x00000004
0x8d814d70
0x00000000
0x00008083
0x00000000
0x00800000
0x40000000
0x80000000
0x0e000000
0x8d8dc0d0
0x80038003
0x00800000
0x00000000
0x00000000
0x00000001
0x00000000
0x8d8274d0
Hardware Leaf Data (0x8d8dc0c0):0x00002000 0x00000000 0x8d8dc0d0 0x8d8274d0

IP Leaf Data:
as:0
prefix_len:32
for_us:0x0
dft_route:0x0
real_intf:0x0
free1: 0x0
hw_use_only: 0x0
lspa_ptr: 0x0
oce_chain_p: 0x8d8274d0
extre_fib_data_ptr: 0x8d8dc0d0
Hardware Extended Leaf Data:
fib_leaf_extension_length: 0
interface_receive: 0x0
traffic_index_valid: 0x0
qos_prec_valid: 0x0
qos_group_valid: 0x0
valid_source: 0x0
traffic_index: 0x0
nat_addr: 0x0
reserved: 0x0
qos_precedence: 0x0
qos_group: 0x0
peer_as_number: 0
path_list_ptr: 0x0
connected_intf_id: 0x0
ipsub_session_uidb: 0xffffffff
Path_list:
urpf loose flag: 0x0
List of interfaces:
OCE Loadbalance Data for ptr 0x8d8274d0:
num_entries:1
level:0x1
pad_1:0x0
l3_lbe_ptr:0x8d8274e0
LBE Array for 0x8d8274e0
Entry 0: oce_chain_p 0x8d8274c0
OCE Adj Data for 0x8d8274c0:
adj:0x50717cc0
base: 6 (CPP HW IPv4 Adjacency Object)
l3_mtu: 1500
adj_flags: 0000
fixup_flags: 0000
output_uidb: 0x1fa0
adj2:0x50588f00
encap: 00008282010200211bfcc2400800
nh_addr: 0x00 0x00 0x00 0x00
oce_chain_p: 0x00000000
counters: 0x893d46f0
byte count: 4447644
packet count: 71732
encap_length: 14
Traffic Loss
This section provides steps for troubleshooting traffic loss.
Step 1
Check for packet loss by examining transmitted packets on the local router and the receive packets on
the destination router.
6-128
OL-23591-02
Chapter 6

RP/0/RSP0/CPU0:router# show interface accounting

Wed Dec 8 13:12:47.627 PST
No accounting statistics available for Bundle-Ether16.10
No accounting statistics available for GigabitEthernet0/1/0/7.210
Protocol
Pkts In
Chars In
Pkts Out
IPV4_UNICAST
2225064
168207595
67521
ARP
29433
1765984
5855
Step 2
Chars Out
3479370
245910
View the hardware data structures involved with the prefix (destination-ip)/(mask). Verify that the RIB
table is consistent with the information that the IGP learned from neighbors. that the CEF tables are
consistent with the RIB. For routes that are learned (not directly connected), the CEF table in the RSP
should be the same as the CEF table in the LC.
show cef {ipv4} [destination ip | destination-ip/mask] hardware egress detail location
node-id
Step 3
View the ARP information on the particular LC or RSP.

Step 4
View any PI code ltrace errors recorded.

RP/0/RSP0/CPU0:router# debug controllers pse qfp forward drop {0 | 1} location 0/2/CPU0

Step 1
Step 2
Verify that the hardware chains for the destination IP address are pointing to either of the following:
COMPLETE adjacencyValid outgoing path exists.
PUNT adjacencyHardware does not know how to send the packet out, it just punts (diverts) the
packet to be switched in software. If the transmit adjacency is PUNT, this could be because ARP is
not resolved yet.
To show if an ARP entry exists for the destination IP, use the show arp location command:
RP/0/RSP0/CPU0:router# show arp location
a.
node-id
If an ARP entry does not exist or is incomplete, add a static ARP entry. Ensure that the Tx adjacency
points to COMPLETE.
RP/0/RSP0/CPU0:router# show cef {ipv4} 192.168.1.1/32 hardware egress detail location
0/4/CPU0
b.
If so, then it means the issue is that of ARP entry not getting updated. Troubleshooting should now
focus on why the ARP entry is not getting added (this includes steps like show arp, show arp idb,
show adjacency gig node-id detail location node-id, show arp trace, and so forth).
c.
If the Tx adjacency still points to PUNT, it means ARP is adding the entry in its database, but
fib_mgr fails to mark the adjacency as COMPLETE.
d.
This could be a fib_mgr, ARP, or AIB problem. Delete and reconfigure the static ARP entry with
AIB and CEF debugs on. The debugs show if ARP is adding the entry inside the AIB and if the AIB
is informing fib_mgr.
OL-23591-02
6-129
Chapter 6
Traceroute Fails
Step 3
Packets could be dropped in the fabric. To verify this, view the fabric counters.
Step 1
Use the shut command (followed by commit) and the no shut command (followed by commit) on the
outgoing interface.
Step 2
Add a static ARP entry for the destination IP.
Workaround
Traceroute Fails
Use traceroute to verify the connectivity to a destination. When traceroute fails to a destination, use
the following commands:
show cef {ipv4} {destination_ip}/(mask} hardware egress detail location node-idView

the hardware data structures involved with the prefix.
show interface location
{outgoing_interface} accountingView input and output packets
from the outgoing interface.

Step 1
Check if the destination IP address has the proper transmit adjacency. See the Tx Adjacency state (it
should be COMPLETE).
RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location
Step 2
node-id
If the transmit adjacency is not complete, there is an issue. If it is pointing to PUNT, that means
probably the mac-address corresponding to the destination IP has not been learned. Try adding a static
arp entry and see if transmit adjacency moves to COMPLETE. If the destination IP is advertised by a
routing protocol such as OSPF, then the transmit adjacency should never show as PUNT.
If the transmit adjacency is shown as DROP, that means there is a static route to the destination IP
explicitly pointing the route to a DROP.
If the transmit adjacency is shown as COMPLETE, it means there is no problem in the hardware chains
that are set up. You should see the counters.
Step 3
See if the output packets are equal to the traceroute packets sent.
RP/0/RSP0/CPU0:router# show interface location outgoing_interface accounting
Workaround
Step 1
Use the shut command (followed by commit) and the no shut command (followed by commit) on the
outgoing interface.
Step 2
Add a static ARP entry for the destination IP.
6-130
OL-23591-02
Chapter 6

Adding Routes Fails
Adding Routes Fails

Perform the steps in this section to troubleshoot failures in adding routes. During Out Of Resource
(OOR), the router does not accept additional routes until existing routes are deleted.
Note
Step 1
The sample commands in this section are applicable to Ethernet LCs, not SIP-700 LCs.
Determine if any resources are experiencing problems. View the state of various data structures. Ideally
the state should be GREEN. If it is either YELLOW or RED, it indicates an OOR condition.
RP/0/RSP0/CPU0:router# show cef resource location node-id
RP/0/RSP0/CPU0:ASR-9010#show cef resource location 0/0/CPU0
Thu Oct 28 09:07:52.405 DST
CEF resource availability summary state: GREEN
CEF will work normally
ipv4 shared memory resource: GREEN
ipv6 shared memory resource: GREEN
mpls shared memory resource: GREEN
common shared memory resource: GREEN
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN
Step 2
Determine which hardware table is OOR. Compare max entries and used entries too see which of the
data structures is using the entries close to the max limit.
RP/0/RSP0/CPU0:router# show cef platform resource location node-id
RP/0/RSP0/CPU0:router# show cef platform resource loc 0/0/CPU0
Thu Oct 28 15:41:47.725 PST
Node: 0/0/CPU0
---------------------------------------------------------------IPV4_LEAF_P usage is same on all NPs
OL-23591-02
6-131
Chapter 6
Adding Routes Fails
NP: 0 struct 23: IPV4_LEAF_P

(maps to ucode stru = 54 in TopSearch1)
Used Entries: 298 Max Entries: 524288
------------------------------------------------------------IPV6_LEAF_P usage is same on all NPs
NP: 0 struct 24: IPV6_LEAF_P
------------------------------------------------------------R_LDI usage is same on all NPs
NP: 0 struct 6: R_LDI
------------------------------------------------------------NR_LDI usage is same on all NPs
NP: 0 struct 7: NR_LDI
------------------------------------------------------------RPF_STRICT usage is same on all NPs
NP: 0 struct 9: RPF_STRICT
------------------------------------------------------------NP: 0 struct 12: TX_ADJ
------------------------------------------------------------NP: 1 struct 12: TX_ADJ
------------------------------------------------------------NP: 2 struct 12: TX_ADJ
------------------------------------------------------------NP: 3 struct 12: TX_ADJ
------------------------------------------------------------RX_ADJ usage is same on all NPs
NP: 0 struct 13: RX_ADJ
------------------------------------------------------------TE_NH_ADJ usage is same on all NPs
NP: 0 struct 14: TE_NH_ADJ
------------------------------------------------------------L2VPN_LDI usage is same on all NPs
NP: 0 struct 16: L2VPN_LDI
------------------------------------------------------------LABEL_UFIB usage is same on all NPs
NP: 0 struct 28: LABEL_UFIB
(maps to ucode stru = 1 in TopParse)
-------------------------------------------------------------
Step 3
After determining which data structure is OOR, verify if it is expected or unexpected. Usually, for each
LEAF (either IPv4), it requires four entries of NR_LDI structure. So if you find the NR_LDI structure
going OOR, see if you have appropriate number of IP LEAFs to take this NR_LDI number to such a
limit.
Step 4
If show cef resource location node-id shows the state in GREEN, it means that the problem is not
caused by an OOR condition. The reason for not being able to add further routes is some thing else.
Enable the following debugs to observe what is happening:
RP/0/RSP0/CPU0:router# debug cef errors location node-id
RP/0/RSP0/CPU0:router# debug cef {ipv4} error location node-id
If you observe any tracebacks, decode the tracebacks by using SBT tool.
6-132
OL-23591-02
Chapter 6

Step 5
View platform ltrace errors for protocols IPv4show cef platform trace {ipv4} error reverse
node-id.
location
Step 6
View platform ltrace common errors for all protocolsshow cef platform trace common error
node-id.
reverse location
Workaround
If it is an OOR condition and expected, delete some existing routes.
When tracebacks appear continuously on the console (typically every 15 seconds), programming of the
entry inside the hardware is not successful. This causes the software to try repeatedly after every 15
seconds. It is possible that the layer just above the hardware or the hardware itself is not up and running.
Step 1
View all platform ltrace common messages. Verify that both CPPs are in ACTIVE_SOLO state.
Note
Step 1 is applicable to SIP-700 line cards only.
show controllers pse qfp system state location node-id
Example
show controllers pse qfp system state location 0/1/CPU0
CPP HA client processes registered (5 of 5)
cpp_sp : Initialized
cpp_cdm : Initialized
cpp_driver1 : Initialized
cpp_driver0 : Initialized
cpp_cp : Initialized
-----------------------------------------CPP 0: dir=INGRESS Role: curr=ACTIVE_SOLO next=ACTIVE_SOLO <<< CPP 0 in ACTIVE_SOLO state
Client State: ENABLE
Image: /pkg/ucode/cpp/cpp-thor-ucode
Image desc: Ucode dir: /nobackup/eruan/thor2/cpp/dp/obj/thor/thor-ingress-hw
Image: thor_ingress
HW: CPP10
Built by: eruan
Host: sjc-lds-447
Time: Tue Sep 28 15:04:57 2010
Component: cpp/dp asr41-9k-cgn/2
Load Cnt: 1 Last load: Oct 01, 2010 21:27:36.488431
Active Threads: 0-159
Stuck Threads: <NONE>
Fault Manager Flags:
ignore_fault:
FALSE
ignore_stuck_thread:
FALSE
crashdump_in_progress: FALSE
------------------------------------------
OL-23591-02
6-133
Chapter 6
fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts
CPP 1: dir=EGRESS Role: curr=ACTIVE_SOLO next=ACTIVE_SOLO <<< CPP 1 in ACTIVE_SOLO state

Client State: ENABLE
Image: /pkg/ucode/cpp/cpp-thor-ucode
Image desc: Ucode dir: /nobackup/eruan/thor2/cpp/dp/obj/thor/thor-egress-hw
Image: thor_egress
HW: CPP10
Built by: eruan
Host: sjc-lds-447
Time: Tue Sep 28 14:55:59 2010
Component: cpp/dp asr41-9k-cgn/2
Load Cnt: 1 Last load: Oct 01, 2010 21:27:36.500431
Active Threads: 0-159
Stuck Threads: <NONE>
Fault Manager Flags:
ignore_fault:
FALSE
ignore_stuck_thread:
FALSE
crashdump_in_progress: FALSE
Step 2
View all platform ltrace protocol messages for IPv4 or IPv6.

show cef platform trace {ipv4 | ipv6} all reverse location
Step 3
node-id
Check that the NP provisioning layer (or PRM) is up. PRM is a layer just above hardware. If PRM is
down, no entry is programmed in hardware, indicating that NP may have had a problem during
initialization.
show controllers NP summary
Step 4
View the NP driver logs to find out if there have been NP initialization errors. If there are NP
initialization errors, it is likely an NP problem.
show controllers NP drvlog location node-id
Step 5
Use the SBT tool to decode the tracebacks. From root of the workspace, use ./util/bin/sbt -p
(process_name) -f (log_file).
Step 1
Restart prm_server process.
Step 2
Reboot LC.
Workaround
fib_mgr Does Not Come Up During LC Reload or After Multiple

Process Restarts
Fib_mgr depends on underlying hardware. If the underlying process or hardware does not come up, it is
likely that fib_mgr will not come up.
show controllers NP summary location node-idCheck
that the NP provisioning layer (or
PRM) is up.
show controllers NP drvlog location node-idView
the NP driver logs.
6-134
OL-23591-02
Chapter 6

show cef platform trace common all reverse location
node-idView platform ltrace common
messages.
show cef platform trace common event reverse location
node-idView platform ltrace
common events.
show cef platform trace {ipv4 | ipv6 | mpls} error reverse location
node-idView
platform ltrace error messages recorded for protocols IPv4, IPv6, or MPLS.
show cef trace all reverse location
node-idView all CEF ltrace messages.
Step 1
Use the show controllers NP summary location and show controllers NP drvlog location commands
to determine if either the PRM or the underlying NP has a problem. If so, the fib_mgr will not come up.
Troubleshoot at the PRM layer or NP layer.
Step 2
If both CPPs are in ACTIVE_SOLO state, the problem is likely a software bug. In this case, collect the
core file and decode the tracebacks using the SBT tool. From root of the workspace, use ./util/bin/sbt -p
(process_name) -f (log_file).
Step 1
Restart the prm_server process.
Step 2
Reboot the LC.
Workaround

The cef entry on RSP may be pointing to the management interface and as a result the traffic originating
from the router may go out on the management interface instead of through the LC interface.
show controllers np drvlog location
node-idShows the PRM view of the Direct Table on the
NP.
show tech-support cefCollects
relevant platform independent traces.
show cef trace events reverse location
node-idView platform independent cef ltrace events.
show cef trace errors reverse location
node-idView platform independent cef ltrace errors.
node-idView CEF platform
common event traces.
show cef platform trace common error reverse location
common error traces.

Step 1
Look for a default route 0.0.0.0/0 configured to go out through the management interface.
Step 2
Look for a static ARP configured for the prefix in question. It is possible that ARP is installing two
entries through both the management interface and also through the LC interface (because the prefix is
reachable by both routes).
OL-23591-02
6-135
Chapter 6
fib_mgr Crashes
Step 3
If the above is not the case, use the show arp command to see if an ARP entry is advertising through the
management interface. If this is the case, clear the ARP and verify the cef entries again.
Workaround
Use the shut command (followed by commit) and the no shut command (followed by commit) on
the management interface.
Use the clear arp-cache command.
Reboot the LC.
fib_mgr Crashes
show cef platform trace common all reverse location
node-idView CEF platform common
traces.
common event traces.
show cef platform trace common error reverse location
common error traces.
show cef platform trace {ipv4 | ipv6 | mpls} error reverse location
node-idView CEF
platform protocol traces for IPv4 or MPLS.

Step 1
If the trigger is a prm restart or crash, this is expected.
Step 2
If the underlying process (prm_server) is down or crashed, it is likely fib_mgr will not come up.
Step 3
Save the core file.
Step 4
Use the SBT to decode the tracebacks. From root of the workspace, use ./util/bin/sbt -p (process_name)
-f (log_file).
Step 5
Save the console logs.
Workaround
Restart fib_mgr or reboot the LC.
Tracebacks Appearing
In this scenario, a few error tracebacks appear on the console because of some trigger (such as interface
shut/no shut, or any other similar trigger).
show cef trace event location
show cef trace errors location
node-idView CEF traces for major events.

node-idView CEF traces for major errors.
show cef platform trace common errors location
node-idView CEF platform traces for
common errors across all protocols.
6-136
OL-23591-02
Chapter 6

show cef platform trace {ipv4 | ipv6 | mpls} errors location
traces for errors in protocols IPv4 or MPLS.
show logging
Step 1
Decode the tracebacks using the SBT tool. From root of the workspace, use ./util/bin/sbt -p
(process_name_ -f (log_file).
Step 2
Save core files.
Workaround
If the tracebacks are impacting service, do the following:
Step 1
Restart the fib_mgr process and check whether that reduces the tracebacks.
Step 2
If the tracebacks continue, reboot the LC.

When traffic is being forwarded through a Layer 3 subinterface and if the encapsulation is changed on
that subinterface, it is sometimes observed that the traffic does not resume until after 15 seconds.
show cef trace event reverse location
node-idView CEF trace messages for major events.
show cef trace error reverse location
node-idView CEF trace messages for major errors.
show cef platform trace common error location
node-idView CEF platform traces for
common errors across all protocols.
show cef platform trace {ipv4 | ipv6 | mpls} event location
traces for major events in protocols IPv4, IPv6, or MPLS.
show cef platform trace {ipv4 | ipv6 | mpls} error location
traces for major errors in protocols IPv4, IPv6, or MPLS.
show arp trace location node-idShows arp related traces.
show arp-gmp trace location node-idShows arp-gmp related traces.
show arp location
node-idView ARP-related information.
This type of traffic loss could happen typically when there is a static arp entry for the prefix which is
experiencing traffic loss. For example, consider the following configuration:
interface GigabitEthernet0/4/0/39.2
ipv4 address 209.165.201.1 255.0.0.0
dot1q vlan 300
When encapsulation changes from dot1q vlan 300 to dot1q vlan 200 on the subinterface, fib_mgr
deletes all prefixes corresponding to this interface and creates them again. It takes 15 seconds to add all
prefixes; traffic does not get forwarded for that time. For example, there is an interface with address
192.0.2.0/8. There is a static ARP entry for 192.0.2.5.
RP/0/RSP0/CPU0:router# show run | inc arp
OL-23591-02
6-137
Chapter 6
The delay is less likely to happen with regular adjacency (not the static ARP).
When VLAN color changes, the following occurs:
Adjacency is deleted, and the adjacency route 192.0.2.5 is deleted.
Connected route is deleted.
Adjacency is added before the connected route is added. The FIB treats adding an adjacency without
a covering connected route as an error, so the route 192.0.2.5 is placed in retry.
Connected route 192.0.2.0/8 is added.
Because the FIB retry timer is 15 seconds, the adjacency route 192.0.2.5 is added after 15 seconds.
Workaround
Remove the static ARP entry.

Sometimes RSP switchover (keyword failover in CLI) causes traffic loss. This may mean the IGP over
which the prefixes are learned is going down. The following assumes OSPF as the IGP.
show process failoverShows process details during failover.
debug ospf haEnables OSPF HA related debugs.
debug ospf instance nsfView before failover and collect the debug log.
show process failoverShows process details after failover.
show redundancyProvides status of the standby node after failover.
Check if the next hop router had a failover.
If so, the OSPF will go down.
If not, verify that nsf cisco is configured under OSPF.

If nsf cisco is configured, see if the next hop is reachable during failover.
If the next hop is not reachable, a link may be going down or having negotiation problems.
If the next hop is reachable, the problem is likely a software bug.
Workaround
Reload the router.

Virtual Router Redundancy Protocol (VRRP) enables a group of routers to form a single virtual router.
This section contains the following subsections:
VRRP Fails to Reach Active State, page 6-140
Tracked Interface Failing, Router State Not Changed, page 6-140
VRRP State Flapping, page 6-140
6-138
OL-23591-02
Chapter 6

More Than One VRRP Router Active, page 6-141
VRRP Active Router Not Forwarding Traffic, page 6-141
Traffic Loss or Unexpected VRRP State After Interface shut/no shut, page 6-142

SUMMARY STEPS
1.
show vrrp [interface type interface-id] [brief]
2.
show vrrp interface type interface-id detail
3.
show vrrp [interface {type interface-id}] statistics [all]
4.
show controllers type interface-id
5.
debug vrrp [ all | edm | events | packets ]
DETAILED STEPS
Step 1
Command or Action
Purpose
show vrrp [interface type interface-id] [brief]
View all VRRP groups status.
Example:
RP/0/0/CPU0:# show vrrp brief
Step 2
show vrrp interface type interface-id detail
View detailed information of VRRP groups.
Example:
RP/0/0/CPU0:# show vrrp gigabitEthernet 0/1/0/1
detail
Step 3
show vrrp [interface {type interface-id}]

statistics [all]
View VRRP statistics.
Example:
RP/0/0/CPU0:# show vrrp statistics
Step 4
show controllers type interface-id
View the VRRP group MAC addresses as part of unicast

filter list.
Example:
RP/0/0/CPU0:# show controllers gigabitEthernet
0/3/0/9
Step 5
debug vrrp [ all | edm | events | packets |

packets ]
Debug the VRRP.
Example:
RP/0/0/CPU0:# debug vrrp packets tengige
0/3/0/9
OL-23591-02
6-139
Chapter 6
VRRP Fails to Reach Active State

Run the following command on both routers:
RP/0/RSP0/CPU0:router# show vrrp detail
Misconfiguration
Step 1
Ensure that the interface with VRRP configured is up.
Step 2
Ensure that an IP address is configured, on the same subnet as the interface, and delay is configured.
Higher Priority Router Already Active

Examine the output of the show vrrp command:
If the Master address for VRRP shows an IP address instead of local, the router with that IP address
is Active.
If preemption is enabled, but the other router has higher priority, then it will remain in the Active
state.
Operational priority may not match the configured priority. If interfaces are down, this negatively
impacts operational priority.
Preemption is Disabled and Another Router Already Active

Examine the output of the show vrrp command. If preemption is disabled, and the router has higher
priority, it will not take over unless preemption is enabled.
Tracked Interface Failing, Router State Not Changed

On both routers:
If preemption is enabled and this router has higher operational priority than the other router, this router
remains in the Active state. Configured priority or the decrement for tracked interfaces needs to be
configured appropriately such that the state transition takes place. If the IP address is the same as the
interface IP address, the router does not change to the Standby state.
VRRP State Flapping

On both routers:
Step 1
6-140
OL-23591-02
Chapter 6

Step 2
RP/0/RSP0/CPU0:router# debug vrrp packets
Check timestamps to determine whether there is a delay in sending or receiving packets. Check the CPU
usage to see if some process is hogging the system resources.
Step 3
RP/0/RSP0/CPU0:router# show spp node-counters location interface-running-vrrp
More Than One VRRP Router Active

Step 1
Verify that the same IP is configured on both ends.

Step 2
Check timestamps to determine whether there is a delay in sending or receiving packets.

Check the CPU usage to see if a process is overusing resources.
Step 3
Enter the debug command for VRRP packets on the peer.

RP/0/RSP0/CPU0:router# debug vrrp packets
Check for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:16:39.217 : vrrp[357]: Gi0/5/0/0: VR1:
Pkt: ADVER: IN: pri 100 src 192.0.2.11. This means advertisement packets are being received by
VRRP. If these are absent, no packets are being received and VRRP becomes active.
Look for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:18:47.876 : vrrp[357]: Gi0/5/0/0: VR1: Pkt:
ADVER: Out: pri 100 src 192.0.2.11. This means the peer is sending VRRP packets.
Step 4
Check the output of the show spp node-counters location interface-running-vrrp on both routers, and
look for packet drops.
RP/0/RSP0/CPU0:router# show spp node-counters location interface-running-vrrp
VRRP Active Router Not Forwarding Traffic

On both routers:
Step 1
Find the virtual MAC address for the group.

Step 2
RP/0/RSP0/CPU0:router# show ether-ctrl trace
Step 3
Ensure that the virtual MAC address is in the unicast address filter list and verify the router is receiving
traffic.
RP/0/RSP0/CPU0:router# show controllers type interface-running-vrrp
OL-23591-02
6-141
Chapter 6
Traffic Loss or Unexpected VRRP State After Interface shut/no shut

In case of shut/no shut on a VRRP-enabled interface, the following has been observed:
If preemption is enabled, recovery times are higher than switchover times. This means higher traffic
loss has occurred when the interface is no shut.
If preemption is disabled, some VRRP groups are preempted after no shut of an interface.
If you observe either of the above conditions after an interface no shut, perform the following steps on
both routers.
Step 1
Step 2
RP/0/RSP0/CPU0:router# show ether-ctrl trace
Step 3
RP/0/RSP0/CPU0:router# show controllers type interface-running-n
Step 4
RP/0/RSP0/CPU0:router# debug vrrp packets interfaceFor the interface on which no shut is being
performed.
Step 5
Enter the no shut command.
Step 6
Observe the console logs and look for lines similar to:
RP/0/RSP0/CPU0:Sep 8 14:16:39.217 : vrrp[357]: Gi0/5/0/0: VR1: Pkt: ADVER: IN: pri 100
src 192.0.2.11.
Note the time lag between the no shut and the first such message seen. For that amount of time, there is
traffic loss between two routers.
Step 7
If there is no traffic flowing between two routers after a no shut event, check the STP configuration on
the Cisco ASR 9000 Series Router. Lowering the fwd delay timer might help in reducing the traffic loss.
Step 8
For preemption disabled case, if the groups still preempt after reducing the fwd delay timer, repeat Step 1
through Step 4, and find the time period of traffic loss between the two routers. The preemption can be
avoided by configuring the minimum delay to be higher than the time period of traffic loss. Minimum
delay can be configured as follows:
RP/0/RSP0/CPU0:router(config)# router vrrp interface gigabitEthernet 0/2/0/10 vrrp delay
minimum 10 reload 5

Use the following guides if you need to review routing configuration commands
Cisco ASR 9000 Series Aggregation Services Router Routing Configuration Guide, Release 4.0
Cisco ASR 9000 Series Aggregation Services Router Routing Command Reference, Release 4.0
6-142
OL-23591-02
CH A P T E R
Troubleshooting Router Switch Fabric and Data

Path
This chapter describes techniques to troubleshoot router switch fabric and data path. It includes the
following sections:
Understanding Switch Fabric Architecture, page 7-143
Getting Started with Fabric Troubleshooting, page 7-145
Troubleshooting Packet Drops, page 7-146
Troubleshooting RSP and LC Crashes, page 7-165
Troubleshooting Complete Loss of Traffic, page 7-168
Gathering Fabric Information Before Calling TAC, page 7-172

Figure 7-1 provides an overview of the switch fabric architecture.
OL-23591-02
7-143
Chapter 7
Figure 7-1
Switch Fabric Architecture
Active Fabric
Switch
Fabric 0
Switch
Fabric 1
Arbiter
Fabric I/O
(LC)
RSP0
Active RP
Active Fabric
Fabric I/O
(LC)
Switch
Fabric 0
Switch
Fabric 1
RSP1
Standby RP
281342
Arbiter
23G fabric channels

Fabric requests
As shown in Figure 7-1, there are two fabric interface ASIC on each RSP. Each fabric interface ASIC
provides 40 GB of throughput. If one RSP is lost, the shelf can still operate at full capacity without loss
of bandwidth.
Each line card (LC) has four 23 GB fabric channels on which to send traffic to the fabric ASICs. The
switch fabric is in an active/active relationship. All four fabric ASICs are active, even though the RSP
cards are in an active/standby relationship. The system performs load balancing on unicast traffic across
these four channels.
The arbiters are in an active/standby relationship (the arbiter on the active RSP card is the active arbiter).
Both the active and standby arbiters receive requests for switch fabric access from the LCs. If there is a
switchover of the active RSP, the standby RSP arbiter has a current copy of switch fabric requests, which
helps to speed up the switchover.
7-144
OL-23591-02
Chapter 7

Figure 7-2 shows the data path from ingress to egress. (Several types of LCs are shown in this example.)
Figure 7-2
Data Path
RSP0
RSP1
Fabric I/O
Fabric
Fabric
System
Timing
CPU
Fabric
arbiter
System
Timing
GE
Switch
Fabric I/O
CPU0
Fabric
Fabric
Fabric
arbiter
GE
Switch
Backplane
GE
PHY
Fabric I/O
8x10GE
Fixed LC
GE
PHY
Fabric I/O
CPU
FPGA
CPU
FPGA
FPGA
NPU
NPU
NPU
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
NPU
10 GE XFP
NPU
10 GE XFP
10x
SFP
NPU
10 GE XFP
10x
SFP
NPU
10 GE XFP
10x
SFP
FPGA
10 GE XFP
10x
SFP
NPU
10 GE XFP
NPU
10 GE XFP
NPU
FPGA
10 GE XFP
NPU
GE
PHY
Fabric I/O
CPU
FPGA
NPU
4x10GE
Fixed LC
280888
40x1GE
Fixed LC
As shown in the drawing, the path travelled by each data packet is:
Incoming interface on LC--> NP mapped to incoming interface on LC --> Bridge3 on LC --> FIA
on LC --> Crossbar switch on RSP --> FIA on LC ---> Bridge3 on LC ---> NP mapped to outgoing
interface ---> Outgoing Interface
Note
In this document, the network processor ASICs are referred to either as network processors (NPs) or
network processor units (NPUs).

To begin troubleshooting problems with the fabric, perform the following steps.
Step 1
Look for active platform fault manager (PFM) alarms on the LCs and RSPs.
Step 2
Check that you have the appropriate version of the bridge field-programmable gate arrays (FPGAs) in
your RSP card.
Step 3
Check that you have the correct software version, board, and FPGA and ASIC versions.
RP/0/RSP0/CPU0:router# show version
OL-23591-02
7-145
Chapter 7
Troubleshooting Packet Drops
RP/0/RSP0/CPU0:router# show inventory raw

RP/0/RSP0/CPU0:router# show hw-module fpd location all
Step 4
Check if there are any errors detected by the system diagnostics.

RP/0/RSP0/CPU0:router# show diag
Step 5
Check that you have the appropriate version of the NPs in your RSP cards.
RP/0/RSP0/CPU0:router# show controllers np summary all
Node: 0/1/CPU0:
---------------------------------------------------------------[total 4 NP] Driver - Version 10.26a Build 9 ( Dec 13 2008, 20:47:03 )
NP 0 : Hardware rev v2 A1
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
Node: 0/2/CPU0: <-- [ LC built with A0 NPU that has known issue ]
---------------------------------------------------------------[total 4 NP] Driver - Version 10.26a Build 9 ( Dec 13 2008, 20:47:03 )

This section explains how to track packets through the system from ingress to egress, and how to
troubleshoot packet drops. It includes the following sections:
Displaying Traffic Status in Line Cards and RSP Cards, page 7-147
Locating Packet Drops by Examining Counters, page 7-148
Locating Drops of Punted Packets, page 7-155
Packet Drop from LC to LC, page 7-157
Packet Drop Between RSP and LC, page 7-158
Packet Drop After Certain Actions, page 7-160
Packet Drop After a Redundancy Switchover, page 7-161
Packet Drop with Unknown Reason, page 7-163
7-146
OL-23591-02
Chapter 7

Displaying Traffic Status in Line Cards and RSP Cards

Figure 7-3 shows the traffic path on the LC and the corresponding CLI commands you use to display the
status at each point in the path.
Figure 7-3
LC Traffic Path and Corresponding CLI Commands
show controllers fabric fia bridge ddr-status location <...>

show controllers fabric fia bridge flow-control location <...>
show controllers fabric fia bridge stats location <...>
show controllers fabric fia bridge sync-status location <...>
show controllers np counters all
show interfaces gigabitEthernet 0/6/0/4
PHY
NPU-0
PHY
NPU-1
PHY
NPU-2
PHY
NPU-3
Bridge
FPGA 0
Fabric I/O
To RSP
fabric I/O
Bridge
FPGA 1
Line Card
show controllers fabric fia link-status location <...>

show controllers fabric fia stats location <...>
show controllers fabric fia drops <ingress | egress> location <...>
show controllers fabric fia errors <ingress | egress> location <...>
281343
show controllers np fabric-counters all np3 location 0/6/CPU
OL-23591-02
7-147
Chapter 7
Figure 7-4 shows the traffic path on the RSP and the corresponding CLI commands you use to display
information at each point in the path.
Figure 7-4
RSP Traffic Path and Corresponding CLI Commands
show controllers fabric arbiter serdes location <...>

show controllers fabric arbiter configstatus location <...> <0..4> <0>
show controllers fabric fia bridge ddr-status location <...>

show controllers fabric fia bridge stats location <...>
Fabric
arbiter
Fabric
XBAR 0
CPU
Fabric I/O
FPGA
Fabric
XBAR 1
RSP-0
show controllers fabric fia link-status location <...>

show controllers fabric fia stats location <...>
show controllers fabric fia drops <ingress | egress> location <...>
show controllers fabric fia errors <ingress | egress> location <...>
Locating Packet Drops by Examining Counters

To locate the source of packet drops, perform the following procedure.
SUMMARY STEPS
1.
Clear the interface counters
2.
Clear the NP counters
3.
Clear the fabric counters
4.
Start the traffic pattern that caused the packet drop
5.
Display the NP-to-interface mapping.
6.
Check the counters at the input interface
7.
Check the NP counters
8.
Check the NP Bridge3 counters
9.
Check the bridge counters
7-148
OL-23591-02
281344
show controllers fabric crossbar serdes instance <0 or 1> location <...>
show controllers fabric crossbar statistics instance <0 or 1> location <...>
show controllers fabric Itrace crossbar all location <...>
Chapter 7

10. Check the fabric interface ASIC (FIA) counters

11. Check the crossbar counters
Note
For the procedure to troubleshoot drops of punted packets, see the Locating Drops of Punted Packets,
page 7-155.
DETAILED STEPS
Step 1
Clear the interface counters.

RP/0/RSP0/CPU0:router# clear counters all
Clear "show interface" counters on all interfaces [confirm]
Step 2
Clear the NP counters.

RP/0/RSP0/CPU0:router# clear controller np counters all
Step 3
Clear fabric counters.

a.
Clear FIA and bridge counters on the LC and RSP.
RP/0/RSP0/CPU0:router# clear controller fabric fia location
b.
Clear fabric crossbar counters.
RP/0/RSP0/CPU0:router# clear controller fabric crossbar-counters location
Step 4
Start the traffic pattern that caused the packet drop.
Step 5
Run the following command to display the NP-to-interface mapping.

RP/0/RSP0/CPU0:router# show controllers np ports all
Step 6
Check the counters at the input interface.

RP/0/RSP0/CPU0:router# show interfaces type location
Step 7
Check the NP counters to verify that traffic is flowing in NP counters along the data path.
RP/0/RSP0/CPU0:router# show controllers np counters {np0|np1|np2|np3|all} location node-id
{| include DROP}
RP/0/RSP0/CPU0:router# show controllers np counters np3 location 0/0/CPU0
| include DROP
The show controllers np command displays information about counters that helps you troubleshoot
drops in the LCs. The names of the internal NP counters have the general format
STAGE_DIRECTION_ACTION, for example, PARSE_FABRIC_RECEIVE_CNT,
RESOLVE_EGRESS_DROP_CNT, and MODIFY_FRAMES_PADDED_CNT.
The values of stage, directon, and action are as follows:
There are five stages in the NP:

Parse
Search-I
Modify
Search-II
OL-23591-02
7-149
Chapter 7
Resolve
Examples of the direction are:

Ingress
Egress
Next_hop
Examples of the action are:

Drop_count
Down
There are additional counters, such as DROP, PUNT, and DIAGS, that provide important information
but are not associated with a specific internal NP stage. Drop and punt counters are kept as an aggregate
total per stage.
Example
RP/0/RSP0/CPU0:router# show controllers np ports all
Thu Jan 1 02:18:48.264 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------NP Bridge Fia
Ports
-- ------ --- --------------------------------------------------0 1
0
1 1
0
2 0
0
3 0
0
RP/0/RSP0/CPU0:router# show interfaces tenGigE 0/1/0/0

Thu Jan 1 01:10:01.908 UTC
TenGigE0/1/0/0 is up, line protocol is up
Hardware is TenGigE, address is 001e.bdfd.1736 (bia 001e.bdfd.1736)
Layer 2 Transport Mode
Encapsulation ARPA,
Full-duplex, 10000Mb/s, LR, link type is force-up
loopback not set,
Maintenance is enabled,
7-150
OL-23591-02
Chapter 7

In the following example, there were some ingress and egress drops in the RESOLVE stage. All of these
drops in the ingress (9 drops) and egress (6 drops) were caused by the next hop being unreachable (a
total of 15 drops for IPv4 next hop down).
Mon Nov 15 12:18:35.289 EST
30
31
295
RESOLVE_INGRESS_DROP_CNT
RESOLVE_EGRESS_DROP_CNT
DROP_IPV4_NEXT_HOP_DOWN
9
6
15
| include DROP
0
0
0
The following example shows a typical output from the same command, but without the modifier
| include DROP.
RP/0/RSP0/CPU0:router# show controllers np counters np3
Mon Nov 15 12:20:35.289 EST
Node: 0/0/CPU0:
Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------23 PARSE_FABRIC_RECEIVE_CNT
417
0
30 RESOLVE_INRESS_DROP_CNT
9
0
6
0
53 MODIFY_FRAMES_PADDED_CNT
3230
0
67 PARSE_MOFRR_SWITCH_MSG_RCVD_FROM_FAB
920
0
70 RESOLVE_INGRESS_L2_PUNT_CNT
1081
0
71 RESOLVE_EGRESS_L3_PUNT_CNT
4613
0
74 RESOLVE_LEARN_FROM_NOTIFY_CNT
3484
0
75 RESOLVE_BD_FLUSH_DELETE_CNT
104
0
83 RESOLVE_MOFRR_HASH_UPDATE_CNT
463
0
87 RESOLVE_MOFRR_SWITCH_MSG_INGNORED
407
0
111 DIAGS
536
0
295 DROP_IPV4_NEXT_HOP_DOWN
15
0
.
.
.
Step 8
Check the NP Bridge3 counters.

RP/0/RSP0/CPU0:router# show controllers np fabric-counters all ?
all
np0
np1
np2
np3
All
NP0
NP1
NP2
NP3
NP instances
instance
instance
instance
instance
RP/0/RSP0/CPU0:router# show controllers np fabric-counters all <np instance or all>

location <location>
RP/0/RSP0/CPU0:router# show controllers np fabric-counters all np3 location 0/5/CPU0
Check the NP-bridge rx/tx counters for each NP on the LC. View the packet sent and received counts,
bytes transferred, packet counters categorized by packet size, and so forth. The fields of interest are:
xaui_a_t_transmited_packets_cnt: The number of packets sent by the NP to the bridge
xaui_a_r_received_packets_cnt: The number of packets sent by the bridge to the NP
Step 9
Check the bridge counters
OL-23591-02
7-151
Chapter 7
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location node-id
Examples
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/RSP0/CPU0
Mon Nov 22 14:14:48.010 PST
Device Rx Interface
Packet
Error
Threshold
Count
Drops
Drops
-------------------------------------------------------------------------------Bridge0 From-Fabric(DDR) 492283
0
0
From CPU
492283
0
0
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/1/CPU0
Mon Nov 22 14:18:54.834 PST
UC - Unicast , MC - Multicast
LP - LowPriority , HP - HighPriority
-------------------------------------------------------------------------------FIA 0
******
Cast/
Packet
Packet
Error
Threshold
Prio
Direction
Count
Drops
Drops
-------------------------------------------------------------------------------Unicast Egress Stats
********************
UC HP
Fabric to NP-0
70329
0
0
UC LP
Fabric to NP-0
0
0
0
UC HP
Fabric to NP-1
70329
0
0
UC LP
Fabric to NP-1
0
0
0
UC HP
Fabric to NP-2
70329
0
0
UC LP
Fabric to NP-2
0
0
0
UC HP
Fabric to NP-3
70329
0
0
UC LP
Fabric to NP-3
0
0
0
---------------------------------------------------------------UC
Total Egress
281316
0
0
Multicast Egress Stats
*********************
MC HP
Fabric to NP-0
0
0
0
MC LP
Fabric to NP-0
0
0
0
MC HP
Fabric to NP-1
0
0
0
MC LP
Fabric to NP-1
0
0
0
MC HP
Fabric to NP-2
0
0
0
MC LP
Fabric to NP-2
0
0
0
MC HP
Fabric to NP-3
0
0
0
MC LP
Fabric to NP-3
0
0
0
--------------------------------------------------------------MC
Total Egress
0
0
0
Cast/
Packet
Packet
Prio
Direction
Count
-------------------------------------------------Unicast Ingress Stats
*********************
UC HP
NP-0 to Fabric
70329
UC LP
NP-0 to Fabric
0
UC HP
NP-1 to Fabric
70329
UC LP
NP-1 to Fabric
0
UC HP
NP-2 to Fabric
70329
7-152
OL-23591-02
Chapter 7

UC LP
NP-2 to Fabric
0
UC HP
NP-3 to Fabric
70329
UC LP
NP-3 to Fabric
0
-------------------------------------------------UC
Total Ingress
281316
Multicast Ingress Stats
***********************
MC HP
NP-0 to Fabric
0
MC LP
NP-0 to Fabric
0
MC HP
NP-1 to Fabric
0
MC LP
NP-1 to Fabric
0
MC HP
NP-2 to Fabric
0
MC LP
NP-2 to Fabric
0
MC HP
NP-3 to Fabric
0
MC LP
NP-3 to Fabric
0
-------------------------------------------------MC
Total Ingress
0
Ingress Drop Stats (MC & UC combined)

**************************************
PriorityPacket
Error
Threshold
Direction
Drops
Drops
-------------------------------------------------LP
NP-0 to Fabric
0
0
HP
NP-0 to Fabric
0
0
LP
NP-1 to Fabric
0
0
HP
NP-1 to Fabric
0
0
LP
NP-2 to Fabric
0
0
HP
NP-2 to Fabric
0
0
LP
NP-3 to Fabric
0
0
HP
NP-3 to Fabric
0
0
-------------------------------------------------Total IngressDrops 0
0
Step 10
Check the FIA counters

RP/0/RSP0/CPU0:router# show controllers fabric fia stats location location
Examples:
RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/RSP0/CPU0
Wed Aug 25 12:36:43.151 DST
FIA:0 DDR Packet counters:
=========================
From Punt 686545
To Punt
582387
FIA:0 SuperFrame counters:
=========================
To Unicast Xbar[0]
821335
To Unicast Xbar[1]
0
To Unicast Xbar[2]
0
To Unicast Xbar[3]
0
To MultiCast Xbar[0]
7758
0
15807
0
From Unicast Xbar[0]
629854
0
1
OL-23591-02
7-153
Chapter 7
From
From
From
From
From
Unicast Xbar[3]
MultiCast Xbar[0]
MultiCast Xbar[1]
MultiCast Xbar[2]
MultiCast Xbar[3]
0
2589
0
2588
0
FIA:0 Total Drop counters:

=========================
Ingress drop: 0
Egress drop: 2
Total drop: 2
RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/2/CPU0
FIA:0 DDR Packet counters:
=========================
From Bridge#[0] 510
To Bridge #[0] 510
From Bridge#[1] 510
To Bridge #[1] 510
FIA:0 SuperFrame counters:
=========================
To Unicast Xbar[0]
19
To Unicast Xbar[1]
20
To Unicast Xbar[2]
0
To Unicast Xbar[3]
0
0
0
0
0
From
From
From
From
From
From
From
From
Unicast Xbar[0]
Unicast Xbar[1]
Unicast Xbar[2]
Unicast Xbar[3]
MultiCast Xbar[0]
MultiCast Xbar[1]
MultiCast Xbar[2]
MultiCast Xbar[3]
19
20
0
0
0
0
0
0
FIA:0 Total Drop counters:

=========================
Ingress drop: 0
Egress drop: 0
Total drop: 0
RP/0/RSP0/CPU0:router# show controllers fabric fia q-depth [location location]
Thu Jan 1 02:16:37.227 UTC
FIA 0
-----Total Pkt queue depth count = 0
Step 11
Check the crossbar counters to make sure there are no dropped packets.
RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance [0|1] location
location
Example:
RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance 0 location
0/RSP0/CPU0
Location: 0/RSP0/CPU0 (physical slot 4)
7-154
OL-23591-02
Chapter 7

Asic Instance: 0
Fabric info for node 0/RSP0/CPU0 (physical slot: 4)
Dropped packets
:
mcast
unicast
+---------------------------------------------------------------+
Input buf bp pkts
:
0
0
Output buf bp pkts
:
0
0
Xbar timeout buf bp pkts
:
0
0
HOL drop pkts
:
0
0
Null POE drop pkts
:
0
0
Locating Drops of Punted Packets

To locate drops of punted packets, perform the following procedure.
SUMMARY STEPS
1.
Clear all packet counters
2.
Start traffic
3.
Check traffic counters at each component
4.
Check NP counters for NP mapping to interface, and check NP0 for inject packet count
5.
Check fabric-related counters
6.
Check punt FPGA counters
DETAILED STEPS
Step 1
Clear all packet counters as described in the Locating Packet Drops by Examining Counters section
on page 7-148.
Step 2
Start traffic.
Step 3
Check traffic counters at each component in the punted packet path. Use a procedure similar to the one
described in the Locating Packet Drops by Examining Counters section on page 7-148. However, for
punted packets, the data path is:
Incoming Interface --> NP --> LC CPU --> NP --> Bridge3 --> LC FIA --> RSP Crossbar--> Punt
FPGA on RSP --> RSP CPU --> RSP FIA --> RSP Crossbar --> LC FIA --> LC CPU --> NP0 --->
LC FIA ---> Crossbar ---> RSP FIA ---> RSP CPU
Step 4
Check the NP counters for NP mapping to interface, and check NP0 for the inject packet count. The
following fields provide information on the NP counters:
820 PARSE_LC_INJECT_TO_FAB_CNT
970 MODIFY_FABRIC_TRANSMIT_CNT
822 PARSE_FAB_INJECT_IPV4_CNT
Step 5
Check the fabric-related counters for any packet drops.

RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance 0 location
0/RSP0/CPU0
OL-23591-02
7-155
Chapter 7
RP/0/RSP0/CPU0:router# show controllers fabric fia stats [location location]

Example:
RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/5/CPU0
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats [location location]
Examples:
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/RSP0/CPU0
Wed Aug 25 14:12:03.916 DST
Device
Rx Interface
Packet
Error
Threshold
Count
Drops
Drops
-------------------------------------------------------------------------------Bridge0 From-Fabric(DDR) 603698
0
0
From CPU
711734
0
0
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/5/CPU0
Wed Aug 25 14:12:20.867 DST
UC - Unicast , MC - Multicast
LP - LowPriority , HP - HighPriority
-------------------------------------------------------------------------------FIA 0
******
Cast/
Packet
Packet
Error
Threshold
Prio
Direction
Count
Drops
Drops
-------------------------------------------------------------------------------Unicast Egress Stats
********************
UC HP
Fabric to NP-0
28
0
0
UC LP
Fabric to NP-0
0
0
0
UC HP
Fabric to NP-1
28
0
0
UC LP
Fabric to NP-1
0
0
0
UC HP
Fabric to NP-2
28
0
0
UC LP
Fabric to NP-2
0
0
0
UC HP
Fabric to NP-3
28
0
0
UC LP
Fabric to NP-3
0
0
0
---------------------------------------------------------------UC
Total Egress
112
0
0
Multicast Egress Stats
*********************
MC HP
Fabric to NP-0
205
0
0
MC LP
Fabric to NP-0
2
0
0
MC HP
Fabric to NP-1
205
0
0
MC LP
Fabric to NP-1
2
0
0
MC HP
Fabric to NP-2
205
0
0
MC LP
Fabric to NP-2
2
0
0
MC HP
Fabric to NP-3
205
0
0
MC LP
Fabric to NP-3
2
0
0
--------------------------------------------------------------MC
Total Egress
828
0
0
--More--
Step 6
To check for packets punted to and injected from the LC or RP CPU, run the following commands.
RP/0/RSP0/CPU0:router# show spp interface location node-id
7-156
OL-23591-02
Chapter 7

RP/0/RSP0/CPU0:router# show spp node-counters location node-id

RP/0/RSP0/CPU0:router# show spp node location node-id
RP/0/RSP0/CPU0:router# show spp sid stats location node-id
RP/0/RSP0/CPU0:router# show spp client location node-id
Note
Step 7
To clear the spp counters, run the command clear spp {client | interface | node-counters}
location node-id. This command clears client statistics, interface statistics, and per-node
counters, depending on the keyword you use.
To query the punt switch for the statistics on the LC CPU, run the following command.
RP/0/RSP0/CPU0:router# show controllers punt-switch switch-stats location node-id
Packet Drop from LC to LC

In this scenario, you have configured the system, RSP and LC have come up and are stable, LC to LC
traffic is going through, but some packets are dropped.
The possible causes are:
Traffic dropped at interface
Traffic dropped at NP3
Traffic dropped at bridge
Traffic dropped at the fabric I/O
Synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
Traffic has wrong vqi
Oversubscribed traffic
Unknown failures
Locate the Problem and Take Corrective Action

Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1
If not already done, perform the procedures in the Getting Started with Fabric Troubleshooting section
on page 7-145 to verify that you have the correct versions of the hardware and software.
Step 2
Collect the sync status of fabric on the LC.

show controllers fabric fia link-status location <0/1/CPU0>
show controllers fabric fia bridge ddr-status location <0/1/cpu0>
show controllers fabric fia bridge sync-status location 0/1/cpu0
Step 3
Collect configuration information.

show run
OL-23591-02
7-157
Chapter 7
Step 4
dump PFM errors on both source and destination LC.

show pfm location <0/1/cpu0>
Step 5
Collect the fabric I/O/Bridge counters on both source and destination card.
show interfaces
show controllers fabric fia stats location 0/1/CPU0
show controllers fabric fia bridge stats location 0/1/CPU0
Step 6
Collect redundancy information.

show redundancy
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution
Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1
Perform reset h at LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2
Pull out the LC and reinsert it to see if it can boot up.
Step 3
Stop other streams of traffic to see if this failed stream can go through.
Step 4
Reduce the rate of the traffic to see if the drop continues.
Packet Drop Between RSP and LC

In this scenario, you have configured the system, RSP and LC have come up and are stable, but one of
the following problems occurred:
Protocol or ping traffic (punt path traffic) has some drops
Initially the ping/protocol packets are not going through, but later recover.
7-158
OL-23591-02
Chapter 7

Sync between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
Traffic drop at Punt FPGA
sn database sync issue
Unknown failures

Step 1
Step 2
Collect the sync status of fabric on the linecard.

Step 3

show run
Step 4
Dump the PFM errors for the card.

show pfm location <0/rsp0/cpu0>
Step 5
Collect the fabric I/O/bridge counters on both RSP and LC.

show interfaces
show controllers fabric fia stats location 0/rsp0/CPU0
Where to Go Next
OL-23591-02
7-159
Chapter 7
Caution
of traffic.
Step 1
Step 2
Step 3
Step 4
Determine whether the drop is a single burst in the beginning or is continuous.
Step 5
Determine if the drop is associated with particular packet size.
Packet Drop After Certain Actions

In this scenario, the system is configured, RSP and LC have come up, and traffic is flowing properly for
some time. However, after certain action such as configuration change, online insertion and removal
(OIR) of LC/RSP, LC reload, or software upgrade, some traffic drop or complete traffic loss is observed.
Unknown failures

Step 1
Perform the procedures in the Getting Started with Fabric Troubleshooting section on page 7-145 to
verify that you have the correct versions of the hardware and software.
Step 2
Collect the sync status of fabric on the linecard.

Step 3
7-160
OL-23591-02
Chapter 7

show run
Step 4

Step 5
Collect the fabric I/O/bridge counters on both the RSP and LC.
show interfaces
Step 6

show redundancy
Where to Go Next
Caution
of traffic.
Step 1
Step 2
Step 3
Step 4
Repeat Step 1 through Step 3 to determine whether the results are reproducible.
Packet Drop After a Redundancy Switchover

In this scenario, you have configured the system, RSP and LC have come up, and traffic is flowing
properly for some time. However, after a switchover (by a command or OIR), you see some traffic drop
or complete traffic loss.
OL-23591-02
7-161
Chapter 7
Fabric is stuck
Unknown failures

Step 1
Step 2
Collect the sync status of fabric on the linecard before and after the switchover.
Step 3

show run
Step 4

Step 5
show interfaces
Step 6

show redundancy
7-162
OL-23591-02
Chapter 7

Where to Go Next
Caution
of traffic.
Step 1
Stop other streams of traffic to see if this failed stream can go through again.
Step 2
Repeat Step 1 several times to determine if the result is reproducible.
Step 3
Perfom a switchover back to the other side to determine whether both directions are having the same
traffic problems.
Step 4
After obtaining the necessary approvals from your network and system administrators (because this step
will stop all traffic on this unit), reboot the entire system and check to see if it recovers.
Packet Drop with Unknown Reason

In this scenario, you have configured the system, RSP and LC have come up, and traffic is flowing
properly for a significant time (at least several days). However, for an unknown reason, the system
experiences traffic drops or complete traffic loss.
Fabric is stuck
Unknown failures

Step 1
OL-23591-02
7-163
Chapter 7
Step 2
Collect the sync status of fabric on the linecard before and after the switchover.
Step 3

Step 4
show interfaces
Step 5

show redundancy
Step 6
Check for drops on the the fabric I/O interface (FIA drop counters) on the LC in both the ingress (to
fabric) and egress (from fabric) directions.
show
show
show
show
Step 7
controllers
controllers
controllers
controllers
fabric
fabric
fabric
fabric
fia
fia
fia
fia
drops
drops
error
error
egress location
ingress location
egress location
ingress location
Check for drops on the bridge. Counters are a combination of high priority (HP), low priority (LP),
unicast, multicast, DDR, and DDR-threshold packets. They are furthur segregated into critical and
informational based on their severity. All Ethernet linecards have 2 bridges. Use the following command
to obtain this information.
show controllers fabric fia bridge stats location <linecard location>
Step 8
Check if there are any drops on Punt FPGA on RSP.

show controllers fabric fia bridge stats location 0/RSP0/CPU0
Where to Go Next
7-164
OL-23591-02
Chapter 7

Caution
of traffic.
Step 1
Stop other streams of traffic to see if this failed stream can go through again.
Step 2
Reboot the LCs one at a time and check if the traffic recovers.
Step 3
After obtaining the necessary approvals from your network and system administrators (because this step
will stop all traffic on this unit), reboot the entire system and check to see if it recovers.
Step 4
Reconfigure the system to see if it recovers.

This section explains how to troubleshoot the following problems:
Active RSP Is Crashing, page 7-165
Standby RSP Is Crashing, page 7-166
LC Is Crashing, page 7-167
Active RSP Is Crashing

In this scenario, the active RSP keeps crashing and the RSP console shows that the active fabric manager
or fia_rsp (the fabric I/O process) terminates repeatedly.
Initialization of the fabric I/O fails for some reason
Fabric self-test fails
The synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
Unknown failures

Step 1
Step 2
Collect the sync status of fabric on the RSP card.

show controllers fabric fia link-status location <0/RSP0/CPU0>
show controllers fabric fia bridge sync-status location
OL-23591-02
7-165
Chapter 7
Step 3

Step 4
Collect the fabric I/O/Punt counters.

show controllers fabric fia stats location <0/rsp0/CPU0>
Where to Go Next
Caution
of traffic.
Step 1
Perform reset h at LC ROMMON and reboot the RSP again to see if this clears the problem.
Step 2
Pull out the RSP and reinsert it to see if it can boot up.
Step 3
Swap the slot (put the RSP card into the other RSP slot) and see if it can boot up properly.
Standby RSP Is Crashing

In this scenario, the active RSP is up and running, but the standby RSP keeps crashing. The RSP console
shows that the standby fabric manager or fia_rsp (the fabric I/O process) terminates repeatedly.
Initialization of the standby fabric I/O fails for some reason
Fabric self-test on the standby card fails
The sync between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
Communication between the active and standby card is not working
Unknown failures

Step 1
7-166
OL-23591-02
Chapter 7

Step 2
Collect the sync status of fabric on the RSP card.

show controllers fabric fia link-status location <0/RSP0/CPU0>
Step 3

Step 4
Dump the redundancy status.

show redundancy
Step 5
Collect the fabric I/O/ punt counters.

show controllers fabric fia stats location <0/1/CPU0>
Where to Go Next
Caution
of traffic.
Step 1
Perform reset h at the ROMMON and reboot the standby RSP again to see if this clears the problem.
Step 2
Pull out the RSP and reinsert it to see if it can boot up.
Step 3
Swap the slot (put the RSP card into the other RSP slot) and see if it can boot up properly.
LC Is Crashing
In this scenario, a LC keeps crashing and the RSP console shows that fia_lc (the fabric I/O process)
terminates repeatedly.
Initialization of the LC fabric I/O fails for some reason
Fabric self-test on the LC fails
The synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
Communication between the LC and the RSP is not working properly
There is a sync problem between the fabric I/O and the bridge
Unknown failures
OL-23591-02
7-167
Chapter 7
Troubleshooting Complete Loss of Traffic

Step 1
Step 2
Collect the sync status of the fabric on the LC.

Step 3

Step 4
Collect the fabric I/O/ bridge counters.

show controllers fabric fia stats location <0/1/CPU0>
show controllers fabric fia bridge stats location <0/1/CPU0>
Where to Go Next
Caution
of traffic.
Step 1
Perform reset h at the LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2
Step 3
Swap the slot (pull out the LC and insert it into another LC slot) and see if it can boot up properly.
Step 4
Put a different LC of same type to see if that card can booting up properly.

This section explains how to troubleshoot scenarios in which the system is active but traffic does not go
through. It includes the following topics:
No Traffic from LC to LC, page 7-169
7-168
OL-23591-02
Chapter 7

No Traffic Between RSP and LC, page 7-170
No Traffic from LC to LC
In this scenario, you have configured the system and the RSP and LC have come up and are stable, but
no LC-to-LC traffic is going through.
Traffic dropped at the interface
Traffic dropped at the bridge
Unknown failures

Step 1
Step 2

Step 3

show run
Step 4

Step 5
Collect the fabric I/O/bridge counters on both the source and destination cards.
show interfaces
Step 6

show redundancy
OL-23591-02
7-169
Chapter 7
Where to Go Next
Caution
of traffic.
Step 1
Step 2
Pull out the LC and reinsert it to see if it can boot up and carry traffic.
Step 3
Step 4
Run online diagnostics to locate errors in the system. For additional information on diagnostics, see the
Using Diagnostic Commands section on page 1-59.
No Traffic Between RSP and LC

In this scenario, you have configured the system and the RSP and LC have come up and are stable, but
no protocol or ping traffic (punt path traffic) is going through.
Traffic dropped at the interface
Traffic dropped at the bridge
Traffic dropped at the punt FPGA
Traffic dropped at the protocol level
Unknown failures

7-170
OL-23591-02
Chapter 7

Step 1
Step 2

Step 3

show run
Step 4

Step 5
show interfaces
Step 6

show redundancy
Where to Go Next
Caution
of traffic.
Step 1
Step 2
Pull out the LC and reinsert it to see if it can boot up and carry traffic.
Step 3
Pull out the RSP card and reinsert it to see if it can boot up and carry traffic.
Step 4
OL-23591-02
7-171
Chapter 7
Step 5
Run online diagnostics to locate errors in the system. For additional information on diagnostics, see the
Using Diagnostic Commands section on page 1-59.

If you need support from Cisco to troubleshoot the fabric, we recommend that you gather the following
information if time permits:
Output of the following commands (this will display software version, and the line card, fabric card,
FPGA, and ASIC versions)
show
show
show
show
version
inventory raw
diag
hw-module fpd location
Information on chassis type

(admin) show inventory
Platform-related information
show platform
Ingress interface(s), egress interface(s), and expected packet path
Drop counters
Logs (capture all logs on the RSP console port)
7-172
OL-23591-02
CH A P T E R

This chapter describes techniques to troubleshoot MultiProtocol Label Switching (MPLS) services.
MPLS carries different kinds of traffic, such as IP packets and Ethernet frames. The general flow of
packets in a Cisco ASR 9000 Aggregation Series Router is as follows:
Incoming interface => Ingress NP => Switch fabric => Egress NP => Outgoing interface.
This chapter contains the following subsections:
Verifying MPLS PIE Activation and MPLS Configuration, page 8-173
Troubleshooting Connectivity Over MPLS, page 8-174
IP Packets Not Forwarded to LSP, page 8-175
IP Packets Not Forwarded to MPLS TE Tunnel, page 8-176
MPLS Packets Not Forwarded to MPLS TE Tunnel, page 8-176
MPLS TE Tunnels Do Not Come Up, page 8-176
FRR-Protected Tunnel Goes Down After Triggering FRR, page 8-177
MPLS TE FRR Database Not Built, page 8-178
MPLS FRR Switch Time Debugging, page 8-178
Verifying MPLS PIE Activation and MPLS Configuration

For operation of MPLS, the MPLS PIE must be active and MPLS must be present in your running
configuration:
Verify that the MPLS PIE is installed, committed, and activated. It is not installed by default.
Verify that MPLS is configured in your running-config. After you install the MPLS PIE, you must
commit it. If you configure MPLS but you have not committed the MPLS PIE, the system deletes
all of your MPLS configuration if you reload the router image.
Caution
Verify that the MPLS PIE is committed before you configure MPLS. Otherwise all of your
MPLS configuration data will be lost if the image is reloaded.
OL-23591-02
8-173
Chapter 8

This section explains how to troubleshoot MPLS connectivity for multipoint Layer 2 services.
Step 1
Ping the opposite interface (on the remote router) on the MPLS interface. Verify that the ping is
successful.
Step 2
Verify that the remote interface shows up as an ospf neighbor.

show ospf neigbor
Step 3
Verify that the remote router ID (typically the remote router loopback) is in the routing table.
show route ipv4
Step 4
Ping the IP address of the remote router (the same IP address that was displayed in Step 3). Verify that
the ping is successful.
Step 5
Verify that label distribution protocol (LDP) is up between the local and remote routers.
show mpls ldp neighbor
Step 6
Verify that you can find the ID of the remote router in an MPLS command. In the case of a PW, this ID
will be theIPv4 address for the PW.
Step 7
Verify that the BGP neighbor is up.
Step 8
If you are using PW in the core, verify that the PWs are properly configured on both PEs.
Step 9
Check that configurations are correct on all peers in the VPLS domain. This includes, for example,
loopbacks, IGP (OSPF or ISIS), LDP, BGP, and L2VPN.
Note
L2VPN services rely on Layer 3 connectivity from the PE through the core. If you need to reconfigure
any routing parameters, use the procedures shown in Cisco ASR 9000 Series Aggregation Services
Router Routing Configuration Guide, Release 4.0.

SUMMARY STEPS
1.
debug mpls ldp transport events
2.
debug mpls ldp transport connections
3.
show mpls forwarding tunnels detail
4.
debug mpls ea platform {all | errors | events | info} [ location ]
5.
show cef platform trace [adj | all } common | fwdwlk | ipfrr | ipv4 | ipv6 | mpls | rpf | te] location
node-id
6.
show cef platform resource location node-id
7.
show mpls forwarding labels label-id hardware egress location node-id
8.
show mpls ldp discovery
8-174
OL-23591-02
Chapter 8

9.
DETAILED STEPS
Command or Action
Purpose
Step 1
debug mpls ldp transport event
Displays and logs discovery and connection setup/shutdown

events.
Step 2
debug mpls ldp transport connection
Displays and logs connection setup/shutdown events.
Step 3
show mpls forwarding tunnels detail
Display MPLS tunnel status.
Step 4
debug mpls ea platform {all | errors | events |

info} [location]
Displays and logs MPLS setup events and errors.
Step 5
show cef platform trace [adj | all | common |

fwdwlk | ipfrr | ipv4 | ipv6 | mpls | rpf | te]
location node-id
Display data path setup event and error logs.
Step 6
show cef platform resource location node-id
Display line card resource event and error logs.
Step 7
show mpls forwarding label label-id hardware

egress location node-id
Display MPLS label status.
Step 8
show mpls ldp discovery
Display MPLS LDP status.
Step 9
Display MPLS LDP neighbor status.

These commands help to troubleshoot IP packets not being forwarded on the label switched path (LSP).
Step 1
Check the prefix information.

RP/0/RSP0/CPU0:router# show cef prefix/length
Step 2
Check the hardware label FIB.

RP/0/RSP0/CPU0:router# show mpls forwarding labels label-id hardware egress location
node-id
Step 3
Find out whether the ARP resolves for the next hop prefix.
RP/0/RSP0/CPU0:router# show arp prefix location node-id
Workaround
If ARP is not resolved, ping the destination and run the show arp command again to see if the ARP
resolves. If it does not resolve, it means there is no path to reach the destination, or the destination is not
operational.
If ARP is resolved, use the clear arp location node-id command to clear the ARP information on the
node. This command clears the current ARP table entries, and the system will refill the ARP entries with
the latest ARP information. This might help if there are stale and incorrect entries in the ARP table.
OL-23591-02
8-175
Chapter 8

Step 1
Check the tunnel adjacency of the prefix.

RP/0/RSP0/CPU0:router# show cef prefix hardware ingress location node-id
Step 2
Ensure that the MPLS traffic tunnel is up.

RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels up
Step 3
Check the hardware TE label FIB by running the following command on the ingress LC for the unicast
traffic.
RP/0/RSP0/CPU0:router# show cef adjacency tunnel-te tunnel-id hardware ingress location
node-id
Workaround:
Enter the shut command (followed by commit) and the no shut command (followed by commit) on the
tunnel interface to reprogram the hardware.
MPLS Packets Not Forwarded to MPLS TE Tunnel

Step 1
Ensure that transmit adjacency is complete.

RP/0/RSP0/CPU0:router# show mpls forwarding labels
Step 2
label-id hardware egress location node-id
Ensure that hardware tunnel adjacency is complete by running the following command on the ingress LC
for the unicast traffic.
RP/0/RSP0/CPU0:router# show cef mpls adjacency tunnel-te
te-id hardware egress location
node-id
Workaround
Perform the shut command (followed by commit) and the no shut command (followed by commit) of
the tunnel interface to reprogram the hardware.
MPLS TE Tunnels Do Not Come Up

Step 1
Ensure that the tunnel egress interface is configured in RSVP.

rsvp
bandwidth 100000
!
bandwidth 100000
!
8-176
OL-23591-02
Chapter 8

interface GigabitEthernet0/4/0/8 <<---- tunnel egress interface

bandwidth 100000
!
bandwidth 100000
!
mpls traffic-eng <<---- Ensure that the tunnel egress interface is configured in mpls
traffic-engineering config
!
!
interface GigabitEthernet0/4/0/8 <<---- tunnel egress interface
!
!
Step 2
Verify that the tunnel egress interface is up, for example:

RP/0/RSP0/CPU0:router# show interface GigabitEthernet0/4/0/8
Step 3
Ensure that traffic engineering is configured in OSPF.

router ospf te
log adjacency changes detail
router-id 192.168.1.30
area 0
mpls traffic-eng
Step 4
Ensure that ping is successful on tunnel destination IP.

Fast ReRoute (FRR) is a mechanism for protecting MPLS Traffic Engineering (TE) label-switched paths
(LSPs) from link and node failures by locally repairing the LSPs at the point of failure, allowing data to
continue to flow on them while their headend routers attempt to establish new end-to-end LSPs to replace
them. FRR locally repairs the protected LSPs by rerouting them over backup tunnels that bypass failed
links or nodes.
Step 1
Ping the address in an updated sender template.

RP/0/RSP0/CPU0:router# ping PLR_Address
Step 2
Ensure that the MP address is reachable. Check forwarding over the backup tunnel is working.
RP/0/RSP0/CPU0:router# ping backup_tunnel_destination
Step 3
Ensure that the backup tunnel is in Up, Up state.

RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels
Step 4
Check RSVP traces to find out why the tunnel went down.
RP/0/RSP0/CPU0:router# show mpls traffic-eng trace event
OL-23591-02
8-177
Chapter 8

Step 1
Ensure that the protected tunnel is fast reroutable.

a.
Step 2
RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute database
Ensure that backup does not pass through a protected interface.

RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnel backup protected-interface
Step 3
Ensure that backup has enough backup bandwidth (if configured).

RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels backup
Step 4
Ensure that backup and protected tunnels have a merge point (check hop information).
RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels
Step 5
Ensure that protected and backup tunnels are in Up, Up state.

RP/0/RSP0/CPU0:router# show mpls traffic-eng tunnels brief
Note
Step 6
Protected tunnels with 0 signaled bandwidth cannot be protected by limited backup-bw tunnels.
Enable debugs, remove, and reapply backup tunnel-te.
RP/0/RSP0/CPU0:router# debug mpls traffic-eng frr
Step 7
Shut and no shut the backup tunnel and/or protected tunnel (if possible). This resets backup tunnel
assignments.
Step 8
Ensure that the fast-reroute option is not configured on the backup tunnel.
RP/0/RSP0/CPU0:router# show running-config interface tunnel-te15
Step 9
Step 10
Find out if backup is assigned to a protected LSP.

a.
b.
RP/0/RSP0/CPU0:router# show mpls traffic-eng forwarding
c.
RP/0/RSP0/CPU0:router# show rsvp fast-reroute
Ensure that the pool-type of the protected LSP bandwidth and backup-bw of the backup tunnel matches.

Step 1
Ensure that the FRR database is built and in ready state.

Step 2
Upon FRR triggered, ensure that FRR is in the active state.

8-178
OL-23591-02
Chapter 8

Step 3
Check FRR switch time of LC that the primary tunnel is failed.

RP/0/RSP0/CPU0:router# show mpls traffic-eng fast-reroute log location node-id
Step 4
Ensure that both primary and backup tunnels on the LC received the FRR trigger.
RP/0/RSP0/CPU0:router# show cef platform trace te all location node-id
OL-23591-02
8-179
Chapter 8
8-180
OL-23591-02
CH A P T E R

This chapter describes techniques to troubleshoot Layer 2 virtual private network (L2VPN) features. In
this document, L2VPN refers to a family of Layer 2 functions and Ethernet services provided by the
Cisco ASR 9000 Aggregation Series Router.
If you are experiencing a problem with L2VPN traffic, the source of the problem could be caused by any
of the following conditions:
Interfaces in the customer edge (CE) router down or configured incorrectly.
Interfaces in the provider edge (PE) router down or configured incorrectly.
MAC address updates not functioning correctly.
Bridge domain not configured correctly.
Routing in the core network down or not configured correctly.
This chapter contains the following sections that explain how to troubleshoot these conditions:
Troubleshooting VLAN Traffic and L2 TCAM Classification, page 9-181
Troubleshooting Multipoint Layer 2 Services, page 9-190
Troubleshooting Point-to-Point Layer 2 Services, page 9-206
Troubleshooting Specific Outage Scenarios In Layer 2 Services, page 9-214
Troubleshooting Dynamic Host Configuration Protocol Snooping, page 9-227
Troubleshooting Multiple Spanning Tree, page 9-230
Additional ReferencesCommand Reference and Configuration Guides, page 9-232
Troubleshooting VLAN Traffic and L2 TCAM Classification

This section explains how to troubleshoot VLAN traffic problems related to Layer 2 TCAM
classification. (TCAM = ternary content addressable memory.) It contains the following topics:
Understanding Problems with VLAN Traffic and L2 TCAM Classification, page 9-182
Verifying the Configuration Is Correct, page 9-182
Verifying Interfaces, Subinterfaces, and Packet Forwarding, page 9-183
OL-23591-02
9-181
Chapter 9
Understanding Problems with VLAN Traffic and L2 TCAM Classification

If traffic on a VLAN is not getting through, the traffic might not be reaching the subinterface for which
it is intended. The problem could be related to any of the following:
The main interface (trunk) or subinterfaceProblems could be caused by physical issues or

configuration errors.
Incorrect classification (tagging) of the trafficIf traffic has the wrong VLAN tag, it cannot reach
the intended subinterface. Furthermore, the main interface cannot route the traffic, because it does
not classify or forward tagged traffic.
A remote peer could be sending messages with an unknown VLAN number or encapsulation type.
Drop counters on the main interface and subinterface indicate where the traffic is being dropped.
If a packet has an incorrect VLAN tag, the main interface drops the packet and the main interface
drop counter increments.
If the packet has a correct VLAN tag, it reaches the intended subinterface, but if the subinterface
drops the packet for any reason, the subinterface drop counter increments.
Verifying the Configuration Is Correct

In many cases, VLAN traffic failures are caused by configuration problems. Some configuration
omissions and errors can go unnoticed, because a bridge domain does not always display a commit
failure when an incorrect configuration is committed. You need to verify that your configuration is
correct by using the show commands listed in this section.
The system allows you to configure and commit a bridge domain with subinterfaces assigned to the ACs,
even if you have not yet created the subinterfaces themselves. However, the ACs will be operationally
down until you configure and commit the necessary subinterfaces.
Verify that your configuration is consistent with the following recommendations and requirements:
We recommend as a best practice that you assign the same VLAN tag to all the ACs in a bridge
domain.
When you create a main interface for the AC (in interface config mode):
You cannot configure an encapsulation statement
You must include the l2transport keyword on a separate command line
Example:
l2transport
When you create a subinterface for the AC (in interface config mode):
You must include the l2transport keyword on the same command line
You must configure an encapsulation statement
Example:
interface GigabitEthernet0/2/0/2.2 l2transport
Review your running configuration to verify that it is complete and the necessary interfaces are up.
(show running-config).
9-182
OL-23591-02
Chapter 9

Ensure that the interfaces and subinterfaces for the ACs are actually up. View the up/down status of
the bridge domain, ACs, and PWs (if present) by means of the show l2vpn bridge-domain
summary command. Verify that the counts are incrementing, which means that the ACs are up.
Make sure that bridge ports (for example, ACs and PWs) are assigned to the bridge domains.
Verify that a unique main or subinterface is assigned to each AC in the bridge domain.
Verifying Interfaces, Subinterfaces, and Packet Forwarding

Perform these steps to verify that the interface and subinterface (if applicable) are up, and that Layer 2
virtual private network (L2VPN) packets are being forwarded on the interface and subinterface.
Correct any problems you discover, then rerun the show commands in this section.
Step 1
Display the main interface state and subinterface state. (The main interface is also called the trunk
interface, and it is identified as trunk in some of the CLI commands.)
RP/0/RSP0/CPU0:router# show running-config interface
RP/0/RSP0/CPU0:router# show ethernet trunk
Verify that the interfaces and subinterfaces are up or down as expected.
Run this command a second time to verify that counters are being incremented.
Verify that the port settings (for example, MTU, duplex) are as expected.
Verify that traffic is being directed to the correct subinterfaces. If it is not, the configuration of the
classification might be incorrect.
Verify that there is no traffic running on the main (trunk) interface; traffic that is misclassified might
default to run on the main interface.
Verify that the encapsulations match what you expect on the subinterfaces.
Use the interface statistics for the subinterface to determine whether packets are being
demultiplexed to the correct subinterface. Use the interface statistics on the parent physical/bundle
interface to determine whether traffic is being sent/received out of the trunk port. The Layer 2
statistics for the physical/bundle interface sum over all of the child/subinterfaces.
The counters on the main interface count packets as they are sent/received physically on the wire.
On the other hand, the subinterface counters are located in the forwarding engine.
Step 2
Check the interface packet drop counters to determine if packets are being dropped and if they are,
where and why.
Display the state of interface as recognized by the L2VPN object. Verify that L2VPN packets are being
forwarded on interface and subinterface (if applicable).
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface gigabitEthernet interface-id
hardware ingress location node-id
Step 3
Display the Ethernet tags and check for any errors or mismatches. This command gives tag information
in a very concise format, if you want to check the encapsulation on multiple subinterfaces.
RP/0/RSP0/CPU0:router# show ethernet tags
OL-23591-02
9-183
Chapter 9
Step 4
Verify that the subinterface matching order is as expected. The match-order option lists the subinterfaces
in the order that they match traffic. If the traffic is being classified to a different interface than you
expect, this command can help you determine why.
RP/0/RSP0/CPU0:router# show ethernet tags match-order
Step 5
Display the interface debug counters for each network processor unit. The following example shows the
NP counters.
RP/0/RSP0/CPU0:router# show controllers np counters {all | np0 | np1 | np2 | np3}
Step 6
If the output of the command in Step 5 shows that the UIDB_TCAM_MISS_AGG_DROP counter is
incrementing, it is possible that the physical port is receiving tagged traffic that does not match the
encapsulation statement of any subinterface. The parent/main interface is an untagged Layer 3 interface,
and rejects any tagged traffic that fails classification against any of its subinterfaces/children.
RP/0/RSP0/CPU0:router# clear controllers np counters all location node-id
RP/0/RSP0/CPU0:router# show controllers np counters {all | np0 | np1 | np2 | np3}
a.
Verify that there is incoming tagged traffic that does not match the encapsulation statement of any
subinterface, and that this traffic is not needed (that is, you do not intend to configure a subinterface
to receive and forward this traffic).
Encapsulation not matched but the traffic is neededCreate the necessary subinterface or
correct the encapsulation statement on the applicable existing subinterface.

Encapsulation not matched, traffic not needed, and no encapsulation default currently
configuredGo to Substep b.
Encapsulation not matched, traffic not needed, and there is an encapsulation default currently
configuredGo to Substep c.
b.
Add an encapsulation default subinterface to receive all of the tagged traffic with unwanted
encapsulation statements. Check whether the UIDB_TCAM_MISS_AGG_DROP goes to zero, and
the default subinterface counters start going up. This process shifts the incrementing of counters
away from the main interface and isolates it on the default subinterface.
c.
Verify that the Layer 2 encapsulation default subinterface is properly configured.
Note
See the example below with the CLI statement encapsulation default.
Example
In this example, the system displays information on the subinterface 0/0/0/0.1.
RP/0/RSP0/CPU0:router# show running-config interface
!
encapsulation dot1q 10 second-dot1q 20
.
.
.
RP/0/0/CPU0:router# show interfaces GigabitEthernet 0/0/0/0.1
GigabitEthernet0/0/0/0.1 is up, line protocol is up <<< This subinterface is up
9-184
OL-23591-02
Chapter 9

Hardware is VLAN sub-interface(s), address is 02fe.08cb.26c5

reliability Unknown, txload Unknown, rxload Unknown
Encapsulation 802.1Q Virtual LAN, <<< Encapsulation is correct
Outer Match: Dot1Q VLAN 10 <<< Encapsulation
Ethertype Any, MAC Match src any, dest any<<< Encapsulation
loopback not set,
Last input never, output never
1400 packets input, 2800 bytes
7000 input drops, 8400 queue drops, 9800 input errors
4200 packets output, 5600 bytes
11200 output drops, 12600 queue drops, 14000 output errors
In this example, Bundle-Ether16 is the main interface (also referred to as the trunk interface or Layer 3
interface), and Bundle-Ether16.160 and Bundle-Ether16.161 are subinterfaces.
Bundle-Ether16 is up, line protocol is up <<< The main interface is up
Hardware is Aggregated Ethernet interface(s), address is 001b.53ff.87f0
Description: Connect to P19_C7609-S Port-Ch 16
Internet address is Unknown
MTU 9216 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability 255/255, txload 0/255,
rxload 0/255
Encapsulation ARPA, loopback not set,
No. of members in this bundle: 2
GigabitEthernet0/1/0/16 Full-duplex
1000Mb/s
Active
GigabitEthernet0/1/0/17 Full-duplex
1000Mb/s
Standby
202037 packets input, 18079605 bytes, 1 total input drops <<< Includes the sum of
packets on all the subinterfaces in addition to the packets on the main interface.
Bundle-Ether16.160 is up, line protocol is up <<< This subinterface is up
Hardware is VLAN sub-interface(s), address is 001b.53ff.87f0
Description: Connect to P19_C7609-S Port-Ch 16 Service Instance 160
reliability Unknown, txload Unknown, rxload Unknown
Encapsulation 802.1Q, loopback not set, <<< Encapsulation is correct
Last input never, output never
5425 packets input, 368952 bytes <<< Traffic is present on this subinterface
1 input drops, 0 queue drops, 0 input errors
161269 packets output, 11611364 bytes
0 output drops, 0 queue drops, 0 output errors
OL-23591-02
9-185
Chapter 9
Bundle-Ether16.161 is up, line protocol is up <<< This subinterface is up

Hardware is VLAN sub-interface(s), address is 001b.53ff.87f0
Description: Connect to P19_C7609-S Port-Ch 16 Service Instance 161
--More-RP/0/RSP0/CPU0:router# show running-config interface
description Connect to C7609-S Port-Ch 16
mtu 9216
bundle maximum-active links 1
!
description Connect to C7609-S Port-Ch 16 Service Instance 160
encapsulation dot1q 160 <<< Encapsulation is correct
!
description Connect to C7609-S Port-Ch 16 Service Instance 161
!
interface Bundle-Ether16.162
description Connect to C7609-S Port-Ch 16.162
ipv4 address 192.0.2.44 255.255.255.0
!
interface Bundle-Ether16.163
description Connect to C7609-S Port-Ch 16.163
ipv4 address 192.0.2.44 255.255.255.0
!
interface Loopback0
--More--
RP/0/RSP0/CPU0:router# show ethernet trunk

Trunk
Sub types
Interface
St Ly
MTU
Subs
L2
BE16
Up L3
9216
4
2
Gi0/1/0/3
Up L3
9014
5
5
Gi0/1/0/7
Up L3
9014
6
6
Gi0/1/0/19
Up L3
9014
2
2
Gi0/1/0/20
Up L3
9014
1
1
Gi0/1/0/30
Up L3
9014
1
1
Summary
19
17
L3
2
0
0
0
0
0
Sub states
Up
Down
4
0
5
0
6
0
2
0
1
0
1
0
19
Ad-Down
0
0
0
0
0
0
0
The following example shows the NP counters. For a description of how to interpret NP counter
information, see the Displaying Traffic Status in Line Cards and RSP Cards section on page 7-147.
Note
If you want to clear counters at any time during this procedure (to make it easier to see which counters
are incrementing), use the command clear controllers np counters all location node-id.
RP/0/RSP0/CPU0:router# show controllers np counters all
Fri Oct 29 10:49:57.377 DST
Node: 0/0/CPU0:
----------------------------------------------------------------
9-186
OL-23591-02
Chapter 9

Show global stats counters for NP0, revision v3

Offset Counter
FrameValue
Rate (pps)
189232
0
165012
0
175313
0
4158
0
48244
0
160848
0
10804
0
4158
0
111 DIAGS
24024
0
223 PUNT_STATISTICS
1193133
1
224 PUNT_STATISTICS_EXCD
1
0
225 PUNT_DIAGS_RSP_ACT
24220
0
468 RESOLVE_MAC_NOTIFY_CTRL_DROP_CNT
160854
0
600 PARSE_FAB_MACN_RECEIVE_CNT
160853
0
601 PARSE_FAB_DEST_MACN_RECEIVE_CNT
1
0
--More--
This example shows that L2VPN packets are being forwarded on the interface and subinterface (if
applicable).
RP/0/RSP0/CPU0:router# show running-config l2vpn
l2vpn
bridge group BG
bridge-domain BD1
interface TenGigE0/1/0/0.0
!
!
!
neighbor 10.100.1.1 pw-id 2
!
!
!
!
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface Te0/1/0/0.0 detail loc 0/1/cpu0
Local interface: TenGigE0/1/0/0.0, Xconnect id: 0x440003, Status: up
Segment 1
AC, TenGigE0/1/0/0.0, status: Bound
Statistics:
packets: received 55749484, sent 1
bytes: received 3567966976, sent 42
packets dropped: PLU 0, tail 0
bytes dropped: PLU 0, tail 0
Segment 2
Bridge id: 0, Split horizon group id: 0
Storm control: disabled
MAC learning: enabled
MAC port down flush: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: none
MAC limit reached: no
MAC Secure: disabled, Logging: disabled
DHCPv4 snooping: profile not known on this node, disabled
Dynamic ARP Inspection: disabled, Logging: disabled
OL-23591-02
9-187
Chapter 9
IP Source Guard: disabled, Logging: disabled

IGMP snooping profile: profile not known on this node
Router guard disabled
.
.
.
Xconnect id: 0xfffc0001, Status: down
Segment 1
MPLS, Destination address: 210.100.1.1, pw-id: 2, status: Not bound
Pseudowire label: UNKNOWN
Control word disabled
Statistics:
packets dropped: PLU 0, tail 0, out of order 0
bytes dropped: PLU 0, tail 0, out of order 0
Segment 2
Flooding:
This example displays detailed tag information for multiple subinterfaces.

RP/0/0/CPU0:router# show ethernet tags
St:
AD - Administratively Down, Dn - Down, Up - Up
Ly:
L2 - Switched layer 2 service, L3 = Terminated layer 3 service,
Xtra
C - Match on Cos, E - Match on Ethertype, M - Match on source MAC
-,+:
Ingress rewrite operation; number of tags to pop and push respectively
Interface
Gi0/0/0/0.1
Gi0/0/0/0.2
St
Up
Up
MTU Ly Outer
1518 L2 .1Q:10
1522 L2 .1Q:10
Inner
.1Q:20
Xtra -,+
0 0
0 0
This example shows the configuration and query of the Ethernet tags.
RP/0/RSP0/CPU0:router# show run interface gig0/0/0/0.1
Thu Oct 14 08:57:16.831 EDT
!
RP/0/RSP0/CPU0:router# show ethernet tags gigabitEthernet 0/0/0/0.1 detail location
0/0/CPU0
GigabitEthernet0/0/0/0.1 is up, service is L2
Interface MTU is 1518, switched L2 MTU is 1518
Outer Match: Dot1Q VLAN 1
Local traffic encap: Dot1Q VLAN 1
Pop 0 tags, push none
In this example, 0.2 is listed before 0.1. Any traffic with outer VLAN .1Q 10, and inner tag .1Q 20 would
match Gi0/0/0/0.2.
9-188
OL-23591-02
Chapter 9

RP/0/0/CPU0:router# show ethernet tags match-order

St:
AD - Administratively Down, Dn - Down, Up - Up
Ly:
L2 - Switched layer 2 service, L3 = Terminated layer 3 service,
Xtra
C - Match on Cos, E - Match on Ethertype, M - Match on source MAC
-,+:
Ingress rewrite operation; number of tags to pop and push respectively
Interface
Gi0/0/0/0.2
Gi0/0/0/0.1
St
Up
Up
MTU Ly Outer
1522 L2 .1Q:10
1518 L2 .1Q:10
Inner
.1Q:20
-
Xtra -,+
0 0
0 0
This example displays the VFI statistics.

Bridge group: 189, bridge-domain: 189, id: 0, state: up, ShgId: 0, MSTi: 0
MAC withdraw: enabled
MAC withdraw for Access PW: enabled
Flooding:
MAC limit: 4000, Action: none, Notification: syslog
Split Horizon Group: none
DHCPv4 snooping: disabled
IGMP Snooping profile: none
Bridge MTU: 9000
MIB cvplsConfigIndex: 1
Filter MAC addresses:
Create time: 22/09/2010 04:16:14 (2w4d ago)
No status change since creation
ACs: 2 (2 up), VFIs: 0, PWs: 0 (0 up), PBBs: 0 (0 up)
List of ACs:
AC: GigabitEthernet0/1/0/3.189, state is up
.
.
.
List of VFIs:
VFI 190
PW: neighbor 10.19.19.19, PW ID 190, state is up ( established )
PW class Use_Tu-44190, XC ID 0xfffc0003
Encapsulation MPLS, protocol LDP
PW type Ethernet, control word disabled, interworking none
PW backup disable delay 0 sec
Sequencing not set
Preferred path
MPLS
-----------Label
Group ID
Interface
MTU
Control word
PW type
VCCV CV type
tunnel TE 44190, fallback disabled

Local
Remote
------------------------------ ------------------------16002
101
0x1
0x0
190
unknown
1998
1998
disabled
disabled
Ethernet
Ethernet
0x2
0x6
(LSP ping verification)
(BFD PW FD only)
VCCV CC type 0x6
0x6
OL-23591-02
9-189
Chapter 9
Troubleshooting Multipoint Layer 2 Services
(router alert label)

(TTL expiry)
------------ -----------------------------MIB cpwVcIndex: 4294705155
Last time status changed: 22/09/2010 04:21:04
MAC withdraw message: send 0 receive 0
Static MAC addresses:
Statistics:
VFI Statistics:
drops: illegal VLAN 0, illegal length 0

(TTL expiry)
-------------------------
(2w4d ago)
This example shows how to set up an encapsulation default subinterface. in this scenario, you expect
incoming traffic on gig0/1/0/1 to be all single-tagged dot1q 100. However, you see some occasional
traffic with other encapsulations being dropped. These drops could be due to a few stray packets (for
example dot1q 200), and they are dropped without being processed on gig0/1/0/1; the
UIDB_TCAM_MISS_AGG_DROP counter is incremented. You can configure one default subinterface
to catch all the stray packets. Then the drops appear as counters on this isolated default interface, not as
UIDB_TCAM_MISS_AGG_DROP on the main interface.
interface gig0/1/0/1
mtu 1500
!
interface gig0/1/0/1.1 l2transport
!
interface gig0/1/0/1.2 l2transport
encapsulation default
!
<=== encapsulation default

This section explains how to troubleshoot multipoint Layer 2 services, and includes these topics:
Basic Bridging: Example, page 9-190
Verifying MAC Address Updates, page 9-192
Troubleshooting Multipoint Layer 2 Bridging Services (VPLS), page 9-195
Troubleshooting Bridge Domains That Use BGP-AD, page 9-201
Basic Bridging: Example

Figure 9-1 shows an example of a bridge domain configuration. The configuration commands are listed
below the drawing. Make sure that your own configuration is consistent with the applicable CLI structure
and syntax shown in this example.
9-190
OL-23591-02
Chapter 9

Figure 9-1
gig0/1/0/1
Example of Bridge Domain Configuration
Router
gig0/5/0/8
bundle-ether1
Bridge port 1
gig0/1/0/1
Bridge port 3
bundle-ether1.1
gig0/2/0/2
Bridge port 2
gig0/2/0/2.2
Bridge domain mybd
gig0/5/0/9
EFPs
255023
EFPs
The configuration for Figure 9-1 is as follows.

l2transport
!
!
!
bundle id 1 mode active
!
bundle id 1 mode active
!
!
!
l2vpn
bridge group bg_example
bridge-domain mybd
!
!
interface Bundle-ether1.1
!
!
!
!
Use show commands to display the status of the network.

Step 1
Verify that bundle members Gig0/5/0/8 and Gig0/5/0/9 are both Active, that is, that Link Aggregation
Control Protocol (LACP) indicates that they are connected with their adjacent neighbors.
RP/0/RSP0/CPU0:router# show bundle bundle-ether1
OL-23591-02
9-191
Chapter 9
Note
For additional information on troubleshooting bundles and LACP, see the Troubleshooting Problems
with Link Bundles section on page 5-118.
Step 2
Follow the steps in the Troubleshooting VLAN Traffic and L2 TCAM Classification section on
page 9-181 for the ACsGig0/1/0/1, Gig0/2/0/2, and Bundle-ether1.1.
Step 3
Display the bridge domain running configuration and ensure that it contains the appropriate commands
for your network.
RP/0/RSP0/CPU0:router# show run l2vpn bridge group bg_example
Step 4
Verify that the bridge domain, bridge ports, and ACs are all in Up state.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name mybd
Step 5
View additional details of the bridge domain, such as the feature settings and verify they are as expected.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name mybd detail
Verifying MAC Address Updates

This section explains how to determine whether MAC addresses are being flooded, learned and updated,
which are all prerequisites for traffic to be switched properly on the bridge domain. Even if traffic is
flowing, you need to verify that the system is continuing to flood, learn, and update MAC address
information appropriately.
You can track MAC learning on a specific MAC address for a node that could be several hops away. This
information helps you evaluate the health of the network:
Determine whether a source MAC address been learned on a specific bridge domain.
Determine the specific bridge port on which the source MAC address was learned (either a PW or
an AC), and provide information about the status of that bridge port.
View the age timer on the learned MAC address, which is a statistic on the traffic stream. The system
periodically checks that it is updating learned MAC addresses, and, if it is updating MAC addresses
successfully, the system restarts the age timer at the initial value (0). This reset occurs at the half-age
time, and the system sends a MAC update notification. If the configured maximum time elapses
(default 5 minutes) without an update, the MAC address ages out, which means there is no
communication and traffic is not getting through.
To find out whether a MAC address is being learned, monitor the age repeatedly, for example, every
10 seconds for five iterations. If the MAC age continues to increment beyond the half-age time, it
means there is no traffic flowing during the time you monitored it.
Step 1
Display the MAC address table for the bridge domain. Verify that MAC addresses are being learned and
resynced. Include the specific bridge domain and MAC address of interest, so the output will display the
specific bridge-port (AC or PW) on which the specific MAC address was learned.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain
mac-address mac-address-id location node-id
If the MAC address was learned on a PW, the output shows the IP address of the neighbor. Otherwise it
shows the MAC address of the AC.
9-192
OL-23591-02
Chapter 9

A bridge domain is an entity that exists on multiple LCs. However, the show command singles out one
LC. If the MAC address was learned on a bridge-port on a different LC, the display output reports the
LC on which it was learnednot the actual bridge-port. To get the bridge-port data, rerun the command
on the actual LC on which it was learned.
Step 2
(Optional) As an alternative to the procedure in Step 1, you can run a more general command without
specifying a specific bridge domain or MAC address. However, the output could flood your terminal
screen.
Caution
Before you run this command without specifying a particular bridge domain and MAC address, take
steps to limit the amount of data that can be output on your terminal screen. Otherwise the amount of
output could be extremely large.
This command displays all the MAC addresses learned on all bridge domains. As a safety mechanism,
before you enter this command, set your terminal length, for example:
RP/0/RSP0/CPU0:router# term length 20
If you need the full display, direct the output to a file, for example:
RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location node-id
Step 3
Display the MAC table for the bridge domain and verify that the MAC address has been learned. Notice
the bridge port (the same as the attachment circuit [AC]) from which the MAC address was learned, and
whether it was learned through a pseudowire (PW).
Caution
Before you run this command without specifying a MAC address ID, take steps to limit the amount of
data that can be output on your terminal screen. Otherwise the amount of output could be extremely
large.
This command displays all the MAC addresses learned on a bridge domain. As a safety mechanism,
One other approach to limit the output is to run the command with a pipe filter and CTRL-C after you
see the output you want.
mac-address detail location node-id [ | begin GigabitEthernet interface-id ]
Step 4
Use the following command to display the data for a specific bridge domain and MAC address.
mac-address mac-address detail location node-id
Example
OL-23591-02
9-193
Chapter 9
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location 0/6/CPU0

Mac Address
Type
Learned from/Filtered on
LC learned Resync Age
Mapped to
----------------------------------------------------------------------------------------0000.0001.0101 dynamic Gi0/6/0/1.1
0/6/CPU0
0d 0h 1m 59s
N/A
0000.0001.0102 dynamic Gi0/6/0/1.1
0/6/CPU0
0d 0h 1m 59s
N/A
0000.0002.0202 dynamic (192.0.2.20, 1:101)
0/6/CPU0
0d 0h 1m 59s
N/A
0000.0003.0303 dynamic (192.0.2.40, 1:101)
0/6/CPU0
0d 0h 1m 59s
N/A
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain test:test mac-address

0000.9999.9999 detail location 0/5/CPU0
Bridge-domain name: test:test, id: 0, state: up
Flooding:
Number of bridge ports: 2
Number of MAC addresses: 1
GigabitEthernet0/5/0/17.60, state: oper up

Number of MAC: 1
Mac Address: 0000.9999.9999, LC learned: 0/5/CPU0 <<< MAC is learned
Age: 0d 0h 0m 7s, Flag: local
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain testgrp:testbr mac-address

0000.8888.8888 detail location 0/5/cpu0
Bridge-domain name: testgrp:testbr, id: 0, state: up MAC learning: enabled
Flooding:
MAC aging time: 300 s, Type: inactivity MAC limit: 4000, Action: none, Notification:
syslog MAC limit reached: no
Security: disabled
DHCPv4 snooping: profile not known on this node IGMP snooping: disabled, flooding:
disabled Bridge MTU: 1500 bytes Number of bridge ports: 2 Number of MAC addresses: 2
Multi-spanning tree instance: 0
Number of MAC: 1
Sent(Packets/Bytes): 8000/800000
Received(Packets/Bytes): 27000/2700000
Storm control drop counters:
Broadcast(Packets/Bytes): 0/0
Multicast(Packets/Bytes): 0/0
Unknown unicast(Packets/Bytes): 0/0
Nbor 8.8.8.8 pw-id 98 <<< MAC is learned on a pseudowire
Number of MAC: 1
Sent(Packets/Bytes): 27000/2592000
Received(Packets/Bytes): 8000/768000
Broadcast(Packets/Bytes): 0/0
Multicast(Packets/Bytes): 0/0
Unknown unicast(Packets/Bytes): 0/0 Mac Address: 0000.8888.8888, LC learned:
0/5/CPU0
Age: 0d 0h 0m 10s, Flag: local
9-194
OL-23591-02
Chapter 9

Troubleshooting Multipoint Layer 2 Bridging Services (VPLS)

This section provides information on troubleshooting multipoint Layer 2 bridging services, also called
virtual private LAN services (VPLS) on the Cisco ASR 9000 Aggregation Services Router. VPLS
enables geographically separated local-area network (LAN) segments to be interconnected as a single
bridged domain over an MPLS network and provides transparent multipoint Layer 2 connectivity for
customers.
This section contains the following topics:
Understanding VPLS Architecture, page 9-195
Verifying MPLS PIE Activation, MPLS Configuration, and MPLS Connectivity, page 9-196
Procedure for Troubleshooting Multipoint Layer 2 Services, page 9-196
Example of Point-To-Point Layer 2 Deployment, page 9-206
Understanding VPLS Architecture

The VPLS architecture allows end-to-end connection between provider edge (PE) routers, which
supports delivery of multipoint Ethernet services. Without VPLS, end-to-end connectivity between PE
routers is achieved by creating a full-mesh of real connections between each PE router. With VPLS, as
shown in Figure 9-2, the full mesh of real connections is replaced by a full mesh of virtual (pseudowire)
connections. In this example, the interconnections between the network provider edge (N-PE) nodes are
made by means of pseudowires (PWs) through an IP/MPLS core network. The PWs can be created either
through manual configuration or autodiscovery.
Figure 9-2 is a partial implementation of a VPLS architecture. In a full VPLS architercture (not shown
here), the full mesh of pseudowires is replaced by a combination of pseudowires and one or more bridge
domains in the P core network. Each PE router would have a single PW connecting the router to a
P router in the core. This core P router would have a bridge domain, and this bridge domain would
terminate all PE router PWs. This would replace the full mesh of Figure 9-2 with a hub-and-spoke, the
hub being the bridge domain in the P router.
Example of VPLS Architecture with Pseudowires in MPLS Core
CE
N-PE
Ethernet
(VLAN/Port/EFP)
Attachment circuit
MPLS Core
Full Mesh PWs + LDP
N-PE
CE
Ethernet
(VLAN/Port/EFP)
Attachment circuit
208684
Figure 9-2
The VPLS network requires the creation of a bridge domain (Layer 2 broadcast domain) on each of the
PE routers. The VPLS PE device holds all the VPLS forwarding MAC tables and bridge domain
information. In addition, it is responsible for all flooding broadcast frames and multicast replications.
OL-23591-02
9-195
Chapter 9
Verifying MPLS PIE Activation, MPLS Configuration, and MPLS Connectivity

This section is applicable to operation of multipoint Layer 2 services over PWs. For PWs to function,
the MPLS PIE must be active and MPLS must be present in your running configuration:
Caution
Verify that the MPLS PIE is installed, committed, and activated. It is not installed by default.
Verify that MPLS is configured in your running-config. After you install the MPLS PIE, you must
commit it. If you configure MPLS but you have not committed the MPLS PIE, the system deletes
all of your MPLS configuration if you reload the router image.
Verify that the MPLS PIE is committed before you configure MPLS. Otherwise all of your MPLS
configuration data will be lost if the image is reloaded.
PWs operate over the MPLS network, therefore, MPLS connectivity is a prerequite for bringing up a PW.
To verify MPLS connectivity, see the Troubleshooting Connectivity Over MPLS section on
page 8-174.
Procedure for Troubleshooting Multipoint Layer 2 Services

Perform these steps if you are having connectivity problems with Layer 2 multipoint services.
Step 1
Check for the following underlying problems, which can cause failure of the multipoint Layer 2 services.
The bridge domain uses an attachment circuit (AC) for which the interfaces have not been created.
The AC interface for the bridge domain is operationally down.
The AC interface for the bridge domain is administratively down.
The AC is not configured as Layer 2 (the l2transport keyword is missing from the configuration
command).
The traffic on the AC interface is not classified properly (wrong encapsulation statement).
There is an MTU mismatch between the local and remote routers.
Step 2
Verify that you can ping the opposite interface (on the remote router) from the MPLS interface.
Step 3
Verify that the remote interface shows up as an ospf neighbor.

show ospf neighbor
Step 4
Verify that the remote router ID, typically the remote router loopback, is in the routing table.
show route ipv4
Step 5
Ping the remote router with the same IP address that is used for the PW (ping x.x.x.x).
Step 6
Verify that you can find the remote router ID in an MPLS command. It should be the ipv4 address for
the PW.
Step 7
Verify that the BGP neighbor is up. (This step is necessary only if BGP autodiscovery has been
configured.)
show bgp neighbors
Step 8
Verify that the VFI is advertized in both PEs, and that PWs are established.
show l2vpn bridge-domain [brief | detail]
9-196
OL-23591-02
Chapter 9

Step 9
Check local and remote labels.

show mpls forwarding [labels]
show l2vpn forwarding detail location
Step 10
Verify that PWs are bound in the Layer 2 forwarding information base (L2FIB) with the proper
cross-connect ID.
show l2vpn forwarding detail location
Step 11
Verify that NLRIs are received and PWs created.

show l2vpn discovery [summary]
Example
The following example shows that autodiscovery is on, the PW is up, and NLRIs have been received from
the peer router. Check the cross-connect ID. Check the local and remote label and compare with the label
binding in the MPLS label switching database (LSD) by means of the show mpls forwarding command.
In this example, the local MPLS label ID is 16005.
Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, state: up, ShgId: 0, MSTi: 0
Flooding:
Security: disabled
Bridge MTU: 1500
List of ACs:
Type VLAN; Num Ranges: 1
VLAN ranges: [2, 2]
MTU 1504; XC ID 0x2040001; interworking none
Flooding:
Security: disabled
Storm Control: disabled
Statistics:
OL-23591-02
9-197
Chapter 9

packets: broadcast 0, multicast 0, unknown unicast 0
bytes: broadcast 0, multicast 0, unknown unicast 0
List of Access PWs:
List of VFIs:
VFI bg1_bd1_vfi
VPN-ID: 101, Auto Discovery: <<< BGP is provisioned, service is connected,
autodiscovery is on, and IP address is advertized
Route Distinguisher: 101:1
Import Route Targets:
101:1
Export Route Targets:
101:1
Signaling protocol: LDP
AS Number: 1
VPLS-ID: 1:101
L2VPN Router ID: 10.10.10.10
PW: neighbor 10.20.20.20, PW ID 1:101, state is up ( established ) <<< PW is up
PW class not set, XC ID 0xfffc0001 <<< cross-connect ID
Encapsulation MPLS, Auto-discovered (BGP), protocol LDP
Sequencing not set
MPLS
Local
Remote
------------ ------------------------------ ------------------------Label
16005
16006
<<< Local and remote labels have been received, which indicates that signaling is up. The
local MPLS label is 16005.
BGP Peer ID 10.10.10.10
10.20.20.20
<<< Received the NLRI from the BGP peer, which means the PW is established.
LDP ID
10.10.10.10
10.20.20.20
AII
10.10.10.10
10.20.20.20
AGI
1:101
1:101
Group ID
0x0
0x0
Interface
bg1_bd1_vfi
bg1_bd1_vfi
MTU
1500
1500
disabled
PW type
Ethernet
Ethernet
VCCV CV type 0x2
0x2
VCCV CC type 0x6
0x6
(TTL expiry)
(TTL expiry)
------------ ------------------------------ ------------------------MIB cpwVcIndex: 1
Create time: 14/04/2010 23:10:51 (00:37:19 ago)
Last time status changed: 14/04/2010 23:10:56 (00:37:14 ago)
Statistics:
RP/0/RSP0/CPU0:router# show mpls forwarding
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------16000 Pop
10.20.20.20/32
Gi0/6/0/21
10.0.0.2
226000292
16001 Pop
10.30.30.30/32
Gi0/6/0/3
10.0.0.2
0
16002 Pop
10.0.0.0/24
Gi0/6/0/3
10.0.0.2
0
16003 16003
10.40.40.40/32
Gi0/6/0/3
10.0.0.2
226000620
16004 Unlabelled 10.0.1.253/32
Mg0/RSP0/CPU0/0 10.2.0.4
0
16005 Pop
PW(10.20.20.20:2814754062073957)
\ <<< The local MPLS label is 16005.
9-198
OL-23591-02
Chapter 9

16006
Pop
BD=0
point2point
PW(10.40.40.40:2814754062073957)
\
BD=0
point2point
214700000
214700000
This example displays the L2VPN fowarding parameters.

RP/0/RSP0/CPU0:router# show running-config l2vpn
l2vpn
bridge group BG
bridge-domain BD1
!
!
!
!
!
!
!
RP/0/RSP0/CPU0:router# show l2vpn forwarding detail location 0/1/CPU0
Segment 1
Statistics:
Segment 2
Flooding:
Segment 1
Statistics:
Segment 2
Flooding:
OL-23591-02
9-199
Chapter 9

Segment 1
Statistics:
Segment 2
Flooding:
Xconnect id: 0xfffc0001, Status: down
Segment 1
MPLS, Destination address: 210.100.1.1, pw-id: 2, status: Not bound
Pseudowire label: UNKNOWN
Statistics:
packets dropped: PLU 0, tail 0, out of order 0
bytes dropped: PLU 0, tail 0, out of order 0
Segment 2
Flooding:
The following example shows that BGP is connected and active, and that there are VPNs and NLRIs on
the bridge domain.
9-200
OL-23591-02
Chapter 9

RP/0/RSP0/CPU0:router# show l2vpn discovery summary

BGP: connected=yes, active=yes, stdby=yes
Services
Bridge domain: registered=yes, Num VPNs=1
Num Local Edges=1, Num Remote Edges=2, Num Received NLRIs=2
Xconnect: registered=yes, Num VPNs=0
The following example shows that the local router ID is advertised and that NLRIs are recieved from the
remote peers.
RP/0/RSP0/CPU0:router# show l2vpn discovery
Service Type: VPLS, Connected
List of VPNs (1 VPNs):
Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, signaling protocol: LDP
VPLS-ID: 1:101
Local L2 router id: 10.10.10.10 <<< advertised
List of Remote NLRI (2 NLRIs): <<< NLRIs received from the remote peer address
Local Addr
Remote Addr
Remote L2 RID
Time Created
--------------- --------------- --------------- ------------------10.10.10.10
10.20.20.20
10.20.20.20
04/14/2010 23:10:51
10.10.10.10
10.40.40.40
10.40.40.40
04/14/2010 23:19:06
Troubleshooting Bridge Domains That Use BGP-AD

Perform this procedure to verify that the configuration is correct for the features you are troubleshooting.
In addition, run this procedure on all peers in the VPLS domain. (For peers that are not ASR 9000 nodes,
run a procedure similar to this one to check the running configurations.)
For detailed configuration procedures, see the Cisco ASR 9000 Series Aggregation Services Routers
Configuration Guides.
Step 1
Verify the configuration of BGP autodiscovery with LDP signaling.

a.
Configure Loopback and Links with IP addresses.
b.
Configure IGP (OSPF or ISIS)
c.
Configure LDP
d.
Configure BGP
e.
Configure L2VPN (VPLS)
Example
####Sample Configuration from WEST:
####CONFIGURE LOOPBACKs and Links
Interface loopback0
Ipv4 address 10.10.10.10 255.255.255.255
!
Interface gig0/6/0/1.1 l2transport
Description Attachment Circuit connected to Customer site
Encapsulation dot1q 2
!
Interface gig0/6/0/21
Description Connected to EAST Node
Ipv4 address 10.0.0.1 255.255.255.0
!
OL-23591-02
9-201
Chapter 9
Interface gig0/6/0/3
Description Connected to CENTRAL Node
Ipv4 address 192.0.2.1 255.255.255.0
!
####CONFIGURE IGP
Router ospf 1
Router-id 10.10.10.10
Nsr
Nsf cisco
Area 0
interface loopback0
####CONFIGURE MPLS LDP
Mpls ldp
graceful-restart
log neighbor
router-id 10.10.10.10
####CONFIGURE BGP
Router bgp 1
bgp router-id 10.10.10.10
bgp graceful-restart
address-family l2vpn vpls-vpws <<< This shows you have configured this family in BGP so
it will be able to handle the discovery of the neighbor.
!
neighbor 192.0.2.20
remote-as 1
update-source loopback0
address-family l2vpn vpls-vpws
neighbor 172.30.30.30
remote-as 1
update-source loopback0
address-family l2vpn vpls-vpws
####CONFIGURE L2VPN
l2vpn
bridge group bg1
bridge-domain bg1_bd1
interface gig0/6/0/1.1
!
vfi bg1_bd1_vfi
vpn-id 101
autodiscovery bgp
rd 101:1
route-target 101:1
signaling-protocol ldp
vpls-id 1:101
Step 2
Verify the configuration of L2VPN parameters.

a.
show l2vpn atom-db
b.
show l2vpn discovery summary
c.
show l2vpn discovery
d.
show l2vpn bridge-domain
9-202
OL-23591-02
Chapter 9

e.
show l2vpn bridge-domain brief
f.
show l2vpn bridge-domain detail
Example
RP/0/RSP0/CPU0:router# show l2vpn atom-db
Wed Apr 14 23:28:41.905 EDT
Peer ID
VC ID
Encap
Signaling
FEC
Discovery
____________________________________________________________________________
192.0.2.20
192.168.40.40
1:101
1:101
MPLS
MPLS
LDP
LDP
129
129
BGP
BGP
RP/0/RSP0/CPU0:router# show l2vpn discovery summary

Wed Apr 14 23:24:46.156 EDT
BGP: connected=yes, active=yes, stdby=yes
Services
Bridge domain: registered=yes, Num VPNs=1
Xconnect: registered=yes, Num VPNs=0

Wed Apr 14 23:23:00.513 EDT
VPLS-ID: 1:101
Local L2 router id: 10.10.10.10 <<< advertized
List of Remote NLRI (2 NLRIs): <<< NLRIs received from those remote peer addresses
Local Addr
Remote Addr
Remote L2 RID
Time Created
--------------- --------------- --------------- ------------------10.10.10.10
192.0.2.20
192.0.2.20
04/14/2010 23:10:51
10.10.10.10
192.168.40.40
192.168.40.40
04/14/2010 23:19:06
Wed Apr 14 23:46:37.190 EDT
Aging: 300 s, MAC limit: 4000, Action: none, Notification: syslog
Filter MAC addresses: 0
List of ACs:
Gi0/6/0/1.1, state: up, Static MAC addresses: 0
List of Access PWs:
List of VFIs:
VFI bg1_bd1_vfi
Neighbor 192.0.2.20 pw-id 1:101, state: up, Static MAC addresses: 0
Neighbor 192.168.40.40 pw-id 1:101, state: up, Static MAC addresses: 0
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain brief

Wed Apr 14 23:47:42.003 EDT
Bridge Group/Bridge-Domain Name ID
State
Num ACs/up
Num PWs/up
-------------------------------- ----- ---------- -------------- -------------bg1/bg1_bd1
0
up
1/1
2/2
OL-23591-02
9-203
Chapter 9

Wed Apr 14 23:48:11.152 EDT
Flooding:
Security: disabled
Bridge MTU: 1500
List of ACs:
Type VLAN; Num Ranges: 1
VLAN ranges: [2, 2]
MTU 1504; XC ID 0x2040001; interworking none
Flooding:
Security: disabled
Storm Control: disabled
Statistics:
List of Access PWs:
List of VFIs:
VFI bg1_bd1_vfi
VPN-ID: 101, Auto Discovery: BGP, state is Provisioned (Service Connected) <<< It is
Advertized
Route Distinguisher: 101:1
Import Route Targets:
101:1
Export Route Targets:
101:1
AS Number: 1
VPLS-ID: 1:101
PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established ) <<< PW is up
PW class not set, XC ID 0xfffc0001
9-204
OL-23591-02
Chapter 9

Sequencing not set

MPLS
Local
Remote
------------ ------------------------------ ------------------------Label
16005
16006 <<< local and remote labels
have been received, which means the signaling is up.
BGP Peer ID 10.10.10.10
192.0.2.20 <<< Received the NLRI,
which means the PW is established.
LDP ID
10.10.10.10
192.0.2.20
AII
10.10.10.10
192.0.2.20
AGI
1:101
1:101
Group ID
0x0
0x0
Interface
bg1_bd1_vfi
bg1_bd1_vfi
MTU
1500
1500
disabled
PW type
Ethernet
Ethernet
VCCV CV type 0x2
0x2
VCCV CC type 0x6
0x6
(TTL expiry)
(TTL expiry)
------------ ------------------------------ ------------------------MIB cpwVcIndex: 1
Create time: 14/04/2010 23:10:51 (00:37:19 ago)
Last time status changed: 14/04/2010 23:10:56 (00:37:14 ago)
Statistics:
Step 3
Verify the configuration of MPLS forwarding and Label Switching Database (LSD) parameters.
a.
show mpls forwarding
b.
show mpls lsd forwarding
Example
Wed Apr 14 23:41:49.325 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------16000 Pop
192.0.2.20/32
Gi0/6/0/21
10.0.0.2
226000292
16001 Pop
172.30.30.30/32
Gi0/6/0/3
192.0.2.2
0
16002 Pop
172.16.0/24
Gi0/6/0/3
192.0.2.2
0
16003 16003
192.168.40.40/32
Gi0/6/0/3
192.0.2.2
226000620
16004 Unlabelled 10.0.1.253/32
Mg0/RSP0/CPU0/0 10.2.0.4
0
16005 Pop
PW(192.0.2.20:2814754062073957)
\ <<< PW has label and traffic is
running
BD=0
point2point
214700000
16006 Pop
PW(192.168.40.40:2814754062073957)
\ <<< PW has label and traffic is
running
BD=0
point2point
214700000
RP/0/RSP0/CPU0:router# show mpls lsd forwarding

Wed Apr 14 23:42:12.259 EDT
In_Label, (ID), Path_Info: <Type>
16000, (IPv4, 'default':4U, 192.0.2.20/32), 1 Paths
1/1: IPv4, 'default':4U, Gi0/6/0/21, nh=10.0.0.2, lbl=3, tun_id=0 flags=(RETAIN)
OL-23591-02
9-205
Chapter 9
Troubleshooting Point-to-Point Layer 2 Services

1/1: IPv4, 'default':4U, Null, nh=10.2.0.4, lbl=None, tun_id=0 flags=()
16005, (PW, (192.0.2.20:2814754062073957)), 1 Paths
1/1: PW, bridge_id=0, shg_id=1, xc_id=0xfffc0001, f=0x4, lbl=Pop-PW-Ether [Attached]
16006, (PW, (192.168.40.40:2814754062073957)), 1 Paths

This section provides information on troubleshooting point-to-point Layer 2 services. It contains the
following subsections:
Example of Point-To-Point Layer 2 Deployment, page 9-206
AC Is Down, page 9-218
Pseudowire Is Down, page 9-219
VPWS Not Forwarding Traffic from AC to Pseudowire, page 9-212
Pseudowire Up but Ping Fails, page 9-213
Traffic Loss During RSP Fail Over, page 9-213
Preferred Path Not Working, page 9-214
Example of Point-To-Point Layer 2 Deployment

This section contains an example of a point-to-point Layer 2 deployment involving a router with a bridge
domain on one side of the network and a router with a cross-connect on the other. The two routers are
connected by a PW. The PW is a virtual point-to-point connection between the two routers. As shown in
Figure 9-3, the traffic for the PW (the virtual connection between Routers 1 and 2) passes through
Router3, but Routers 1 and 2 behave as if they are directly connected over the PW.
9-206
OL-23591-02
Chapter 9

Figure 9-3
Example of Deployment with Bridge Domain and XConnect Joined by Pseudowire

Pseudowire
gig0/2/0/1
gig0/1/0/1
Bridge port #1
gig0/1/0/1.5
xconnect
gig0/2/0/1.7
gig0/1/0/2
gig0/1/0/3
Bridge port #2
gig0/1/0/2.6
Bridge domain
mybd
gig0/2/0/2
gig0/3/0/1
gig0/3/0/2
Router3
Router2
MPLS/OSPF router ID
(loopback) 10.2.2.2
281922
Router1
MPLS/OSPF router ID
(loopback) 10.1.1.1
Figure 9-3 Notes

Router1 has a bridge domain (mybd) with three bridge portsTwo ACs and one PW:
(AC/EFP) gig0/1/0/1.5
(AC/EFP) gig0/1/0/2.6
(PW) neighbor 10.2.2.2 pw-id 1
Router2 has an xconnect. The xconnect has two membersOne AC and one PW. The xconect is
represented by the dotted line inside the Router2 box. The dotted line also includes the AC/EFP:
(AC/EFP) gig0/2/0/1.7
(PW) neighbor 10.1.1.1 pw-id 1
The PW is represented by the circles (one in Router1 and one in Router2) connected by a dotted line. It
is a virtual point-to-point connection from Router1 to Router2. In reality, the traffic for the PW passes
through Router3, but Router1 and Router2 behave as if they are directly connected over the PW. The port
at the right of Router1 and the port at the left of Router2 are the MPLS connections to Router3.
The configurations for this deployment example are as follows. Make sure that your own configuration
is consistent with the applicable CLI structure and syntax shown in this example.
Router1
!
!
!
ipv4 address 10.0.13.1 255.255.255.0
!
interface Loopback0
ipv4 address 10.1.1.1 255.255.255.255
!
router ospf 1
OL-23591-02
9-207
Chapter 9
log adjacency changes

router-id 10.1.1.1
area 0
!
interface Loopback0
!
!
!
mpls ldp
router-id 10.1.1.1
log
neighbor
!
!
!
l2vpn
bridge group examples
bridge-domain mybd
!
!
!
!
!
!
Router2
!
!
ipv4 address 10.0.23.1 255.255.255.0
!
interface Loopback0
ipv4 address 10.2.2.2 255.255.255.255
!
router ospf 1
router-id 10.2.2.2
area 0
!
interface Loopback0
!
!
!
mpls ldp
router-id 10.2.2.2
log
neighbor
!
!
!
l2vpn
xconnect group examples
p2p myxc
9-208
OL-23591-02
Chapter 9

!
!
!
!
Router 3
ipv4 address 10.0.13.2 255.255.255.0
!
ipv4 address 10.0.23.2 255.255.255.0
!
interface Loopback0
ipv4 address 10.3.3.3 255.255.255.255
!
router ospf 1
router-id 10.3.3.3
area 0
!
!
interface Loopback0
!
!
!
mpls ldp
router-id 10.3.3.3
log
neighbor
!
!
!
!
Use the following procedure to locate any problems with traffic flow in this network. The IP addresses
are based on the sample configurations for Routers 1, 2, and 3 (above).
Step 1
Step 2
Verify ping connectivity over the MPLS links.
From Router1 gig0/1/0/3 to Router3 gig0/3/0/1ping 10.0.13.2
From Router2 gig0/2/0/2 to Router3 gig0/3/0/2ping 10.0.23.2
Verify that OSPF neighbor links are up on the links (the same links listed in Step 1).
Step 3
Verify that the Router1 routing table contains the loopback address of Router2 (10.2.2.2). Also verify
that the Router2 routing table contains the loopback address of Router1 (10.1.1.1).
Step 4
Verify that Router1 can ping the Router2 loopback address, and Router2 can ping the Router1 loopback
address.
From Router1ping 10.2.2.2
From Router2ping 10.1.1.1
OL-23591-02
9-209
Chapter 9
Step 5
Verify that MPLS neighbors are established in the links (the same links listed in Step 1).
RP/0/RSP0/CPU0:router# show mpls ldp neighbor
Step 6
Verify that Router1 has an MPLS label to reach the Router2 loopback address. Also verify that Router2
has an MPLS label to reach the Router1 loopback address.
Note
The output of this command contains one additional MPLS label. This additional label
represents the pseudowire between Router1 and Router2.
Step 7
Verify that the status of the Router1 bridge domain is UP, and that all all ACs are up.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain mybd
Step 8
Verify that the status of the Router1 PW is UP.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain mybd
Step 9
Verify that the Router2 xconnect is UP, and all ACs are UP.
RP/0/RSP0/CPU0:router# show l2vpn xconnect group examples
Step 10
Verify that the Router2 PW is UP.

RP/0/RSP0/CPU0:router# show l2vpn xconnect group examples

SUMMARY STEPS
1.
show l2vpn xconnect [detail | group | interface | neighbor | state | summary | type | state
unresolved]
2.
show l2vpn forwarding {detail | hardware | interface | location | message | resource | summary
| unresolved} location node-id
3.
show mpls forwarding [detail | {label label number} | interface interface-id | labels value |
location | prefix [network/mask | length] | summary | tunnels tunnel-id]
9-210
OL-23591-02
Chapter 9

DETAILED STEPS
Step 1
Command or Action
Purpose
show l2vpn xconnect [detail | group | interface

| neighbor | state | summary | type | state
unresolved]
View brief information on configured cross-connects. Filter

results using the following parameters and keywords:
Example:
detailDetailed information
groupAll cross-connects in a specified group
interfaceInterface and subinterface
neighborNeighbor
stateXconnect state types: up, down
summaryAC information from the AC Manager

database
typeXconnect types: ac-pw, locally switched
state unresolvedUnresolved cross-connects
RP/0/RSP0/CPU0:router# show l2vpn xconnect
Step 2
show l2vpn forwarding {detail | hardware |

interface | location | message | resource |
summary | unresolved} location node-id
View the matching AC subinterface.
Example:
RP/0/RSP0/CPU0:router# show l2vpn forwarding
location 0/2/cpu0
Step 3
show mpls forwarding [detail | {label label

number} | interface interface-id | labels value
| location | prefix [network/mask | length] |
summary | tunnels tunnel-id]
View the MPLS Label Forwarding Information Base (LFIB)

entries with a local labels range.
AC Is Down
Step 1
View the interface state.

Step 2
View the state of the xconnect.

RP/0/RSP0/CPU0:router# show l2vpn xconnect detail
Step 3
Ensure that the AC interface has l2transport configured.
Step 4
Ensure that the AC interface is up.
Step 5
Ensure that the MTUs match.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain interface type interface-name detail
OL-23591-02
9-211
Chapter 9
Pseudowire Is Down
Step 1
View the pseudowire state.

RP/0/RSP0/CPU0:router# show l2vpn xconnect neighbor
Step 2
On the MPLS-enabled interface that connects to the router at the remote end of the PW, view MPLS LDP
neighbor information. Check these conditions:
a.
Ensure that, if the MPLS router-id uses a loopback interface (it usually does), the loopback interface
is present in the OSPF configuration, so that a route to its address is advertised for the other router
to reach.
b.
Ensure that an LDP session is established with the PE peer.
RP/0/RSP0/CPU0:router# show mpls ldp neighbor neighbor
Step 3
Ensure that the MPLS infrastructure has allocated a label for the mpls-id IP address on the opposite
router, and an additional label for the PW tunnel itself.
Step 4
(Perform this step if the MPLS LSP does not come up.) On the MPLS-enabled interface that connects to
the router at the remote end of the PW, view OSPF neighbor information. Verify that the IP address of
the MPLS router ID is reachable:
a.
Ensure that this IP address appears in the routing table.
b.
Ping this IP address and verify that it replies successfully.
c.
Ensure that the PW ID (keyword "pw-id" in the configuration syntax) is identical on both ends of
the PW.
Step 5
Ensure that pseudowires are properly configured on both PEs.
Step 6
Ensure that the MPLS package is installed.
Step 7
Ensure that the core interface is up.
Step 8
Ensure that OSPF is the routing protocol.
Step 9

RP/0/RSP0/CPU0:router# show l2vpn xconnect neighbor
VPWS Not Forwarding Traffic from AC to Pseudowire

This section provides information on troubleshooting forwarding of traffic from the AC to the PW over
virtual private wire services (VPWS). VPWS connects to endpoints defined by physical interfaces or
subinterfaces by emulating a virtual wire between them using the underlying MPLS technology.
Step 1
View pseudowire hardware information.

RP/0/RSP0/CPU0:router# show l2vpn forwarding neighbor 192.168.12.5 pw-id 100 hardware
egress location node-id0
9-212
OL-23591-02
Chapter 9

Step 2
View the bridge information about Broadcast, Multicast and Unknown Unicast.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name 1 det
Step 3
Ensure that the MAC limit has not been exceeded.

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain 1:1 detail location
Step 4
Ensure that the pseudowires and AC are up.
Step 5
Verify that the hardware is programmed for both ACs.
Step 6
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEtherne0/5/0/2 hardware

ingress detail location node-id
Step 7
Verify that the hardware is programmed for pseudowires.
Pseudowire Up but Ping Fails

Step 1
View the bridge domain state.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name node-id detail
Step 2
Ensure that both CEs are on the same subnet.
Step 3
Step 4
Ensure that the end-to-end encapsulations match.
Traffic Loss
Step 1

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name
Step 2
bd-name-id detail
View segment counters to see if the packet and byte switched count increased.
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet node-id detail
location node-id
Step 3
Ensure that the bandwidth rates match between the CEs.
Traffic Loss During RSP Fail Over

When RSP fail over is performed, some times it is seen that the traffic loss is experienced. This may be
because the IGP over which the prefixes are learned is going down. The following assumes OSPF as the
IGP.
show process failoverView
process details during failover
OL-23591-02
9-213
Chapter 9
Troubleshooting Specific Outage Scenarios In Layer 2 Services
debug ospf haEnables
OSPF HA related debugs
debug ospf instance nsfBefore
show process failoverAfter
FO (Fail Over) and collect the debug log
FO
Step 1
One thing to check immediately is if the next hop router also experienced an FO mechanism (Similar to
what is done on this router). If so, the OSPF may go down.
Step 2
If not, verify that nsf cisco is configured under the OSPF. If nsf cisco is configured, see if the next
hop is reachable during FO. If not, there may be a reachability issue like a link going down or negotiation
problems.
Preferred Path Not Working

Step 1
View the state of the bridge domain.

Step 2
bd-name-id detail
View ingress UIDB.

RP/0/RSP0/CPU0:router# show l2vpn forwarding interface interface hardware ingress detail
location node-id

This section contains the following topics, which help you troubleshoot specific Layer 2 outages:
L2VPN Discovery Not Working, page 9-217
AC Is Down, page 9-218
Pseudowire Is Down, page 9-219
VPLS Not Forwarding Flooding Traffic, page 9-220
VPLS Not Forwarding Flooding Traffic from AC to Pseudowire, page 9-224
VPLS Not Forwarding Flooding Traffic from Pseudowire to AC, page 9-224
VPLS Not Forwarding Unicast Traffic from AC to AC, page 9-225
VPLS Not Forwarding Unicast Traffic from AC to Pseudowire, page 9-225
VPLS Not Forwarding Flooding Traffic from Pseudowire to AC, page 9-225
Pseudowire Up but Ping Fails, page 9-226
Pseudowire Flap Causing Traffic Loss, page 9-226
Traffic Loss During RSP Fail Over, page 9-227
Preferred Path Not Working, page 9-227
9-214
OL-23591-02
Chapter 9


SUMMARY STEPS
1.
show l2vpn bridge-domain summary
2.
show l2vpn bridge-domain [bd-name bridge-domain name | brief | detail | group bridge-domain
group name | interface {type interface-id} | neighbor IP address [pw-id value] | summary]
3.
4.
show l2vpn forwarding bridge-domain [ bridge-domain-name] {detail | hardware {egress |

ingress}} {location node-id}
DETAILED STEPS
Step 1
Command or Action
Purpose
View the bridge-domain bridge-ports, which will be

identified in the output as attachment circuits (ACs) and/or
pseudowires (PWs) as applicable.
Example:
summary
Verify that the bridge-domains, ACs, and PWs (as

applicble) are up.
Tip
Repeat this command periodically. Check that

traffic counts are going up over time on the PWs and
ACs in the bridge-domain.
OL-23591-02
9-215
Chapter 9
Step 2
Command or Action
Purpose
show l2vpn bridge-domain neighbor
Look for the status of any bridge-domains that might be

experiencing problems.
show l2vpn bridge-domain group group-id
bd-name bridge-domain name(Optional) Displays

the bridges by the bridge ID. The bridge-domain name
argument is used to name a bridge domain.
brief(Optional) Displays brief information about the

bridges.
detail(Optional) Displays the output for the Layer 2

VPN (L2VPN) to indicate whether or not the MAC
withdrawal feature is enabled and the number of MAC
withdrawal messages that are sent or received from the
pseudowire.
group bridge-domain group name(Optional)

Displays filter information on the bridge-domain group
name. The bridge-domain group name argument is used
to name the bridge domain group.
interface(Optional) Displays the filter information

for the interface on the bridge domain.
typeInterface type.
interface-idIdentifies a physical interface or a virtual

interface.
neighbor IP address(Optional) Displays only the

bridge domain that contains the pseudowires to match
the filter for the neighbor. The IP address argument is
used to configure IP address of the neighbor.
pw-id value(Optional) Displays the filter for the

pseudowire ID. The range is from 1 to 4294967295.
show l2vpn bridge-domain interface type node-id
Example:
neighbor
group 12
show l2vpn bridge-domain interface
gigabitethernet 0/1/0/5
Step 3
Example:
summary
View the BGP autodiscovery status and results. This display

shows the network layer reachability information (NLRI)
that has been sent by the local router and received from the
remote router.
Verify that BGP is active, and that the bridge domain and
cross-connect are registered.
9-216
OL-23591-02
Chapter 9

Step 4
Command or Action
Purpose
show l2vpn forwarding bridge-domain
View forwarding bridge domain information. Filter results

using the following parameters and keywords:
bridge-domain-name(Optional) Name of a bridge

domain.
detailDisplays all the detailed information on the

attachment circuits and pseudowires.
hardwareDisplays the hardware location entry.
egressReads information from the egress PSE.
ingressReads information from the ingress PSE.
location node-idDisplays the bridge-domain

information for the specified location.
Example:
bridge-domain ABC mac-address interface
Gi0/1/2/1.2 detail hardware location 0/4/CPU0
bridge
Step 5
show l2vpn forwarding bridge-domain detail

location
View the display to see which direction is experiencing a

traffic loss. If you have PWs in the core, the PWs should be
in the bound state and traffic should be flowing in the bound
PWs.
Example:
bridge-domain detail location 0/1/CPU0
L2VPN Discovery Not Working

Step 1
Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp).
Step 2
Check that the BGP output shows the remote prefix has been received (show bgp).
Step 3
Check L2VPN discovery to verify that the local router received the LDP NLRI update from the remote
VPLS router (show l2vpn discovery private).
Example
These examples show the output from the show bgp commands.
RP/0/RSP0/CPU0:router# show bgp l2vpn vpls
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network
Next Hop
Rcvd Label
Local Label
Route Distinguisher: 101:1 (default for vrf bg1:bg1_bd1)
*> 10.10.10.10/32
0.0.0.0
nolabel
nolabel
*>i192.0.2.20/32
192.0.2.20
nolabel
nolabel
*>i192.168.40.40/32
192.168.40.40
nolabel
nolabel
Processed 3 prefixes, 3 paths
RP/0/RSP0/CPU0:router# show bgp l2vpn vpls rd 101:1 192.168.40.40

Thu Apr 15 00:00:21.930 EDT
BGP routing table entry for 10280:10280/32, Route Distinguisher: 101:1
Versions:
Process
bRIB/RIB SendTblVer
Speaker
6
6
OL-23591-02
9-217
Chapter 9
Last Modified: Apr 14 23:19:06.805 for 00:41:15

Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Local
192.168.40.40 (metric 3) from 172.30.30.30 (192.168.40.40)
Origin IGP, localpref 0, valid, internal, best, import-candidate, imported
Extended community: RT:101:1 L2VPN AGI:1:101
Originator: 40.40.40.40, Cluster list: 30.30.30.30
This example shows the output from the show l2vpn discovery command.
RP/0/RSP0/CPU0:router# show l2vpn discovery private
AD event trace history [Total events: 3]
----------------------------------------Time
Event
====
=====
04/14/2010 23:09:42 Add edge edge_id/type
04/14/2010 23:10:51 Rcv LDP nlri upd l2rid/nh
04/14/2010 23:19:06 Rcv LDP nlri upd l2rid/nh
VPLS-ID: 1:101
Local L2 router id: 10.10.10.10
List of Remote NLRI (2 NLRIs):
Local Addr
Remote Addr
--------------- --------------10.10.10.10
192.0.2.20
10.10.10.10
192.168.40.40
Remote L2 RID
--------------192.0.2.20
192.168.40.40
Status/PWID
Flags/PeerID
=============== ============
10.10.10.10
0
192.0.2.20
192.0.2.20
192.168.40.40
192.168.40.40
Time Created
------------------04/14/2010 23:10:51
04/14/2010 23:19:06
AD event trace history [Total events: 4]

----------------------------------------Time
Event
====
=====
04/14/2010 23:09:42 Snd LDP nlri l2rid
04/14/2010 23:09:42 Snd refresh
04/14/2010 23:10:51 Upd bmgr ledge_id/nh
04/14/2010 23:19:06 Upd bmgr ledge_id/nh
Status/PWID
===============
10.10.10.10
0
10.10.10.10
10.10.10.10
Flags/PeerID
============
0
0x0
192.0.2.20
192.168.40.40
AC Is Down
Step 1
Step 2
RP/0/RSP0/CPU0:router# show l2vpn bridge interface detail
Step 3
Ensure that the AC interface has l2transport configured.
Step 4
Ensure that the AC interface is up.
Step 5

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain interface type interface-name detail
9-218
OL-23591-02
Chapter 9

Pseudowire Is Down
A pseudowire (PW) is both an L2VPN component and an MPLS component. If a PW is down in one
node, it could be caused by a problem in the local or remote node. Follow these steps to troubleshoot a
problem with a PW in an L2VPN network.
Note
For PW troubleshooting in point-to-point networks, see the Troubleshooting Point-to-Point Layer 2

Services section on page 9-206.
Step 1
Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp).
Step 2
Verify that L2VPN discovery shows the received NLRI (show l2vpn discovery). If the NLRI is not
received, follow the procedure in the L2VPN Discovery Not Working section on page 9-217.
Step 3
View the local and remote labels in the bridge-domain (show l2vpn bridge-domain detail) and compare
these labels with the label binding in LSD (show mpls lsd forwarding labels). See the example below.
Step 4
View OSPF neighbor information.

Step 5
View MPLS LDP neighbor information.

Step 6
View the bridge neighbor state.

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain neighbor
Step 7
If PWs are involved, be sure they are properly configured on both PEs. See the Troubleshooting
Point-to-Point Layer 2 Services section on page 9-206.
Step 8
Step 9
Step 10
Ensure that an IGP (for example OSPF) is up.
Step 11
Step 12

Example
These commands allow you to view the local and remote labels in the bridge-domain and compare them
with the label binding in LSD.
AS Number: 1
VPLS-ID: 1:101
PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established )
OL-23591-02
9-219
Chapter 9
Sequencing not set

MPLS
-----------Label
BGP Peer ID
LDP ID
AII
AGI
Group ID
Interface
MTU
Control word
PW type
VCCV CV type
Local
-----------------------------16005
10.10.10.10
10.10.10.10
10.10.10.10
1:101
0x0
bg1_bd1_vfi
1500
disabled
Ethernet
0x2
VCCV CC type 0x6
(TTL expiry)
------------ ------------------------------
Remote
------------------------16006
192.0.2.20
192.0.2.20
192.0.2.20
1:101
0x0
bg1_bd1_vfi
1500
disabled
Ethernet
0x2
0x6
(TTL expiry)
-------------------------
RP/0/RSP0/CPU0:router# show mpls lsd forwarding labels 16005

Thu Apr 15 00:07:39.888 EDT
16005, (PW, (192.0.2.20:2814754062073957)), 1 Paths
Thu Apr 15 00:09:10.067 EDT
Local Outgoing
Prefix
Outgoing
Label Label
or ID
Interface
------ ----------- ------------------ -----------16005 Pop
PW(192.0.2.20:2814754062073957)
BD=0
16005
Next Hop
Bytes
Switched
--------------- -----------\
point2point
214700000
VPLS Not Forwarding Flooding Traffic

Step 1
Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp, show run
interface).
Step 2
Make sure the PW is up and verify the XC ID.
Step 3
View the local and remote label in the bridge-domain (show l2vpn bridge-domain detail) and compare
these labels with the abel binding in LSD (show mpls lsd forwarding labels). If the NLRI is not
received, follow the procedure in the L2VPN Discovery Not Working section on page 9-217. See the
example below.
Step 4
View the forwarding bridge-domain parameters (show l2vpn forwarding bridge-domain detail
location) to see which direction is experiencing a traffic loss. If you have PWs in the core, the PWs
should be in the bound state and traffic should be flowing in the bound PWs. See the example below.
Step 5
Display the MAC table for the bridge domain and verify that the MAC address has been learned. Notice
the bridge port (the same as the attachment circuit [AC]) from which the MAC address was learned, and
whether it was learned through a pseudowire (PW).
9-220
OL-23591-02
Chapter 9

Caution
Before you run this command without specifying a MAC address ID, take steps to limit the amount of
data that can be output on your terminal screen. Otherwise the amount of output could be extremely
large.
This command displays all the MAC addresses learned on a bridge domain. As a safety mechanism,
One other approach to limit the output is to run the command with a pipe filter and CTRL-C after you
see the output you want.
mac-address detail location node-id [ | begin GigabitEthernet interface-id ]
Step 6
View the NP counters. Capture this output for both ingress and egress line cards. For a description of
how to interpret NP counter information, see the Displaying Traffic Status in Line Cards and RSP
Cards section on page 7-147.
RP/0/RSP0/CPU0:router# show controllers np counters all location
Step 7
View OSPF neighbor information.

Step 8
View MPLS LDP neighbor information.

Step 9
If PWs are involved, be sure they are properly configured on both PEs. See the Troubleshooting
Point-to-Point Layer 2 Services section on page 9-206.
Step 10
Step 11
Step 12
Ensure that OSPF is the routing protocol.
Step 13
Step 14

Example
These commands allow you to view the local and remote labels in the bridge-domain and compare them
with the label binding in LSD.
AS Number: 1
VPLS-ID: 1:101
PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established )
OL-23591-02
9-221
Chapter 9

Sequencing not set
MPLS
-----------Label
BGP Peer ID
LDP ID
AII
AGI
Group ID
Interface
MTU
Control word
PW type
VCCV CV type
Local
-----------------------------16005
10.10.10.10
10.10.10.10
10.10.10.10
1:101
0x0
bg1_bd1_vfi
1500
disabled
Ethernet
0x2
VCCV CC type 0x6
(TTL expiry)
------------ ------------------------------
Remote
------------------------16006
192.0.2.20
192.0.2.20
192.0.2.20
1:101
0x0
bg1_bd1_vfi
1500
disabled
Ethernet
0x2
0x6
(TTL expiry)
-------------------------
RP/0/RSP0/CPU0:router# show mpls lsd forwarding labels 16005

Thu Apr 15 00:07:39.888 EDT
16005, (PW, (192.0.2.20:2814754062073957)), 1 Paths
Thu Apr 15 00:09:10.067 EDT
Local Outgoing
Prefix
Outgoing
Label Label
or ID
Interface
------ ----------- ------------------ -----------16005 Pop
PW(192.0.2.20:2814754062073957)
BD=0
16005
Next Hop
Bytes
Switched
--------------- -----------\
point2point
214700000
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain detail location 0/6/CPU0

Fri Jan 7 13:54:45.740 PST
Bridge-domain name: 189:189, id: 0, state: up
Flooding:
DHCPv4 snooping: profile not known on this node
IGMP snooping: disabled, flooding: enabled
Bridge MTU: 9000 bytes
Number of MAC: 2
9-222
OL-23591-02
Chapter 9

Statistics:
Dynamic arp inspection drop counters:
packets: 0, bytes: 0
IP source guard drop counters:
packets: 0, bytes: 0
.
.
.
RP/0/RSP0/CPU0:router# show controllers np counters all
Mon Nov 15 12:20:35.289 EST
Node: 0/0/CPU0:
Offset Counter
FrameValue
Rate (pps)
417
0
30 RESOLVE_INRESS_DROP_CNT
9
0
6
0
3230
0
920
0
1081
0
71 RESOLVE_EGRESS_L3_PUNT_CNT
4613
0
3484
0
104
0
83 RESOLVE_MOFRR_HASH_UPDATE_CNT
463
0
407
0
111 DIAGS
536
0
295 DROP_IPV4_NEXT_HOP_DOWN
15
0
.
.
.
The following command allows you to view the bridge domain forwarding data.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain location 0/6/CPU0
Thu Apr 15 00:15:29.581 EDT
Bridge
MAC
Bridge-Domain Name
ID
Ports addr
Flooding Learning State
-------------------------------- ------ ----- ------ -------- -------- --------bg1:bg1_bd1
0
3
4
Enabled Enabled UP
The following command allows you to view the bridge domain MAC details. The output from this
command can be very large, so you should limit the terminal screen output or send the data to a file.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location 0/6/CPU0
Mac Address
Type
Learned from/Filtered on
LC learned Resync Age
Mapped to
-------------------------------------------------------------------------------0000.0001.0101 dynamic Gi0/6/0/1.1
0/6/CPU0
0d 0h 1m 59s
N/A
0000.0001.0102 dynamic Gi0/6/0/1.1
0/6/CPU0
0d 0h 1m 59s
N/A
0000.0002.0202 dynamic (192.0.2.20, 1:101)
0/6/CPU0
0d 0h 1m 59s
N/A
0000.0003.0303 dynamic (192.168.40.40, 1:101)
0/6/CPU0
0d 0h 1m 59s
N/A
OL-23591-02
9-223
Chapter 9
VPLS Not Forwarding Flooding Traffic from AC to Pseudowire

Step 1
View ingress UIDB and XID for the segment.

RP/0/RSP0/CPU0:router# show l2vpn forwarding interface hardware ingress detail location
Step 2
If PWs are invloved, view PW hardware information.

RP/0/RSP0/CPU0:router# show l2vpn forwarding neighbor 192.168.12.5 pw-id 100 hardware
egress location node-id0
Step 3
View MPLS leaf information.

RP/0/RSP0/CPU0:router# show mpls forwarding labels hardware egress detail location
Step 4
View bridge information about Broadcast, Multicast and Unknown Unicast.

Step 5

Step 6
View PI event traces.

RP/0/RSP0/CPU0:router# show l2vpn trace location
Step 7
Ensure that the pseudowires (as applicable) and AC are up.
Step 8
Verify the hardware is programmed for both ACs.
Step 9

Step 10
Verify the hardware is programmed for pseudowires.
VPLS Not Forwarding Flooding Traffic from Pseudowire to AC

Step 1
View ingress UIDB and XID for the segment.

RP/0/RSP0/CPU0:router# show l2vpn forwarding interface hardware ingress detail location
Step 2
View MPLS leaf information.

RP/0/RSP0/CPU0:router# show mpls forwarding labels hardware egress detail location
Step 3
View bridge information about Broadcast, Multicast and Unknown Unicast.

Step 4

Step 5

9-224
OL-23591-02
Chapter 9

Step 6
Ensure that the pseudowires (as applicable) and AC are up.
Step 7
Verify that the hardware is programmed for both ACs.

Step 8
Verify that the hardware is programmed for PW (if applicable).
VPLS Not Forwarding Unicast Traffic from AC to AC

Step 1

bd-name-id detail
Step 2
Ensure that the hardware is programmed for both ACs.
Step 3
Ensure that the destination MAC entry is programmed for the LCs destination interface.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location
node-id
VPLS Not Forwarding Unicast Traffic from AC to Pseudowire

Step 1

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name bd-name-id detail
Step 2
Ensure that the hardware is programmed for both AC and PW (as applicable).
Step 3
Ensure that the destination MAC entry is programmed for the LCs destination interface.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location node-id
VPLS Not Forwarding Flooding Traffic from Pseudowire to AC

Step 1

Step 2
bd-name-id detail
Ensure that the hardware is programmed for both AC and PW (as applicable).
OL-23591-02
9-225
Chapter 9
Pseudowire Up but Ping Fails

Step 1
Determine where the ping packets are being dropped, view the xconnect AC interface counters and the
L2VPN counters for the PW. For information on ping procedures, see the Troubleshooting Ping and
ARP Connectivity section on page 3-75.
Step 2

RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name bd-name-id detail
Step 3
Ensure that both CEs are on the same subnet.
Step 4
Step 5
Ensure that the end-to-end encapsulations match.
Traffic Loss
Step 1
Determine where the packets are being dropped, view the xconnect AC interface counters and the
L2VPN counters for the PW. For information on ping procedures, see the Troubleshooting Ping and
ARP Connectivity section on page 3-75.
Step 2

Step 3
bd-name-id detail
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet
location
Step 4
interface-id detail
node-id
Ensure that the bandwidth rates match between the CEs.
Pseudowire Flap Causing Traffic Loss

Step 1

Step 2
bd-name-id detail
location
Step 3
interface-id detail
node-id

9-226
OL-23591-02
Chapter 9

Traffic Loss During RSP Fail Over

Step 1
View the state of the xconnect.

RP/0/RSP0/CPU0:router# show l2vpn bridge detail
Step 2
View the counter for the segment.

location
Step 3

Step 4
interface-id detail
node-id
bd-name-id detail
View ingress UIDB.

location
Step 5
Step 6
node-id
Check all routers in the MPLS path to ensure the following are configured:
a.
MPLS LDP graceful restart
b.
OSPF NSF
View the segment counters to see if the packet and byte switched count increased.
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEthernet node-id detail
location node-id
Preferred Path Not Working

Step 1

Step 2
bd-name-id detail
View ingress UIDB.

location
node-id

Dynamic Host Configuration Protocol Snooping (DHCP snooping) provides DHCP security by filtering
untrusted DHCP messages, and by building and maintaining a DHCP snooping binding table. An
untrusted message is a message that is received from outside the network or firewall and that can cause
traffic attacks within your network. This section describes the following commands:
Show Commands, page 9-228
OL-23591-02
9-227
Chapter 9
Trace Commands, page 9-228
Syslog Commands, page 9-228
Tech-support Commands, page 9-229
Action Commands, page 9-229
L2VPN Commands, page 9-229
L2Snoop Commands, page 9-229
Interface Controller Commands, page 9-230
Show Commands
The DHCP application runs on the RSP. It has several EXEC mode CLI show commands that present the
application's configuration state, DHCP client state, and DHCP packet statistics.
show dhcp ipv4 snoop bindingView
the state of DHCP clients in a table.
show dhcp ipv4 snoop binding mac-address macaddressView
detailed state of DHCP Clients
with the specified MAC Address.
show dhcp ipv4 snoop binding summaryView
show dhcp ipv4 snoop profileView
show dhcp ipv4 snoop profile name
show dhcp ipv4 snoop statisticsView
the total number of DHCP Clients.
a list of DHCP snoop profiles.
nameView details of a specific DHCP snoop profile.

aggregate DHCP snoop Rx, Tx, and drop packets for
each bridge domain.
show dhcp ipv4 snoop statistics bridge-domain nameView detailed DHCP snoop Rx, Tx,
and drop packets for each message type in a bridge domain.
Trace Commands
The DHCP application has over 1200 Trace logs. The Trace logs record significant events that occur in
the application. Trace logs that are associated with a specific DHCP client will contain the client MAC
address.
show dhcp ipv4 trace errorsView
error traces.
show dhcp ipv4 trace eventsView
event traces.
show dhcp ipv4 trace packetsView
show dhcp ipv4 trace snoop errorsView
error traces for DHCP snoop feature.
show dhcp ipv4 trace snoop eventsView
event traces for the DHCP snoop feature.
show dhcp ipv4 trace snoop internalView
packet processing traces.
internal debug traces for the DHCP snoop feature.
Syslog Commands
The DHCP application has over 1600 syslog logs. These logs record events that occur in the application.
debug dhcp ipv4 errorsView
error logs.
debug dhcp ipv4 eventsView
event logs.
9-228
OL-23591-02
Chapter 9

debug dhcp ipv4 packetView
packet processing logs.
debug dhcp ipv4 snoop errorsView
error logs for DHCP snoop feature.
debug dhcp ipv4 snoop eventsView
event logs for the DHCP snoop feature.
debug dhcp ipv4 snoop internalView
internal debug logs for the DHCP snoop feature.
Tech-support Commands
The DHCP application has four tech-support commands that call groups of DHCP CLI commands. Use
tech-support commands for information about the DHCP application for debugging.
filename
show tech-support dhcp ipv4 snoop file
show tech-support dhcp ipv4 snoop bridge-domain-name
bridge-domain-id file
filenameView information for the specified bridge domain.
show tech-support dhcp ipv4 snoop profile-name
profilename file filenameView
information for the specified profile.
Action Commands
Use the following CLI commands to clear DHCP snoop binding states:
clear dhcp ipv4 snoop bindingClears
all DHCP snoop client bindings.
clear dhcp ipv4 snoop binding bridge-domain
bridge-domain-nameClears all DHCP snoop
client bindings in the specified bridge domain.
clear dhcp ipv4 snoop binding mac-address
macaddressClears the DHCP snoop client
bindings with the specified MAC address.
L2VPN Commands
DHCP snoop is enabled on L2VPN ACs by attaching a DHCP snoop profile to a bridge domain or AC.
The DHCP snoop trusted attribute is configured on an AC according to the value of the trusted attribute
in the DHCP snoop profile. L2VPN CLI commands are used to display the status of DHCP snoop
attributes on L2VPN bridge domains and ACs.
show l2vpn bridge-domain bd-name bridgename detailView the L2VPN DHCP snoop
configuration for the specified bridge domain.
show l2vpn forwarding interface interface detail location locationView the L2VPN
DHCP snoop configuration for a specific interface.
L2Snoop Commands
L2Snoop receives and transmits DHCP snoop packets between NETIO and the DHCP snoop application
on the RSP.
show l2snoop statistics pcb allView
the L2SNOOP DHCP packet Rx/Tx statistics to and from
the DHCP snoop application on the RSP.
OL-23591-02
9-229
Chapter 9
Troubleshooting Multiple Spanning Tree
Interface Controller Commands

Interface controllers receive and send DHCP snoop packets between the wire and the network
processors.
show controllers interface statsView the interface controller statistics that include DHCP packets
that are sent and received from the wire.

Multiple Spanning Tree (MST) is an IEEE standard based on the Cisco proprietary Multiple Instances
Spanning Tree Protocol (MISTP) implementation. This section explains how to troubleshoot MST and
contains the following subsections:
MSTP Incorrectly or Inconsistently Formed, page 9-230
MSTP Correctly Formed, but Traffic Flooding, page 9-231
Packet Forwarding Does Not Match MSTP State, page 9-231
MSTAG Access Network Does Not Recognize MSTAG Node as Root, page 9-231
Traffic Not Switching Through MSTAG Node(s), page 9-232

For a complete list of MST show and debug commands, see the Multiple Spanning Tree Protocol
Commands chapter in the Cisco ASR 9000 Series Aggregation Services Router L2VPN and Ethernet
Services Command Reference module.
MSTP Incorrectly or Inconsistently Formed

When the spanning tree is misformed, it is often because of misconfiguration or BPDU loss. This
generally manifests as more than one node showing itself as ROOT, but can also result in disagreement
on which nodes are ROOT.
MSTP Incorrectly or Inconsistently FormedMisconfiguration

Ensure that the following match on the nodes:
Configuration name
Bridge revision
Provider-bridge mode
Instance to VLAN mapping
Run the following command to check that the configuration is consistent across multiple devices.
RP/0/RSP0/CPU0:router# show spanning-tree mst protocol-instance-id configuration
9-230
OL-23591-02
Chapter 9

MSTP Incorrectly or Inconsistently FormedBPDU Loss

Determine if node A is sending BPDUs to node B. The output of the following command includes a
count of the number of BDPUs being sent and received. Run the command several times for each
interface connecting the nodes.
RP/0/RSP0/CPU0:router# show spanning-tree mst protocol-instance-id interface interface-name
Only designated ports will send periodic BPDUs, but non-designated ports send updates on topology
changes and startup. Ensure that BPDUs sent and received are going up as appropriate.
MSTP Correctly Formed, but Traffic Flooding

Intermittent BPDU loss may mean the spanning tree will not show up incorrectly in the show commands,
but will send out topology change notifications. These notifications cause a MAC flush, forcing traffic
to flood until the MAC addresses are re-learned.
Run the following commend to check whether there have been any flushes.
RP/0/RSP0/CPU0:router# show spanning-tree mst <protocol-instance> topology-change flushes
Look for topology change notifications. Run the following command and look for TC 1:
Note
This option is verbose.
Packet Forwarding Does Not Match MSTP State

Step 1
Shut down redundant links, remove MSTP configuration, and ensure that basic bridging works.
RP/0/RSP0/CPU0:router# show spanning-tree mst name
RP/0/RSP0/CPU0:router# show interface interface-name
Step 2
Check the state of each port as calculated by MSTP, and compare it with packet transmit and receive
counts on ports and Ethernet flow points (EFPs) that are controlled by MSTP. Normal data packets
should be sent/received only on ports that are in forwarding (FWD) state. In steady state operation,
BPDUs are sent if there is at least one MSTI that is in Designated role.
Step 3
Ensure that BPDUs are flowing and that root bridge selection is correct. Check those related scenarios
first.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain [detail]
This command will show the status of members of the bridge domain. Ensure that the relevant bridge
domain members are up.
Step 4
Check forwarding state as programmed in hardware.
MSTAG Access Network Does Not Recognize MSTAG Node as Root

If multiple spanning tree access gateway (MSTAG) is not recognized as root, check that the two MSTAG
devices are configured with the same root-id and root-priority for every MST instance (MSTI), and that
the root-priority is lower than any of the access devices (preferably 0).
Also check (on the access devices) for any disputes; disputes are an indication of a misconfiguration.
OL-23591-02
9-231
Chapter 9
Additional ReferencesCommand Reference and Configuration Guides
Step 1
To view the BPDUs being sent by MSTAG, run the following command.
RP/0/RSP0/CPU0:router# show spanning-tree mstag protocol-instance-id bpdu interface
interface-name
There are two ways of configuring MSTAG:
Advertise as though both nodes are separaterequires each node have a unique bridge id and the
configurations complement each other.
Advertise as though each node is a different port on the same nodeconfiguration is identical
except for the port id.
Commands for MSTAG must target the untagged EFP instead of the base interface. Perform the
following steps to verify your configuration and debug MSTAG.
Step 2
Verify the running configuration.

RP/0/RSP0/CPU0:router# show running-config spanning-tree {mst | mstag | repag} name
Traffic Not Switching Through MSTAG Node(s)

Step 1
Collect L2VPN and UIDB data to verify the data path is healthy.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain [detail]
Step 2
Ensure that the forwarding state is set as it was programmed in the hardware.
Additional ReferencesCommand Reference and

Configuration Guides
The following documents provide information on the commands and configuration procedures for
L2VPN and Ethernet Services:
Cisco ASR 9000 Series Aggregation Services Router L2VPN and Ethernet Services Command
Reference
Cisco ASR 9000 Series Aggregation Services Router L2VPN and Ethernet Services Configuration
Guide
Use the following guide when you configure routing. L2VPN services rely on Layer 3 connectivity from
the provider edge (PE) through the core:
Cisco ASR 9000 Series Aggregation Services Router Routing Configuration Guide, Release 4.0
9-232
OL-23591-02
CH A P T E R
10
Troubleshooting Quality of Service and Access

Control Lists
This chapter describes techniques for troubleshooting quality of service (QoS) and access control list
(ACL) features.
The system supports the following QoS features:
Multilevel priority scheduling for voice and video applications with minimal jitter, latency and
packet loss.
Priority propagation to ensure service integrity for voice and video throughout all hierarchy layers,
even at peak hours with high traffic load.
Differentiated Service Code Point (DSCP), MPLS experimental bit (EXP) and IEEE 802.1p IP
Precedence bit classification with marking, policing and scheduling, ingress and egress.
This chapter includes the following sections:
Service-Policy Configuration Is Rejected, page 10-235
Packets are Incorrectly Classified, page 10-235
Packets in Wrong Queue, page 10-236
Packets Incorrectly Marked, page 10-236
Packets Incorrectly Policed, page 10-237
Shaping Incorrect, page 10-237
Weighted Random Early Detection Incorrect, page 10-237
Bandwidth Not Guaranteed, page 10-238
Bandwidth Ratio Not Working, page 10-238
Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands,

page 10-239
Unable to Modify or Delete policy-map or class-map, page 10-240
Unable to Modify or Delete class-map ACL, page 10-240
Unable to Delete service-policy, page 10-240
After QoS EA Restarts, show policy-map interface Fails, page 10-240
After QoS EA Restarts, service-policy config Fails, page 10-241
show policy-map interface Output Error, page 10-241
OL-23591-02
10-233
Chapter 10
Bundle Members Not Configured with service-policy, page 10-241
Troubleshooting Access Control Lists, page 10-241

SUMMARY STEPS
1.
show run policy-map
2.
show run classmap
3.
show run interface
4.
show policy-map interface type interface-name [output | input]
5.
show qos interface type interface-name [output | input]
6.
show qos-ea interface type interface-name [output | input]
7.
show qos-ea km
8.
debug qos-ea ?
DETAILED STEPS
Step 1
Command or Action
Purpose
show run policy-map
View policy-map with name.
Example:
RP/0/RSP0/CPU0:router# show run policy-map
l1-all
Step 2
show run classmap
View class-map configuration with name.
Example:
RP/0/RSP0/CPU0:router# show run class-map c2
Step 3
show run interface
View the service-policy binding for a given

port/subinterface.
Example:
RP/0/RSP0/CPU0:router# show run interface
g0/2/0/0
Step 4
show policy-map interface type interface-name

[output|input]
View all the statistics, queue IDs and class information.
Example:
RP/0/RSP0/CPU0:router# show policy-map
interface g0/2/0/0
10-234
OL-23591-02
Chapter 10

Step 5
Command or Action
Purpose
show qos interface type interface-name

[output|input]
View all the configuration of each class in hardware.
Example:
RP/0/RSP0/CPU0:router# show qos int g0/2/0/0
out
Step 6
show qos-ea interface type interface-name

[output|input]
View all the class information structures.
Example:
RP/0/RSP0/CPU0:router# show qos-ea int g0/2/0/0
out
Step 7
View the key manager (TCAM key manager) related fields

associated to a policy-map/interface binding.
show qos-ea km
Example:
RP/0/RSP0/CPU0:router# show qos-ea km policy
l2-all vmr interface g0/2/0/0 sw
Step 8
debug qos-ea ?
Example:
RP/0/RSP0/CPU0:router# debug qos-ea ?

Step 1
If the service-policy configuration is rejected or failed to commit, check the error message with the show
configuration failed command.
Step 2
If resource usage is more than what is configured, verify how many are checkpointed.
RP/0/RSP0/CPU0:router# show qos-ea ha chkpt all info location node-id
Step 3
Check the OOR.
Step 4
Verify resources used.
Step 5
Verify summary information.

RP/0/RSP0/CPU0:router# show qos summary {queue | police | policy}
Packets are Incorrectly Classified

Step 1
Verify packets are arriving on the correct interface.
Step 2
Verify the packet fields are as expected.
Step 3
Note the packet type.
OL-23591-02
10-235
Chapter 10
Step 4
Verify the KM policy information matches UIDB configuration.

RP/0/RSP0/CPU0:router# show qos-ea km policy policy info location filename
RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename hw detail
Step 5
Verify VMR entries for each class.

RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename hw detail
Step 6
Verify which class the packets are actually matching. If packet fields should match different class, then
NP Microcode needs to debug this further.
RP/0/RSP0/CPU0:router# show policy-map interface filename {output | input} [member
filename]
Step 7
Verify if in ingress QoS lookup occurs before Layer 2 ingress rewrite and that in egress Layer 2 rewrite
occurs before QoS lookup.
RP/0/RSP0/CPU0:router# show qos type interface {input | output} location node-id
Step 8
RP/0/RSP0/CPU0:router# show run interface type node-id
Step 9
RP/0/RSP0/CPU0:router# show run policy-map policy
Step 10
RP/0/RSP0/CPU0:router# show run class-map classmap

Step 1
RP/0/RSP0/CPU0:router# show qos-ea type interface {input | output} location node-id
Step 2
Verify the packets are correctly classified.
Step 3
Verify hash structure.

Step 4
Verify the hash key for the class and hash result of the class has correct Queue ID.
Packets Incorrectly Marked

Step 1
Verify packets are classified correctly.

Step 2
RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename hw
Step 3
RP/0/RSP0/CPU0:router# show qos-ea km policy policy vmr interface filename sw
Step 4
RP/0/RSP0/CPU0:router# show policy-map interface filename {input | output} [member

filename]
10-236
OL-23591-02
Chapter 10

Step 5
Verify marking value.


Step 1
Ensure that packets are correctly classified.
Step 2
Verify whether policer CIR/CBS/PIR/PBS are set correctly as per configured service-policy. Also verify
the rate at which traffic is coming to match against policed rate.
Step 3
Get the token bucket and police node index of the class.
Step 4

filename]
Shaping Incorrect
Step 1

Step 2
Verify whether shaper CIR/CBS/PIR/PBS are set correctly as per configured service-policy. Get the
shape profile ID and entity handle information (np, tm, level, index, offset).
Step 3
Verify the shaper profiles in hardware if they are correctly configured.
Step 4

filename]
Weighted Random Early Detection Incorrect

Step 1
Step 2
Verify whether the weighted random early detection (WRED) curves are correctly configured with
minimum and maximum thresholds of each curve are as per the configured service policy.
OL-23591-02
10-237
Chapter 10
Step 3
Get the WRED profile ID and entity handle information (np, tm, level, index, offset).
Step 4

filename]

Step 1
Step 2
Verify whether the weights of each class are configured correctly as per the bandwidth ratio among
classes.
Step 3
Step 4
If its correctly configured, then get the WFQ profile ID and entity handle information (np, tm, level,
index, offset) of the class.
Step 5

filename]
Bandwidth Ratio Not Working

Step 1
Step 2
Step 3
Verify whether the commit weights of each class is configured correctly as per the bandwidth ratio
among classes. Also verify that excess weights are configured as per the bandwidth remaining ratio
configuration.
Step 4
Step 5

filename]
Step 6
Get the WFQ profile ID and entity handle information (np, tm, level, index, offset) of the class.
Step 7
If commit and excess weights are correct:

a.
Check queue size of each class.
b.
Increase the queue size.
10-238
OL-23591-02
Chapter 10

Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands
Non-zero Queue(conform) and Queue(exceed) Counters In show

policy-map Commands
This section explains what to do if the show policy-map command displays non-zero values for
Queue(conform) and Queue(exceed) counters.
On the ASR 9000, every hardware queue has a configured committed information rate (CIR) and peak
information rate (PIR) value. CIR corresponds to the guaranteed bandwidth for the queue and PIR
corresponds to the maximum bandwidth (also known as the shape rate) for the queue. To configure the
CIR and PIR, use the police rate command. The syntax is:
RP/0/RSP0/CPU0:router# police rate {value [units] | percent percentage} [burst burst-size
[burst-units]] [peak-rate value [units]] [peak-burst peak-burst [burst-units]]
In this command, CIR is the police rate value and PIR is the police peak-rate value.
Example
RP/0/RSP0/CPU0:router# show policy-map location 0/7/0/30
GigabitEthernet0/7/0/30.1001 output: STS-1
Class class-default
Classification statistics
(packets/bytes)
(rate - kbps)
Matched
:
4167078/4179237900
79024
Transmitted
:
2224974/2229365484
42017
Total Dropped
:
1942095/1949863380
36801
Policy SHAPE-OUT Class BFD-OUT
Classification statistics
(packets/bytes)
(rate - kbps)
Matched
:
4786/296732
5
Transmitted
:
4786/296732
5
Total Dropped
:
0/0
0
Policing statistics
(packets/bytes)
(rate - kbps)
Policed(conform)
:
4786/296732
5
Policed(exceed)
:
0/0
0
Policed(violate)
:
0/0
0
Policed and dropped :
0/0
Policed and dropped(parent policer) : Un-determined
Queueing statistics
Queue ID
: 8
High watermark (Unknown)
Inst-queue-len (packets)
: 0
Avg-queue-len
(Unknown)
Taildropped(packets/bytes)
: 0/0
Queue(conform)
:
0/0
0
Queue(exceed)
:
4786/296732
5
RED random drops(packets/bytes)
: 0/0
A non-zero value displayed for Queue(exceed) does not mean that there is a packet drop, but rather the
number of packets above the configured (or system selected) CIR rate on that queue. Although you could
change the Queue(exceed) behavior by explicitly configuring a bandwidth and/or a shape rate on each
queue, it is not necessary to do so. You can treat these counters as informational or simply ignore them.
In the police rate command, if you do not explicitly configure a value for the police rate (the CIR), the
system automatically assigns one. The Queue(conform) counter in the show policy-map command is
the number of packets/bytes that were transmitted within this CIR value, and the Queue(exceed) value
is the number of packets/bytes that were transmitted within the PIR value. The Queue(exceed) counter
is based on whether the parent bandwidth is exceed or conform. If there is no parent bandwidth, all
traffic is counted as excess.
OL-23591-02
10-239
Chapter 10

Step 1
Verify the policy is applied on an interface.

Step 2
Remove service-policy on the interfaces.
Step 3
Modify the policy-map.
Unable to Modify or Delete class-map ACL
show config failed
show running-config
Step 1
Verify the ACL is part of a match statement in a class-map.
Step 2
Verify the class-map is part of any policy-map that is applied on an interface.
Step 3
If the policy-map is applied on interface, ACL modification/deletion is not allowed.
Step 4
Remove all the service-policy configuration of this policy-map and modify ACLs.
Unable to Delete service-policy

Step 1
RP/0/RSP0/CPU0:router# show config failed
Step 2
Restart the qos_ma_ea process.
After QoS EA Restarts, show policy-map interface Fails
Step 1
show running-config
show qos-ea ha chkpt all info location
show qos-ea ha chkpt if-qos all location
node-id
node-id
Verify if the state of QoS EA is in in_sync (state = 2).

RP/0/RSP0/CPU0:router# show qos-ea ha state location node-id
Step 2
If there is no error, do the following:

a.
RP/0/RSP0/CPU0:router# debug generic
10-240
OL-23591-02
Chapter 10

b.
Collect debugs by performing the failing command.

Step 1
Verify the state of QoS EA is in in_sync (state = 2).

RP/0/RSP0/CPU0:router# show qos-ea ha state location node-id
Step 2
If there is no error, do the following:

a.
RP/0/RSP0/CPU0:router# debug generic
b.
Collect the debugs by performing the failing command.
show policy-map interface Output Error

For bundles, specify member interface. Policy information for bundle-interface is not available in the
current release.
show policy-map interface {output | input} member
show {qos | qos-ea} interface {output | input} location
node-id
Bundle Members Not Configured with service-policy

For bundles, specify member interface. Policy information for bundle-interface is not available in the
current release.
show policy-map interface {output | input} member
show {qos | qos-ea} interface {output | input} location
node-id

This section explains how to troubleshooting problems with access control lists (ACLs). ACLs are used
for packet filtering and selecting traffic types to be analyzed, forwarded, or influenced in some way.
Access control entries (ACEs) are individual permit or deny statement within an ACL. Each ACE
includes an action element (permit or deny) and a filter element based upon criteria such as source
address, destination address, protocol, protocol-specific parameters, and so on. This section contains the
following topics:
ACL Messages Not Appearing, page 10-243
Fragmented Packets Being Accepted, page 10-243
Egress Counter Incorrect or Not Working, page 10-244
OL-23591-02
10-241
Chapter 10
ACL Interface Bind Rejected, page 10-244
Single ACE Using Many TCAMs, page 10-244
ACL Using Varying TCAM Space, page 10-245
ACL Logs Not Working for Ethernet Services, page 10-245
Ethernet Services ACL Bind on Interface Rejected, page 10-245
Changing ACL Exhausts TCAM, page 10-245
Cannot Delete ACL, page 10-246
DF Bit Not Supported, page 10-246
Max ACL Limit Reached, page 10-246
Unsupported Combinations in ACL, page 10-246
No Statistics Counters, page 10-246
TCAMs Out of Resources, page 10-246

SUMMARY STEPS
1.
show access-lists ipv4 [rp-access [hardware {ingress | egress} {sequence-number | location

node-id | summary [rp-access] | maximum [detail] [usage {pfilter location node-id}]
2.
debug feature-ea-dll {all | error | info | resmgr | vmr}
DETAILED STEPS
Step 1
Command or Action
Purpose
show access-lists ipv4 [rp-access [hardware

{ingress | egress} {sequence-number | location
node-id | summary [rp-access] | maximum
[detail] [usage {pfilter location node-id}]
View all IPv4 ACL contents. Filter results using the

access-list-name-IPv4 ACL name.
hardwareIngress specifies an inbound interface,

egress specifies an outbound interface.
sequence-numberACL number, 1 to 2147483646.
location node-idRack/slot/module notation of ACL.
summarySummary of all current IPv4 ACLs.
maximumMaximum configurable IPv4 ACLs and

ACEs.
detailOut-of-resource (OOR) details, OOR limits the

number of ACLs and ACEs configured.
usageView the usage of the ACL on a given line card

(LC).
pfilterPacket filtering for the LC.
Example:
RP/0/RSP0/CPU0:router# show access-lists ipv4
dtho 10 ipv4 access-list dtho 10 permit ipv4
any any
Step 2
debug feature-ea-dll {all | error | info |

resmgr | vmr}
View error messages at various levels.
10-242
OL-23591-02
Chapter 10

ACL Messages Not Appearing

Step 1
View ACEs in the ACLs.

RP/0/RSP0/CPU0:router# show access-lists ipv4
Step 2
View TCAM entries in the ACLs..

RP/0/RSP0/CPU0:router# show access-lists ipv4 hardware {ingress | egress} detail ...
Step 3
Configure the logs in the ACL.

RP/0/RSP0/CPU0:router# ipv4 access-lists log-update threshold
Workaround
If an entry with the fragment flag is not present, remove the access-list from all the interfaces and reapply
it.
Note
Fragmented packets are not matched against the deny ACE without fragment keyword. Add the explicit
fragment keyword in the ACE to deny the fragment packet. See the workaround commands in the
Fragmented Packets Being Accepted section on page 10-243.
Fragmented Packets Being Accepted

Step 1

RP/0/RSP0/CPU0:router# show access-list ipv4
Step 2
View IPv4 counters, for example, fragment.

RP/0/RSP0/CPU0:router# show ipv4 traffic
Step 3
View TCAM entries in the ACLs.
Step 4
Ensure that the fragment keyword is in the ACE.

RP/0/RSP0/CPU0:router# deny ipv4 any any fragments
Step 5
Check the fragment packet count received by the device.
Step 6
View TCAM entries.
Workaround
Fragmented packets are not matched against the deny ACE without the fragment keyword. If there is
not an entry with the fragment flag, perform the following procedure.
Step 1
Remove the ACL from all interfaces.
OL-23591-02
10-243
Chapter 10
Step 2
Add the explicit fragment keyword in the ACE to deny the fragment packet.
Step 3
Reapply the ACL to all interfaces.
Egress Counter Incorrect or Not Working

Step 1
View known routes.

Step 2
View ARP table entries. Look for the next hop.

Step 3
View TCAM entries for the ACL.

RP/0/RSP0/CPU0:router# show access-list ipv4 hardware
Workaround
Step 1
If the route is missing or the ARP is incomplete, use the no shut command to recover.
Step 2
If the UIDB table or TCAM entry is incorrect, remove the ACL from all of the interfaces and reapply it.
ACL Interface Bind Rejected

View errors encountered when the configuration was applied.
Workaround
If the error is related to TCAM space, remove the ACEs from the ACL. There is a limit of 64 TCAM
entries per ACL.
Single ACE Using Many TCAMs

Step 1

Step 2
Check the number of ranges in the ACE.
10-244
OL-23591-02
Chapter 10

ACL Using Varying TCAM Space

View Pre-Internal Forwarding Information Base (Pre-IFIB) hardware statistic entries.
RP/0/RSP0/CPU0:router# show lpts pifib brief
ACL Logs Not Working for Ethernet Services

Ethernet services logging is not supported.
Ethernet Services ACL Bind on Interface Rejected

Step 1
View any errors encountered when the configuration was applied.

Step 2

Step 3
View trace log for pfilter_ea.
Step 1
If a field in the ACL is not supported, remove it from the ACE.
Step 2
If the TCAM is out of space, reduce the ACEs in the ACL.
Step 3
Reduce the ranges in the ACL.
Workaround
Changing ACL Exhausts TCAM

View ACEs configured for the ACL.
RP/0/RSP0/CPU0:router# show access-list {ethernet-service/ipv4}
Workaround
Remove the old ACL before applying the new one.
OL-23591-02
10-245
Chapter 10
Cannot Delete ACL

Step 1
View any errors encountered when the configuration was applied.

RP/0/RSP0/CPU0:router# show configuration error
Step 2
View interfaces using the ACL.

RP/0/RSP0/CPU0:router# show access-list {ethernet-services | ipv4} usage pfilter
Step 3
View IPv4 trace information.

RP/0/RSP0/CPU0:router# show access-list ipv4 trace
Step 4
View the Ethernet services trace.

RP/0/RSP0/CPU0:router# show access-list ethernet-services trace
DF Bit Not Supported

The Do Not Fragment (DF) bit is not supported as match criteria in the current release.
Max ACL Limit Reached

The maximum number of ACL IDs per network processor (NP) is 2048. Interfaces share TCAM entries
for the ACL name and direction.
Unsupported Combinations in ACL

There may be unsupported field combinations in the access-list. Verify that the combinations in the
access-list are currently supported. The current release supports the following combinations:
VLAN OUT + L2 PROTO + MAC SA + MAC DA
VLAN OUT + VLAN IN + MAC SA + MAC DA
VLAN OUT + VLAN IN + L2 PROTO + MAC DA
No Statistics Counters
Statistics counters are not supported in the current release.
TCAMs Out of Resources

The TCAMs Out of Resources message means you have attempted to provision more than the available
number of TCAM entries.
10-246
OL-23591-02
CH A P T E R
11

This chapter describes techniques that you can use to troubleshoot multicast services. It includes the
following sections:
Troubleshooting IGMP Snooping (Layer 2 Multicast), page 11-247
Troubleshooting Native Multicast Routing (Layer 3), page 11-256
Troubleshooting IGMP Snooping (Layer 2 Multicast)

This section explains how to troubleshoot problems with Internet Group Management Protocol (IGMP)
snooping. IGMP snooping restricts multicast flows at Layer 2 to only those segments with at least one
interested receiver.
A prerequisite for implementing IGMP snooping is that the network must be configured with a Layer 2
VPN (L2VPN). IGMP snooping is supported only under L2VPN bridge domains.
This section covers the following topics:
Using show Commands, page 11-247
Using the debug, trace, and show tech-support Commands, page 11-249
Troubleshooting Missing Routes and Forwarding Errors, page 11-250
Using show Commands

SUMMARY STEPS
1.
Confirm correct topology and configuration

a. show l2vpn bridge-domain summary
b. show igmp snooping bridge-domain
c. show l2vpn bridge-domain bd-name bd-name
d. show l2vpn bridge-domain bd-name bd-name detail
e. show igmp snooping bridge-domain bd-name detail
f. show igmp snooping port bridge-domain bd-name
g. show igmp snooping profile
2.
Confirm that IGMP Snooping is sending and receiving control traffic
OL-23591-02
11-247
Chapter 11
a. show igmp snooping summary statistics

b. show igmp snooping bridge-domain bd-name detail statistics
c. show igmp snooping port [if-type if-name] detail statistics
3.
Confirm that IGMP Snooping is creating group state as expected

a. show igmp snooping group
b. show igmp snooping group port
c. show igmp snooping group source
d. show igmp snooping group bridge-domain bd-name
e. show igmp snooping port if-type if-name group [detail]
4.
Confirm that the forwarding state matches the IGMP Snooping state
a. show l2vpn forwarding bridge-domain [bridge group name:bd-name] mroute ipv4 location
lc-name
b. show l2vpn forwarding bridge-domain [bridge group name:bd-name] mroute ipv4 hardware
[ingress | egress] location lc-name

5.
View the number of packets sent, received, and failed for IGMP from and to l2snoop; determine
whether packets are being dropped. Use the command show l2snoop statistics summary.
DETAILED STEPS
Command or Action
Step 1
Step 2
Purpose
a.
Confirm correct topology and configuration:
b.
show igmp snooping bridge-domain
a.
List all L2VPN bridge domains
c.
show l2vpn bridge-domain bd-name bd-name
b.
View IGMP Snooping state in all bridge domains.
d.
show l2vpn bridge-domain bd-name bd-name

detail
c.
View information about the specified bridge domain,

including the list of interfaces and VFIs.
e.
show igmp snooping bridge-domain bd-name

detail
d.
f.
show igmp snooping port bridge-domain

bd-name
g.
show igmp snooping profile
View detailed information about the specified bridge

domain, including the IGMP Snooping profile. Verify
that the L2VPN and IGMP configurations are
consistent. The profile, bridge-domain, and bridge
group specified in the IGMP configuration must be
consistent with that specified in the L2VPN
configuration.
e.
View detailed IGMP Snooping information in the

specified bridge domain.
f.
See IGMP Snooping view of the ports and port state in

the specified bridge domain.
g.
View the snooping profile information.
a.
show igmp snooping summary statistics
b.
show igmp snooping bridge-domain bd-name

detail statistics
c.
show igmp snooping port [if-type if-name] detail

statistics
Confirm that IGMP Snooping is sending and receiving

control traffic:
a.
View global traffic statistics.
b.
View traffic at the bridge domain level.
c.
View traffic at the port level.
11-248
OL-23591-02
Chapter 11

Command or Action
Step 3
a.
show igmp snooping group
b.
show igmp snooping group port
c.
show igmp snooping group source
d.
show igmp snooping group bridge-domain

bd-name
Step 5
Confirm that IGMP Snooping is creating group state as

expected:
a.
View group state in all bridge domains.
b.
View the groups created for a given port.
c.
View the group created for a given source.
show igmp snooping port if-type if-name group

[detail]
d.
View group state in the specified bridge domain.
e.
View group state on the specified interface.
a.
show l2vpn forwarding bridge-domain [bridge

group name:bd-name] mroute ipv4 location
lc-name
Confirm that the forwarding state matches the IGMP

Snooping state:
b.
show l2vpn forwarding bridge-domain [bridge

group name:bd-name] mroute ipv4 hardware
[ingress | egress] location lc-name
e.
Step 4
Purpose
a.
show l2snoop statistics summary
a.
View forwarding state in the L2FIB on the specified

line card.
b.
View forwarding state installed in the hardware on the

specified line card.
View the number of packets sent, received, and failed for

IGMP from and to l2snoop; determine whether packets are
being dropped.
Using the debug, trace, and show tech-support Commands

The following commands are useful for debugging IGMP snooping.
Note
These commands might cause a large amount of information to be displayed on your output terminal.
This command gathers information about the IGMP control packets in the system, for example, whether
JOIN and QUERY packets are being received and transmitted.
debug igmp snooping {all | error | event | init | management | packet | packet-error | proto | topo}
allAll debugging
errorError debugging
eventEvent debugging
initInit debugging
managementManagement debugging
packetPacket debugging
packet-errorPacket error debugging
protoProto debugging
topoTopology debugging
If IGMP control packets are not being received and transmitted as expected, use this command to help
locate the cause of the problem.
debug l2snoop {call | error | events | init | packet}
callL2snoop function call related debugging
OL-23591-02
11-249
Chapter 11
errorL2snoop error debugging
eventsL2snoop events debugging
initL2snoop init debugging
packetDecode L2snoop packet sends and receives
The following commands provide additional information for advanced troubleshooting.

show igmp snooping trace
allShow IGMP snoop trace all
errorShow IGMP snooping trace error
fileSpecific file
hexdumpDisplay traces in hexadecimal
lastDisplay last <n> entries
locationCard location
packet-errorShow IGMP snooping trace packet error
reverseDisplay latest traces first
statsDisplay statistics
tailfDisplay new traces as they are added
uniqueUnique entries with counts
verboseDisplay internal debugging information
wrappingWrapping entries
show tech-support igmp snooping {file | terminal}
fileSpecify a valid file name (for example, disk0:tmp.log)
terminalSend output to terminal
Troubleshooting Missing Routes and Forwarding Errors

This section explains what to do if packets are not being directed to the expected routes or are not being
received by the RSP card.
Step 1
Check the L2VPN configuration to verify that the IGMP snooping profile is configured on the bridge
domain and optionally on one or more bridge ports. The snooping profile must be present in the bridge
domain for IGMP snooping to be enabled.
RP/0/RSP0/CPU0:router# show run l2vpn
Step 2
Verify that IGMP snooping is enabled, that is, attached to the bridge-domain and optionally to one or
more bridge ports.
Step 3
Verify that Layer 2 multicast routes (mroutes) are present in the bridge-domain.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain [bridge-domain-id] mroute ipv4
summary location node-id
Step 4
Verify the status of flood forwarding. Confirm that the forwarding state matches the snooping state.
11-250
OL-23591-02
Chapter 11

RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain [bridge-domain-id] detail

location node-id
Step 5
Verify that the querier is enabled in the snooped domain. Without a querier, the system drops all IGMP
reports with reason No Querier.
RP/0/RSP0/CPU0:router# show igmp snooping summary statistics
include-zeroes | include Reports No
Thu Jan 6 12:03:10.715 MEZ
Reports No Querier:
0
V3 Reports No Querier:
0
V3 Reports No Sources:
0
Tip
Step 6
Use variations of this command (for example, omitting the include modifier) to show other types
of errors, such as time to live errors (TTL not 1), which are useful in troubleshooting.
Verify that the bridge domain, number of mrouters, ports, and IP addresses are as expected. Run the
snooping statistics several times to notice any trends and the corresponding locations.
RP/0/RSP0/CPU0:router# show igmp snooping bridge-domain [bridge-domain-id] detail
statistics
Step 7
Verify that packets are being received and transmitted as expected, and that there are no failed packets.
Clear the counters and rerun them several times to notice any trends and the corresponding locations.
RP/0/RSP0/CPU0:router# show l2snoop statistics pcb all location active-RSP-location
The output of this command is an aggregate of all Layer 2 snoop counters, therefore, further
investigation is needed to determine if any observed failures are related to IGMP snooping. (In this
command, PCB = protocol control block.)
Step 8
Check whether the IGMP_SNOOP packet counters are incrementing on the network processors (NPs).
If so, this indicates that the packets are being punted. Clear the counters and rerun them several times to
notice any trends and the corresponding locations.
RP/0/RSP0/CPU0:router# show controller np counters <np number or all> location <LC
location where punts are expected>
Examples
1.
This example shows that the IGMP snooping profile is configured in the bridge domain.
RP/0/RSP0/CPU0:router# show run l2vpn

Tue Jan 4 09:59:49.849 PST
l2vpn
router-id 10.144.144.144
pw-class CW_enable
encapsulation mpls
control-word
!
!
xconnect group g1
p2p p1
pw-class c1
!
!
!
OL-23591-02
11-251
Chapter 11
bridge group 215

bridge-domain 215
mtu 9000
igmp snooping profile default
storm-control multicast pps 500
storm-control broadcast pps 4500
!
!
!
vfi 215
!
!
!
!
.
.
.
2.
This example shows that IGMP snooping profile is present in the bridge domain, therefore IGMP
snooping is enabled.

.
.
.
Bridge group: 215, bridge-domain: 215, id: 4, state: up, ShgId: 0, MSTi: 0
MAC withdraw for Access PW: enabled
Flooding:
IGMP Snooping profile: default
Bridge MTU: 9000
MIB cvplsConfigIndex: 5
Filter MAC addresses:
No status change since creation
List of ACs:
.
.
.
3.
This example shows that Layer 2 multicast routes (mroutes) are present in the bridge-domain.
RP/0/RSP0/CPU0: router# show l2vpn forwarding bridge-domain 215:215 mroute ipv4 summary
location 0/1/CPU0
Tue Jan 4 10:20:57.264 PST
Global Mroute Stats:
-----------------------------------------------------------------------------------
11-252
OL-23591-02
Chapter 11

Message
=======
mcast route count
mcast route update dropped
mcast route delete dropped
mcast route del all drop
mcast route add
mcast route delete
mcast route delete all
mcast xid add
mcast xid delete
mcast stale xid delete
mcast stale delete
mcast bulk messages
Count
=====
7
0
0
0
27
20
0
124
15
0
0
48
Info1
=====
0x4
0x0
0x0
0x0
0x4
0x4
0x0
0x4
0x4
0x0
0x0
0x0
Info2
=====
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
Time
====
Jun 28
Jun 28
Jan 9
Jun 28
Jan 17
Jun 28
01:48:14.324
01:48:14.324
14:49:08.761
01:48:14.324
10:41:06.123
01:48:14.324
Per Bridge Mroute Stats:

Bridge-domain: 215:215 ,id: 4
L2 Multicast Route entries: 7
mroute add: 27, delete: 20
xid add: 124, delete: 15
mroute delete all: 0
mroute update dropped: 0
mroute delete dropped: 0
mroute delete all dropped: 0
Deleted stale mroute entries: 0
Deleted stale xid entries: 0
4.
This example shows that IGMP snooping is enabled and flooding is disabled.

0/1/CPU0
Mon Jan 3 14:49:50.332 PST
Bridge-domain name: 215:215, id: 4, state: up
Flooding:
DHCPv4 snooping: profile not known on this node
IGMP snooping: enabled, flooding: disabled
Bridge MTU: 9000 bytes
.
.
.
5.
This example shows the bridge domain, number of mrouters, ports, and IP addresses.
RP/0/RSP0/CPU0:router# show igmp snooping bridge-domain 215:215 detail statistics

Mon Jan 3 10:56:17.534 PST
Bridge Domain
------------215:215
Profile
------default
Act
--Y
Ver
--v2
#Ports
-----4
#Mrtrs
-----3
#Grps
----6
#SGs
---0
Profile Configured Attributes:
OL-23591-02
11-253
Chapter 11
System IP Address:
10.144.144.144
Minimum Version:
2
Report Suppression:
Enabled
Unsolicited Report Interval:
1000 (milliseconds)
TCN Query Solicit:
Enabled
TCN Membership Sync:
Disabled
TCN Flood:
Enabled
TCN Flood Query Count:
2
Router Alert Check:
Disabled
TTL Check:
Disabled
Internal Querier Support:
Enabled
Internal Querier Version:
3
Internal Querier Timeout:
0 (seconds)
Internal Querier Interval:
60 (seconds)
Internal Querier Max Response Time: 10.0 (seconds)
Internal Querier Robustness:
2
Internal Querier TCN Query Interval: 10 (seconds)
Internal Querier TCN Query Count:
2
Internal Querier TCN Query MRT:
0 (seconds)
Querier Query Interval:
60 (seconds)
Querier LMQ Interval:
1000 (milliseconds)
Querier LMQ Count:
2
Querier Robustness:
2
Startup Query Interval:
15 seconds
Startup Query Count:
2
Startup Query Max Response Time:
10.0 seconds
Mrouter Forwarding:
Enabled
Querier:
IP Address:
10.161.161.161
Port:
Neighbor 10.19.19.19 pw-id 215
Version:
v2
Query Interval:
60 seconds
Robustness:
2
Max Resp Time:
10.0 seconds
Time since last G-Query:
30 seconds
Internal Querier Statistics (elapsed time since last cleared 5w4d):
Rx General Queries:
142074
Rx General Queries When Disabled:
142072
Rx General Queries As Querier:
1
Rx General Queries As Non Querier:
0
Rx General Queries As Winner:
0
Rx General Queries As Loser:
0
Rx Global Leaves:
1590
Rx Global Leaves When Disabled:
1590
Rx Global Leaves As Non Querier:
0
Rx Global Leaves Ignored:
0
Rx Pim Enabled Notifications:
0
Rx Pim Disabled Notifications:
0
Rx Local Query Solicitations:
0
Tx General Queries:
0
Mrouter Ports:
3
Dynamic:
Neighbor 10.19.19.19 pw-id 215
Dynamic:
Dynamic:
STP Forwarding Ports:
0
ICCP Group Ports:
0
Groups:
6
Member Ports:
6
V3 Source Groups:
0
Static/Include/Exclude:
0/0/0
Member Ports (Include/Exclude):
0/0
Traffic Statistics (elapsed time since last cleared 5w4d):
Received Reinjected
Generated
Messages:
827226
411703
1599
11-254
OL-23591-02
Chapter 11

IGMP General Queries:

IGMP Group Specific Queries:
IGMP G&S Specific Queries:
IGMP V2 Reports:
IGMP V3 Reports:
IGMP V2 Leaves:
IGMP Global Leaves:
PIM Hellos:
Rx Packet Treatment:
Packets Flooded:
Packets Forwarded To Members:
Packets Forwarded To Mrouters:
Packets Consumed:
Reports Suppressed:
Rx Errors:
Packets Missing Router Alert:
Leaves Non-Member:
Rx Other:
None
Tx Errors:
No Querier in BD:
Startup Query Sync Statistics:
None
6.
142074
12
0
452290
0
159
2
232689
124872
0
0
286719
0
112
0
4
0
0
0
3
4
1588
-
124874
0
286841
398308
165570
452268
109
This example shows that packets are being received and transmitted, and that there are no failed
packets.
RP/0/RSP0/CPU0:router# show l2snoop statistics pcb all location 0/RSP0/CPU0

Tue Jan 4 10:48:17.423 PST
Statistics for PCB 0x50026960
Send:
0 packets received from application
0 xipc pulse received from application
0 packets sent to network (NetIO)
0 packets failed getting queued to network (NetIO)
Rcvd:
238504 packets received from network
238504 packets queued to application
0 packets failed queued to application
Statistics for PCB 0x50024fe0
Send:
424768 packets received from application
423062 xipc pulse received from application
424768 packets sent to network (NetIO)
0 packets failed getting queued to network (NetIO)
Rcvd:
611715 packets received from network
611715 packets queued to application
0 packets failed queued to application
7.
This example shows the IGMP_SNOOP control packet counters on the network processors (NPs).

Mon Jan 3 12:10:40.215 PST
Node: 0/1/CPU0:
Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------22 PARSE_ENET_RECEIVE_CNT
163222344
43
77489373
22
24 PARSE_LOOPBACK_RECEIVE_CNT
114662
0
OL-23591-02
11-255
Chapter 11
Troubleshooting Native Multicast Routing (Layer 3)
29
30
31
33
34
112
134
148
149
170
172
224
225
226
230
313
MODIFY_FABRIC_TRANSMIT_CNT
MODIFY_ENET_TRANSMIT_CNT
PARSE_INGRESS_DROP_CNT
RESOLVE_INGRESS_DROP_CNT
RESOLVE_EGRESS_DROP_CNT
DIAGS
IGMP_SNOOP
IPV4MC_DO_ALL
IPV4MC_DO_ALL_EXCD
PUNT_IFIB
PUNT_ADJ
PUNT_STATISTICS
PUNT_STATISTICS_EXCD
PUNT_DIAGS_RSP_ACT
NETIO_RP_TO_LC_CPU_PUNT
BFD_NOT_ENABLED
75222687
146294912
17654881
44
407520
56853
3
5860
13590
69828282
2
5667050
1
57337
1011
24
22
42
1
0
0
0
0 <<< snoop
0
0
20
0
2
0
0
0
0
.
.
.

This section explains how to troubleshoot native multicast. Native multicast routing (also called IP
multicast) is a bandwidth-conserving technology that reduces traffic by simultaneously delivering a
single stream of information to thousands of corporate recipients and homes.
Multicast PIE Installation Fails, page 11-262
Multicast CLI Unavailable Although PIE Is Installed, page 11-263
This command not authorized Error Message, page 11-263
Dynamic IGMP Failure, page 11-263
Traffic Fails on Some Interfaces, page 11-267
Traffic Fails on Some InterfacesMGID, page 11-268
Throughput Loss at Receiver Interfaces, page 11-268
Reverse Path Forwarding IP Address Problems, page 11-268

This section explains how to use the show and debug commands.
Figure 11-1 illustrates the flow of information in the native multicast process, along with some of the
important show commands.
On the RSP card, the IGMP and PIM send control packet information, including joins, queries,
reports, leaves and join-prune, to the MRIB. The MRIB populates the global routing tables and
allocates multicast group IDs (MGIDs).
On the LC, the MFIB receives routing information from the MRIB and programs the necessary
tables and structures in the hardware (network processors or NP) and sets up the multicast routes
and groups.
11-256
OL-23591-02
Chapter 11

Figure 11-1
Native Multicast Process and Corresponding CLI Commands
show pim group-map

show pim topology route-count
show pim neighbor
show pim summary
show igmp traffic

show igmp group summary
show igmp interface
IGMP
PIM
show mrib client filter
show mrib route summary
show mrib nsf
MRIB
RSP-0
show mfib route
show mfib connections
show mfib counter
show mfib nsf
show mfib hardware route olist loc <>
show mfib hardware connection loc <>
show mfib hardware ltrace loc <>
show mfib hardware interface loc <>
MFIB
show controller np counters all
show controller np struct <id> all
show controller np summary all
282276
NP
Line-Card
SUMMARY STEPS
1.
show igmp {global-interface | groups | | interface | nsf | old-output | snooping | ssm | summary
| traffic | vrf name}
2.
show pim [vrf vrf-name] {bgp-safi | bsr | context | df | global | group-map | interface | ipv4 | ipv6
| join-prune | ma | mdt | mstatic | multicast | neighbor | nsf | old-output | range-list | rpf | safi-all
| summary | table-context | topology | traffic | tunnel | unicast}
3.
show mrib [vrf vrf-name] {client | ipv4 | ipv6 | label-table-info | mdt-interface | mpls | nsf |
old-output | platform | route | route-collapse | table-info |tlc }
4.
show mfib [vrf vrf-name] {bundle-hash | connections | counter | encap-info | hardware |

interface | ipv4 | ipv6 | lsm | mdt | nsf | route | svd | table-info}
5.
show mfib hardware {adjacency | connection | interface | ltrace | resource-counters | route |

table} location node-id
6.
show mfib hardware route { accept-bitmap | internal | mofrr | olist | statistics | summary } {* |
A.B.C.D | A.B.C.D/length | detail | hex-dump} location node-id
7.
show mfib hardware route summary location node-id
8.
debug mrib errors
9.
debug mrib events
10. debug mfib warning

11. debug mfib errors
OL-23591-02
11-257
Chapter 11
12. debug mlib errors

13. debug mlib warning
DETAILED STEPS
Step 1
Command or Action
Purpose
show igmp {global-interface | groups |

interface | nsf | old-output | ranges |
snooping | ssm | summary | traffic | vrf name}
View all Internet Group Management Protocol

(IGMP)-related information in the control plane. IGMP is a
protocol used by IPv4 systems to report IP multicast
memberships to neighboring multicast routers.
Example:
RP/0/RSP0/CPU0:router# show igmp groups
Filter results using the following parameters and keywords:
global-interfaceIGMP global Interface Descriptor

Block (IDB) data structures, IDBs have information
like IP addresses, interface states, and packet statistics.
There is one IDB for each interface and one for each
subinterface.
groupsIGMP group memberships.
interfaceIGMP interface information.
nsfCurrent multicast NSF state for IGMP, either

normal or activated for NSF. The latter state indicates
that recovery is in progress due to an IGMP failure. The
total NSF timeout and time remaining are displayed
until NSF expiration.
old-outputProvides backward compatibility.
rangesIGMP group-map ranges
snoopingIGMP snooping parameters.
ssmSource Specific Multicast (SSM)-related

information.
summaryIGMP summary.
trafficIGMP traffic counters.
vrf nameSpecify a Virtual Private Network (VPN)

routing and forwarding (VRF).
11-258
OL-23591-02
Chapter 11

Step 2
Command or Action
Purpose
show pim [vrf vrf-name]{bgp-safi | bsr |

context | df | global | group-map | interface |
ipv4 | ipv6 | join-prune | ma | mdt | mstatic |
multicast | neighbor | nsf | old-output |
range-list | rpf | safi-all | summary |
table-context | topology | traffic | tunnel |
unicast}
View Protocol Independent Multicast (PIM)-related

information in the control plane. Filter results using the
bgp-safiBorder Gateway Protocol (BGP) secondary

address family (SAFI) database.
bsrPIM Bootstrap Router (BSR) information.
Example:
contextPIM VRF Contexts.
RP/0/RSP0/CPU0:router# show pim neighbor
dfBidirectional Designated Forwarder (DF).
globalPIM global summary
group-mapPIM group-to-protocol mapping

information.
interfacePIM interface information.
ipv4IPv4 Address Family.
join-prunePIM Join/Prune information.
maPIM Management Agent information
mdtData MDT information.
mstatic Multicast Static Route information.
multicastSAFI Multicast.
neighborPIM neighbor information.
nsfNon-stop forwarding.
old-outputProvides backward compatibility.
range-listPIM range-list information.
rpfRPF information.
safi-allSAFI wildcard.
summaryPIM summary information.
table-contextPIM Table context.
topologyPIM topology table information.
trafficPIM traffic counters.
tunnelTunnel interfaces.
unicastSAFI Unicast.
vrfVRF. If you include vrf, you must include it

immediately after show pim, and also specify a
vrf-name.
OL-23591-02
11-259
Chapter 11
Step 3
Command or Action
Purpose
show mrib [vrf vrf-name] {client | ipv4 | ipv6

| label-table-info | mdt-interface | mpls | nsf
| old-output | platform | route |
route-collapse | table-info |tlc}
View Multicast Routing Information Base (MRIB)

information. Filter results using the following parameters
and keywords:
Example:
RP/0/RSP0/CPU0:router# show mrib client
clientMRIB client connections.
ipv4IPv4 address family
ipv6IPv6 address family
label-table-infoMRIB label table information
mdt-interfaceMDT interface handle DB
mplsMRIB MPLS related information
nsfNon-stop forwarding
old-outputDisplay the old show output
platformPlatform-specific data
routeRouting database.
route-collapseMRIB route collapse database
table-infoMRIB VRF table information.
tlcMRIB table-linecard database

immediately after show mrib, and also specify a
vrf-name.
11-260
OL-23591-02
Chapter 11

Step 4
Command or Action
Purpose
show mfib [vrf vrf-name] {bundle-hash |

connections | counter | encap-info | hardware |
interface | ipv4 | ipv6 | lsm | mdt | nsf |
route | svd | table-info}
View Multicast Forwarding Information Base (MFIB)

information in the control plane. Filter results using the
Example:
RP/0/RSP0/CPU0:router# show mfib nsf
Step 5
show mfib hardware {interface | ltrace |

resource-counters | route} location node-id
Example:
RP/0/RSP0/CPU0:router# show mfib hardware route
olist location 0/4/CPU0
bundle-hashBundle hash for given Interface,S,G

tuple
connectionsStatus of MFIB connections to servers.
counterMFIB global counters.
encap-infoMulticast Virtual Private Network

(MVPN) Encap information.
hardwareCisco ASR 9000 Series Router hardware.
interfaceMFIB interface specific information.
lsmLabel Switched Multicast.
mdtMDT tunnel information.
nsfMulticast NSF status.
routeRouting database.
svdSingular Value Decomposition (SVD) events.
table-infoTable information.

immediately after show mfib, and also specify a
vrf-name or all.
View all hardware data in the Multicast PD. Filter results

using the following parameters and keywords:
interfaceCisco ASR 9000 Series Router hardware.
ltraceIP Multicast platform specific trace

information.
resource-countersAllocated and freed hardware

resources.
routePlatform-specific information for the routing

database.
locationSpecify the MFIB location.
Note
The output of these commands can be large when

there are a large number of routes and output
interface lists (olists) configured.
OL-23591-02
11-261
Chapter 11
Step 6
Command or Action
Purpose
show mfib hardware route {accept-bitmap |

internal | mofrr | olist | statistics |
summary} {* | A.B.C.D | A.B.C.D/length | detail
| hex-dump} location node-id
View all hardware ROUTE data in the multicast PD. Filter

accept-bitmapAccepting interface list for bidir

routes.
Example:
internalDisplay route internal structures

olist location 0/4/CPU0
mofrrDisplay per-route MoFRR information
olistOutput interface list (olist) stored in the

hardware.
statisticsPer route packets and bytes counters.
summarySummary of routes.
*Shared tree entries.
A.B.C.DSource/group IP address.
A.B.C.D/lengthGroup IP address/prefix length.
detailDetails of each route (requires 140 columns).
hex-dumpHex dump of the PLU and TLU.
locationSpecify the MFIB location.
Note
Step 7
show mfib hardware route summary location

node-id
Example:
The output of these commands can be large when

there are a large number of routes and olists
configured.
View all hardware ROUTE data in the multicast PD. Filter

summarySummary of routes.
locationMFIB location.

summary location 0/4/CPU0
Step 8
debug mrib errors
Example:
RP/0/RSP0/CPU0:router# debug mrib errors
Step 9
debug mrib events
Step 10
debug mfib warning
Step 11
debug mfib errors
Step 12
debug mlib errors
Step 13
debug mlib warning
To monitor Multicast Routing Information Base (MRIB)

internal errors, use the debug mrib errors command in
EXEC mode. To disable debugging output, use the no form
of this command.
Use these debug commands to obtain additional
information.
Multicast PIE Installation Fails

Step 1
View detailed information for the specified install id.

RP/0/RSP0/CPU0:router# show install log [1-4294967295] detail
11-262
OL-23591-02
Chapter 11

Step 2
Ensure that the PIE file name is correct and reissue the command.
Step 3
Ensure that the location of the PIE file is correct and reissue the command
Step 4
Ensure that the PIE file has proper permissions (755) and reissue the command.
Step 5
If you are loading from the TFTP directory, ensure that the following are true:
a.
Router has network connectivity.
b.
TFTP address is properly configured.
c.
TFTP server has connectivity.

RP/0/RSP0/CPU0:router# ping
d.
Step 6
tftp-server-addr
If loading locally from a router, ensure that the PIE file is stored on the router.
Verify all nodes are in the IOS XR RUN State

Multicast CLI Unavailable Although PIE Is Installed

show install activeView
active package information
Ensure that the following are correct:
PIE file name.
PIE file location.
PIE file installation on ALL nodes.
This command not authorized Error Message

While issuing certain commands in config or EXEC mode, the This command not authorized error
message appears, disabling further access. This means the user does not have the appropriate privileges.
Check to see that you have Cisco-Support and root privileges to use the desired command.
show config run(from
admin mode) View current operating admin configuration of the system.
Dynamic IGMP Failure

A dynamic IGMP failure occurs when the dynamic source and group states (*,G) are timing out. There
are two scenarios that suggest you need to troubleshoot this problem:
Step 1
IGMP join messages sent from a host does not resul in creation of a new route or addition of an
OLIST member; similarly, IGMP leave messages sent from a host do not result in deletion of an
existing route or removal of an existing OLIST member.
The groups and routes are configured and set up correctly, but when traffic is sent to the
Cisco ASR 9000 Aggregation Series Router from the tester, it is not received at the Rx tester port.
Verify that IGMP packets are being received by the IGMP process on the RSP.
OL-23591-02
11-263
Chapter 11
RP/0/RSP0/CPU0:router# show ipv4 traffic

Thu Jan 6 10:28:24.622 PST
IP statistics:
Rcvd: 294930104 total, 3052259 local destination
0 format errors, 0 bad hop count
593296 unknown protocol, 0 not a gateway
0 security failures, 0 bad source, 369473 bad header
1453418 with options, 0 bad, 0 unknown
Opts: 0 end, 0 nop, 0 basic security, 0 extended security
0 strict source rt, 0 loose source rt, 0 record rt
0 stream ID, 0 timestamp, 1453418 alert, 0 cipso
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble, 0 fragments received
0 fragmented, 0 fragment count, 0 fragment max drop
Bcast: 0 sent, 0 received
Mcast: 9878922 sent, 291744628 received
Drop: 0 encapsulation failed, 3 no route, 0 too big
Sent: 13116263 total
.
.
.
RP/0/RSP0/CPU0:router# show igmp traffic
Thu Jan 6 10:30:27.821 PST
IGMP Traffic Counters
Elapsed time since counters cleared: 6w0d
Valid IGMP Packets

Queries
Reports
Leaves
Mtrace packets
DVMRP packets
PIM packets
Received
491447
368364
123083
0
0
0
0
Sent
777437
246191
531246
0
0
0
0
Errors:
Malformed Packets
Bad Checksums
Socket Errors
Bad Scope Errors
Auxiliary Data Len Errors
Packets dropped due to invalid socket
Packets which couldn't be accessed
Packet allocation failure
Other packets drops
Step 2
0
0
0
0
0
0
0
0
0
Verify that IGMP has the specified group/source on the interface.

RP/0/RSP0/CPU0:router# show igmp interface gigabitEthernet 0/1/0/2
Thu Jan 6 10:34:30.664 PST
GigabitEthernet0/1/0/2 is up, line protocol is up
IGMP is enabled on interface
Current IGMP version is 3
IGMP query interval is 60 seconds
IGMP querier timeout is 125 seconds
IGMP max query response time is 10 seconds
Last member query response interval is 1 seconds
IGMP activity: 6 joins, 0 leaves
IGMP querying router is 10.147.4.44 (this system)
11-264
OL-23591-02
Chapter 11

Step 3
Verify that the multicast routing information base (MRIB) has the expected route and outgoing interface
list (OLIST).
RP/0/RSP0/CPU0:router# show mrib route
Thu Jan 6 10:38:42.800 PST
IP Multicast Routing Information Base
Entry flags: L - Domain-Local Source, E - External Source to the Domain,
C - Directly-Connected Check, S - Signal, IA - Inherit Accept,
IF - Inherit From, D - Drop, MA - MDT Address, ME - MDT Encap,
MD - MDT Decap, MT - MDT Threshold Crossed, MH - MDT interface handle
CD - Conditional Decap, MPLS - MPLS Decap, MF - MPLS Encap, EX - Extranet
MoFE - MoFRR Enabled, MoFS - MoFRR State
Interface flags: F - Forward, A - Accept, IC - Internal Copy,
NS - Negate Signal, DP - Don't Preserve, SP - Signal Present,
II - Internal Interest, ID - Internal Disinterest, LI - Local Interest,
LD - Local Disinterest, DI - Decapsulation Interface
EI - Encapsulation Interface, MI - MDT Interface, LVIF - MPLS Encap,
EX - Extranet, A2 - Secondary Accept
(*,10.66.66.66) RPF nbr: 10.114.8.11 Flags: C
Up: 6w0d
Incoming Interface List
TenGigE0/4/0/0 Flags: A, Up: 6w0d
Outgoing Interface List
GigabitEthernet0/1/0/8 Flags: F NS, Up: 6w0d
(10.191.4.1,232.166.166.166) RPF nbr: 10.114.8.11 Flags: MoFE MoFS
Up: 3w5d
MOFRR State: Active Sequence No 2856
GigabitEthernet0/1/0/23 Flags: A2, Up: 3w5d
GigabitEthernet0/1/0/8 Flags: F NS, Up: 3w5d
(*,10.60.0.0/16) RPF nbr: 10.144.144.144 Flags: IF
Up: 6w0d
Loopback0 Flags: F A, Up: 6w0d
GigabitEthernet0/1/0/18 Flags: A, Up: 3w5d
Loopback0 Flags: F A, Up: 6w0d
Step 4
Verify that the multicast forwarding information base (MFIB) on the LC has the interface as an OLIST
member.
RP/0/RSP0/CPU0:router# show mfib route location 0/1/CPU0
Thu Jan 6 10:47:06.989 PST
IP Multicast Forwarding Information Base
Entry flags: C - Directly-Connected Check, S - Signal, D - Drop,
IA - Inherit Accept, IF - Inherit From, MA - MDT Address,
ME - MDT Encap, MD - MDT Decap, MT - MDT Threshold Crossed,
MH - MDT interface handle, CD - Conditional Decap,
DT - MDT Decap True, EX - Extranet
MoFE - MoFRR Enabled, MoFS - MoFRR State
Interface flags: F - Forward, A - Accept, IC - Internal Copy,
NS - Negate Signal, DP - Don't Preserve, SP - Signal Present,
EG - Egress, EI - Encapsulation Interface, MI - MDT Interface,
EX - Extranet, A2 - Secondary Accept
OL-23591-02
11-265
Chapter 11
Forwarding/Replication Counts: Packets in/Packets out/Bytes out

Failure Counts: RPF / TTL / Empty Olist / Encap RL / Other
(*,10.0.0.0/4),
Flags: C
Up: 6w0d
Last Used: never
SW Forwarding Counts: 0/0/0
SW Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0/0
(*,10.66.66.66),
Flags: C
Up: 6w0d
Last Used: never
SW Forwarding Counts: 0/0/0
SW Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0/0
TenGigE0/4/0/0 Flags: A, Up:6w0d
GigabitEthernet0/1/0/8 Flags: NS EG, Up:6w0d
Step 5
Verify that the MFIB in the hardware has the interface as an OLIST member.
RP/0/RSP0/CPU0:router# show mfib hardware interface gigabitEthernet 0/1/0/27 location
0/1/CPU0
Thu Jan 6 10:55:27.723 PST
LC Type: A9K-40GE-L
-------------------------------------------------------------------Interface
Handle
RefCnt TTL Routes uIDB Enbld Comment
-------------------------------------------------------------------Gi0/1/0/27
0x2000740 8
0
3
55
True
success
-------------------------------------------------------------------ROUTE INFORMATION:
Legend:
S: Source, G: Group, P: Prefix length, PI: Packets cn, PO: packets out,
RF: RPF failures, TF: TTL failures, OF: OLIST failures, F: Other failures
Route flags - (Ingress)
C: Chip ID, IC: BACL check, IP: Punt this packet to LC CPU,
ID: Directly connected, IS: RPF interface signal, IU: Punt copy to RP,
IF: Punt to LC CPU if forwarded, IM: Result match, IV: Valid entry,
IR: RPF IF, IA: Fabric slotmask, IG: Mulicast group ID
Route flags - (Egress)
ET: Table ID to be used for OLIST lookup, EO: OLIST count bit,
ER: Route MGID to be used for OLIST/NRPF lookup, EM: Result match,
EV: Valid entry, EC: Count of OLIST members on this chip,
BS: Base of the statistics pointer
Interface: Gi0/1/0/27
S:0.0.0.0 G:10.60.0.0 P:16 PI:1784 PO:0 RF:670 TF:0 OF:0 F:0
--------------------------------------------------------------------------------C IC IP ID IS IU IF IM IV IR
IA
IG
ET EO ER
EM EV EC
BS
--------------------------------------------------------------------------------0 T F F F F F T T 0x2000740 0x0
0x4208 0 F 4
T T 0
0x36c6c
1 T F F F F F T T 0x2000740 0x0
0x4208 0 F 4
T T 0
0x36b04
2 T F F F F F T T 0x2000740 0x0
0x4208 0 F 4
T T 0
0x36b1c
3 T F F F F F T T 0x2000740 0x0
0x4208 0 F 4
T T 0
0x36d44
--------------------------------------------------------------------------------S:0.0.0.0 G:10.60.66.66 P:32 PI:113837 PO:999688227 RF:198 TF:0 OF:0 F:0
IA
IG
ET EO ER
EM EV EC
BS
--------------------------------------------------------------------------------0 T F F F F F T T 0x0
0x2
0x420b 0 F 7
T T 0
0x36c7b
1 T F F F F F T T 0x0
0x2
0x420b 0 T 7
T T 1
0x36b13
0x2
0x420b 0 F 7
T T 0
0x36b2b
11-266
OL-23591-02
Chapter 11

0x2
0x420b 0 F 7
T T 0
0x36d53
--------------------------------------------------------------------------------S:0.0.0.0 G:10.60.64.64 P:32 PI:84205 PO:973945464 RF:200 TF:0 OF:0 F:0
IA
IG
ET EO ER
EM EV EC
BS
--------------------------------------------------------------------------------0 T F F F F F T T 0x0
0x2
0x420a 0 F 6
T T 0
0x36c76
0x2
0x420a 0 T 6
T T 1
0x36b0e
0x2
0x420a 0 F 6
T T 0
0x36b26
0x2
0x420a 0 F 6
T T 0
0x36d4e
---------------------------------------------------------------------------------
Step 6
One possible cause could be that the IGMP group is timing out. One way to check is to create a static
route for the (*,G) and see if traffic is now received. If it is, it means that the groups are timing out.
Step 7
To confirm the result from Step 6, remove the static route and decrease the query interval (resulting in
more queries per minute) to make it clearer:
RP/0/RSP0/CPU0:router# conf
RP/0/RSP0/CPU0:router(config)# router igmp
RP/0/RSP0/CPU0:router(config-igmp)# query-interval 1
RP/0/RSP0/CPU0:router(config-igmp)# commit
Step 8
Ensure that packets are going out of interface at the interval set.
Step 9
Check that the tester responds with an IGMP membership report. If the packets are received at the tester,
the result from Step 6 is confirmed. Use the workaround.
Workaround
Configure static groups as a temporary workaround.
Traffic Fails on Some Interfaces

Traffic is failing on some interfaces or channels. You determine that
Groups and routes are configured and set up correctly.
Traffic is sent to the ASR 9000 from the tester.
Traffic is received correctly on some interfaces but not on others or some video channels are
received correctly on an interface while others are not.
Possible causes could be:
Step 1
OLIST may not be properly configured.
UIDB values not correctly set in hardware.
MGID not correctly set up.
Ensure that packets are going from the ingress network processor (NP) through the fabric to the egress
NP.
RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id |
egress node-id }
OL-23591-02
11-267
Chapter 11
Step 2
View olist interfaces for the route.

RP/0/RSP0/CPU0:router# show mfib hardware route olist location {ingress node-id | egress
node-id }
Step 3
View the statistics for the specific route and source.

RP/0/RSP0/CPU0:router# show mfib hardware route stat [src ip addr] location {ingress
node-id | egress node-id }
Traffic Fails on Some InterfacesMGID

Step 1
Ensure that packets are going from the ingress NP through the fabric to the egress NP.
egress node-id }
Step 2
View olist interfaces for the route.

RP/0/RSP0/CPU0:router# show mfib hardware route olist location {ingress node-id | egress
node-id }
Step 3
Ensure that packets are transmitted out of ingress NP to fabric and received by egress NP from the fabric.
egress node-id }
Step 4
View the MGID for the route.
Throughput Loss at Receiver Interfaces

Traffic is sent and received on routes but there is a loss of throughput at the receiver.
Step 1
Ensure that packets are going from the ingress NP through the fabric to the egress NP.
egress node-id}
Step 2
The preceding command checks if if packets are punted to the RP. If so, check if the source of that
channel is setting some IP options or not.
Reverse Path Forwarding IP Address Problems

Reverse Path Forwarding (RPF) ensures loop-free forwarding of multicast packets in multicast routing.
Packets from Wrong IP AddressLoose RPF, page 11-269
11-268
OL-23591-02
Chapter 11

Packets Forwarded with Wrong IP AddressStrict RPF, page 11-269

show cef ipv4 interfaceView IPv4 Cisco Express Forwarding (CEF)-related information for an
interface.
Packets from Wrong IP AddressLoose RPF

In loose RPF, the packets incoming on that particular interface are checked to determine if the source IP
of the packet is reachable through some interface on the box. If not, the packet is dropped.
RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id
Workaround
Unconfigure and configure loose RPF on the interface.
Packets Forwarded with Wrong IP AddressStrict RPF

In strict RPF, the packets incoming on that particular interface are checked to determine if the source IP
of the packet is reachable through the same interface on the box on which the packet came in. If not, the
packet is dropped.
Verify non-null rpf_ptr and uidb list with the show cef command.
RP/0/RSP0/CPU0:router# show cef {ipv4 | ipv6} prefix hardware egress detail location
node-id
Workaround
Unconfigure and configure strict RPF on the interface.
OL-23591-02
11-269
Chapter 11
11-270
OL-23591-02
INDEX
access control list (ACL)

access privileges
adjacency
1-49
describe hostname command
1-2
diagnostic commands
4-101
alarm indication signal (AIS)

ARP
describe command
10-241
1-21
1-59
documentation
3-93
prerequisite for troubleshooting
3-75
ASIC errors
dynamic IGMP failure
1-54
1-1
11-263
bidirectional forwarding detection
Ethernet CFM
3-81
3-85
Ethernet Connectivity Fault Management
3-85
C
CCM
3-89
CEF troubleshooting
CFM
fabric
4-95
forwarding information base (FIB)
3-85
cfs check command

cisco-support task ID
1-3
1-58
1-2
gathering information
commit confirmed command

Continuity check messages
3-89
4-109
counters, NP
3-85
I
IGMP reports dropped
control plane Ethernet network

overview
1-58
1-20
connectivity fault management (CFM)

control plane
4-95
1-22
Cisco Technical Support

CLI access
4-109, 7-143
1-41
IGMP snooping
11-251
11-247
IGMP snooping querier
7-148
installation, software
crashes, RSP and LC
7-165
install verify command
crosscheck for MEPs
3-92
interfaces
11-251
1-7, 1-10
1-10
2-61
interfaces, connectivity
interfaces, optical
3-75
2-68
OL-23591-02
IN-1
Index
IP multicast
point-to-point Layer 2 deployment
11-256
prerequisite documentation
L
L2VPN
prompt, router
1-2
punted packets
6-129, 7-155
1-1
9-181
Layer 3 routing problems
6-125
Q
QoS features
M
MAC address updates
9-192
maintenance domains
3-85
Management Ethernet interface

man command
memory
RDI
1-4
3-91
remote defect indication

router prompt
1-38
MEP defects
RSP switchover
3-92
3-91
1-2
6-138
3-90
MEPs and MIPs

MIB download
3-88
1-57
8-173
multicast
10-233
1-46
MEP crosscheck
MPLS
9-206
show adjacency command
11-247
4-102
show adjacency detail hardware command
Multiple Spanning Tree (MST)
9-230
multipoint Layer 2 bridging services (VPLS)
4-102
show adjacency hardware trace location command

9-195
show adjacency ipv4 nexthop command
4-103
4-102
show adjacency ipv4 nexthop detail hardware

command 4-102
show adjacency remote detail command
native multicast routing
show adjacency remote detail hardware command

show adjacency trace client command
network
documenting
NP counters
11-256
1-1
7-148
show adjacency trace command

show arp command
4-103
1-5, 4-101
show asic-errors command
4-101
1-54
show cef adjacency tunnel-te command
optical line card
2-68
show cef ipv4 detail command
4-103
4-97
show cef ipv4 detail location command
4-98
show cef ipv4 hardware egress command
show cef ipv4 hardware ingress command
packet drops
ping
3-75
7-158
4-102
4-103
show arp traffic location command
4-101
show cef ipv4 interface command
4-98
show cef ipv4 summary command
4-99
show cef ipv4 trace command
4-98
4-98
4-99
IN-2
OL-23591-02
Index
show cef platform trace ipv4 all command

show cfgmgr trace command
show uidb data command
1-22
show configuration commit changes command

show configuration commit list command
show configuration failed command
1-19, 1-22
software installation
1-22
show configuration history commit command
software version
1-22
show controllers backplane ethernet clients all

command 1-44
startup failed
show controllers backplane ethernet local clients statistics

command 1-44
show environment command
show hw-module subslot command
show install command
8-176
7-168
1-4
8-176
user access
2-62, 4-110
show platform command
1-27, 1-44, 1-49
4-97
version
1-21
1-8
VLAN traffic
show running-config command
1-2
1-28
show netio idb command
1-16, 1-27, 1-28
1-18
show sysdb trace verification command

show system verify command
4-106
1-28
show memory summary command
show sysdb trace command
traffic drop, transient
1-5
1-38
show running-config
6-130
tunnel, TE
1-27, 1-59
show route ipv4 command
1-56
transport input telnet command
1-3
1-5
show memory heap command
6-133, 6-136
traffic loss
2-62
show ipv4 interface command
1-28
traffic engineering tunnel
1-29
show interfaces brief command
9-182
1-56
traceroute
1-12, 1-14
1-58
1-50
trace commands
1-12
show interface brief command
show memory
trace
2-62
tracebacks
show install committed command
show logging command
gathering information
3-87
top processes command
1-8
show interfaces command
top command
4-110
show interface command
2-62
1-24
TCAM classification
show hw-module subslot address status

pluggable-optics 2-68
show install active command
1-8
TAC
1-28
1-52, 1-53
1-7, 1-10
2-62
show ethernet cfm configuration-errors command

show history command
1-8, 1-27
sohw hw-module subslot counters framer command
1-28, 1-50
show controllers stats command
4-110
1-52
show version command
1-24
1-59
4-110
show uidb index command

show users command
1-20
show configuration failed startup command

show context command
show tech-support command
4-99
VPLS
9-182
9-195
VPWS deployment
1-22
VRRP
9-206
6-138
1-29, 1-59
OL-23591-02
IN-3
Index
W
WRED weighted random early detection
10-237
IN-4
OL-23591-02

TR 40 Asr 9 Kbook

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

TR 40 Asr 9 Kbook

Încărcat de

Drepturi de autor:

Formate disponibile

Cisco IOS XR Troubleshooting Guide for

the Cisco ASR 9000 Aggregation Services

Text Part Number: OL-23591-02

Changes to This Document

Obtaining Documentation and Submitting a Service Request

General Troubleshooting Procedures

Prerequisite Documentation for Troubleshooting

Verifying and Troubleshooting CLI Access 1-2

Validating and Troubleshooting Installation of the Cisco IOS XR Software Package

Basic Cisco IOS XR Verification and Troubleshooting Commands

Troubleshooting the Backplane Ethernet Control System

Displaying ASIC Errors

Gathering Information Before You Call Cisco TAC 1-58

Verifying and Troubleshooting Interface Status

Verifying and Troubleshooting Gigabit Ethernet Interfaces

Verifying and Troubleshooting Pluggable Optical Line Card Interfaces

Troubleshooting Interface Connectivity

Troubleshooting Ping and ARP Connectivity

Troubleshooting Bidirectional Forwarding Detection 3-81

Troubleshooting Packet Forwarding

Troubleshooting Adjacency Information

Troubleshooting Transient Traffic Drop

Troubleshooting Packet Drop in the Fabric

Troubleshooting Control Plane Information

Troubleshooting Bundles and Load Balancing

Troubleshooting Problems with Link Bundles 5-118

Troubleshooting Layer 3 Connectivity

Packets Are Punted and Switched in Software

Adding Routes Fails

Traffic Loss Because of Changing encap on a Subinterface

Troubleshooting Virtual Router Redundancy Protocol

Tracked Interface Failing, Router State Not Changed 6-140

Troubleshooting Router Switch Fabric and Data Path

Getting Started with Fabric Troubleshooting

Troubleshooting Packet Drops 7-146

Troubleshooting Complete Loss of Traffic 7-168

Troubleshooting MPLS Services

Verifying MPLS PIE Activation and MPLS Configuration

IP Packets Not Forwarded to MPLS TE Tunnel

MPLS Packets Not Forwarded to MPLS TE Tunnel

FRR-Protected Tunnel Goes Down After Triggering FRR

MPLS FRR Switch Time Debugging

Troubleshooting L2VPN and Ethernet Services

Troubleshooting VLAN Traffic and L2 TCAM Classification 9-181

Troubleshooting Point-to-Point Layer 2 Services 9-206

Show Commands 9-228

Troubleshooting Quality of Service and Access Control Lists

Packets in Wrong Queue

Packets Incorrectly Marked

Packets Incorrectly Policed

Service-Policy Configuration Is Rejected

Weighted Random Early Detection Incorrect

Bandwidth Ratio Not Working

Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands

After QoS EA Restarts, show policy-map interface Fails

Bundle Members Not Configured with service-policy

Using show and debug Commands 10-242

Troubleshooting Multicast Services

Troubleshooting IGMP Snooping (Layer 2 Multicast) 11-247

Changes to This Document, page xi

Obtaining Documentation and Submitting a Service Request, page xii

Changes to This Document

Changes to This Document