Sunteți pe pagina 1din 112

Troubleshooting Cisco Catalyst

3750, 3560, 2960-S and 2960-X


Series Switches
BRKCRS-3141

John Wu
BU Escalation

Administrators spend most of their time


Monitoring and troubleshooting

Security-related configuration
Initial install, config & testing
Upgrade of older equipment
Traffic optimization
Other

0%
Source: The Total Economic Impact of Cisco Catalyst Access Switching,
A Commissioned Study Conducted by Forrester Consulting On Behalf of Cisco Systems, January 2012
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

10%

20%

30%

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

Compact

Product Overview
3750-X

3560-X
2960-X/2960-XR

3750v2

3560-E
2960-S/SF
3560v2

3750 Stack Rear View

2960-S Stack Rear View


BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Compact

Product Overview

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Catalyst 3750-X : Architecture Overview


Switch Fabric

Stack PHY

TCAM

TCAM

Port ASIC

Two
Stack
Cables

TCAM

Port ASIC

3
Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular
2 PHY

Flash
Serial
10/100

24X1G POE

12X1G

12X1G

24X1G POE

12X1G

12X1G

StackWise,
StackWise
Plus

10G or 1G

Packet Flow across switch components

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Switch Components: Troubleshooting Areas of Focus


Stack
PHY

Switch Fabric
TCAM

TCAM

Port ASIC

Two
Stack
Cables
Stack
errors

TCAM

Port ASIC

CPU

Port ASIC

SDRAM
12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial

TCAM

12X1G

High ?

10/100

24X1G POE

12X1G

24X1G POE

12X1G

StackWise,
StackWise
Plus

10G or 1G

12X1G

Buffers?
QoS

TCAM
Resources?
BRKCRS-3141

Interface
Issue?

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

No PoE ?
8

Running
out?

Before We Start
The outputs in this presentation are from 3750, 3750x
Troubleshooting the 2960, 3560, and 3750 series switches are basically the same
Differences called out

Caution!!!

debug and show platform commands to follow in the slides


Excessive debug output to console may disable switch
show platform commands are intended for in-depth troubleshooting by Cisco engineers
Use debug and show platform commands as advised by TAC only

TroubleShooting Basics
Check the syslog for warnings and errors
Use common sense
Some TS techniques impact switch operation

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

10

Compact

Troubleshooting Link Issues


Switch Fabric
TCAM

TCAM

Port ASIC

Stack PHY
TCAM

Port ASIC

Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial
10/100

Link issues

When are we concerned about the link issues?


Connectivity
Traffic

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

11

Troubleshooting Link Issues


Is the link up/up?
Are packets being sent and received ?
Are there any errors ?
Collect data multiple times to confirm
Switch# show interface GigabitEthernet 1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
....
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Output queue: 0/40 (size/max)
7539 packets input, 9856 bytes, 0 no buffer
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 4059 multicast, 0 pause input
0 input packets with dribble condition detected
3508 packets output, 3560 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

12

Ethernet Controller Stats

Details about errors

Clear Stats with clear controller ethernet-controller command

Switch# show controller ethernet-controller GigabitEthernet 1/0/1

Transmit GigabitEthernet4/0/1
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
..
0 Excessive collisions
0 Late collisions
0 VLAN discard frames
0 Excess defer frames

Receive

..

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

13

0
0
0
0
0

Alignment errors
FCS errors
Oversize frames
Undersize frames
Collision fragments

0
0
0
0

Symbol error frames


Invalid frames, too large
Valid frames, too large
Invalid frames, too small

PHY Registers
Information from PHY registers
False carrier and symbol errors are reported

Switch#show controllers ethernet-controller g0/46 phy detail


GigabitEthernet0/46 (gpn: 46, port-number: 46)
0000: 3100 Control Register
0001: 7969 Control STATUS
<removed>
0015: 646B Receive Error Counter

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

14

:
:

0011 0001 0000 0000


0111 1001 0110 1001

0110 0100 0110 1011

Link Issues: What Kind of Errors?


Switch# show int gi1/0/1 counters
Port
Gi1/0/1

InOctets
9856

InUcastPkts InMcastPkts
7539
4059
14

InBcastPkts

Port
Gi1/0/1

OutOctets OutUcastPkts OutMcastPkts OutBcastPkts


3560
3508
3056
23

Switch# show interfaces GigabitEthernet 1/0/1 counters errors


Port
Align-Err
Gi1/0/1
0

FCS-Err Xmit-Err
0
0
0

Rcv-Err UnderSize OutDiscards


0
0

Port
Single-Col Multi-Col Late-Col Excess-Col Carri-Sen
Gi1/0/1
0
0
0
0
0
0
0

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

15

Runts

Giants

Link Issues: Link Not Coming Up


Check configured duplex and speed on both switch and host
Upgrade the NIC drivers on the host to the latest version
Try a different cable/NIC and switchport to exclude faulty hardware

Switch# show interfaces status | inc connected


Gi1/0/1
connected trunk
a-full 10 10/100/1000BaseTX
Gi1/0/2
connected 101
a-full a-100 10/100/1000BaseTX
Gi1/0/24
connected 1
a-full a-1000 10/100/1000BaseTX

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

16

SFP Link Issue Prevention EEM TCL Use Case


1000 Base-T (copper) and 100
Base-FX SFPs have embedded
PHYs, allowing speed and duplex
to be configured on their respective
interface
1

LinkUpApplyConfig is a Tcl policy


that monitors an SFP link-up event

EEM
Tcl
Policy

However, these settings are


cleared as soon as the SFP is
unplugged

Speed and duplex settings (in


startup-config) are automatically reapplied to the SFP interface

If the same SFP type is re-inserted,


its configuration is not recovered

LinkUpApplyConfig.tcl can be downloaded at the following hyperlink:


https://supportforums.cisco.com/docs/DOC-23267
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

17

Link Issues: Checking Physical Cabling


TDR feature helps determine possible cabling issues

Switch# test cable-diagnostics tdr interface GigabitEthernet4/0/1


TDR test started on interface Gi4/0/1
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.
Switch#
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/1, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet4/0/1, changed state to down
*%LINK-3-UPDOWN: Interface GigabitEthernet4/0/1, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/1, changed state to upw
Switch# show cable-diagnostics tdr interface GigabitEthernet4/0/1
TDR test last run on: March 01 03:11:11
Interface Speed Local pair Pair length
Remote pair Pair status
--------- ----- ---------- ------------------ ----------- -------------------Gi4/0/1
100M Pair A
N/A
Pair A
Normal
Pair B
9
+/- 10 meters Pair B
Open
Pair C
8
+/- 10 meters Pair C
Short
Pair D
9
+/- 10 meters Pair D
Short

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

18

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

20

Compact

Switch Hardware Components: CPU


Switch Fabric
TCAM

TCAM

Port ASIC

Stack PHY
TCAM

Port ASIC

Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial
10/100

CPU Functions
-

BRKCRS-3141

Runs the IOS


Processes Control Plane traffic (LACP / STP / CDP / etc.)
Processes packets that are not switched in Hardware
Communicates with controller(s)

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

21

High
Slow

CPU: Troubleshooting Processes

High CPU utilization is due to

Processes taking up resources

Forwarded Network Traffic

Normal CPU utilization varies by

Switch Model

Number of connected ports, modules, switches in stack

Feature set (LANBASE, IP BASE, IP SERVICES)

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

22

CPU: Troubleshooting Processes

Is CPU high an issue?

HW switched traffic not impacted by high CPU

Its an issue when these issues occur because control packets not sent or received

Spanning Tree Protocol (STP) reconverges

Routing protocol flaps

CLI is slow or unresponsive

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

23

CPU: Troubleshooting Processes


Use show process cpu history to display the history of CPU utilization
Switch# show processes cpu history
7466466455553535356639
4814199847367790442069
100
*
90
*
80
*
70 *
**
*
60 * ** ** * * *
** *
50 **** ******* * * *** *
40 **************** *****
30 **********************
20 **********************
10 ######################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0
5
0
5
0
5
0
5
0
5
0
5
0
CPU% per hour (last 72 hours)
* = maximum CPU%
# = average CPU%

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

24

CPU: Troubleshooting Processes


Configure the CPU threshold
(config)# process cpu threshold type {total | process | interrupt}
\ rising percentage interval seconds [falling fall-percentage interval seconds]
Syslog message
*Mar 1 01:03:15.601: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilisation
(Total/Intr): 18%/0%, Top 3 processes(Pid/Util): 4/10%, 75/1%, 164/0%

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

25

CPU: Troubleshooting Processes


Use show process cpu sorted to display current CPU utilization
Switch# show processes cpu sorted
CPU utilization for five seconds: 43%/7%; one minute: 28%; five minutes: 22%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
196
897835293 538983117
1665 6.05% 6.74% 10.05%
0 IP Input
102
46542612 69782387
666 2.33% 1.79% 1.61%
0 hpm main process
8
7967710
67451
118127 2.33% 0.29% 0.19%
0 Licensing Auto U
141
48894294 114699852
426 1.24% 1.01% 1.18%
0 Hulc LED Process
68
45347109
1374466
32992 1.24% 0.85% 0.86%
0 Adjust Regions

Total CPU utilization 43%

Processes could cause high CPU

Hulc running con, SFF8472, IP Input


Interrupt based CPU
utilization 7%

Hulc LED Process, Exec/Virtual Exec Process


SNMP Engine Process, etc.

Process based CPU


utilization 36%

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Use EEM script to monitor

Cisco Public

26

16 CPU Queues and Port ASIC queues


16 different CPU Queues
Packets to CPU Queues first stored on port ASIC
Switch# show platform port-asic stats drop
Supervisor TxQueue Drop Statistics
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue

BRKCRS-3141

0:
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:

0
0
0
0
0
0
0
1000
0
0
0
0
0
0
0
0

packets dropped
before reaching
to the CPU

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Switch# show controllers cpu-interface


cpu-queue-frames retrieved dropped
----------------- ---------- ---------rpc
132917740 0
stp
31879262
0
ipc
10746915
0
routing protocol 267
0
L2 protocol
424610
0
remote console
1121711
0
sw forwarding
0
0
host
345
0
broadcast
13931
0
cbt-to-spt
0
0
igmp snooping
0
0
icmp
0
0
logging
0
0
rpf-fail
0
0
dstats
132935598 0
cpu heartbeat
82903147
0

27

16 CPU Queues & CPU Buffer Pools


Each queue reserves buffers
for specific traffic

Switch# show buffer | in RxQ

RxQ0 buffers (rpc)


RxQ1 buffers (stp)
RxQ2 buffers (ipc)
RxQ3 buffers (routing protocol)
RxQ4 buffers (L2 protocol)
RxQ5 buffers (remote console)
RxQ6 buffers (sw forwarding)
RxQ7 buffers (host)
RxQ8 buffers (broadcast)
RxQ9 buffers (cbt-to-spt)
RxQ10 buffers (igmp snooping)
RxQ11 buffers (icmp)
RxQ12 buffers (logging)
RxQ13 buffers (rpf-fail)

CPU buffer pools are named


RxQ0 to RxQ15

RxQ15 buffers (cpu heartbeat)

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

28

CPU: Software Forwarding Queue (Q6)


For Traffic that hardware cannot process

- SW forwarding performance is much lower than HW

To debug any CPU Q

Switch# debug platform cpu-queues software-fwd-q


*Mar 1 10:37:33.205 AEDT: SW-FWD-Q:IP packet: Local Port Fwding L3If:Vlan1
L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1
SrcGPN:56, SrcGID:56, ACLLogIdx:0x0,
MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0
IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1
IP Opts

Incoming physical
interface

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

SMAC of the host sending


the traffic

Cisco Public

29

CPU: Routing Protocol Queue (Q3)

Receives all traffic for routing protocols (BGP, OSPF, EIGRP, HSRP, etc.)

Switch# debug platform cpu-queues routing-protocol-q


Switch# debug standby

HSRP debugging is on
*Mar

6 00:47:39.260: RT-Q:Queued: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1


DI:0x12FC, LT:7, Vlan:100
SrcGPN:1, SrcGID:1, ACLLogIdx:0x0, MacDA:0100.5e00.0002, MacSA:
0018.ba88.1fc1
IP_SA:10.1.1.2 IP_DA:224.0.0.2 IP_Proto:17

*Mar

BRKCRS-3141

6 00:47:39.260: HSRP: Vl100 Grp 0 Hello

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

in

30

10.1.1.2 Standby pri 100 vIP 10.1.1.55

CPU: Host Queue (Q7)


Used for all unicast traffic sent to the switch
TACACS, SSH, telnet, ping, SNMP

Show buffer shows current buffer usage


Switch# debug platform cpu-queues host-q
*Mar

6 00:01:46.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100


L2If:GigabitEthernet1/0/1 DI:0xB0, LT:7, Vlan:100
SrcGPN:489, SrcGID:488,
ACLLogIdx:0x0, MacDA:000f.f7e8.e041, MacSA: 0018.ba88.1fc1
IP_SA:10.1.1.2
IP_DA:10.1.1.1 IP_Proto:1

TPFFD:DC0001E9_00000064_00B00076-000000B0_A68A0000_00000000
Switch# show buffer | begin RxQ7
RxQ7 buffers, 2040 bytes (total 192, permanent 192):

64 in free list (0 min, 192 max allowed)


294 hits, 0 misses

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

31

CPU: ICMP Queue (Q11)

Receives all traffic for which an ICMP message needs to be generated


Excluding PING
Routed port only
Switch# debug ip icmp
Switch# debug platform cpu-queues icmp-q
*Mar

9 21:34:30.695: ICMP-Q:Queued to Process, use GW:10.1.1.3: Remote Port Blocked


L3If:Vlan100 L2If:GigabitEthernet4/0/1 DI:0xB4, LT:7, Vlan:100
SrcGPN:163,
SrcGID:163, ACLLogIdx:0x0, MacDA:0018.ba88.1fc1, MacSA: 000f.f7e8.e041
IP_SA:10.1.1.1
IP_DA:77.1.1.1 IP_Proto:1

*Mar

BRKCRS-3141

9 21:34:30.695: ICMP: redirect sent to 10.1.1.1 for dest 77.1.1.1, use gw 10.1.1.3

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

32

CPU: ICMP Queue (Q11)


ICMP Unreachables Example
Symptoms:
Relatively high CPU
Low processor utilization
ICMP Queue heavily utilized
Switch# show processes cpu sorted
CPU utilization for five seconds: 53%/47%; one minute: 31%; five minutes: 18%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
149
397089
3879429
102 0.63% 0.34% 0.45%
0 Spanning Tree
112
325474
117735
2764 0.31% 0.15% 0.09%
0 HRPC qos request
Switch# clear controllers cpu
Switch# show controllers cpu-interface | include icmp
icmp
133148
0
0
0
0

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

33

CPU: ICMP Queue (Q11)


ICMP Unreachables Example
Causes
High amount of traffic is dropped because of a deny statement in an ACL
CPU is interrupted to send ICMP unreachable packets back to the source

Solution
Disable ICMP unreachables on the ingress interface

Switch(config)# interface GigabitEthernet1/0/2


Switch(config-if)# no ip unreachable
Switch(config-if)# end

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

34

CPU: Best Practices


Storm Control can help to protect CPU
(config-if)#storm-control broadcast level level[.level]
(config-if)#storm-control action ?
shutdown Shutdown this interface if a storm occurs
send SNMP trap if a storm occurs

Protocol Storm Protection (PSP)


(config-if)#psp ?
arp Set rate limit value for ARP Packets
dhcp Set rate limit value for DHCP Packets
igmp Set rate limit value for IGMP Packets

The switch drops all traffic on the VP for 30 seconds


Enable parser config cache interface

http://www.cisco.com/c/en/us/td/docs/ios/fundamentals/configuration/guide/15_1s/cf_15_1s_book/config_cache.html#wp1057005

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

35

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

36

Compact

Switch Hardware Components: Memory


Switch Fabric
Port ASIC

Port ASIC

Stack PHY

Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial
10/100

Two Types of Memory


-

Processor memory is used by IOS Processes

I/O memory is used for CPU traffic

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

37

Running
out?

Troubleshooting Memory Utilization


Syslog messages most common indication
%SYS2MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4, pool Processor, alignment 0
Process= "IP Input", ipl= 2, pid= 21
%PLATFORM_RPC-0-RESOURCE_CRASH: System is unable to allocate memory for RPC

Switch not accessible, any CLI output becomes show process memory, etc.
Switch# show memory statistics

Processor
I/O

Head
2641D6C
7400000

Total(b)
81519252
12574720

Memory available now

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Used(b)
31192204
8532852

Free(b)
50327048
4041868

The lowest free


since boot up
Cisco Public

38

Lowest(b)
49241540
3821068

Largest(b)
48621848
4039616

Largest block switch


can allocate

Troubleshooting Memory Utilization


Two main reasons of Processor memory problems
-

Is any process steadily


increasing held memory?

Process does not release the memory after use


Process does not limit the amount of memory it allocates

Switch# show processes memory sorted


PID TTY Allocated
Freed
Holding
0
0
74539888
23738156
47199076
0
0
3399716
17490880
1590292
65
0
712620
27424
594488
324
0
19794764
19262624
539264
11
0
228060
14940
226488

Getbufs
0
10657136
0
0
0

Retbufs
0
553112
0
0
0

Process
*Init*
*Dead*
Stack Mgr Notifi
hulc running con
ARP Input

Run commands multiple times to benchmark


show processes memory sorted
show memory summary
show memory allocating totals

Use Threshold notifications for periodic monitoring


Switch(config)# memory free low-watermark processor 20000
Switch(config)# memory free low-watermark io 20000
%SYS-4-FREEMEMLOW: Free Memory has dropped below 20000k
Pool: Processor Free: 66814056 freemem_lwm: 204800000

Reference:
http://www.cisco.com/en/US/docs/ios/12_2s/feature/guide/fs_memnt.html

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

39

Memory Leak Example (using show commands)


Switch#show clock
09:34:41.300 UTC Wed Apr 3 2013
switch#show proc mem sorted
Processor Pool Total: 78964596 Used: 36942892 Free:
42021704
I/O Pool Total: 12574720 Used: 8583916 Free: 3990804
PID TTY Allocated Freed
Holding Getbufs Retbufs Process
204 0
25509496 21347536 3957470
0
0 HTTP CORE
Switch#show clock
11:34:41.300 UTC Wed Apr 3 2013

Switch#show proc mem 204


Process ID: 204
Process Name: HTTP CORE
Total Memory Held: 4175420 bytes
Processor memory Holding = 4175420 bytes
pc = 0x015E5430, size = 2143156, count = 5265
pc = 0x0166F148, size = 1643716, count = 11583
pc = 0x01685C18, size = 231660, count = 351
pc = 0x004F4B60, size = 35136, count = 61
pc = 0x015E55C8, size = 27976, count = 351
pc = 0x015DB614, size = 25720, count = 351
pc = 0x01141F34, size = 2408, count = 2

switch#show proc mem sorted


Processor Pool Total: 78964596 Used: 36942892 Free:
42021704
I/O Pool Total: 12574720 Used: 8583916 Free: 3990804
PID TTY Allocated Freed
Holding Getbufs Retbufs Process
204 0
25727446 21347536 4175420 0
0 HTTP CORE

Provide captures to TAC if no bugs are found in bug tool kit


A quick search leads to :
CSCsk34832 Memory leak in HTTP CORE

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

40

IOS Memory Leak Detector


Inbuilt Memory Leak Detector can be used to detect memory leaks
show memory debug leaks [chunks | largest | lowmem | summary]
Switch#show memory debug leak
Adding blocks for GD...
I/O memory
Address Size Alloc_pc PID Alloc-Proc Name
Processor memory
Address Size Alloc_pc
28D91E8 72
16EF0B4
28D927C 124
16EF0B4
28D9358 68
16EF0B4

PID Alloc-Proc Name


0 *Dead* SSH2 String
0 *Dead* SSH2 String
0 *Dead* SSH2 String

A quick search leads to :


CSCsm89128 Memory Leak when invalid password entered for SSHv2 session
Reference:
http://www.cisco.com/en/US/partner/docs/ios/fundamentals/configuration/guide/cf_mem-leak-detect.html
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

41

Troubleshooting: I/O Memory Buffers


Switch# show buffers
Buffer elements:
1679 in free list (500 max allowed)
27109526 hits, 0 misses, 1641 created

I/O memory for incoming CPU bound


packets
Classified into two major pools :

Public buffer pools:


Small buffers, 104 bytes (total 50, permanent
50, peak 181 @ 3w5d):
49 in free list (20 min, 150 max allowed)
129877853 hits, 141 misses, 390 trims, 390
created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent
25, peak 94 @ 7w0d):
25 in free list (10 min, 150 max allowed)
616791 hits, 54 misses, 162 trims, 162
created
0 failures (0 no memory)

- Public Buffer Pools (pre-assigned based on


the packet size)
- Interface buffer pools (pre-assigned for 15
CPU queues)

Look for incrementing Failures and No


Memory in show buffers output
show memory debug leak can detect I/O
memory leaks as well

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

:(truncated)

42

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

44

Compact

Switch Hardware Components: Port ASIC Issues


Switch Fabric
TCAM

TCAM

Port ASIC

Stack PHY
TCAM

Port ASIC

CPU

Port ASIC

SDRAM
12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial
10/100

ASIC Forwarding cannot be accomplished?

High CPU Utilization

Is it a performance problem?

BRKCRS-3141

Packet drops on a port


2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

45

Forwarding?
Buffers?

Troubleshooting ASIC Issues : HW Forwarding


L2 and L3 forwarding decision based on
-

Routing/Switching tables

ACL Redirection (PBR/WCCP)


Switch# show plat forward <src intf> <srcmac> <dstmac> [ip <srcip> <dstip>
<protocol>]
Switch# show platform forward Gi0/1 0000.0000.0001 0000.0000.0010 ip 192.168.1.242
192.168.10.242 0
[..]
Egress: Asic 0, switch 1
Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 0
portMap 0x200, non-SPAN portMap 0x200
Destination Interface
Output Packets:
[..]
Port
Vlan
Gi0/10
0020

BRKCRS-3141

SrcMac
0000.0000.0020

2014 Cisco and/or its affiliates. All rights reserved.

DstMac
Cos
0000.0000.0002
Cisco Public

46

Dscpv

Troubleshooting ASIC Issues: Software Forwarding


ASIC cannot process data packets and forwards them to CPU using Q6
Switch# clear controllers cpu
Switch# show controllers cpu-interface | include sw forwarding
sw forwarding
71558
0
0
0

Switch# debug platform cpu-queues software-fwd-q


SW-FWD-Q:Consumed by SW-Bridging: Remote Port Blocked L3If:Vlan101 L2If:GigabitEthernet1/0/2
DI:0x2FD, LT:7, Vlan:101
SrcGPN:2, SrcGID:2, ACLLogIdx:0x0, MacDA:000f.f7e8.e042, MacSA:
0000.00bb.87df
IP_SA:10.101.1.100 IP_DA:10.99.1.100 IP_Proto:255

Switch# show plat forward Gi1/0/2 00.00bb.87df 000f.f7e8.e042 ip 10.101.1.100 10.99.1.100 255
Redirected by Input ACL. New destIndex is 0x02C7.
==========================================
Egress: ASIC 0, switch 1
CPU queues: 6 14.

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

47

ASIC Issues: Mapping Interfaces to Port-ASIC


Show platform pm if-number shows this mapping
Physical and ASIC port numbers may not match
This command shows all members

Switch# show platform pm if-numbers


interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
---------------------------------------------------------------------Gi3/0/1
109 109 1
1/1 3
1
1
local
Yes
Yes
Gi3/0/2
110 110 2
1/0 3
2
2
local
Yes
Yes
Gi3/0/3
111 111 3
1/3 3
3
3
local
Yes
Yes
Gi3/0/4
112 112 4
1/2 3
4
4
local
Yes
Yes
Gi3/0/5
113 113 5
1/5 3
5
5
local
Yes
Yes
Gi3/0/6
114 114 6
1/4 3
6
6
local
Yes
Yes
Gi3/0/7
115 115 7
1/7 3
7
7
local
Yes
Yes

ASIC/Port
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

48

ASIC Issues: Port-ASIC Statistics


Provides overview of possible drops/issues on the switch
Local and Member switches
Switch# remote command 2 show controller ethernet-controller port-asic statistics

Switch# show controllers ethernet-controller port-asic statistics


===========================================================================
Switch 2, PortASIC 0 Statistics
--------------------------------------------------------------------------0 RxQ-0, wt-0 enqueue frames
0 RxQ-0, wt-0 drop frames
8811506 RxQ-0, wt-1 enqueue frames
0 RxQ-0, wt-1 drop frames
0 RxQ-0, wt-2 enqueue frames
0 RxQ-0, wt-2 drop frames
0 RxQ-1, wt-0 enqueue frames
<snip>
100 TxBufferFull Drop Count
...
0 SneakQueue Drop Count
...
0 Sup Queue 0 Drop Frames
<snip>
0 Sup Queue 7 Drop Frames
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

0 RxQ-1, wt-0 drop frames

0 Rx Fcs Error Frames


0 Tx Too Old Frames
0 Sup Queue 8 Drop Frames
0 Sup Queue 15 Drop Frames
49

ASIC Issues: Egress Queue Drops


Queue and weight are 0-based
Tuning of buffers is only possible when QoS is enabled
Drops on egress indicate oversubscription
Switch# show platform port-asic stats drop gigabitEthernet 1/0/3
Interface Gi1/0/3 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
...
Queue 3
Weight 0 Frames 100000
Weight 1 Frames 0
Weight 2 Frames 0

More information
in the upcoming
QOS section

Switch# show platform port-asic stats enqueue gi1/0/3

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

50

Switch Hardware Components: Port ASIC QoS

Switch Fabric
TCAM

Stack PHY

TCAM

Port ASIC

TCAM

Port ASIC

Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial
10/100

Ingress QoS
Egress QoS

QoS

Traffic Classification Maps


BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

52

Cisco Catalyst 3750 QoS Overview


Policer

Marker
Queue 1
Marker

Queue 1

Classify
Input
Traffic

SRR
Queue 2
Policer

Marker

Policer

Marker

StackWise

Policer

Queue 2

SRR
Queue 3
Queue 4

Classification
Inspect incoming
packets
Assign QOS Label
to grouped packet
Use ACL, or other
configuration to
determine QOS
labels

BRKCRS-3141

Policing

Marking

Compares incoming Act on policer


traffic rate w/
decision
configured policer
Reclass or drop
and determine if
out-of-profile
packet is IN or Out of
Profile.
Either aggregate or
individual flow basis
256 policers/ASIC

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Ingress Queue/
Schedule
Congestion
Control
Two queues/port ASIC
shared servicing
One queue is
configurable for strict
priority servicing
WTD for congestion
control (three
thresholds per queue)
SRR is performed
53

Egress Queue/
Schedule
Congestion
Control
Four SRR queues/port shared
or shaped servicing
One queue is configurable
for strict priority servicing
WTD for congestion
control (three thresholds
per queue)
Egress queue shaping
Egress port rate limiting

Cisco Catalyst 2960S/X QoS Model


Policer

Marker
Queue 1

Policer

Marker
Queue 2

Classify
Input
Traffic

SRR
Queue 3
Policer

Marker

Policer

Marker

Queue 4

Classification
Inspect incoming
packets
Assign QOS Label
to grouped packet
Use ACL, or other
configuration to
determine QOS
labels

BRKCRS-3141

Policing

Marking

NO
Ingress Queues

Compares incoming Act on policer


traffic rate w/
decision
configured policer
Reclass or drop
and determine if
out-of-profile
packet is IN or Out of
Profile.
Either aggregate or
individual flow basis
256 policers/ASIC

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

54

Egress Queue/
Schedule
Congestion
Control
Four SRR queues/port shared
or shaped servicing
One queue is configurable
for strict priority servicing
WTD for congestion
control (three thresholds
per queue)
Egress queue shaping
Egress port rate limiting

Why Ingress QoS ?


Ingress QoS responsibilities

Ensure traffic classified correctly


Police traffic via Service Policy with traffic profiles
Security ACLs
Prioritize traffic during Stack congestion

Symptoms for ingress QOS problems


Packets unexpectedly dropped due to Access Service Policy, or stack congestion
Packets improperly marked for priority

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

55

QoS Troubleshooting Ingress


access

dot1q

Gi1/0/2

3750

10000 IP packets
with DSCP 34

Ingress policer with


trust DSCP

Switch# show mls qos interface gigabit 1/0/2 statistics


GigabitEthernet1/0/2 (All statistics are in packets)
dscp: incoming
------------------------------0 - 4 :
30 - 34 :
...
Policer: Inprofile:

0
0

0
0

0
0

1467 OutofProfile:

10,000 packets were received, DSCP value 34


1,467 packets were in profile
8,533 were dropped due to exceeding the policer
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

56

0
0
8533

0
10000

Causes of Egress Congestion

Congestion is the biggest QoS issue


Slower speed link
Oversubscription

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

57

Why Egress QoS? Rate Transition


Slower speed interfaces take longer to transmit packets
Introduction of Gigabit servers pushes congestion to the edge
QoS drops lowest priority packets

Buffers up on 100Mb interfaces


Packets take longer to egress

Traffic Burst on 10 Gig interface


3

5 43

3 5 2 4 1 3 2 1
Fat 10 Gig Pipe
with pkts ingressing

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Egress
Buffer
Cisco Public

58

Thin 100 Mbps pipes


with pkts egressing

Egress Queuing
Policer

Marker
Queue 1
Marker

Queue 1

Classify
Input
Traffic

SRR
Queue 2
Policer

Marker

Policer

Marker

StackWise

Policer

Queue 2

SRR
Queue 3
Queue 4

The Cisco Catalyst 3750/2960 have four egress queues


Queue 1 is optionally the priority queue
Port-based bandwidth rate limiting can be configured from 10% to 90%
These Egress queues, perform Shaped Round Robin SRR in queue sharing and queue
shaping mode
Weighted Tail Drop (WTD) for congestion management
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

59

Queues share Bandwidth and Buffers


Boarding on Overbooked Flight

The passenger capacity defines the


available interface bandwidth

Bandwith share per queue (class)

First
Business
Economy U.
Economy L.

2%
15%
20%
63%

Boarding Lines and waiting lists are


managed for each queue (class),
allowing different buffer sizes
(depths) and drop thresholds

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

60

What is an Egress Queue-set


Buffer Allocation and Drop Strategy
Two available queue-sets
Switch#show mls qos queue-set 1
Queueset: 1
Queue
:
1
2
3
4
--------------------------------------------buffers
:
20
20
30
30
threshold1:
33
33
33
33
threshold2:
66
66
77
50
reserved :
92
92
100
67
maximum
:
138
300
300
300

Each interface belongs to one queue-set


4 Egress Queues per port
3 drop thresholds per Queue
Threshold defines drop precedence for a
class of traffic
Threshold values over 100% dip into
common pool (MAX).

All values in Percentages of 100

Queue-set does not define bandwidth

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

61

Mapping Classes to Egress Queues


Maps available for DSCP and COS.
64 DSCP default values shown.
Each DSCP value maps to an egress Queue, and threshold
Queues range: 1-4, Threshold range:01-03
DSCP:0
Queue 4:
Threshold 3

Switch# show mls qos maps dscp-output-q


Dscp-outputq-threshold map:
d1 :d2
0
1
2
3
4
5
6
7
8
-----------------------------------------------------------0 :
04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-01
1 :
04-02 04-02 04-02 04-02 04-02 04-02 03-03 03-03 03-03
2 :
03-03 03-03 03-03 03-03 02-03 02-03 02-03 02-03 02-03
3 :
02-03 02-03 03-03 03-03 03-03 03-03 03-03 03-03 03-03
4 :
01-03 01-03 01-03 01-03 01-03 01-03 01-03 01-03 02-03
5 :
02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03
6 :
02-03 02-03 02-03 02-03
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

62

9
04-02
03-03
02-03
03-03
02-03
02-03

DSCP:46
Queue 1
Threshold 3

DSCP:63
Queue 2
Threshold 3

QoS Troubleshooting - Ingress


access

Remember this from a


few slides ago??

Gi1/0/2

Gi1/0/1

3750

10000 IP packets
with DSCP 34

Ingress policer with


trust DSCP

Switch# show mls qos interface gigabit 1/0/2 statistics


GigabitEthernet1/0/2 (All statistics are in packets)
dscp: incoming
------------------------------0 - 4 :
30 - 34 :
...
Policer: Inprofile:

0
0

0
0

0
0

0
0

1467 OutofProfile:

0
10000

8533

1,467 packets were in profile, and forwarded to egress interface

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

63

dot1q

QoS Troubleshooting - Egress


access

Gi1/0/2

dot1q

Gi1/0/1

3750

10000 IP packets
with DSCP 34

Ingress policer with


trust DSCP

Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics


GigabitEthernet1/0/1 (All statistics are in packets)
<output removed>
dscp: outgoing
------------------------------<output removed>
25 - 29 :
30 - 34 :
<output removed>

0
0

0
0

0
0

0
0

1467 packets were in profile and made it to the egress port

DSCP is 34
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

64

0
1467

QoS Troubleshooting Egress (2)


access

Gi1/0/2

Gi1/0/1

dot1q

3750

10000 IP packets
with DSCP 34

Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics


GigabitEthernet1/0/1 (All statistics are in packets)
<output removed>
0 4 :
1467
0
0
0
30 - 34 :
0
0
0
0

0
0

1467 packets were in profile and made it to the egress port but with DSCP 0 instead
of 34
Possible reasons
Attached service policy does not mark or trust dscp value
Traffic is being routed via the CPU
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

65

QoS Troubleshooting Egress Q Maps


100Mb/s

10Mb/s
3750

10000 IP packets
with DSCP 34

Gig 1/0/1

Gig 1/0/2

Switch# show mls qos interface gi 1/0/2 statistics


GigabitEthernet1/0/2 (All statistics are in packets)
dscp: incoming
------------------------------0 - 4 :
30 - 34 :

0
0

0
0

0
0

0
0

Switch# show mls qos maps dscp-output-q


Dscp-outputq-threshold map:
d1 :d2
0
1
2
3
4
5
6
7
8
-----------------------------------------------------------0 :
02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01
1 :
02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01
2 :
03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01
3 :
03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
4 :
01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01
5 :
04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
6 :
04-01 04-01 04-01 04-01
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

66

0
10000

9
02-01
03-01
03-01
04-01
04-01
04-01

10000 packets are


received and will
egress on Q4,
threshold 1

QoS Troubleshooting Egress Queue Thresholds


100Mb/s
10000 IP packets
with DSCP 34

10Mb/s
3750

Gig 1/0/2

Gig 1/0/1

Switch# show mls qos interface Gig 1/0/1 statistics


dscp: outgoing
------------------------------0 - 4 :
0
0
0
30 - 34 :
0
0
0
...
output queues enqueued:
queue:
threshold1 threshold2 threshold3
----------------------------------------queue 0:
2
0
0
queue 1:
0
6
260
queue 2:
0
0
0
queue 3:
1080
0
0

0
0

output queues dropped:


queue:
threshold1 threshold2 threshold3
----------------------------------------queue 0:
0
0
0
queue 1:
0
0
0
queue 2:
0
0
0
queue 3:
8920
0
0
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

0
1080

1080 packets will


egress on Q4,
threshold 1

Remaining 8920 pkts


dropped because of
congestion

CPU Generated Packets Egress


Queue 2

67

QoS Troubleshooting - Port-ASIC


100Mb/s

10Mb/s
3750

10000 IP packets
with DSCP 34

Gig 1/0/1

Gig 1/0/2

Viewing Egress Congestion (another way) with port-asic command


Switch# show platform port-asic stats drop gigabitEthernet 1/0/1
Interface Gi1/0/1 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 8920
Weight 1 Frames 0
Weight 2 Frames 0
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

68

Command works on all


Catalyst IOS versions

10000 packets were


received, 8920 were
dropped on egress

QoS Troubleshooting - Buffer Tuning


Tuning Buffers and Thresholds to fix Congestion
Queue-sets define the buffer allocation
Default values can be modified
2 Queue-sets are available
Reserved - how many buffers will be reserved for this port
Default Queue-set values listed below
Switch# show mls qos int gi1/0/1 buffers
GigabitEthernet1/0/1
Identifies Queue-set assigned to interface
The port is mapped to qset : 1
The allocations between the queues are : 25 25 25 25
Switch# show mls qos queue-set
Queueset: 1
Queue
:
1
2
3
4
---------------------------------------------buffers
:
25
25
25
25
threshold1:
200
200
100
100
threshold2:
200
200
100
100
reserved :
50
50
50
50
maximum
:
400
400
400
400
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Dropped on this Queue


and Threshold
69

QoS Troubleshooting - Buffer Tuning (2)


100Mb/s
10000 IP packets
with DSCP 34

10Mb/s
3750

Switch# show platform port-asic stats drop gigabitEthernet 1/0/1


Interface Gi1/0/1 TxQueue Drop Statistics
Queue 3
Packet drops with current Queue-set configuration
Weight 0 Frames 8920
Switch(config)# mls qos queue-set output 1 threshold 4 300 300 50 400
Switch# show mls qos queue-set
Queueset: 1
Queue
:
1
2
3
4
---------------------------------------------buffers
:
25
25
25
25
threshold1:
100
100
100
300
threshold2:
100
100
100
300
reserved :
50
50
50
50
maximum
:
400
400
400
400

Threshold increased to 300

Switch# show platform port-asic stats drop gigabitEthernet 1/0/1


Interface Gi1/0/1 TxQueue Drop Statistics
Queue 3
No additional Packet drops after Queue-set change
Weight 0 Frames 8920

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

70

Egress QoS Summary


Packet drops dont always indicate a problem
For ex, Gigabit servers can easily oversubscribe 100M clients
Most protocols react well to drop and will slow down so
maximum performance can be achieved

Analyze traffic patterns

Tune buffers as needed increasing thresholds has minimal side effects


Take advantage of both queue-sets
E.g.: use Queue-set 1 on downlinks, Queue-set 2 on uplinks

Map queues to distribute traffic according to the Plan


Set thresholds to optimize high priority traffic
Auto QoS
QoS is not easy, but Auto QOS makes it easy
Auto QoS produces consistent configurations across all 2K and 3K switch models

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

71

Switch Hardware Components: TCAM


Switch Fabric
TCAM

TCAM

Port ASIC

Stack PHY
TCAM

Port ASIC

Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial
10/100

TCAM Resources?

The TCAM stores Forwarding database


-

ACLs

IPv4, IPv6 and MAC addresses


Service policies and security

Multicast Addresses and Groups

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

73

TCAM: Switch Database Manager (SDM)


SDM defines how TCAM
resources are allocated
Changing SDM template requires
reboot

Switch# show sdm prefer default


"desktop default" template:
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

All stack members must use same


SDM template
Switch# show sdm prefer ?
access
Access bias
default
Default bias
dual-ipv4-and-ipv6 Support both
IPv4 and IPv6
routing
Unicast bias
vlan
VLAN bias
Switch# show sdm prefer dual-ipv4-andipv6 ?
default Default bias
routing Unicast bias
vlan
VLAN bias
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

number of unicast mac addresses:


number of IPv4 IGMP groups + multicast routes:
number of IPv4 unicast routes:
number of directly-connected IPv4 hosts:
number of indirect IPv4 routes:
number of IPv4 policy based routing aces:
number of IPv4/MAC qos aces:
number of IPv4/MAC security aces:

List of available SDM Types

74

6K
1K
8K
6K
2K
0
0.5K
1K

TCAM Utilization

Route entries not


forwarded in H/W

TCAM space is limited

Switch# show platform ip unicast failed route

Problem when Used


Masks/Values = MAX

Switch# show platform tcam utilization

Optimize ACE/Routing
entries
Change SDM Template

Layer 3 Routing

CAM Utilization for ASIC# 0

Masks/Values
Unicast mac addresses:

784/6272

14/40

IPv4 IGMP groups + multicast routes:

144/1152

7/27

IPv4 unicast directly-connected routes:

784/6272

14/40

IPv4 unicast indirectly-connected routes:

272/2176

11/55

IPv4 policy based routing aces:


IPv4 security aces:

0/0

0/0

768/768

260/260

1024/1024

723/723

Note: Allocation of TCAM entries per feature uses


a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

BRKCRS-3141

Used

Masks/values

IPv4 qos aces:

Security ACLs
Permit/deny

Max

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

75

TCAM Overload

An error message will get generated


Traffic forwarding will be done (partly) in Software
CPU utilization will go up packets punted to CPU for processing
%ACLMGR-4-UNLOADING: Unloading ACL input label 1 VLAN interfaces 101 IPv4/Mac feature
%ACLMGR-4-ACLTCAMFULL: ACL TCAM Full. Software Forwarding packets on Input label 1 on L3 L2
Switch# sh platform acl oacltcamfull
Vlan

oacl_tcam_full_bitmap

101
Vlan

0x

notify_apps

NOT-FULL

ipv6_oacl_tcam_full_bitmap

notify_apps

Switch# sh platform acl label 1 detail

IPv4/MAC ACL label


-----------------Unloaded due to lack of space:
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

76

Means ACL Not Fully


Programmed in TCAM

TCAM: Switch Database Manager (SDM)


Strategies to choose SDM

SDM Template

Use Case

access

L2 & L3, fewer L2 & L3 addresses than default,


Supports Policy Based Routing, more security ACEs

default

L2 & L3, more L2 & L3 addresses than access

routing

L2 & L3, weighted towards L3 space, Supports


Policy Based Routing

vlan

L2 only, 12K MAC Addresses

dual-ipv4-and-ipv6

Required for IPv6 functionality

default, routing, vlan

same distribution as above, but with IPv6 resources

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

77

TCAM Hardware Summary


TCAM Partition based on SDM Template
L2 and L3 overload of TCAM resource: punt to CPU
Number of ACEs depend on
Switch Model
SDM Template different Templates for Layer 3 capable switches

If ACL does not fit in TCAM, will be processed in SW (CPU)


CPU processing is much slower than TCAM

Switch reboot required when SDM template changed

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

78

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

80

Compact

Troubleshooting Power Over Ethernet


Stack
PHY

Switch Fabric
TCAM

TCAM

Port ASIC

Two
Stack
Cables

TCAM

Port ASIC

CPU

Port ASIC

SDRAM
12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash
Serial

TCAM

TCAM

24X1G POE

12X1G

12X1G

10/100

24X1G POE

12X1G

12X1G

StackWise,
StackWise
Plus

10G or 1G

What is the Power requirement of PD ?


Is it a Cisco PD or a Third party PD ?
Does the PD work fine on a different port or a different
switch ?
Or all POE ports affected ?
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

81

PoE

Troubleshooting PoE: CDP/LLDP Negotiation


Inline power available. If not, this
Stack-1# show power inline
log would be seen:
Module Available Used Remaining
%ILPOWER-5(Watts) (Watts) (Watts)
ILPOWER_POWER_DENY: Interface
<interface>: inline power denied
------ --------- -------- --------1
420.0
22.2
397.8
Power Allocated
2
370.0
18.2
351.8
Interface Admin Oper
Power Device
Class Max
(Watts)
--------- ------ ---------- ------- ------------------- ----- ---Gi1/0/2 auto on
10.3
IP Phone 7970
3 15.4
Gi2/0/2 auto on
8.5 AIR-AP1220-IOS
n/a 15.4

Power Negotiation can occur via CDP or LLDP Power-via- MDI protocol.
CDP PD requests the worst-case power (including the link loss)
LLDP PD requests only power required, the PSE adds the link loss values
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

82

Troubleshooting PoE: PD Drawing Too Much Power


Imax or Tstart error reported when a PD misbehaves and draws more power
- Imax error is an operating fault and reported after PD power up
-

Tstart is a start up fault before PD reported Power Good

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface gig1/0/1 Power Controller reports power Imax error detected
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface: Power Controller reports power Tstart error detected

Switch(config)# int gig 1/0/1


Switch(config-if)# power inline port 2x-mode
Switch(config-if)# shut
Switch(config-if)# no shut

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

83

Configure 2x-mode on the affected


interface as a workaround. The mode
raises the thresholds for I(cut), I(short)
current.
Bug ID CSCsw18530

Troubleshooting PoE: Power given but Power Good


not reported
PD detected. Power was granted but the PD was not up

PD cannot provide MPS (10mA)

%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi8/0/1: Power given, but Power Controller does not report Power
Good

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

84

Troubleshooting PoE: Multiple ports not providing


power

Check the controller status and Error counters

%FRNTEND_CTRLR-2-SUB_INACTIVE: The front end controller 0 is inactive Traceback= 1CA91E8 1CA95B8 1E7CD8C 1E73818
%FRNTEND_CTRLR-1-SUB_I2C_ERR: Sub 0 reported 36B5B98 I2C errors

Switch# show platform frontend-controller subordinate <0-2>


State

OK

Last Reset Reason

UNKNOWN REASON

:
SLE Poe No Port

SLE I2C Busy

SLE I2C Error

SLE I2C Timeout

SLE Invalid Reg Len

SLE Msg Underrun

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Check for incrementing error


counters

Cisco Public

85

Troubleshooting PoE: Useful Debugs


Specific debug conditions can
be used where available
Switch#debug condition interface gigabitEthernet 1/0/15
Condition 1 set
Switch#debug ilpower powerman
ILPOWER powerman debugging is onilpower_power_assgn_handle_event: event 0,
pwr assign is done by proto CDPPort Gi1/0/9: Selected Protocol CDP
Ilpower interface (Gi1/0/9) process tlv from cdp INPUT:
power_request_level[] = 12000 0 0 0 0

Switch#debug ilpower event


%ILPOWER-7-DETECT: Interface Gi1/0/15: Power Device detected: IEEE PD
ILP uses DC Disconnect(Gi1/0/15): state=ILP_IEEE_PD_DETECTED_S, event=
ILP uses DC Disconnect(Gi1/0/15): state=ILP_LINK_UP_S, event=PHY_LINK_UP_EV
ILP uses DC Disconnect(Gi1/0/15): state=ILP_LINK_UP_S, event=
ILP_POWER_POLICE_DISABLE_EV

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

86

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

88

Compact

Switch Hardware Components: Stacking


Switch Fabric
TCAM

TCAM

Port ASIC

Stack PHY
TCAM

Port ASIC

Port ASIC

CPU
SDRAM

12 Port
PHY

12 Port
PHY

12 Port
PHY

12 Port
PHY

Modular PHY

Flash

Serial
10/100

Only 3750, 3750G, 3750-E, 3750-X support stacking with StackWise or StackWise Plus
C2960-S, C2960-SF, 2960-X, 2960-XR support FlexStack or FlexStack Plus

Conditions that can prevent a switch from joining a stack:


- Incompatible IOS Versions between the stack members
- Stack cable issue
- SDM Template mismatch

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

89

Stack
errors

Troubleshooting Stacks: Version Mismatch


Software Version Mismatch
IOS version of all stack switches (show version) should be either the same or
compatible
Switches with different Major Version numbers
Occurs on switch member addition, or RMA replacement
3750X# show version
Switch Ports Model
SW Version
------ ----- ----------------------1 54 WS-C3750X-48P 15.2(1)E
2 54 WS-C3750X-48P 15.2(1)E

SW Image
C3750E-UNIVERSALK9-M
C3750E-UNIVERSALK9-M

3750X# show platform stack manager all


IOS Versions should match
Switch Master/ Mac Address
Version Current
Number Member
(maj.min) State
----------------------------------------------------------1
Member f866.f2ab.7180
1.51
Ready
2
Member f866.f2af.3b00
1.51
Ready
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Major versions must match

90

Troubleshooting Stacks, Stack Cables

A Switch can join a stack with only one Stackwise interface connected to
another active stack member.
Important precautions for connecting Stackwise cables
- Retainer screws on the connector should not be loose
- Retainer screws on the connector should not be too tight
- Retainer screws should be tightened finger tight and no more

Retainer not fully engaged

Retainer fully engaged


BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

91

Troubleshooting: Stack Commands


3750# show switch detail
Current
Switch# Role
Mac Address
Priority
State
-----------------------------------------------------1
Slave
000c.30ae.4f00
9
Ready
*2
Master
000d.bd5c.1680
15
Ready
Stack Port Status
Neighbors
Switch# Port 1
Port 2
Port 1
Port 2
-----------------------------------------------------1
Ok
Ok
2
2
2
Ok
Ok
1
1

3750# show switch stack-ring activity


Switch
Frames sent to stack ring (approximate)
-----------------------------------------------1
5781
2
4928
Total frames sent to stack ring : 10709
Note: these counts do not include frames sent to the ring
by certain output features such as output SPAN and output
ACLs.

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

92

3750E# show switch stack-ring speed


Stack Ring Speed
: 32G
Stack Ring Configuration: Full
Stack Ring Protocol
: StackWisePlus

Use the mode button on the


switch to determine its
switch number

Troubleshooting: Stack Commands


Details on the stack ports, members 1 and 3 active
3750# show switch
Switch/Stack Mac Address : 001b.545f.2800
Mac persistency wait time: 4 mins

H/W
Current
Switch# Role
Mac Address
Priority Version State
---------------------------------------------------------*1
Master 001b.545f.2800
12
1
Ready
2
Member 0000.0000.0000
0
1
Provisioned
3
Member 001d.46be.7500
8
1
Ready
3750# show switch stack-ports summary
Switch#/ Stack
Neighbor
Cable
Port#
Port
Length
Status
-------- ------ -------- -------1/1
OK
3
50 cm
1/2
Down
None
50 cm
3/1
Down
None
50 cm
3/2
OK
1
50 cm

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Link
OK

Link
Active

Sync
OK

---Yes
No
No
Yes

-----Yes
No
No
Yes

---Yes
No
No
Yes

Cisco Public

93

#
Changes
To LinkOK
--------1
0
0
1

In
Loopback
-------No
No
No
No

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

95

Compact

Troubleshooting L2 Unicast Forwarding

Server

Symptom: Host cannot reach server


Steps

Layer 1 operational between host/Phone and switch?


Switch receiving traffic on that interface?
Congestion between host and switch?
MAC address learned?
MAC address of next hop correct?
Spanning tree state forwarding?
Other features preventing traffic flow?
Errored packets on the interface
Check HW programming

Consider possibilities
Create and execute action plan
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Distribution
and Core

C3750

Host
96

L2 Forwarding: Troubleshooting - 1
Server

Step 1: Verify if the link is up


Switch# show interface Gi1/0/3 status
Port
Name
Status
Gi1/0/3
connected

Vlan
10

Duplex
a-full

Speed Type
a-100 10/100/
1000BaseTX

Step 2: Verify if the port is in the right vlan and

is forwarding
Switch# show spanning-tree interface Gi1/0/3
Vlan
Role Sts Cost
Prio.Nbr Type
------------------- ---- --- --------- -------- ----------------------VLAN010
Desg FWD 19
128.2
P2p

Distribution
and Core

Step 3: Check if the packets are being received/sent


C3750

on the port
Switch# show interfaces gigabitEthernet 1/0/3 counters
Port
InOctets
InUcastPkts
InMcastPkts
Gi1/0/3
2108289
48
0
Port
Gi1/0/3
BRKCRS-3141

OutOctets
36817803

OutUcastPkts
48229

2014 Cisco and/or its affiliates. All rights reserved.

OutMcastPkts
252940

Cisco Public

97

InBcastPkts
6813
OutBcastPkts
72564

Host

L2 Forwarding: Troubleshooting 2
MAC Address Learning

Server

Step 4a: Verify if the Mac-address is correctly learned on the


port
Switch# show mac address-table interface gigabitEthernet 1/0/3
Mac Address Table
-------------------------------------------

Vlan
Mac Address
Type
Ports
------------------------10
00b1.a3d3.4321
DYNAMIC
Gi1/0/3
Total Mac Addresses for this criterion: 1

Distribution
and Core

Step 4b: Verify if the destination Mac-address is learned on the


switch on the expected port
C3750

Switch# show mac address-table dynamic address 00b1.a3d3.1234


Mac Address Table
------------------------------------------Vlan
Mac Address
Type
Ports
------------------------10
00b1.a3d3.1234
DYNAMIC
Gi1/0/4
Total Mac Addresses for this criterion: 1

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Host
98

L2 Forwarding: Troubleshooting 3
Spanning Tree

Server

Step 5: Spanning tree state forwarding in software?


Switch#show spanning-tree vlan 10

Interfaces are FWDing

VLAN0010
Spanning tree enabled protocol ieee
Root ID
Priority
32778
Address
0003.fd6b.0700
This bridge is the root
Hello Time
2 sec Max Age 20 sec
Bridge ID

Priority
Address
Hello Time
Aging Time

Interface
------------------Gi1/0/3
Gi1/0/4

BRKCRS-3141

Role
---Desg
Desg

Distribution
and Core
Forward Delay 15 sec

32778 (priority 32768 sys-id-ext 10)


0003.fd6b.0700
2 sec Max Age 20 sec Forward Delay 15 sec
300 sec
Sts
--FWD
FWD

Cost
--------4
4

2014 Cisco and/or its affiliates. All rights reserved.

Prio.Nbr
-------128.3
128.4

Cisco Public

Type
-----------------------P2p
P2p Edge

99

C3750

Host

L2: Mac-Address Disappears From a Port


Check for spanning tree topology changes
Does the link remain up?
Link down causes MAC
Addresses to be flushed

Is it learned on another port?


Switch# show spanning-tree vlan 10 detail
.
.
.

Number of topology changes 5 last change occurred 18:45:22 ago


from GigabitEthernet1/0/3
...

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

100

L2 Forwarding: Troubleshooting 4
Interface

Server

Step 6a: Check Interfaces for Error-Disabled


Switch# show interface status err-disabled

Nothing in list.
No interfaces are Disabled

Switch#

Step 6b: Check Interface counters for errors

Distribution
and Core

Switch#show interface gi1/0/3 counters errors


Port
Gi1/0/3

Align-Err
0

FCS-Err
0

Xmit-Err
0

Rcv-Err
0

Port
Single-Col Multi-Col
Late-Col Excess-Col
Gi1/0/3
0
0
0
0
Switch#
Switch#show interface gi1/0/4 counters errors
Port
Gi1/0/4
Port
Gi1/0/4
BRKCRS-3141

Align-Err
0
Single-Col
0

FCS-Err
0
Multi-Col
0

Xmit-Err
0
Late-Col
0

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Carri-Sen
0

Rcv-Err
0

Excess-Col
0
101

UnderSize
0

UnderSize
0

Carri-Sen
0

OutDiscards
0
Runts
0

Giants
0

C3750

OutDiscards
0
Runts
0

Giants
0

Host

Layer 2 Forwarding: Troubleshooting 5


Advanced Techniques
Step 7: Use show platform forward to find Egress Interface programmed in H/W
Switch# show platform forward <ingress intf> <srcmac> <dstmac>
Switch# show platform forward gigabitEthernet 1/0/3 0000.0000.4321 0000.0000.1234
Ingress:
Global Port Number: 3, lpn: 1 ASIC Number: 6
Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 3
Hashes: L2Src 0x00 L2Dst 0x0B L3Src 0x00 L3Dst 0x0B
Lookup
Key-Used
Index-Hit A-Data
Classify 68_00F00000_00001234-02_00000000_00004321 0102E 00000002
InputACL 20_00F00000_00001234-00_00000000_00004321 01FF8 01000000
L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000
L2FwdMsk FF_03FFFFFF_FFFFFFFF
L2Fwd 83_00020000_00001234
00EB6 000000B5
Station Descriptor: F004F002, DestIndex: F004, RewriteIndex: F002
==========================================
Egress: ASIC 6, switch 1
Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 3
portMap 0x4, non-SPAN portMap 0x4
Destination Interface
Output Packets:
-----------------------------------------GigabitEthernet1/0/4 Packet 1
Lookup
Key-Used
Index-Hit A-Data
OutptACL 30_00F00000_00001234-00_00000000_00004321 01FFC 01000000
Port
Vlan SrcMac
DstMac Cos Dscpv
Gi1/0/4 0010 0000.0000.4321 0000.0000.1234

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

102

Checklist: Interface Troubleshooting


Are packets being received?
Is the expected Mac-address learned on another port?
Check if dot1x is in use, if so, is the port authorized?
Does port security allow more Mac-addresses?
Is the port in spanning tree forwarding?
Other features preventing traffic flow?
ACLs
PVLAN

Show logging is there a history of instability

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

103

Layer 3 IP Unicast Routing


Use the switch to debug end to end IP issues
- Verify source reachability from the switch
- Verify destination reachability from the switch
- Verify hardware forwarding from source to destination (and back)

VLAN:101
IP: 100.1.1.1
Mac: 000f.f7e8.e042

3750

3750
3750

Gi1/0/1
Destination
IP: 172.16.100.100

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Vlan:100
IP: 10.1.1.1
Mac :000f.f7e8.e041

Cisco Public

105

Gi1/0/2

Source
IP: 100.1.1.2
Mac: 0018.ba88.1fc1

L3: Verify Source Reachability


Troubleshooting Steps
Source IP = 100.1.1.2
PING the source
PING the source with a loopback

Verify the ARP table


Verify the MAC table
VLAN:101
IP: 100.1.1.1
Mac: 000f.f7e8.e042

3750

3750
3750

Gi1/0/1

Destination
IP: 172.16.100.100
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Vlan:100
IP: 10.1.1.1
Mac :000f.f7e8.e041
Cisco Public

106

Gi1/0/2

Source
IP: 100.1.1.2
Mac: 0018.ba88.1fc1

L3: Verify Source Reachability


3750# ping 100.1.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
Change source IP to loopback
3750# ping 100.1.1.2 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds:
Packet sent with a source address of 99.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/maz = 1/4/9 ms
3750# sh ip arp vlan 101
Protocol Address
Age (min) Hardware Addr
Type
Interface
Internet 100.1.1.1
000f.f7e8.e042 ARPA
Vlan101
Internet 100.1.1.2
23
0018.ba88.1fc1 ARPA
Vlan101

3750# sh mac address-table address 0018.ba88.1fc1


Mac Address Table
------------------------------------------Vlan
Mac Address
Type
Ports
------------------------101
0018.ba88.1fc1
DYNAMIC
Gi1/0/2
Total Mac Addresses for this criterion: 1
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

107

L3: Verify Source Reachability - 2


Verify packets from the source are getting to the CPU
Switch# show plat for <ingress intf> <srcmac> <dstmac> ip <srcip> <dstip> icmp <0-255> <0-255>
3750#show platform for Gi1/0/2 0018.ba88.1fc1 000f.f7e8.e042 ip 100.1.1.2 100.1.1.1 icmp 0 0
Ingress:
Global Port Number: 1, lpn: 3 Asic Number: 1
Source Vlan Id: Real 101, Mapped 9. L2EncapType 0, L3EncapType 0
Hashes: L2Src 0x03 L2Dst 0x05 L3Src 0x09 L3Dst 0x03
Lookup
Key-Used
Index-Hit
A-Data
Classify 78_64010101_64010102-00_01000000_00000100
017FE
00000000
InputACL 40_64010101_64010102-00_01000000_00000100
01FFA
03000000
L3Local C0_00302401_64010101
01CF0
00000000
L3Scndr 10_64010101_64010102-00_00000000_00000100
008AA
000A0008_00000000
Lookup Used: Secondary
Station Descriptor: 00B00000, DestIndex: 00B0, RewriteIndex: 0000
==========================================
<output removed>
Output Packets:
Packet arriving on CPU
==========================================
(host) & 14 (dstats)
Egress: Asic 0, switch 2
CPU queues: 7 14.
Source Vlan Id: Real 101, Mapped 9. L2EncapType 0, L3EncapType 0
portMap 0x0, non-SPAN portMap 0x0
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

108

queue 7

L3: Verify Destination Reachability


Troubleshooting Steps
Destination IP = 172.16.100.100
Verify there is a route to the destination
Verify there is a valid ARP for the next hop

PING the destination (repeat w/ source VLAN as source address)


VLAN:101
IP: 100.1.1.1
Mac: 000f.f7e8.e042

3750

3750
3750

Gi1/0/1

Destination
IP: 172.16.100.100

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Vlan:100
IP: 10.1.1.1
Mac :000f.f7e8.e041

Cisco Public

109

Gi1/0/2

Source
IP: 100.1.1.2
Mac: 0018.ba88.1fc1

L3: Verify Destination Reachability - 1


Resolved next hop
Switch# sh ip route 172.16.100.100
Show next hop to final destination
Routing entry for 172.16.100.0/24
Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 1
Last update from 10.1.1.2 on Vlan100, 00:08:54 ago
Routing Descriptor Blocks:
* 10.1.1.2, from 100.1.1.2, 00:08:54 ago, via Vlan100
Route metric is 20, traffic share count is 1
Switch# sh ip arp 10.1.1.2
Protocol Address
Age (min)
Internet
9

verify next hop is known


Hardware Addr
0018.ba88.1fc1

Type
ARPA

Interface
Vlan100

Switch# ping 172.16.100.100


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.100.100, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Switch# ping 172.16.100.100 source vlan 101


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.100.100, timeout is 2 seconds:
Packet sent with a source address of 192.168.100.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

110

L3:
Verify
Destination
Reachability
2
Unresolved next hop
Switch# sh ip route 172.16.100.0 255.255.255.0
Routing entry for 172.16.100.0/24
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
* 10.13.13.3
Route metric is 0, traffic share count is 1
Switch# sh run | include ip route
ip route 172.16.100.0 255.255.255.0 10.13.13.3
Switch# sh ip arp 10.13.13.3
Protocol Address
Age (min) Hardware Addr
Internet
0
Incomplete

Show next hop to final destination

verify next hop is known: FAIL

Type
Interface
ARPA

Switch# sh platform ip unicast failed arp


ARP Throttle Queue trying to resolve next
Total of 1 arp entries waiting on ARP-HRPC ThrottleQ
hop
========================
ARP throttled IP Address
========================
10.13.13.3/32 Table:0
------------------------Switch# sh platform ip unicast route 172.16.100.0 255.255.255.0
Fib 172.16.100.0/24 Tbl:0 Bucket:0
IOS Path 0 Spl Adj glean
HL3UFlags:0x80
SFT Entry:hdl:0xA5 HwFL:0x4
Adjacency not programmed in ASIC
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

111

L3: Verify Hardware Forwarding


Show platform forward to verify HW programming
Output Packets:
==========================================
GigabitEthernet 1/0/2 Packet 1
Dropped due to failed deja vu check

Packet not forwarded to the interface it


was received from

Input ACL:
==========================================
Addr 0x7E7, tcam(6), mapRam (0x2)
framAddr (0x48400E14) Adata (0X200000)
Denied by Input ACL

Traffic denied by ACL

Ingress:
Dont use vlan option if it is not a trunk
Global Port Number: 5, lpn: 5 Asic Number: 1
Dropping the frame due to VLAN mode filtering,(allowDotOneQ) is set to FALSE

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

112

Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples

3750-X

3560X

2960-X/2960-XR

3560E

Tools and Tricks


2960-S/SF

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

3750v2

Cisco Public

114

Compact

Tools and Tricks


Enable NTP to troubleshoot across switches
Include date and time for debug and log messages
service timestamps log [datetime|uptime] localtime msec show-timezone
service timestamps debug [datetime|uptime] localtime msec show-timezone

Session to another switch member


C3750#session <member #>
C3750#remote command <1-9|all> IOS command

Automate successful troubleshooting using EEM


Review open caveats sections in release notes
Search Bug Toolkit for known issues
Reference Output Interpreter to decode command output
Reference System Message Guide for mitigation recommendations
Check the documentation and online guides
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

115

References
Troubleshooting Catalyst 3750:
http://www.cisco.com/en/US/products/hw/switches/ps5023/prod_troubleshooting_guides_list.html

Online Resources on http://www.cisco.com:

Troubleshooting High CPU Utilization


Troubleshooting Power over Ethernet (PoE)
Troubleshooting Switch Stacks
Cisco Catalyst 3750 QoS Configuration Examples (Doc 91862)
Auto Negotiation issues: (Document 17053)

Want to learn more? Check out CCNP Practical Studies: Troubleshooting by Donna Harrington.
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

116

Cisco Beyond - Product Extension Community


EEM Scripting Community
Open source scripts,
share, upload,
download, learn by
example

Categories include:
Network Management,
Diagnostics, Routing,
QoS, High availability,
User interface, Security
Comments, ratings,
community managed
forum

http://cisco.com/go/ciscobeyond
BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

117

Complete Your Online Session Evaluation


Give us your feedback and you
could win fabulous prizes. Winners
announced daily.
Complete your session evaluation
through the Cisco Live mobile app
or visit one of the interactive kiosks
located throughout the convention
center.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

118

Continue Your Education


Demos in the Cisco Campus
Walk-in Self-Paced Labs
Table Topics

Meet the Engineer 1:1 meetings

BRKCRS-3141

2014 Cisco and/or its affiliates. All rights reserved.

Cisco Public

119