Designing MPLS in Next Generation Data Center: A Case Study

Troubleshooting Cisco Catalyst
3750, 3560, 2960-S and 2960-X

Series Switches
BRKCRS-3141
John Wu
BU Escalation
Administrators spend most of their time

Monitoring and troubleshooting
Security-related configuration
Initial install, config & testing
Upgrade of older equipment
Traffic optimization
Other
0%
Source: The Total Economic Impact of Cisco Catalyst Access Switching,
A Commissioned Study Conducted by Forrester Consulting On Behalf of Cisco Systems, January 2012
BRKCRS-3141
2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
10%
20%
30%
Agenda
Product Overview
Areas of Troubleshooting
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
Compact
Product Overview
3750-X
3560-X
2960-X/2960-XR
3750v2
3560-E
2960-S/SF
3560v2
3750 Stack Rear View
2960-S Stack Rear View

BRKCRS-3141
Cisco Public
Compact
Product Overview
BRKCRS-3141
Cisco Public
Catalyst 3750-X : Architecture Overview

Switch Fabric
Stack PHY
TCAM
TCAM
Port ASIC
Two
Stack
Cables
TCAM
Port ASIC
3
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular
2 PHY
Flash
Serial
10/100
24X1G POE
12X1G
12X1G
24X1G POE
12X1G
12X1G
StackWise,
StackWise
Plus
10G or 1G
Packet Flow across switch components
BRKCRS-3141
Cisco Public
Switch Components: Troubleshooting Areas of Focus

Stack
PHY
Switch Fabric
TCAM
TCAM
Port ASIC
Two
Stack
Cables
Stack
errors
TCAM
Port ASIC
CPU
Port ASIC
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
TCAM
12X1G
High ?
10/100
24X1G POE
12X1G
24X1G POE
12X1G
StackWise,
StackWise
Plus
10G or 1G
12X1G
Buffers?
QoS
TCAM
Resources?
BRKCRS-3141
Interface
Issue?
Cisco Public
No PoE ?
8
Running
out?
Before We Start
The outputs in this presentation are from 3750, 3750x
Troubleshooting the 2960, 3560, and 3750 series switches are basically the same
Differences called out
Caution!!!
debug and show platform commands to follow in the slides

Excessive debug output to console may disable switch
show platform commands are intended for in-depth troubleshooting by Cisco engineers
Use debug and show platform commands as advised by TAC only
TroubleShooting Basics
Check the syslog for warnings and errors
Use common sense
Some TS techniques impact switch operation
BRKCRS-3141
Cisco Public
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
10
Compact
Troubleshooting Link Issues

Switch Fabric
TCAM
TCAM
Port ASIC
Stack PHY
TCAM
Port ASIC
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
Link issues
When are we concerned about the link issues?

Connectivity
Traffic
BRKCRS-3141
Cisco Public
11
Troubleshooting Link Issues

Is the link up/up?
Are packets being sent and received ?
Are there any errors ?
Collect data multiple times to confirm
Switch# show interface GigabitEthernet 1/0/1
GigabitEthernet1/0/1 is up, line protocol is up (connected)
....
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Output queue: 0/40 (size/max)
7539 packets input, 9856 bytes, 0 no buffer
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 4059 multicast, 0 pause input
0 input packets with dribble condition detected
3508 packets output, 3560 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
BRKCRS-3141
Cisco Public
12
Ethernet Controller Stats
Details about errors
Clear Stats with clear controller ethernet-controller command
Switch# show controller ethernet-controller GigabitEthernet 1/0/1
Transmit GigabitEthernet4/0/1
0 1 collision frames
..
0 Excessive collisions
0 Late collisions
0 VLAN discard frames
0 Excess defer frames
Receive
..
BRKCRS-3141
Cisco Public
13
0
0
0
0
0
Alignment errors
FCS errors
Oversize frames
Undersize frames
Collision fragments
0
0
0
0
Symbol error frames

Invalid frames, too large
Valid frames, too large
Invalid frames, too small
PHY Registers
Information from PHY registers
False carrier and symbol errors are reported
Switch#show controllers ethernet-controller g0/46 phy detail

GigabitEthernet0/46 (gpn: 46, port-number: 46)
0000: 3100 Control Register
0001: 7969 Control STATUS
<removed>
0015: 646B Receive Error Counter
BRKCRS-3141
Cisco Public
14
:
:
0011 0001 0000 0000

0111 1001 0110 1001
0110 0100 0110 1011
Link Issues: What Kind of Errors?

Switch# show int gi1/0/1 counters
Port
Gi1/0/1
InOctets
9856
InUcastPkts InMcastPkts
7539
4059
14
InBcastPkts
Port
Gi1/0/1
OutOctets OutUcastPkts OutMcastPkts OutBcastPkts

3560
3508
3056
23
Switch# show interfaces GigabitEthernet 1/0/1 counters errors

Port
Align-Err
Gi1/0/1
0
FCS-Err Xmit-Err
0
0
0
Rcv-Err UnderSize OutDiscards

0
0
Port
Single-Col Multi-Col Late-Col Excess-Col Carri-Sen
Gi1/0/1
0
0
0
0
0
0
0
BRKCRS-3141
Cisco Public
15
Runts
Giants
Link Issues: Link Not Coming Up

Check configured duplex and speed on both switch and host
Upgrade the NIC drivers on the host to the latest version
Try a different cable/NIC and switchport to exclude faulty hardware
Switch# show interfaces status | inc connected

Gi1/0/1
connected trunk
a-full 10 10/100/1000BaseTX
Gi1/0/2
connected 101
a-full a-100 10/100/1000BaseTX
Gi1/0/24
connected 1
a-full a-1000 10/100/1000BaseTX
BRKCRS-3141
Cisco Public
16
SFP Link Issue Prevention EEM TCL Use Case

1000 Base-T (copper) and 100
Base-FX SFPs have embedded
PHYs, allowing speed and duplex
to be configured on their respective
interface
1
LinkUpApplyConfig is a Tcl policy

that monitors an SFP link-up event
EEM
Tcl
Policy
However, these settings are

cleared as soon as the SFP is
unplugged
Speed and duplex settings (in

startup-config) are automatically reapplied to the SFP interface
If the same SFP type is re-inserted,

its configuration is not recovered
LinkUpApplyConfig.tcl can be downloaded at the following hyperlink:

https://supportforums.cisco.com/docs/DOC-23267
BRKCRS-3141
Cisco Public
17
Link Issues: Checking Physical Cabling

TDR feature helps determine possible cabling issues
Switch# test cable-diagnostics tdr interface GigabitEthernet4/0/1

TDR test started on interface Gi4/0/1
A TDR test can take a few seconds to run on an interface
Use 'show cable-diagnostics tdr' to read the TDR results.
Switch#
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/1, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet4/0/1, changed state to down
*%LINK-3-UPDOWN: Interface GigabitEthernet4/0/1, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/1, changed state to upw
Switch# show cable-diagnostics tdr interface GigabitEthernet4/0/1
TDR test last run on: March 01 03:11:11
Interface Speed Local pair Pair length
Remote pair Pair status
--------- ----- ---------- ------------------ ----------- -------------------Gi4/0/1
100M Pair A
N/A
Pair A
Normal
Pair B
9
+/- 10 meters Pair B
Open
Pair C
8
+/- 10 meters Pair C
Short
Pair D
9
+/- 10 meters Pair D
Short
BRKCRS-3141
Cisco Public
18
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
20
Compact
Switch Hardware Components: CPU

Switch Fabric
TCAM
TCAM
Port ASIC
Stack PHY
TCAM
Port ASIC
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
CPU Functions
-
BRKCRS-3141
Runs the IOS

Processes Control Plane traffic (LACP / STP / CDP / etc.)
Processes packets that are not switched in Hardware
Communicates with controller(s)
Cisco Public
21
High
Slow
CPU: Troubleshooting Processes
High CPU utilization is due to
Processes taking up resources
Forwarded Network Traffic
Normal CPU utilization varies by
Switch Model
Number of connected ports, modules, switches in stack
Feature set (LANBASE, IP BASE, IP SERVICES)
BRKCRS-3141
Cisco Public
22
Is CPU high an issue?
HW switched traffic not impacted by high CPU
Its an issue when these issues occur because control packets not sent or received
Spanning Tree Protocol (STP) reconverges
Routing protocol flaps
CLI is slow or unresponsive
BRKCRS-3141
Cisco Public
23

Use show process cpu history to display the history of CPU utilization
Switch# show processes cpu history
7466466455553535356639
4814199847367790442069
100
*
90
*
80
*
70 *
**
*
60 * ** ** * * *
** *
50 **** ******* * * *** *
40 **************** *****
30 **********************
20 **********************
10 ######################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0
5
0
5
0
5
0
5
0
5
0
5
0
CPU% per hour (last 72 hours)
* = maximum CPU%
# = average CPU%
BRKCRS-3141
Cisco Public
24

Configure the CPU threshold
(config)# process cpu threshold type {total | process | interrupt}
\ rising percentage interval seconds [falling fall-percentage interval seconds]
Syslog message
*Mar 1 01:03:15.601: %SYS-1-CPURISINGTHRESHOLD: Threshold: Process CPU Utilisation
(Total/Intr): 18%/0%, Top 3 processes(Pid/Util): 4/10%, 75/1%, 164/0%
BRKCRS-3141
Cisco Public
25

Use show process cpu sorted to display current CPU utilization
Switch# show processes cpu sorted
CPU utilization for five seconds: 43%/7%; one minute: 28%; five minutes: 22%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
196
897835293 538983117
1665 6.05% 6.74% 10.05%
0 IP Input
102
46542612 69782387
666 2.33% 1.79% 1.61%
0 hpm main process
8
7967710
67451
118127 2.33% 0.29% 0.19%
0 Licensing Auto U
141
48894294 114699852
426 1.24% 1.01% 1.18%
0 Hulc LED Process
68
45347109
1374466
32992 1.24% 0.85% 0.86%
0 Adjust Regions
Total CPU utilization 43%
Processes could cause high CPU
Hulc running con, SFF8472, IP Input

Interrupt based CPU
utilization 7%
Hulc LED Process, Exec/Virtual Exec Process

SNMP Engine Process, etc.
Process based CPU

utilization 36%
BRKCRS-3141
Use EEM script to monitor
Cisco Public
26
16 CPU Queues and Port ASIC queues

16 different CPU Queues
Packets to CPU Queues first stored on port ASIC
Switch# show platform port-asic stats drop
Supervisor TxQueue Drop Statistics
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
Queue
BRKCRS-3141
0:
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
0
0
0
0
0
0
0
1000
0
0
0
0
0
0
0
0
packets dropped
before reaching
to the CPU
Cisco Public
Switch# show controllers cpu-interface

cpu-queue-frames retrieved dropped
----------------- ---------- ---------rpc
132917740 0
stp
31879262
0
ipc
10746915
0
routing protocol 267
0
L2 protocol
424610
0
remote console
1121711
0
sw forwarding
0
0
host
345
0
broadcast
13931
0
cbt-to-spt
0
0
igmp snooping
0
0
icmp
0
0
logging
0
0
rpf-fail
0
0
dstats
132935598 0
cpu heartbeat
82903147
0
27
16 CPU Queues & CPU Buffer Pools

Each queue reserves buffers
for specific traffic
Switch# show buffer | in RxQ
RxQ0 buffers (rpc)

RxQ1 buffers (stp)
RxQ2 buffers (ipc)
RxQ3 buffers (routing protocol)
RxQ4 buffers (L2 protocol)
RxQ5 buffers (remote console)
RxQ6 buffers (sw forwarding)
RxQ7 buffers (host)
RxQ8 buffers (broadcast)
RxQ9 buffers (cbt-to-spt)
RxQ10 buffers (igmp snooping)
RxQ11 buffers (icmp)
RxQ12 buffers (logging)
RxQ13 buffers (rpf-fail)
CPU buffer pools are named

RxQ0 to RxQ15
RxQ15 buffers (cpu heartbeat)
BRKCRS-3141
Cisco Public
28
CPU: Software Forwarding Queue (Q6)

For Traffic that hardware cannot process
- SW forwarding performance is much lower than HW
To debug any CPU Q
Switch# debug platform cpu-queues software-fwd-q

*Mar 1 10:37:33.205 AEDT: SW-FWD-Q:IP packet: Local Port Fwding L3If:Vlan1
L2If:GigabitEthernet2/0/2 DI:0x2F, LT:7, Vlan:1
SrcGPN:56, SrcGID:56, ACLLogIdx:0x0,
MacDA:c471.fe1e.f0c0, MacSA: 0007.7d75.88c0
IP_SA:14.160.38.1 IP_DA:14.160.38.130 IP_Proto:1
IP Opts
Incoming physical
interface
BRKCRS-3141
SMAC of the host sending

the traffic
Cisco Public
29
CPU: Routing Protocol Queue (Q3)
Receives all traffic for routing protocols (BGP, OSPF, EIGRP, HSRP, etc.)
Switch# debug platform cpu-queues routing-protocol-q

Switch# debug standby
HSRP debugging is on
*Mar
6 00:47:39.260: RT-Q:Queued: Local Port Fwding L3If:Vlan100 L2If:GigabitEthernet1/0/1

DI:0x12FC, LT:7, Vlan:100
SrcGPN:1, SrcGID:1, ACLLogIdx:0x0, MacDA:0100.5e00.0002, MacSA:
0018.ba88.1fc1
*Mar
BRKCRS-3141
6 00:47:39.260: HSRP: Vl100 Grp 0 Hello
Cisco Public
in
30
10.1.1.2 Standby pri 100 vIP 10.1.1.55
CPU: Host Queue (Q7)

Used for all unicast traffic sent to the switch
TACACS, SSH, telnet, ping, SNMP
Show buffer shows current buffer usage

Switch# debug platform cpu-queues host-q
*Mar
6 00:01:46.648: Host-Q:Queued L3If: Local Port Fwding L3If:Vlan100

L2If:GigabitEthernet1/0/1 DI:0xB0, LT:7, Vlan:100
SrcGPN:489, SrcGID:488,
ACLLogIdx:0x0, MacDA:000f.f7e8.e041, MacSA: 0018.ba88.1fc1
IP_SA:10.1.1.2
IP_DA:10.1.1.1 IP_Proto:1
TPFFD:DC0001E9_00000064_00B00076-000000B0_A68A0000_00000000
Switch# show buffer | begin RxQ7
RxQ7 buffers, 2040 bytes (total 192, permanent 192):
64 in free list (0 min, 192 max allowed)

294 hits, 0 misses
BRKCRS-3141
Cisco Public
31
CPU: ICMP Queue (Q11)
Receives all traffic for which an ICMP message needs to be generated

Excluding PING
Routed port only
Switch# debug ip icmp
Switch# debug platform cpu-queues icmp-q
*Mar
9 21:34:30.695: ICMP-Q:Queued to Process, use GW:10.1.1.3: Remote Port Blocked

L3If:Vlan100 L2If:GigabitEthernet4/0/1 DI:0xB4, LT:7, Vlan:100
SrcGPN:163,
SrcGID:163, ACLLogIdx:0x0, MacDA:0018.ba88.1fc1, MacSA: 000f.f7e8.e041
IP_SA:10.1.1.1
IP_DA:77.1.1.1 IP_Proto:1
*Mar
BRKCRS-3141
9 21:34:30.695: ICMP: redirect sent to 10.1.1.1 for dest 77.1.1.1, use gw 10.1.1.3
Cisco Public
32

ICMP Unreachables Example
Symptoms:
Relatively high CPU
Low processor utilization
ICMP Queue heavily utilized
Switch# show processes cpu sorted
CPU utilization for five seconds: 53%/47%; one minute: 31%; five minutes: 18%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
149
397089
3879429
102 0.63% 0.34% 0.45%
0 Spanning Tree
112
325474
117735
2764 0.31% 0.15% 0.09%
0 HRPC qos request
Switch# clear controllers cpu
Switch# show controllers cpu-interface | include icmp
icmp
133148
0
0
0
0
BRKCRS-3141
Cisco Public
33

ICMP Unreachables Example
Causes
High amount of traffic is dropped because of a deny statement in an ACL
CPU is interrupted to send ICMP unreachable packets back to the source
Solution
Disable ICMP unreachables on the ingress interface
Switch(config)# interface GigabitEthernet1/0/2

Switch(config-if)# no ip unreachable
Switch(config-if)# end
BRKCRS-3141
Cisco Public
34
CPU: Best Practices

Storm Control can help to protect CPU
(config-if)#storm-control broadcast level level[.level]
(config-if)#storm-control action ?
shutdown Shutdown this interface if a storm occurs
send SNMP trap if a storm occurs
Protocol Storm Protection (PSP)

(config-if)#psp ?
arp Set rate limit value for ARP Packets
dhcp Set rate limit value for DHCP Packets
igmp Set rate limit value for IGMP Packets
The switch drops all traffic on the VP for 30 seconds

Enable parser config cache interface
http://www.cisco.com/c/en/us/td/docs/ios/fundamentals/configuration/guide/15_1s/cf_15_1s_book/config_cache.html#wp1057005
BRKCRS-3141
Cisco Public
35
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
36
Compact
Switch Hardware Components: Memory

Switch Fabric
Port ASIC
Port ASIC
Stack PHY
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
Two Types of Memory

-
Processor memory is used by IOS Processes
I/O memory is used for CPU traffic
BRKCRS-3141
Cisco Public
37
Running
out?
Troubleshooting Memory Utilization

Syslog messages most common indication
%SYS2MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x601617A4, pool Processor, alignment 0
Process= "IP Input", ipl= 2, pid= 21
%PLATFORM_RPC-0-RESOURCE_CRASH: System is unable to allocate memory for RPC
Switch not accessible, any CLI output becomes show process memory, etc.
Switch# show memory statistics
Processor
I/O
Head
2641D6C
7400000
Total(b)
81519252
12574720
Memory available now
BRKCRS-3141
Used(b)
31192204
8532852
Free(b)
50327048
4041868
The lowest free

since boot up
Cisco Public
38
Lowest(b)
49241540
3821068
Largest(b)
48621848
4039616
Largest block switch

can allocate
Troubleshooting Memory Utilization

Two main reasons of Processor memory problems
-
Is any process steadily

increasing held memory?
Process does not release the memory after use

Process does not limit the amount of memory it allocates
Switch# show processes memory sorted

PID TTY Allocated
Freed
Holding
0
0
74539888
23738156
47199076
0
0
3399716
17490880
1590292
65
0
712620
27424
594488
324
0
19794764
19262624
539264
11
0
228060
14940
226488
Getbufs
0
10657136
0
0
0
Retbufs
0
553112
0
0
0
Process
*Init*
*Dead*
Stack Mgr Notifi
hulc running con
ARP Input
Run commands multiple times to benchmark

show processes memory sorted
show memory summary
show memory allocating totals
Use Threshold notifications for periodic monitoring

Switch(config)# memory free low-watermark processor 20000
Switch(config)# memory free low-watermark io 20000
%SYS-4-FREEMEMLOW: Free Memory has dropped below 20000k
Pool: Processor Free: 66814056 freemem_lwm: 204800000
Reference:
http://www.cisco.com/en/US/docs/ios/12_2s/feature/guide/fs_memnt.html
BRKCRS-3141
Cisco Public
39
Memory Leak Example (using show commands)

Switch#show clock
09:34:41.300 UTC Wed Apr 3 2013
switch#show proc mem sorted
Processor Pool Total: 78964596 Used: 36942892 Free:
42021704
I/O Pool Total: 12574720 Used: 8583916 Free: 3990804
PID TTY Allocated Freed
Holding Getbufs Retbufs Process
204 0
25509496 21347536 3957470
0
0 HTTP CORE
Switch#show clock
11:34:41.300 UTC Wed Apr 3 2013
Switch#show proc mem 204

Process ID: 204
Process Name: HTTP CORE
Total Memory Held: 4175420 bytes
Processor memory Holding = 4175420 bytes
pc = 0x015E5430, size = 2143156, count = 5265
pc = 0x0166F148, size = 1643716, count = 11583
pc = 0x01685C18, size = 231660, count = 351
pc = 0x004F4B60, size = 35136, count = 61
pc = 0x015E55C8, size = 27976, count = 351
pc = 0x015DB614, size = 25720, count = 351
pc = 0x01141F34, size = 2408, count = 2
switch#show proc mem sorted

Processor Pool Total: 78964596 Used: 36942892 Free:
42021704
I/O Pool Total: 12574720 Used: 8583916 Free: 3990804
PID TTY Allocated Freed
Holding Getbufs Retbufs Process
204 0
25727446 21347536 4175420 0
0 HTTP CORE
Provide captures to TAC if no bugs are found in bug tool kit

A quick search leads to :
CSCsk34832 Memory leak in HTTP CORE
BRKCRS-3141
Cisco Public
40
IOS Memory Leak Detector

Inbuilt Memory Leak Detector can be used to detect memory leaks
show memory debug leaks [chunks | largest | lowmem | summary]
Switch#show memory debug leak
Adding blocks for GD...
I/O memory
Address Size Alloc_pc PID Alloc-Proc Name
Processor memory
Address Size Alloc_pc
28D91E8 72
16EF0B4
28D927C 124
16EF0B4
28D9358 68
16EF0B4
PID Alloc-Proc Name

0 *Dead* SSH2 String
A quick search leads to :

CSCsm89128 Memory Leak when invalid password entered for SSHv2 session
Reference:
http://www.cisco.com/en/US/partner/docs/ios/fundamentals/configuration/guide/cf_mem-leak-detect.html
BRKCRS-3141
Cisco Public
41
Troubleshooting: I/O Memory Buffers

Switch# show buffers
Buffer elements:
1679 in free list (500 max allowed)
27109526 hits, 0 misses, 1641 created
I/O memory for incoming CPU bound

packets
Classified into two major pools :
Public buffer pools:

Small buffers, 104 bytes (total 50, permanent
50, peak 181 @ 3w5d):
129877853 hits, 141 misses, 390 trims, 390
created
0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent
25, peak 94 @ 7w0d):
616791 hits, 54 misses, 162 trims, 162
created
0 failures (0 no memory)
- Public Buffer Pools (pre-assigned based on

the packet size)
- Interface buffer pools (pre-assigned for 15
CPU queues)
Look for incrementing Failures and No

Memory in show buffers output
show memory debug leak can detect I/O
memory leaks as well
BRKCRS-3141
Cisco Public
:(truncated)
42
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
44
Compact
Switch Hardware Components: Port ASIC Issues

Switch Fabric
TCAM
TCAM
Port ASIC
Stack PHY
TCAM
Port ASIC
CPU
Port ASIC
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
ASIC Forwarding cannot be accomplished?
High CPU Utilization
Is it a performance problem?
BRKCRS-3141
Packet drops on a port

Cisco Public
45
Forwarding?
Buffers?
Troubleshooting ASIC Issues : HW Forwarding

L2 and L3 forwarding decision based on
-
Routing/Switching tables
ACL Redirection (PBR/WCCP)

Switch# show plat forward <src intf> <srcmac> <dstmac> [ip <srcip> <dstip>
<protocol>]
Switch# show platform forward Gi0/1 0000.0000.0001 0000.0000.0010 ip 192.168.1.242
192.168.10.242 0
[..]
Egress: Asic 0, switch 1
Source Vlan Id: Real 10, Mapped 2. L2EncapType 0, L3EncapType 0
portMap 0x200, non-SPAN portMap 0x200
Destination Interface
Output Packets:
[..]
Port
Vlan
Gi0/10
0020
BRKCRS-3141
SrcMac
0000.0000.0020
DstMac
Cos
0000.0000.0002
Cisco Public
46
Dscpv
Troubleshooting ASIC Issues: Software Forwarding

ASIC cannot process data packets and forwards them to CPU using Q6
Switch# clear controllers cpu
Switch# show controllers cpu-interface | include sw forwarding
sw forwarding
71558
0
0
0
Switch# debug platform cpu-queues software-fwd-q

SW-FWD-Q:Consumed by SW-Bridging: Remote Port Blocked L3If:Vlan101 L2If:GigabitEthernet1/0/2
DI:0x2FD, LT:7, Vlan:101
SrcGPN:2, SrcGID:2, ACLLogIdx:0x0, MacDA:000f.f7e8.e042, MacSA:
0000.00bb.87df
Switch# show plat forward Gi1/0/2 00.00bb.87df 000f.f7e8.e042 ip 10.101.1.100 10.99.1.100 255
Redirected by Input ACL. New destIndex is 0x02C7.
==========================================
Egress: ASIC 0, switch 1
CPU queues: 6 14.
BRKCRS-3141
Cisco Public
47
ASIC Issues: Mapping Interfaces to Port-ASIC

Show platform pm if-number shows this mapping
Physical and ASIC port numbers may not match
This command shows all members
Switch# show platform pm if-numbers

interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
---------------------------------------------------------------------Gi3/0/1
109 109 1
1/1 3
1
1
local
Yes
Yes
Gi3/0/2
110 110 2
1/0 3
2
2
local
Yes
Yes
Gi3/0/3
111 111 3
1/3 3
3
3
local
Yes
Yes
Gi3/0/4
112 112 4
1/2 3
4
4
local
Yes
Yes
Gi3/0/5
113 113 5
1/5 3
5
5
local
Yes
Yes
Gi3/0/6
114 114 6
1/4 3
6
6
local
Yes
Yes
Gi3/0/7
115 115 7
1/7 3
7
7
local
Yes
Yes
ASIC/Port
BRKCRS-3141
Cisco Public
48
ASIC Issues: Port-ASIC Statistics

Provides overview of possible drops/issues on the switch
Local and Member switches
Switch# remote command 2 show controller ethernet-controller port-asic statistics
Switch# show controllers ethernet-controller port-asic statistics

===========================================================================
Switch 2, PortASIC 0 Statistics
--------------------------------------------------------------------------0 RxQ-0, wt-0 enqueue frames
0 RxQ-0, wt-0 drop frames
8811506 RxQ-0, wt-1 enqueue frames
<snip>
100 TxBufferFull Drop Count
...
0 SneakQueue Drop Count
...
0 Sup Queue 0 Drop Frames
<snip>
BRKCRS-3141
Cisco Public
0 Rx Fcs Error Frames

0 Tx Too Old Frames
49
ASIC Issues: Egress Queue Drops

Queue and weight are 0-based
Tuning of buffers is only possible when QoS is enabled
Drops on egress indicate oversubscription
Switch# show platform port-asic stats drop gigabitEthernet 1/0/3
Interface Gi1/0/3 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
...
Queue 3
Weight 0 Frames 100000
Weight 1 Frames 0
Weight 2 Frames 0
More information
in the upcoming
QOS section
Switch# show platform port-asic stats enqueue gi1/0/3
BRKCRS-3141
Cisco Public
50
Switch Hardware Components: Port ASIC QoS
Switch Fabric
TCAM
Stack PHY
TCAM
Port ASIC
TCAM
Port ASIC
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
Ingress QoS
Egress QoS
QoS
Traffic Classification Maps

BRKCRS-3141
Cisco Public
52
Cisco Catalyst 3750 QoS Overview

Policer
Marker
Queue 1
Marker
Queue 1
Classify
Input
Traffic
SRR
Queue 2
Policer
Marker
Policer
Marker
StackWise
Policer
Queue 2
SRR
Queue 3
Queue 4
Classification
Inspect incoming
packets
Assign QOS Label
to grouped packet
Use ACL, or other
configuration to
determine QOS
labels
BRKCRS-3141
Policing
Marking
Compares incoming Act on policer

traffic rate w/
decision
configured policer
Reclass or drop
and determine if
out-of-profile
packet is IN or Out of
Profile.
Either aggregate or
individual flow basis
256 policers/ASIC
Cisco Public
Ingress Queue/
Schedule
Congestion
Control
Two queues/port ASIC
shared servicing
One queue is
configurable for strict
priority servicing
WTD for congestion
control (three
thresholds per queue)
SRR is performed
53
Egress Queue/
Schedule
Congestion
Control
Four SRR queues/port shared
or shaped servicing
One queue is configurable
for strict priority servicing
WTD for congestion
control (three thresholds
per queue)
Egress queue shaping
Egress port rate limiting
Cisco Catalyst 2960S/X QoS Model

Policer
Marker
Queue 1
Policer
Marker
Queue 2
Classify
Input
Traffic
SRR
Queue 3
Policer
Marker
Policer
Marker
Queue 4
Classification
Inspect incoming
packets
Assign QOS Label
to grouped packet
Use ACL, or other
configuration to
determine QOS
labels
BRKCRS-3141
Policing
Marking
NO
Ingress Queues
Compares incoming Act on policer

traffic rate w/
decision
configured policer
Reclass or drop
and determine if
out-of-profile
packet is IN or Out of
Profile.
Either aggregate or
individual flow basis
256 policers/ASIC
Cisco Public
54
Egress Queue/
Schedule
Congestion
Control
Four SRR queues/port shared
or shaped servicing
One queue is configurable
for strict priority servicing
WTD for congestion
control (three thresholds
per queue)
Egress queue shaping
Egress port rate limiting
Why Ingress QoS ?

Ingress QoS responsibilities
Ensure traffic classified correctly

Police traffic via Service Policy with traffic profiles
Security ACLs
Prioritize traffic during Stack congestion
Symptoms for ingress QOS problems

Packets unexpectedly dropped due to Access Service Policy, or stack congestion
Packets improperly marked for priority
BRKCRS-3141
Cisco Public
55
QoS Troubleshooting Ingress

access
dot1q
Gi1/0/2
3750
10000 IP packets
with DSCP 34
Ingress policer with

trust DSCP
Switch# show mls qos interface gigabit 1/0/2 statistics

GigabitEthernet1/0/2 (All statistics are in packets)
dscp: incoming
------------------------------0 - 4 :
30 - 34 :
...
Policer: Inprofile:
0
0
0
0
0
0
1467 OutofProfile:
10,000 packets were received, DSCP value 34

1,467 packets were in profile
8,533 were dropped due to exceeding the policer
BRKCRS-3141
Cisco Public
56
0
0
8533
0
10000
Causes of Egress Congestion
Congestion is the biggest QoS issue

Slower speed link
Oversubscription
BRKCRS-3141
Cisco Public
57
Why Egress QoS? Rate Transition

Slower speed interfaces take longer to transmit packets
Introduction of Gigabit servers pushes congestion to the edge
QoS drops lowest priority packets
Buffers up on 100Mb interfaces

Packets take longer to egress
Traffic Burst on 10 Gig interface

3
5 43
3 5 2 4 1 3 2 1
Fat 10 Gig Pipe
with pkts ingressing
BRKCRS-3141
Egress
Buffer
Cisco Public
58
Thin 100 Mbps pipes

with pkts egressing
Egress Queuing
Policer
Marker
Queue 1
Marker
Queue 1
Classify
Input
Traffic
SRR
Queue 2
Policer
Marker
Policer
Marker
StackWise
Policer
Queue 2
SRR
Queue 3
Queue 4
The Cisco Catalyst 3750/2960 have four egress queues

Queue 1 is optionally the priority queue
Port-based bandwidth rate limiting can be configured from 10% to 90%
These Egress queues, perform Shaped Round Robin SRR in queue sharing and queue
shaping mode
Weighted Tail Drop (WTD) for congestion management
BRKCRS-3141
Cisco Public
59
Queues share Bandwidth and Buffers

Boarding on Overbooked Flight
The passenger capacity defines the

available interface bandwidth
Bandwith share per queue (class)
First
Business
Economy U.
Economy L.
2%
15%
20%
63%
Boarding Lines and waiting lists are

managed for each queue (class),
allowing different buffer sizes
(depths) and drop thresholds
BRKCRS-3141
Cisco Public
60
What is an Egress Queue-set

Buffer Allocation and Drop Strategy
Two available queue-sets
Switch#show mls qos queue-set 1
Queueset: 1
Queue
:
1
2
3
4
--------------------------------------------buffers
:
20
20
30
30
threshold1:
33
33
33
33
threshold2:
66
66
77
50
reserved :
92
92
100
67
maximum
:
138
300
300
300
Each interface belongs to one queue-set

4 Egress Queues per port
3 drop thresholds per Queue
Threshold defines drop precedence for a
class of traffic
Threshold values over 100% dip into
common pool (MAX).
All values in Percentages of 100
Queue-set does not define bandwidth
BRKCRS-3141
Cisco Public
61
Mapping Classes to Egress Queues

Maps available for DSCP and COS.
64 DSCP default values shown.
Each DSCP value maps to an egress Queue, and threshold
Queues range: 1-4, Threshold range:01-03
DSCP:0
Queue 4:
Threshold 3
Switch# show mls qos maps dscp-output-q

Dscp-outputq-threshold map:
d1 :d2
0
1
2
3
4
5
6
7
8
-----------------------------------------------------------0 :
04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-01
1 :
04-02 04-02 04-02 04-02 04-02 04-02 03-03 03-03 03-03
2 :
03-03 03-03 03-03 03-03 02-03 02-03 02-03 02-03 02-03
3 :
02-03 02-03 03-03 03-03 03-03 03-03 03-03 03-03 03-03
4 :
01-03 01-03 01-03 01-03 01-03 01-03 01-03 01-03 02-03
5 :
02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03
6 :
02-03 02-03 02-03 02-03
BRKCRS-3141
Cisco Public
62
9
04-02
03-03
02-03
03-03
02-03
02-03
DSCP:46
Queue 1
Threshold 3
DSCP:63
Queue 2
Threshold 3
QoS Troubleshooting - Ingress

access
Remember this from a

few slides ago??
Gi1/0/2
Gi1/0/1
3750
10000 IP packets
with DSCP 34

trust DSCP
Switch# show mls qos interface gigabit 1/0/2 statistics

dscp: incoming
------------------------------0 - 4 :
30 - 34 :
...
Policer: Inprofile:
0
0
0
0
0
0
0
0
1467 OutofProfile:
0
10000
8533
1,467 packets were in profile, and forwarded to egress interface
BRKCRS-3141
Cisco Public
63
dot1q
QoS Troubleshooting - Egress

access
Gi1/0/2
dot1q
Gi1/0/1
3750
10000 IP packets
with DSCP 34

trust DSCP
Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics

<output removed>
dscp: outgoing
------------------------------<output removed>
25 - 29 :
30 - 34 :
<output removed>
0
0
0
0
0
0
0
0
1467 packets were in profile and made it to the egress port
DSCP is 34
BRKCRS-3141
Cisco Public
64
0
1467
QoS Troubleshooting Egress (2)

access
Gi1/0/2
Gi1/0/1
dot1q
3750
10000 IP packets
with DSCP 34
Switch#sh mls qos interface gigabitEthernet 1/0/1 statistics

<output removed>
0 4 :
1467
0
0
0
30 - 34 :
0
0
0
0
0
0
1467 packets were in profile and made it to the egress port but with DSCP 0 instead
of 34
Possible reasons
Attached service policy does not mark or trust dscp value
Traffic is being routed via the CPU
BRKCRS-3141
Cisco Public
65
QoS Troubleshooting Egress Q Maps

100Mb/s
10Mb/s
3750
10000 IP packets
with DSCP 34
Gig 1/0/1
Gig 1/0/2
Switch# show mls qos interface gi 1/0/2 statistics

dscp: incoming
------------------------------0 - 4 :
30 - 34 :
0
0
0
0
0
0
0
0
Switch# show mls qos maps dscp-output-q

Dscp-outputq-threshold map:
d1 :d2
0
1
2
3
4
5
6
7
8
-----------------------------------------------------------0 :
02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01
1 :
02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01
2 :
03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01
3 :
03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
4 :
01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01
5 :
04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
6 :
04-01 04-01 04-01 04-01
BRKCRS-3141
Cisco Public
66
0
10000
9
02-01
03-01
03-01
04-01
04-01
04-01
10000 packets are

received and will
egress on Q4,
threshold 1
QoS Troubleshooting Egress Queue Thresholds

100Mb/s
10000 IP packets
with DSCP 34
10Mb/s
3750
Gig 1/0/2
Gig 1/0/1
Switch# show mls qos interface Gig 1/0/1 statistics

dscp: outgoing
------------------------------0 - 4 :
0
0
0
30 - 34 :
0
0
0
...
output queues enqueued:
queue:
threshold1 threshold2 threshold3
----------------------------------------queue 0:
2
0
0
queue 1:
0
6
260
queue 2:
0
0
0
queue 3:
1080
0
0
0
0
output queues dropped:

queue:
threshold1 threshold2 threshold3
----------------------------------------queue 0:
0
0
0
queue 1:
0
0
0
queue 2:
0
0
0
queue 3:
8920
0
0
BRKCRS-3141
Cisco Public
0
1080
1080 packets will

egress on Q4,
threshold 1
Remaining 8920 pkts

dropped because of
congestion
CPU Generated Packets Egress

Queue 2
67
QoS Troubleshooting - Port-ASIC

100Mb/s
10Mb/s
3750
10000 IP packets
with DSCP 34
Gig 1/0/1
Gig 1/0/2
Viewing Egress Congestion (another way) with port-asic command

Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 1 Frames 0
Weight 2 Frames 0
BRKCRS-3141
Cisco Public
68
Command works on all

Catalyst IOS versions
10000 packets were

received, 8920 were
dropped on egress
QoS Troubleshooting - Buffer Tuning

Tuning Buffers and Thresholds to fix Congestion
Queue-sets define the buffer allocation
Default values can be modified
2 Queue-sets are available
Reserved - how many buffers will be reserved for this port
Default Queue-set values listed below
Switch# show mls qos int gi1/0/1 buffers
GigabitEthernet1/0/1
Identifies Queue-set assigned to interface
The port is mapped to qset : 1
The allocations between the queues are : 25 25 25 25
Switch# show mls qos queue-set
Queueset: 1
Queue
:
1
2
3
4
---------------------------------------------buffers
:
25
25
25
25
threshold1:
200
200
100
100
threshold2:
200
200
100
100
reserved :
50
50
50
50
maximum
:
400
400
400
400
BRKCRS-3141
Cisco Public
Dropped on this Queue

and Threshold
69
QoS Troubleshooting - Buffer Tuning (2)

100Mb/s
10000 IP packets
with DSCP 34
10Mb/s
3750

Queue 3
Packet drops with current Queue-set configuration
Switch(config)# mls qos queue-set output 1 threshold 4 300 300 50 400
Switch# show mls qos queue-set
Queueset: 1
Queue
:
1
2
3
4
---------------------------------------------buffers
:
25
25
25
25
threshold1:
100
100
100
300
threshold2:
100
100
100
300
reserved :
50
50
50
50
maximum
:
400
400
400
400
Threshold increased to 300

Queue 3
No additional Packet drops after Queue-set change
BRKCRS-3141
Cisco Public
70
Egress QoS Summary

Packet drops dont always indicate a problem
For ex, Gigabit servers can easily oversubscribe 100M clients
Most protocols react well to drop and will slow down so
maximum performance can be achieved
Analyze traffic patterns
Tune buffers as needed increasing thresholds has minimal side effects

Take advantage of both queue-sets
E.g.: use Queue-set 1 on downlinks, Queue-set 2 on uplinks
Map queues to distribute traffic according to the Plan

Set thresholds to optimize high priority traffic
Auto QoS
QoS is not easy, but Auto QOS makes it easy
Auto QoS produces consistent configurations across all 2K and 3K switch models
BRKCRS-3141
Cisco Public
71
Switch Hardware Components: TCAM

Switch Fabric
TCAM
TCAM
Port ASIC
Stack PHY
TCAM
Port ASIC
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
TCAM Resources?
The TCAM stores Forwarding database

-
ACLs
IPv4, IPv6 and MAC addresses

Service policies and security
Multicast Addresses and Groups
BRKCRS-3141
Cisco Public
73
TCAM: Switch Database Manager (SDM)

SDM defines how TCAM
resources are allocated
Changing SDM template requires
reboot
Switch# show sdm prefer default

"desktop default" template:
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.
All stack members must use same

SDM template
Switch# show sdm prefer ?
access
Access bias
default
Default bias
dual-ipv4-and-ipv6 Support both
IPv4 and IPv6
routing
Unicast bias
vlan
VLAN bias
Switch# show sdm prefer dual-ipv4-andipv6 ?
default Default bias
routing Unicast bias
vlan
VLAN bias
BRKCRS-3141
Cisco Public
number of unicast mac addresses:

number of IPv4 IGMP groups + multicast routes:
number of IPv4 unicast routes:
number of directly-connected IPv4 hosts:
number of indirect IPv4 routes:
number of IPv4 policy based routing aces:
number of IPv4/MAC qos aces:
number of IPv4/MAC security aces:
List of available SDM Types
74
6K
1K
8K
6K
2K
0
0.5K
1K
TCAM Utilization
Route entries not

forwarded in H/W
TCAM space is limited
Switch# show platform ip unicast failed route
Problem when Used

Masks/Values = MAX
Switch# show platform tcam utilization
Optimize ACE/Routing
entries
Change SDM Template
Layer 3 Routing
CAM Utilization for ASIC# 0
Masks/Values
Unicast mac addresses:
784/6272
14/40
IPv4 IGMP groups + multicast routes:
144/1152
7/27
IPv4 unicast directly-connected routes:
784/6272
14/40
IPv4 unicast indirectly-connected routes:
272/2176
11/55
IPv4 policy based routing aces:

IPv4 security aces:
0/0
0/0
768/768
260/260
1024/1024
723/723
Note: Allocation of TCAM entries per feature uses

a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization
BRKCRS-3141
Used
Masks/values
IPv4 qos aces:
Security ACLs
Permit/deny
Max
Cisco Public
75
TCAM Overload
An error message will get generated

Traffic forwarding will be done (partly) in Software
CPU utilization will go up packets punted to CPU for processing
%ACLMGR-4-UNLOADING: Unloading ACL input label 1 VLAN interfaces 101 IPv4/Mac feature
%ACLMGR-4-ACLTCAMFULL: ACL TCAM Full. Software Forwarding packets on Input label 1 on L3 L2
Switch# sh platform acl oacltcamfull
Vlan
oacl_tcam_full_bitmap
101
Vlan
0x
notify_apps
NOT-FULL
ipv6_oacl_tcam_full_bitmap
notify_apps
Switch# sh platform acl label 1 detail
IPv4/MAC ACL label

-----------------Unloaded due to lack of space:
BRKCRS-3141
Cisco Public
76
Means ACL Not Fully

Programmed in TCAM
TCAM: Switch Database Manager (SDM)

Strategies to choose SDM
SDM Template
Use Case
access
L2 & L3, fewer L2 & L3 addresses than default,

Supports Policy Based Routing, more security ACEs
default
L2 & L3, more L2 & L3 addresses than access
routing
L2 & L3, weighted towards L3 space, Supports

Policy Based Routing
vlan
L2 only, 12K MAC Addresses
dual-ipv4-and-ipv6
Required for IPv6 functionality
default, routing, vlan
same distribution as above, but with IPv6 resources
BRKCRS-3141
Cisco Public
77
TCAM Hardware Summary

TCAM Partition based on SDM Template
L2 and L3 overload of TCAM resource: punt to CPU
Number of ACEs depend on
Switch Model
SDM Template different Templates for Layer 3 capable switches
If ACL does not fit in TCAM, will be processed in SW (CPU)

CPU processing is much slower than TCAM
Switch reboot required when SDM template changed
BRKCRS-3141
Cisco Public
78
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
80
Compact
Troubleshooting Power Over Ethernet

Stack
PHY
Switch Fabric
TCAM
TCAM
Port ASIC
Two
Stack
Cables
TCAM
Port ASIC
CPU
Port ASIC
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
TCAM
TCAM
24X1G POE
12X1G
12X1G
10/100
24X1G POE
12X1G
12X1G
StackWise,
StackWise
Plus
10G or 1G
What is the Power requirement of PD ?

Is it a Cisco PD or a Third party PD ?
Does the PD work fine on a different port or a different
switch ?
Or all POE ports affected ?
BRKCRS-3141
Cisco Public
81
PoE
Troubleshooting PoE: CDP/LLDP Negotiation

Inline power available. If not, this
Stack-1# show power inline
log would be seen:
Module Available Used Remaining
%ILPOWER-5(Watts) (Watts) (Watts)
ILPOWER_POWER_DENY: Interface
<interface>: inline power denied
------ --------- -------- --------1
420.0
22.2
397.8
Power Allocated
2
370.0
18.2
351.8
Interface Admin Oper
Power Device
Class Max
(Watts)
--------- ------ ---------- ------- ------------------- ----- ---Gi1/0/2 auto on
10.3
IP Phone 7970
3 15.4
Gi2/0/2 auto on
8.5 AIR-AP1220-IOS
n/a 15.4
Power Negotiation can occur via CDP or LLDP Power-via- MDI protocol.
CDP PD requests the worst-case power (including the link loss)
LLDP PD requests only power required, the PSE adds the link loss values
BRKCRS-3141
Cisco Public
82
Troubleshooting PoE: PD Drawing Too Much Power

Imax or Tstart error reported when a PD misbehaves and draws more power
- Imax error is an operating fault and reported after PD power up
-
Tstart is a start up fault before PD reported Power Good
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface gig1/0/1 Power Controller reports power Imax error detected
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface: Power Controller reports power Tstart error detected
Switch(config)# int gig 1/0/1

Switch(config-if)# power inline port 2x-mode
Switch(config-if)# shut
Switch(config-if)# no shut
BRKCRS-3141
Cisco Public
83
Configure 2x-mode on the affected

interface as a workaround. The mode
raises the thresholds for I(cut), I(short)
current.
Bug ID CSCsw18530
Troubleshooting PoE: Power given but Power Good

not reported
PD detected. Power was granted but the PD was not up
PD cannot provide MPS (10mA)
%ILPOWER-3-CONTROLLER_PORT_ERR: Controller port error, Interface Gi8/0/1: Power given, but Power Controller does not report Power
Good
BRKCRS-3141
Cisco Public
84
Troubleshooting PoE: Multiple ports not providing

power
Check the controller status and Error counters
%FRNTEND_CTRLR-2-SUB_INACTIVE: The front end controller 0 is inactive Traceback= 1CA91E8 1CA95B8 1E7CD8C 1E73818
%FRNTEND_CTRLR-1-SUB_I2C_ERR: Sub 0 reported 36B5B98 I2C errors
Switch# show platform frontend-controller subordinate <0-2>

State
OK
Last Reset Reason
UNKNOWN REASON
:
SLE Poe No Port
SLE I2C Busy
SLE I2C Error
SLE I2C Timeout
SLE Invalid Reg Len
SLE Msg Underrun
BRKCRS-3141
Check for incrementing error

counters
Cisco Public
85
Troubleshooting PoE: Useful Debugs

Specific debug conditions can
be used where available
Switch#debug condition interface gigabitEthernet 1/0/15
Condition 1 set
Switch#debug ilpower powerman
ILPOWER powerman debugging is onilpower_power_assgn_handle_event: event 0,
pwr assign is done by proto CDPPort Gi1/0/9: Selected Protocol CDP
Ilpower interface (Gi1/0/9) process tlv from cdp INPUT:
power_request_level[] = 12000 0 0 0 0
Switch#debug ilpower event

%ILPOWER-7-DETECT: Interface Gi1/0/15: Power Device detected: IEEE PD
ILP uses DC Disconnect(Gi1/0/15): state=ILP_IEEE_PD_DETECTED_S, event=
ILP uses DC Disconnect(Gi1/0/15): state=ILP_LINK_UP_S, event=PHY_LINK_UP_EV
ILP uses DC Disconnect(Gi1/0/15): state=ILP_LINK_UP_S, event=
ILP_POWER_POLICE_DISABLE_EV
BRKCRS-3141
Cisco Public
86
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
88
Compact
Switch Hardware Components: Stacking

Switch Fabric
TCAM
TCAM
Port ASIC
Stack PHY
TCAM
Port ASIC
Port ASIC
CPU
SDRAM
12 Port
PHY
12 Port
PHY
12 Port
PHY
12 Port
PHY
Modular PHY
Flash
Serial
10/100
Only 3750, 3750G, 3750-E, 3750-X support stacking with StackWise or StackWise Plus
C2960-S, C2960-SF, 2960-X, 2960-XR support FlexStack or FlexStack Plus
Conditions that can prevent a switch from joining a stack:

- Incompatible IOS Versions between the stack members
- Stack cable issue
- SDM Template mismatch
BRKCRS-3141
Cisco Public
89
Stack
errors
Troubleshooting Stacks: Version Mismatch

Software Version Mismatch
IOS version of all stack switches (show version) should be either the same or
compatible
Switches with different Major Version numbers
Occurs on switch member addition, or RMA replacement
3750X# show version
Switch Ports Model
SW Version
------ ----- ----------------------1 54 WS-C3750X-48P 15.2(1)E
2 54 WS-C3750X-48P 15.2(1)E
SW Image
C3750E-UNIVERSALK9-M
C3750E-UNIVERSALK9-M
3750X# show platform stack manager all

IOS Versions should match
Switch Master/ Mac Address
Version Current
Number Member
(maj.min) State
----------------------------------------------------------1
Member f866.f2ab.7180
1.51
Ready
2
Member f866.f2af.3b00
1.51
Ready
BRKCRS-3141
Cisco Public
Major versions must match
90
Troubleshooting Stacks, Stack Cables
A Switch can join a stack with only one Stackwise interface connected to
another active stack member.
Important precautions for connecting Stackwise cables
- Retainer screws on the connector should not be loose
- Retainer screws on the connector should not be too tight
- Retainer screws should be tightened finger tight and no more
Retainer not fully engaged
Retainer fully engaged

BRKCRS-3141
Cisco Public
91
Troubleshooting: Stack Commands

3750# show switch detail
Current
Switch# Role
Mac Address
Priority
State
-----------------------------------------------------1
Slave
000c.30ae.4f00
9
Ready
*2
Master
000d.bd5c.1680
15
Ready
Stack Port Status
Neighbors
Switch# Port 1
Port 2
Port 1
Port 2
-----------------------------------------------------1
Ok
Ok
2
2
2
Ok
Ok
1
1
3750# show switch stack-ring activity

Switch
Frames sent to stack ring (approximate)
-----------------------------------------------1
5781
2
4928
Total frames sent to stack ring : 10709
Note: these counts do not include frames sent to the ring
by certain output features such as output SPAN and output
ACLs.
BRKCRS-3141
Cisco Public
92
3750E# show switch stack-ring speed

Stack Ring Speed
: 32G
Stack Ring Configuration: Full
Stack Ring Protocol
: StackWisePlus
Use the mode button on the

switch to determine its
switch number
Troubleshooting: Stack Commands

Details on the stack ports, members 1 and 3 active
3750# show switch
Switch/Stack Mac Address : 001b.545f.2800
Mac persistency wait time: 4 mins
H/W
Current
Switch# Role
Mac Address
Priority Version State
---------------------------------------------------------*1
Master 001b.545f.2800
12
1
Ready
2
Member 0000.0000.0000
0
1
Provisioned
3
Member 001d.46be.7500
8
1
Ready
3750# show switch stack-ports summary
Switch#/ Stack
Neighbor
Cable
Port#
Port
Length
Status
-------- ------ -------- -------1/1
OK
3
50 cm
1/2
Down
None
50 cm
3/1
Down
None
50 cm
3/2
OK
1
50 cm
BRKCRS-3141
Link
OK
Link
Active
Sync
OK
---Yes
No
No
Yes
-----Yes
No
No
Yes
---Yes
No
No
Yes
Cisco Public
93
#
Changes
To LinkOK
--------1
0
0
1
In
Loopback
-------No
No
No
No
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
95
Compact
Troubleshooting L2 Unicast Forwarding
Server
Symptom: Host cannot reach server

Steps
Layer 1 operational between host/Phone and switch?

Switch receiving traffic on that interface?
Congestion between host and switch?
MAC address learned?
MAC address of next hop correct?
Spanning tree state forwarding?
Other features preventing traffic flow?
Errored packets on the interface
Check HW programming
Consider possibilities
Create and execute action plan
BRKCRS-3141
Cisco Public
Distribution
and Core
C3750
Host
96
L2 Forwarding: Troubleshooting - 1
Server
Step 1: Verify if the link is up

Switch# show interface Gi1/0/3 status
Port
Name
Status
Gi1/0/3
connected
Vlan
10
Duplex
a-full
Speed Type
a-100 10/100/
1000BaseTX
Step 2: Verify if the port is in the right vlan and
is forwarding
Switch# show spanning-tree interface Gi1/0/3
Vlan
Role Sts Cost
Prio.Nbr Type
------------------- ---- --- --------- -------- ----------------------VLAN010
Desg FWD 19
128.2
P2p
Distribution
and Core
Step 3: Check if the packets are being received/sent

C3750
on the port
Switch# show interfaces gigabitEthernet 1/0/3 counters
Port
InOctets
InUcastPkts
InMcastPkts
Gi1/0/3
2108289
48
0
Port
Gi1/0/3
BRKCRS-3141
OutOctets
36817803
OutUcastPkts
48229
OutMcastPkts
252940
Cisco Public
97
InBcastPkts
6813
OutBcastPkts
72564
Host
L2 Forwarding: Troubleshooting 2
MAC Address Learning
Server
Step 4a: Verify if the Mac-address is correctly learned on the

port
Switch# show mac address-table interface gigabitEthernet 1/0/3
Mac Address Table
-------------------------------------------
Vlan
Mac Address
Type
Ports
------------------------10
00b1.a3d3.4321
DYNAMIC
Gi1/0/3
Total Mac Addresses for this criterion: 1
Distribution
and Core
Step 4b: Verify if the destination Mac-address is learned on the

switch on the expected port
C3750
Switch# show mac address-table dynamic address 00b1.a3d3.1234

Mac Address Table
------------------------------------------Vlan
Mac Address
Type
Ports
------------------------10
00b1.a3d3.1234
DYNAMIC
Gi1/0/4
BRKCRS-3141
Cisco Public
Host
98
Spanning Tree
Server
Step 5: Spanning tree state forwarding in software?

Switch#show spanning-tree vlan 10
Interfaces are FWDing
VLAN0010
Spanning tree enabled protocol ieee
Root ID
Priority
32778
Address
0003.fd6b.0700
This bridge is the root
Hello Time
2 sec Max Age 20 sec
Bridge ID
Priority
Address
Hello Time
Aging Time
Interface
------------------Gi1/0/3
Gi1/0/4
BRKCRS-3141
Role
---Desg
Desg
Distribution
and Core
Forward Delay 15 sec
32778 (priority 32768 sys-id-ext 10)

0003.fd6b.0700
2 sec Max Age 20 sec Forward Delay 15 sec
300 sec
Sts
--FWD
FWD
Cost
--------4
4
Prio.Nbr
-------128.3
128.4
Cisco Public
Type
-----------------------P2p
P2p Edge
99
C3750
Host
L2: Mac-Address Disappears From a Port

Check for spanning tree topology changes
Does the link remain up?
Link down causes MAC
Addresses to be flushed
Is it learned on another port?

Switch# show spanning-tree vlan 10 detail
.
.
.
Number of topology changes 5 last change occurred 18:45:22 ago

from GigabitEthernet1/0/3
...
BRKCRS-3141
Cisco Public
100
Interface
Server
Step 6a: Check Interfaces for Error-Disabled

Switch# show interface status err-disabled
Nothing in list.
No interfaces are Disabled
Switch#
Step 6b: Check Interface counters for errors
Distribution
and Core
Switch#show interface gi1/0/3 counters errors

Port
Gi1/0/3
Align-Err
0
FCS-Err
0
Xmit-Err
0
Rcv-Err
0
Port
Single-Col Multi-Col
Late-Col Excess-Col
Gi1/0/3
0
0
0
0
Switch#
Switch#show interface gi1/0/4 counters errors
Port
Gi1/0/4
Port
Gi1/0/4
BRKCRS-3141
Align-Err
0
Single-Col
0
FCS-Err
0
Multi-Col
0
Xmit-Err
0
Late-Col
0
Cisco Public
Carri-Sen
0
Rcv-Err
0
Excess-Col
0
101
UnderSize
0
UnderSize
0
Carri-Sen
0
OutDiscards
0
Runts
0
Giants
0
C3750
OutDiscards
0
Runts
0
Giants
0
Host
Layer 2 Forwarding: Troubleshooting 5

Advanced Techniques
Step 7: Use show platform forward to find Egress Interface programmed in H/W
Switch# show platform forward <ingress intf> <srcmac> <dstmac>
Switch# show platform forward gigabitEthernet 1/0/3 0000.0000.4321 0000.0000.1234
Ingress:
Global Port Number: 3, lpn: 1 ASIC Number: 6
Hashes: L2Src 0x00 L2Dst 0x0B L3Src 0x00 L3Dst 0x0B
Lookup
Key-Used
Index-Hit A-Data
Classify 68_00F00000_00001234-02_00000000_00004321 0102E 00000002
InputACL 20_00F00000_00001234-00_00000000_00004321 01FF8 01000000
L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000
L2FwdMsk FF_03FFFFFF_FFFFFFFF
L2Fwd 83_00020000_00001234
00EB6 000000B5
Station Descriptor: F004F002, DestIndex: F004, RewriteIndex: F002
==========================================
Egress: ASIC 6, switch 1
Destination Interface
Output Packets:
-----------------------------------------GigabitEthernet1/0/4 Packet 1
Lookup
Key-Used
Index-Hit A-Data
OutptACL 30_00F00000_00001234-00_00000000_00004321 01FFC 01000000
Port
Vlan SrcMac
DstMac Cos Dscpv
Gi1/0/4 0010 0000.0000.4321 0000.0000.1234
BRKCRS-3141
Cisco Public
102
Checklist: Interface Troubleshooting

Are packets being received?
Is the expected Mac-address learned on another port?
Check if dot1x is in use, if so, is the port authorized?
Does port security allow more Mac-addresses?
Is the port in spanning tree forwarding?
Other features preventing traffic flow?
ACLs
PVLAN
Show logging is there a history of instability
BRKCRS-3141
Cisco Public
103
Layer 3 IP Unicast Routing

Use the switch to debug end to end IP issues
- Verify source reachability from the switch
- Verify destination reachability from the switch
- Verify hardware forwarding from source to destination (and back)
VLAN:101
IP: 100.1.1.1
Mac: 000f.f7e8.e042
3750
3750
3750
Gi1/0/1
Destination
IP: 172.16.100.100
BRKCRS-3141
Vlan:100
IP: 10.1.1.1
Mac :000f.f7e8.e041
Cisco Public
105
Gi1/0/2
Source
IP: 100.1.1.2
Mac: 0018.ba88.1fc1
L3: Verify Source Reachability

Troubleshooting Steps
Source IP = 100.1.1.2
PING the source
PING the source with a loopback
Verify the ARP table

Verify the MAC table
VLAN:101
IP: 100.1.1.1
Mac: 000f.f7e8.e042
3750
3750
3750
Gi1/0/1
Destination
IP: 172.16.100.100
BRKCRS-3141
Vlan:100
IP: 10.1.1.1
Mac :000f.f7e8.e041
Cisco Public
106
Gi1/0/2
Source
IP: 100.1.1.2
Mac: 0018.ba88.1fc1
L3: Verify Source Reachability

3750# ping 100.1.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 100.1.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
Change source IP to loopback
3750# ping 100.1.1.2 source lo0
Packet sent with a source address of 99.1.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/maz = 1/4/9 ms
3750# sh ip arp vlan 101
Protocol Address
Age (min) Hardware Addr
Type
Interface
Internet 100.1.1.1
000f.f7e8.e042 ARPA
Vlan101
Internet 100.1.1.2
23
0018.ba88.1fc1 ARPA
Vlan101
3750# sh mac address-table address 0018.ba88.1fc1

Mac Address Table
------------------------------------------Vlan
Mac Address
Type
Ports
------------------------101
0018.ba88.1fc1
DYNAMIC
Gi1/0/2
BRKCRS-3141
Cisco Public
107
L3: Verify Source Reachability - 2

Verify packets from the source are getting to the CPU
Switch# show plat for <ingress intf> <srcmac> <dstmac> ip <srcip> <dstip> icmp <0-255> <0-255>
3750#show platform for Gi1/0/2 0018.ba88.1fc1 000f.f7e8.e042 ip 100.1.1.2 100.1.1.1 icmp 0 0
Ingress:
Global Port Number: 1, lpn: 3 Asic Number: 1
Hashes: L2Src 0x03 L2Dst 0x05 L3Src 0x09 L3Dst 0x03
Lookup
Key-Used
Index-Hit
A-Data
Classify 78_64010101_64010102-00_01000000_00000100
017FE
00000000
InputACL 40_64010101_64010102-00_01000000_00000100
01FFA
03000000
L3Local C0_00302401_64010101
01CF0
00000000
L3Scndr 10_64010101_64010102-00_00000000_00000100
008AA
000A0008_00000000
Lookup Used: Secondary
Station Descriptor: 00B00000, DestIndex: 00B0, RewriteIndex: 0000
==========================================
<output removed>
Output Packets:
Packet arriving on CPU
==========================================
(host) & 14 (dstats)
Egress: Asic 0, switch 2
CPU queues: 7 14.
BRKCRS-3141
Cisco Public
108
queue 7
L3: Verify Destination Reachability

Troubleshooting Steps
Destination IP = 172.16.100.100
Verify there is a route to the destination
Verify there is a valid ARP for the next hop
PING the destination (repeat w/ source VLAN as source address)

VLAN:101
IP: 100.1.1.1
Mac: 000f.f7e8.e042
3750
3750
3750
Gi1/0/1
Destination
IP: 172.16.100.100
BRKCRS-3141
Vlan:100
IP: 10.1.1.1
Mac :000f.f7e8.e041
Cisco Public
109
Gi1/0/2
Source
IP: 100.1.1.2
Mac: 0018.ba88.1fc1
L3: Verify Destination Reachability - 1

Resolved next hop
Switch# sh ip route 172.16.100.100
Show next hop to final destination
Routing entry for 172.16.100.0/24
Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 1
Last update from 10.1.1.2 on Vlan100, 00:08:54 ago
Routing Descriptor Blocks:
* 10.1.1.2, from 100.1.1.2, 00:08:54 ago, via Vlan100
Route metric is 20, traffic share count is 1
Switch# sh ip arp 10.1.1.2
Protocol Address
Age (min)
Internet
9
verify next hop is known

Hardware Addr
0018.ba88.1fc1
Type
ARPA
Interface
Vlan100
Switch# ping 172.16.100.100

!!!!!
Switch# ping 172.16.100.100 source vlan 101

Packet sent with a source address of 192.168.100.1
!!!!!
BRKCRS-3141
Cisco Public
110
L3:
Verify
Destination
Reachability
2
Unresolved next hop
Switch# sh ip route 172.16.100.0 255.255.255.0
Routing entry for 172.16.100.0/24
Known via "static", distance 1, metric 0
Routing Descriptor Blocks:
* 10.13.13.3
Route metric is 0, traffic share count is 1
Switch# sh run | include ip route
ip route 172.16.100.0 255.255.255.0 10.13.13.3
Switch# sh ip arp 10.13.13.3
Protocol Address
Age (min) Hardware Addr
Internet
0
Incomplete
Show next hop to final destination
verify next hop is known: FAIL
Type
Interface
ARPA
Switch# sh platform ip unicast failed arp

ARP Throttle Queue trying to resolve next
Total of 1 arp entries waiting on ARP-HRPC ThrottleQ
hop
========================
ARP throttled IP Address
========================
10.13.13.3/32 Table:0
------------------------Switch# sh platform ip unicast route 172.16.100.0 255.255.255.0
Fib 172.16.100.0/24 Tbl:0 Bucket:0
IOS Path 0 Spl Adj glean
HL3UFlags:0x80
SFT Entry:hdl:0xA5 HwFL:0x4
Adjacency not programmed in ASIC
BRKCRS-3141
Cisco Public
111
L3: Verify Hardware Forwarding

Show platform forward to verify HW programming
Output Packets:
==========================================
GigabitEthernet 1/0/2 Packet 1
Dropped due to failed deja vu check
Packet not forwarded to the interface it

was received from
Input ACL:
==========================================
Addr 0x7E7, tcam(6), mapRam (0x2)
framAddr (0x48400E14) Adata (0X200000)
Denied by Input ACL
Traffic denied by ACL
Ingress:
Dont use vlan option if it is not a trunk
Global Port Number: 5, lpn: 5 Asic Number: 1
Dropping the frame due to VLAN mode filtering,(allowDotOneQ) is set to FALSE
BRKCRS-3141
Cisco Public
112
Agenda
Product Overview
PHY Local Link
CPU
Memory
Port ASIC
Power Over Ethernet
Stacking
Advanced Examples
3750-X
3560X
2960-X/2960-XR
3560E
Tools and Tricks

2960-S/SF
BRKCRS-3141
3750v2
Cisco Public
114
Compact
Tools and Tricks

Enable NTP to troubleshoot across switches
Include date and time for debug and log messages
service timestamps log [datetime|uptime] localtime msec show-timezone
service timestamps debug [datetime|uptime] localtime msec show-timezone
Session to another switch member

C3750#session <member #>
C3750#remote command <1-9|all> IOS command
Automate successful troubleshooting using EEM

Review open caveats sections in release notes
Search Bug Toolkit for known issues
Reference Output Interpreter to decode command output
Reference System Message Guide for mitigation recommendations
Check the documentation and online guides
BRKCRS-3141
Cisco Public
115
References
Troubleshooting Catalyst 3750:
http://www.cisco.com/en/US/products/hw/switches/ps5023/prod_troubleshooting_guides_list.html
Online Resources on http://www.cisco.com:
Troubleshooting High CPU Utilization

Troubleshooting Power over Ethernet (PoE)
Troubleshooting Switch Stacks
Cisco Catalyst 3750 QoS Configuration Examples (Doc 91862)
Auto Negotiation issues: (Document 17053)
Want to learn more? Check out CCNP Practical Studies: Troubleshooting by Donna Harrington.
BRKCRS-3141
Cisco Public
116
Cisco Beyond - Product Extension Community

EEM Scripting Community
Open source scripts,
share, upload,
download, learn by
example
Categories include:
Network Management,
Diagnostics, Routing,
QoS, High availability,
User interface, Security
Comments, ratings,
community managed
forum
http://cisco.com/go/ciscobeyond
BRKCRS-3141
Cisco Public
117
Complete Your Online Session Evaluation

Give us your feedback and you
could win fabulous prizes. Winners
announced daily.
Complete your session evaluation
through the Cisco Live mobile app
or visit one of the interactive kiosks
located throughout the convention
center.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
BRKCRS-3141
Cisco Public
118
Continue Your Education

Demos in the Cisco Campus
Walk-in Self-Paced Labs
Table Topics
Meet the Engineer 1:1 meetings
BRKCRS-3141
Cisco Public
119

Designing MPLS in Next Generation Data Center: A Case Study

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Designing MPLS in Next Generation Data Center: A Case Study

Încărcat de

Drepturi de autor:

Formate disponibile

Troubleshooting Cisco Catalyst

3750, 3560, 2960-S and 2960-X

Administrators spend most of their time

2014 Cisco and/or its affiliates. All rights reserved.

Tools and Tricks

2014 Cisco and/or its affiliates. All rights reserved.

3750 Stack Rear View

2960-S Stack Rear View

2014 Cisco and/or its affiliates. All rights reserved.

2014 Cisco and/or its affiliates. All rights reserved.

Catalyst 3750-X : Architecture Overview

Packet Flow across switch components

2014 Cisco and/or its affiliates. All rights reserved.

Switch Components: Troubleshooting Areas of Focus

2014 Cisco and/or its affiliates. All rights reserved.

debug and show platform commands to follow in the slides

2014 Cisco and/or its affiliates. All rights reserved.

Tools and Tricks

2014 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Link Issues

When are we concerned about the link issues?

2014 Cisco and/or its affiliates. All rights reserved.

Troubleshooting Link Issues

2014 Cisco and/or its affiliates. All rights reserved.

Ethernet Controller Stats

Details about errors

Clear Stats with clear controller ethernet-controller command

Switch# show controller ethernet-controller GigabitEthernet 1/0/1

2014 Cisco and/or its affiliates. All rights reserved.

Symbol error frames

Switch#show controllers ethernet-controller g0/46 phy detail

2014 Cisco and/or its affiliates. All rights reserved.

0011 0001 0000 0000

0110 0100 0110 1011

Link Issues: What Kind of Errors?

OutOctets OutUcastPkts OutMcastPkts OutBcastPkts

Switch# show interfaces GigabitEthernet 1/0/1 counters errors

Rcv-Err UnderSize OutDiscards

2014 Cisco and/or its affiliates. All rights reserved.

Link Issues: Link Not Coming Up

Switch# show interfaces status | inc connected

2014 Cisco and/or its affiliates. All rights reserved.

SFP Link Issue Prevention EEM TCL Use Case

LinkUpApplyConfig is a Tcl policy

However, these settings are

Speed and duplex settings (in

If the same SFP type is re-inserted,

LinkUpApplyConfig.tcl can be downloaded at the following hyperlink:

2014 Cisco and/or its affiliates. All rights reserved.

Link Issues: Checking Physical Cabling

Switch# test cable-diagnostics tdr interface GigabitEthernet4/0/1

2014 Cisco and/or its affiliates. All rights reserved.

Tools and Tricks

2014 Cisco and/or its affiliates. All rights reserved.

Switch Hardware Components: CPU

Runs the IOS

2014 Cisco and/or its affiliates. All rights reserved.

CPU: Troubleshooting Processes

High CPU utilization is due to

Processes taking up resources

Forwarded Network Traffic

Normal CPU utilization varies by

Number of connected ports, modules, switches in stack

Feature set (LANBASE, IP BASE, IP SERVICES)