Advanced Troubleshooting Cisco WAAS

Advanced Troubleshooting Cisco Wide Area Application Services
BRKAPP-3006
BRKAPP-3006
2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Agenda
WAE Overview Diagnostic Reports Physical Components Platform Transport Optimizations Application Acceleration Packet Capture Debugs Summary
BRKAPP-3006
Cisco Public
> WAE Overview Diagnostic Reports Physical Components Platform Transport Optimizations Application Acceleration Packet Capture Debugs Summary
WAE Overview
Wide Area Application Engine (WAE)

Wide Area Application Services Version 4.5 and above
IOS Platform with Services and CLI

CIFS AO MAPI AO HTTP AO SSL AO Video AO NFS AO ICA AO EPM Virtual Virtual WoW Blade Blade Configuration Management #2 #3 System (CMS) Virtual Blades
TCP Proxy with Scheduler Optimizer (SO) DRE, LZ, TFO
Cisco Linux Kernel Policy Engine, Filter-Bypass, Egress Method, Directed Mode, Auto-Discovery
Flash IOS Shell Linux
BRKAPP-3006
Application Storage
Object Storage
DRE Storage
Virtual Blade Storage /vbspace

Cisco Public
Ethernet Network I/O

5
WAE Overview > Diagnostic Reports Physical Components Platform Transport Optimizations Application Acceleration Packet Capture Debugs Summary
Diagnostic Reports
Self Diagnostic Tool

A good place to start
BRKAPP-3006
Cisco Public
Self Diagnostic Tool
BRKAPP-3006
Cisco Public
WAAS System Report

WAAS system report (sysreport)
Compressed archive Relevant support and system health information
The sysreport includes the following:

Command output Logs Configurations State Information
The sysreport can be generated from the WAE Manager GUI or CLI:
WAE# copy sysreport <disk | ftp | tftp> ...
BRKAPP-3006
Cisco Public
WAE Overview Diagnostic Reports > Physical Components Platform Transport Optimizations Application Acceleration Packet Capture Debugs Summary
Physical Components


BRKAPP-3006
Application Storage
Object Storage
DRE Storage

Cisco Public

11
Disk Health and Status
Online Defunct Missing <null> Rebuilding
BRKAPP-3006
Cisco Public
12
Replacing Failed Disks

Disk failures are automatically detected by the system
Failed disks are automatically removed from service Not present or not responding (Shutdown)
Administrator can also shutdown disk for scheduled replacement:

WAE(config)# disk disk-name disk01 shutdown Device maybe busy while going offline ... please wait! mdadm: set /dev/sdb1 faulty in /dev/md0 mdadm: set /dev/sdb2 faulty in /dev/md1
RAID-1
< snip >

WAE# disk disk-name disk01 replace Controllers found: 1
RAID-5
Command completed successfully.

BRKAPP-3006 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Disk Partitions
WAE# show disks details
Physical disk information: disk00: Present 3LN2T3LG000098183V64 140011MB(136.7GB) disk01: Present 3LN2RN2P00009818KUM7 140011MB(136.7GB) Mounted file systems: MOUNT POINT TYPE /sw internal /swstore internal /state internal /local/local1 SYSFS /disk00-04 CONTENT .../local1/spool PRINTSPOOL DEVICE /dev/md0 /dev/md1 /dev/md2 /dev/md5 /dev/md4 /dev/md6 (h00 c00 i128 l00 - Int DAS-SAS) (h00 c01 i128 l00 - Int DAS-SAS)
SIZE 991MB 991MB 5951MB 6943MB 117798MB 991MB
INUSE 912MB 879MB 196MB 978MB 1786MB 16MB
FREE USE% 79MB 92% 112MB 88% 5755MB 3% 5965MB 14% 116012MB 1% 975MB 1%
BRKAPP-3006
Cisco Public
14
Important Directories and Log Files

The following directories are used by Cisco WAAS for log files
/local1 Root directory for all log files /local1/logs Service log files (aka admin logs) /local1/errorlog Service log files (aka debug logs) /local1/core_dir Process core dump files
File system navigation commands

cd dir find-pattern pwd type type-tail <filename> <lines>
BRKAPP-3006
Cisco Public
15
Ethernet Interface Errors
WAE# show int gi 1/0 Type:Ethernet Ethernet address:00:14:5E:AC:2D:79 Internet address:10.88.80.135 Broadcast address:10.88.80.255 Netmask:255.255.255.128 Maximum Transfer Unit Size:1500 Metric:1 Packets Received: 160661 Input Errors: 0 Input Packets Dropped: 0 Input Packets Overruns: 0 Input Packets Frames: 0 Packet Sent: 122371 Output Errors: 0 Output Packets Dropped: 0 Output Packets Overruns: 0 Output Packets Carrier: 0 Output Queue Length:1000 Collisions: 0 Interrupts:16 Flags:UP BROADCAST RUNNING MULTICAST Link State: Interface is up,line protocol up Mode: autoselect, full-duplex, 1000baseTX WAE#
BRKAPP-3006
Verify Interface Statistics for Errors
Cisco Public
16
Ethernet Interface Speed / Duplex

WAE# show int gi 1/0 Type:Ethernet Ethernet address:00:14:5E:AC:2D:79 Internet address:10.88.80.135 Broadcast address:10.88.80.255 Netmask:255.255.255.128 Maximum Transfer Unit Size:1500 Metric:1 Packets Received: 160661 Input Errors: 0 Input Packets Dropped: 0 Input Packets Overruns: 0 Input Packets Frames: 0 Packet Sent: 122371 Output Errors: 0 Output Packets Dropped: 0 Output Packets Overruns: 0 Output Packets Carrier: 0 Output Queue Length:1000 Collisions: 0 Interrupts:16 Flags:UP BROADCAST RUNNING MULTICAST Link State: Interface is up,line protocol up Mode: autoselect, full-duplex, 1000baseTX WAE#
BRKAPP-3006
Verify Interface State, Speed and Duplex
Cisco Public
17
CPU Utilization (Historical)
BRKAPP-3006
Cisco Public
18
CPU Utilization (Real Time)

WAE# show processes cpu
CPU utilization: CPU name User cpu: 0.06% cpu0: 0.05% cpu1: 0.08% cpu2: 0.05% cpu3: 0.08% Average: N/A Peak: N/A Nice 1.51% 1.51% 1.45% 1.46% 1.64% N/A N/A System 0.46% 0.42% 0.51% 0.40% 0.51% N/A N/A Idle 97.90% 97.95% 97.89% 98.03% 97.71% N/A N/A IOwait 0.05% 0.05% 0.05% 0.05% 0.05% N/A N/A IRQ 0.00% 0.00% 0.00% 0.00% 0.00% softIRQ 0.01% 0.03% 0.01% 0.01% 0.01% Steal 0.00% 0.00% 0.00% 0.00% 0.00% N/A N/A
Overall current CPU utilization (100 - (Idle + IOwait))% : 2.05% User Nice System Idle IOwait IRQ softIRQ Steal Average --More-Percent Percent Percent Percent Percent Percent Percent Percent Average of CPU time that the system spent in user mode. of CPU time that the system spent on low priority tasks. of CPU time that the system spent in system mode. of CPU time when the system is idle. of CPU time when the system is waiting for I/O to complete. of CPU time when the system is servicing interrupts. of CPU time when the system is servicing softirqs. of CPU time that the system spent on involuntary wait. CPU utilization since reboot.
BRKAPP-3006
Cisco Public
19
WAE Overview Diagnostic Reports Physical Components > Platform Transport Optimizations Application Acceleration Packet Capture Debugs Summary
Platform


BRKAPP-3006
Application Storage
Object Storage
DRE Storage

Cisco Public

21
WCCP Configuration Recommendations

WCCP on a software-based platform
GRE Forwarding (Default) Hash Assignment (Default) Inbound or Outbound Interception "ip wccp redirect exclude in" on WCCP client interface (outbound interception only) WAAS Egress Method: IP Forwarding, Negotiated Return
WCCP on a hardware-based platform

L2 Forwarding Mask Assignment Inbound Interception No "ip wccp redirect exclude in" WAAS Egress Method: IP Forwarding, Generic GRE (Cat6k PFC-based systems only)
BRKAPP-3006
Cisco Public
22
WCCPv2 Interception Verification

show ip wccp [detail] provides WCCP inventory
Number of routers and WAEs or search service group, packets re-directed in software, forwarding and return method
L2 Forwarding and GRE Forwarding Hash assignment

On software IOS routers only Verify hash allotment
Mask assignment
On L3 switches and some IOS versions Mask is configurable
BRKAPP-3006
Cisco Public
23
WCCPv2 Interception Verification (IOS)

Router# show ip wccp
Global WCCP information: Router information: Router Identifier: Protocol Version: Service Identifier: 61 Number of Service Group Clients: Number of Service Group Routers: Total Packets s/w Redirected: Process: CEF: Service mode: Service access-list: Total Packets Dropped Closed: Redirect access-list: Total Packets Denied Redirect: Total Packets Unassigned: Group access-list: Total Messages Denied to Group: Total Authentication failures: Total Bypassed Packets Received: --More-10.88.81.242 2.0 1 1 68755 2 68753 Open -none0 -none0 0 -none0 0 0
Client = WAE
BRKAPP-3006
Cisco Public
24

Global WCCP information: Router information: Router Identifier: Protocol Version: Service Identifier: 61 Number of Service Group Clients: Number of Service Group Routers: Total Packets s/w Redirected: Process: CEF: Service mode: Service access-list: Total Packets Dropped Closed: Redirect access-list: Total Packets Denied Redirect: Total Packets Unassigned: Group access-list: Total Messages Denied to Group: Total Authentication failures: Total Bypassed Packets Received: --More-BRKAPP-3006
10.88.81.242 2.0 1 1 68755 2 68753 Open -none0 -none0 0 -none0 0 0
Verify That Counters Are Incrementing on SoftwareBased Platforms (e.g. ISR)
Cisco Public
25

Verify That Counters Are Not Incrementing on HardwareBased Platforms (e.g. Cat6k)
Cisco Public
26

Counter Will Increment for Packets That Match Service Group but Do Not Match Redirect-List
Cisco Public
27

Increments for Every Packet Received with Incorrect Service Group Password
Cisco Public
28
WCCPv2 Interception Verification (IOS) - SW

Router# show ip wccp 61 detail
WCCP Client information: WCCP Client ID: Protocol Version: State: Redirection: Packet Return: Assignment: Initial Hash Info: Assigned Hash Info: Hash Allotment: Packets s/w Redirected: Connect Time: Bypassed Packets Process: CEF: Errors:
Verify WAE State in Service Group
10.88.81.242 2.0 Usable GRE GRE HASH FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 256 (100.00%) 68755 3w6d 2 68753
% of Hash Buckets Assigned
Current Time in the Service Group

WCCPv2 Interception Verification (IOS) - HW

Cat6k# sh ip wccp 61 det
WCCP Client information: WCCP Client ID: Protocol Version: State: Redirection: Packet Return: Packets Redirected: Connect Time: Assignment: 10.88.80.135 2.0 Usable L2 GRE 0 1d18h MASK
Mask SrcAddr DstAddr SrcPort DstPort ---- ------------------- ------0000: 0x00000f00 0x00000000 0x0000 0x0000 Value ----0000: 0001: 0002: 0003: SrcAddr ------0x00000000 0x00000001 0x00000040 0x00000041 DstAddr ------0x00000000 0x00000000 0x00000000 0x00000000 SrcPort ------0x0000 0x0000 0x0000 0x0000 DstPort ------0x0000 0x0000 0x0000 0x0000 CE-IP ----0x0A585087 0x0A585087 0x0A585087 0x0A585087
Service group mask (10.88.80.135)

(10.88.80.135) (10.88.80.135) (10.88.80.135)
BRKAPP-3006
Cisco Public
30
Interception Verification (IOS) - HW

Catalyst 6500 / 7600 platforms are capable of WCCP in both software and hardware forwarding paths Inspecting TCAM programming shows whether WCCP is handled in software or hardware
Cat6k# show tcam interface Vlan900 acl in ip * Global Defaults not shared Entries from Bank 0 Entries from Bank 1 permit punt tcp host 10.88.80.135 any ip any any (8 matches)
Punt entries caused by: Hash Assignment Outbound Redirection Redirect Exclude In Unknown WAE MAC
BRKAPP-3006
Cisco Public
31

Cat6k# sh tcam int vlan 900 acl in ip
Global Defaults not shared Entries from Bank 0 Entries from Bank 1 permit policy-route policy-route policy-route policy-route policy-route policy-route policy-route policy-route policy-route policy-route policy-route policy-route policy-route
BRKAPP-3006
policy-route entries = full hardware redirection

host 10.88.80.135 any any 0.0.0.0 255.255.232.190 (60 matches) any 0.0.0.1 255.255.232.190 (8 matches) any 0.0.0.64 255.255.232.190 (16 matches) any 0.0.0.65 255.255.232.190 (19 matches) any 0.0.1.0 255.255.232.190 any 0.0.1.1 255.255.232.190 any 0.0.1.64 255.255.232.190 any 0.0.1.65 255.255.232.190 any 0.0.2.0 255.255.232.190 any 0.0.2.1 255.255.232.190 any 0.0.2.64 255.255.232.190 any 0.0.2.65 255.255.232.190 (75 matches) any 0.0.3.0 255.255.232.190 (222195 matches)
2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
tcp tcp tcp tcp tcp tcp tcp tcp tcp tcp tcp tcp tcp tcp

Cat6k# configure terminal
Enter configuration commands, one per line. Cat6k(config)# service internal Cat6k(config)# end End with CNTL/Z.
Cat6k# show ip wccp 61 internal

Internal WCCP client information (1): Index: 0 WCCP Client ID: 10.88.80.135 Protocol Version: 2.0 State: 0007 (AUV ) Connect Time: 00:00:05 Redirection: L2 MAC: 0000.0000.0000 Packet Return: GRE L2 Address Changes: 0 Assignment: MASK Redirect Assignments: Received: 0 Invalid: 0 Duplicate: 0 < snip >
BRKAPP-3006
HIA from WAE must enter same interface that WAE MAC is known through
Cisco Public 33
WCCPv2 Interception Verification (WAE)

WAE# show wccp services
Services configured on this File Engine TCP Promiscuous 61 TCP Promiscuous 62
WAE# show wccp status WCCP version 2 is enabled and currently active WAE# show wccp routers
Verify WCCP Is Configured and Enabled
Router Information for Service: TCP Promiscuous 61 Routers Seeing this Wide Area Engine(1) Router Id Sent To Recv ID AssKeyIP 44.77.22.3 10.88.80.129 00090C46 10.88.80.133 Routers not Seeing this Wide Area Engine -NONERouters Notified of from other WAE's -NONEMulticast Addresses Configured -NONE-
AssKeyCN 1
MemberCN 5
Verify Bi-Directional Communication with WCCPEnabled Routers

Cisco Public 34
BRKAPP-3006
show wccp gre

WAE# show wccp gre
Transparent GRE packets received: 5531561 Transparent non-GRE packets received: 0 Transparent non-GRE non-WCCP packets received: 0 Total packets accepted: 5051 Invalid packets received: 0 Packets received with invalid service: 0 Packets received on a disabled service: 0 Packets received too small: 0 Packets dropped due to zero TTL: 0 Packets dropped due to bad buckets: 0 Packets dropped due to no redirect address: 0 Packets dropped due to loopback redirect: 0 Pass-through pkts dropped on assignment update:0 Connections bypassed due to load: 0 Packets sent back to router: 0 GRE packets sent to router (not bypass) 0 Packets sent to another WAE: 0 GRE fragments redirected: 0 GRE encapsulated fragments received: 0 Packets failed encapsulated reassembly: 0 Packets failed GRE encapsulation: 0 --More-BRKAPP-3006 2012 Cisco and/or its affiliates. All rights reserved.
Either of These Counters Should Be Incrementing If WCCP Redirection Is Working
Cisco Public
35
show wccp gre

WAE# show wccp gre
For Packets Redirected Using WCCP L2Redirect Forwarding Method
Cisco Public
36
show wccp gre

WAE# show wccp gre
For Packets L2 Redirected Using NonWCCP (L4, PBR, Etc.) Interception Method
Cisco Public
37
show wccp gre

WAE-612# show wccp gre
Packets Accepted for Optimization (I.E. AutoDiscovery Found Peer WAE)
Cisco Public
38
show wccp gre
WAE-612# show wccp gre

Only Includes Packets Handled Using WCCP Return Egress Method
Cisco Public
39
show wccp gre

WAE# show wccp gre
Packets Forwarded Directly Between WAEs Due to WCCP Flow Protection
Cisco Public
40
Egress Method Mismatch

Only WCCP GRE Return is supported as a negotiated-return egress method Generic GRE egress method only supports WCCP GRE as the intercept-method Alarm cleared when mismatch is resolved by altering the egress method and/or WCCP configuration Verify configured and used egress method with sh egress-method CLI command
BRKAPP-3006
Cisco Public
41
Egress Method
Intercept method : WCCP
WAE# show egress-methods
TCP Promiscuous 61 : WCCP negotiated return method : WCCP GRE Destination ----------any Egress Method Configured ---------------------WCCP Negotiated Return Egress Method Used ------------WCCP GRE
TCP Promiscuous 62 : WCCP negotiated return method : WCCP GRE Destination ----------any < snip > WAE674#
BRKAPP-3006 2012 Cisco and/or its affiliates. All rights reserved.
Egress Method Configured ---------------------WCCP Negotiated Return
Egress Method Used ------------WCCP GRE
IP Forwarding, WCCP GRE, or Generic GRE
Cisco Public
42
Egress Method Mismatch

WAE#
sh egress-methods
Intercept method : WCCP TCP Promiscuous 61 : WCCP negotiated return method : WCCP GRE Egress Method Egress Method Destination Configured Used ----------- ---------------------- ------------any Generic GRE IP Forwarding WARNING: WCCP has negotiated WCCP L2 as the intercept method for which generic GRE is not supported as an egress method in this release. This device uses IP forwarding as the egress method instead of the configured generic GRE egress method. TCP Promiscuous 62 : WCCP negotiated return method : WCCP GRE Egress Method Egress Method Destination Configured Used ----------- ---------------------- ------------any Generic GRE IP Forwarding WARNING: WCCP has negotiated WCCP L2 as the intercept method for which generic GRE is not supported as an egress method in this release. This device uses IP forwarding as the egress method instead of the configured generic GRE egress method.
BRKAPP-3006
Cisco Public
43
Inline Interception
WAE-612# show interface inlineGroup 1/0 Interface is in intercept operating mode. Standard NIC mode is off. Disable bypass mode is off. VLAN IDs configured for inline interception: All Watchdog timer is enabled. Timer frequency: 1600 ms. Autoreset frequency 500 ms. The watchdog timer will expire in 1452 ms. WAE-612#
Intercept Operating Mode or Bypass Operating Mode
Intercept operating mode Packet are passed to WAAS for (potential) optimization Bypass operating mode Mechanical bypass between ports in InlineGroup during failure or admin shutdown
BRKAPP-3006
Cisco Public
44
show interface inlinePort

WAE# show interface inlinePort 1/0/wan
Device name : eth4. Bypass master interface. Packets Received : 54231 Packets Intercepted: 0 Packets Bridged : 54231 Packets Forwarded : 0 Packets Dropped : 0 Packets Received on native : 0 Active flows for this interface : 0 ... WAE# show interface inlinePort 1/0/lan Device name : eth5. Bypass slave interface. Packets Received : 334602 Packets Intercepted: 0 Packets Bridged : 334599 Packets Forwarded : 0 Packets Dropped : 3 Packets Received on native : 0 Active flows for this interface : 0 ... WAE-612#
Use sh int inlinep to Determine Device Name for Any Inlineport The Device Name Is Needed for Packet Captures
Cisco Public 45
BRKAPP-3006
Auto Discovery - Refresher

Clients SYN forwarded by WAE1 with Auto Discovery Options SYN forwarded again by WAE2 with Auto Discovery Options SYN/ACK processed by WAE2, Auto Discovery Options added SYN/ACK processed by WAE1, to determine Auto Discovery success, SYN/ACK sent to Client and ACK with options sent to WAE2 ACK with options processed by WAE2 to determine Auto Discovery success, ACK sent to server.
WCCPv2 or PBR
WAN
WCCPv2 or PBR
WAE1
WAE2
Cisco Public 46
Automatic Discovery Verification

Client Client:Server TCP SYN WAE WAE Server
BRKAPP-3006
Cisco Public
47

Client Client:Server TCP SYN WAE Client:Server TCP SYN+OPT WAE Server
BRKAPP-3006
Cisco Public
48

Client WAE WAE Client:Server TCP SYN+OPT Server
BRKAPP-3006
Cisco Public
49

Client WAE Server:Client TCP SYN+ACK+OPT WAE Server:Client TCP SYN+ACK Server
BRKAPP-3006
Cisco Public
50
Auto-Discovery Statistics
WAE# show stat auto-discovery ...
Auto discovery failure: No peer or asymmetric route: Insufficient option space: Invalid connection state: Missing Ack conf: Intermediate device: ... Auto discovery success TO: Internal server: External server: Auto discovery success FOR: Internal client: External client: Auto discovery success SYN retransmission: Zero retransmit: One retransmit: Two+ retransmit: Auto discovery Miscellaneous: RST received: SYNs found with our device id: SYN retransmit count resets:
BRKAPP-3006
100 0 0 0 0 40 0 0 2902 0 0 0 0 0
Cisco Public 51
Could not find a peer WAE 2902
WAE# show stat auto-discovery ... Auto discovery failure: No peer or asymmetric route: Insufficient option space: Invalid connection state: Missing Ack conf: Intermediate device: ... Auto discovery success TO: Internal server: External server: Auto discovery success FOR: Internal client: External client: Auto discovery success SYN retransmission: Zero retransmit: One retransmit: Two+ retransmit: Auto discovery Miscellaneous: RST received: SYNs found with our device id: SYN retransmit count resets:
0 0 0 0 0 40 0
Not enough space in TCP 2902 Options header for AD 0 0 option

0 0 0
Cisco Public 52
2902 0
0 0 0 0 1 40 0
The WAE is not closest to 2902 server or client the

0 0 0 0 0
Cisco Public 53
2902 0
0 0 0 0 0 40 Packet received with our own 0 TCP option - indicates a 2902 redirection loop 0 2902 0 0 0 0 0
Cisco Public 54
Redirection Problems
Proper network interception configuration is critical to the success of a WAAS deployment Failure to insure that egress packets from the WAE are not re-intercepted can lead to a redirection loop: You can search syslog.txt for instances of the above message using the following command:
WAE# find match Routing Loop syslog.txt WAE# show stat filtering .. Syn packets dropped with our own id in the options: ..
%WAAS-SYS-3-900000: 137.34.79.11:1192 - 137.34.77.196:139 - opt_syn_rcv: Routing Loop detected - Packet has our own devid. Packet dropped.
As well as in the TFO filtering statistics:
Cisco Public
55
During normal operation, there are 3 different connection segments for an optimized flow
WAN
In the event of an interception failure, packets from an original connection could be received on the optimized segment by a peer WAE
%WAAS-SYS-3-900000:src=10.56.46.183:4386 dst=10.56.46.164:445 Unoptimised packet received on optimized socket: seq=2796284443 end_seq=279628444

WAN
vPath Interception
vWAAS# show statistics vn-service vpath VPATH Statistics ***************** Packet Statistics ----------------VPATH Enabled VPATH Packet received Optimized TCP Packets VPATH returned WAAS Bypassed VPATH packets returned VPATH encapsulated IP pkts(excluding TCP) returned VPATH encapsulated Non-IP packets returned VPATH Fragments received VPATH Fragments returned VPATH Packets returned when VPATH not configured Non-VPATH Packets received Error Statistics ----------------VPATH intercepted packets dropped VPATH Packet CRC failures VPATH packets with unsupported Version VPATH packets with wrong request type
BRKAPP-3006
= = = = = = = = = = = = = =
YES 4783472 918762 15537 0 26 0 0 0 810022 0 0 0 0

Cisco Public 57
WCCP Group 1
WCCP Group 2
BRKAPP-3006
Cisco Public
58
WCCP Group 1
BRKAPP-3006
Cisco Public
59
Transport Optimizations WAE Overview

Diagnostic Reports Physical Components Platform > Transport Optimizations Application Acceleration Packet Capture Debugs Summary


BRKAPP-3006
Application Storage
Object Storage
DRE Storage

Cisco Public

61
WAE# show stat tfo Total number of connections No. of active connections No. of pending (to be accepted) connections No. of bypass connections No. of normal closed conns No. of reset connections Socket write failure Socket read failure WAN socket close while waiting to write AO socket close while waiting to write WAN socket error close while waiting to read AO socket error close while waiting to read DRE decode failure DRE encode failure Connection init failure WAN socket unexpected close while waiting to read Exceeded maximum number of supported connections Buffer allocation or manipulation failed Peer received reset from end host DRE connection state out of sync Memory allocation failed for buffer heads Unoptimized packet received on optimized side <snip>
Verify Global TFO Status
Total number of TCP optimized connections Total number of TCP connections that were optimized since the last TFO statistics reset.
: : : : : : : : : : : : : : : : : : : : : :
108 1 0 1 96 11 4 0 1 0 0 1 0 0 0 5 0 0 0 0 0 0
Cisco Public
62
Pass Through Statistics

WAE#
show statistics pass-through
Outbound ---------------------PT Client: Bytes Packets PT Server: Bytes Packets PT In Progress: Bytes Packets Overall No Peer Rjct Capabilities Rjct Resources App Config Global Config Asymmetric In Progress Intermediate Overload Internal Error App Override Server Black List AD Version Mismatch AD AO Incompatible AD AOIM Progress DM Version Mismatch
BRKAPP-3006
0 0 530626936 829570 93109057 253459 Active ---------------------0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Active and historical passthrough connection counts

Completed ---------------------98343 16598 0 0 10 0 60111 21624 0 0 0 0 0 0 0 0 0
Cisco Public 63
CM Connection Statistics
BRKAPP-3006
Cisco Public
64
BRKAPP-3006
Cisco Public
65
Client
WAE
OptimizedReadBytes OptimizedWriteBytes CompressionRatio OriginalReadBytes OriginalWriteBytes
BRKAPP-3006
Cisco Public
66
Connection Statistics
WAE# show stat connection Current Active Optimized Flows: Current Active Optimized TCP Plus Flows: Current Active Optimized TCP Only Flows: Current Active Optimized TCP Preposition Flows: Current Active Auto-Discovery Flows: Current Active Pass-Through Flows: Historical Flows: 3 3 0 0 2 0 100
D:DRE,L:LZ,T:TCP Optimization, A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO ConnID 26527 26528 26529 ... Source IP:Port 212.1.4.10:1884 212.1.4.10:1886 212.1.4.10:1887 Dest IP:Port 212.1.5.24:135 212.1.5.22:80 212.1.5.24:1026 PeerID 00:16:9d:38:8e:a3 00:16:9d:38:8e:a3 00:16:9d:38:8e:a3 Accel TE THDL TDL
BRKAPP-3006
Cisco Public
67
WAE# show stat connection conn-id 26528 Connection Id: 26528 Peer Id: 00:16:9d:38:8e:a3 Connection Type: EXTERNAL CLIENT Start Time: Mon Dec 14 04:17:49 2009 Source IP Address: 212.1.4.10 Source Port Number: 1886 Destination IP Address: 212.1.5.22 Destination Port Number: 80 Application Name: Web Classifier Name: HTTP Map Name: basic Directed Mode: FALSE Preposition Flow: FALSE Policy Details: Configured: TCP_OPTIMIZE + DRE + LZ Derived: TCP_OPTIMIZE + DRE + LZ Peer: TCP_OPTIMIZE + DRE + LZ Negotiated: TCP_OPTIMIZE + DRE + LZ Applied: TCP_OPTIMIZE + DRE + LZ Accelerator Details: Configured: HTTP Derived: HTTP Applied: HTTP Hist: None Original Optimized -------------------- -------------------Bytes Read: 301 5235140 Bytes Written: 5570619 5657
Detailed Connection Statistics
Matched application and classifier
Configured and negotiated policies
AO Applied
Bytes read and written on optimized and nonoptimized sockets
BRKAPP-3006
Cisco Public
68

<continued> ----------------- Flow 26528 dre stats ----------------Conn-ID: 26528 212.1.4.10:1886 -- 212.1.5.22:80 Peer No: 0 Status: Active -----------------------------------------------------------------------------Open at 12/14/2009 04:17:49, Still active Encode: Overall: msg: 1, in: 305 B, out: 336 B, ratio: 0.00% DRE: msg: 1, in: 301 B, out: 362 B, ratio: 0.00% DRE Bypass: msg: 1, in: 4 B LZ: msg: 1, in: 372 B, out: 336 B, ratio: 9.68% LZ Bypass: msg: 0, in: 0 B Avg latency: 0.168 ms Delayed msg: 0 Encode th-put: 1772 KB/s Message size distribution: 0-1K=0% 1K-5K=0% 5K-15K=0% 15K-25K=0% 25K-40K=0% >40K=0% Decode: Overall: msg: 1138, in: 5101 KB, out: 5440 KB, ratio: 6.23% DRE: msg: 1138, in: 5413 KB, out: 5440 KB, ratio: 0.50% DRE Bypass: msg: 1, in: 4 B LZ: msg: 696, in: 2460 KB, out: 2772 KB, ratio: 11.25% LZ Bypass: msg: 442, in: 2640 KB Avg latency: 0.124 ms Decode th-put: 38662 KB/s Message size distribution: 0-1K=3% 1K-5K=65% 5K-15K=29% 15K-25K=1% 25K-40K=0% >40K=0%
BRKAPP-3006
Cisco Public
69
Encode and Decode LZ and DRE
Decode
Encode
Encode
Decode
BRKAPP-3006
Cisco Public
70

WAE# show statistics dre Cache: Status: Usable, Oldest Data (age): 109d Total usable disk size: 55456 MB, Used: 2.51% Hash table RAM size: 211 MB, Used: 2.00% Connections: Total (cumulative): 8 Active: 0 24861 KB, ratio: 25496 KB, ratio: 3237 KB, ratio: 1924 >40K=85% 83.03% 82.59% 16.40%
Encode: Overall: msg: 1906, in: DRE: msg: 1906, in: DRE Bypass: msg: 9, in: LZ: msg: 83, in: LZ Bypass: msg: 1823, in: Avg latency: 2.701 ms Encode th-put: 28449 KB/s Message size distribution: 0-1K=0% 1K-5K=0% 5K-15K=1% Decode: ...
143 MB, out: 143 MB, out: 36 B 3872 KB, out: 21624 KB Delayed msg: 15K-25K=3%
25K-40K=7%
BRKAPP-3006
Cisco Public
71
Transaction Logs
Every transaction generates log Multiple transaction attributes recorded
TCP connection start time TCP connection end time Optimization done (AO, DRE, LZ, TFO, or PT) Flow identification information (L3/L4/L5) Bytes
Origin received/sent Optimized received/sent
BRKAPP-3006
Cisco Public
72
Transaction Logs CM
Enable transaction logging on WAEs Written to /local1/logs/tfo/ Archive log schedule The archive filenames use this format: tfo_log_IPADDRESS_YYYY MMDD_HHMMSS.txt Export log schedule
BRKAPP-3006
Cisco Public
73
TFO Transaction Logs

WAE# cd logs/tfo WAE# WAE# ls ftp_export.status tfo_log_22.1.43.10_20090508_190000.txt tfo_log_22.1.43.10_20090508_200000.txt tfo_log_22.1.43.10_20090508_210000.txt working.log WAE# WAE# type-tail working.log
Location of logs The archive files and the current log files
Log entries (fields delimited by :)
Fri May 8 21:08:19 2009 :22.1.43.10 :42029 :22.1.43.20 :443 :BP :NO_PEER : (TFO) (TFO) (None) :<None> :(None) (None) :<None> :<None> Fri May 8 21:08:34 2009 :22.1.41.10 :13113 :22.1.43.20 :443 :BP :ASYMMETRIC : (TFO) (None) (TFO) :<None> :(None) (None) :<None> :<None> .. <date and time>:<src IP>:<src port>:<dst IP>:<port>:BP:<pass-through reason>::
Sawmill Transaction Log Analysis

Sawmill understands WAAS transaction logs Syslog or FTP/SFTP transfer Extensive reports
BRKAPP-3006
Cisco Public
75
Application Acceleration
WAE Overview Diagnostic Reports Physical Components Platform Transport Optimizations > Application Acceleration Packet Capture Debugs Summary


BRKAPP-3006
Application Storage
Object Storage
DRE Storage

Cisco Public

77
AO Troubleshooting Approach
Verify AO configuration and operational state Verify Application Traffic Policy configuration for AO Check global and AO-specific statistics Verify connections are handled / optimized by AO Check connection specific AO statistics
BRKAPP-3006
Cisco Public
78
Accelerator Status
Displays the configuration and operational state of each accelerator
WAE# show accelerator Accelerator ----------cifs epm http mapi nfs ssl video ica Licensed -------Yes Yes Yes Yes Yes Yes Yes Yes Config State -----------Enabled Enabled Enabled Enabled Enabled Enabled Enabled Enabled Operational State ----------------Running Running Running Running Running Running Running Running
WAE# show license License Name Status Activation Date Activated By -------------- ----------- --------------- -------------Transport active 07/14/2011 admin Enterprise not active Video not active Virtual-Blade not active
Accelerator Status
WAE#
show accelerator cifs

Licensed -------Yes Config State -----------Enabled Value ----Registered Use Policy 2000 1990 5.0 seconds Operational State ----------------Running
Accelerator ----------cifs
CIFS: Policy Engine Config Item ------------------------State Default Action Connection Limit Effective Limit Keepalive timeout WAE#
show accelerator nfs

Licensed -------Yes Config State -----------Enabled Value ----Registered Use Policy 2000 1990 5.0 seconds
Accelerator ----------nfs
Operational State ----------------Running
NFS: Policy Engine Config Item ------------------------State Default Action Connection Limit Effective Limit Keepalive timeout
BRKAPP-3006
Global AO Statistics
Common statistics for all accelerators
WAE#
show statistics accelerator http
HTTP: Global Statistics ----------------Time Accelerator was started: Mon Jun 22 02:25:53 2009 Time Statistics were Last Reset/Cleared: Mon Jun 22 02:25:53 2009 Total Handled Connections: 52 Total Optimized Connections: 52 Total Connections Handed-off with Compression Policies Unchanged: 0 Total Dropped Connections: 0 Current Active Connections: 0 Current Pending Connections: 0 Maximum Active Connections: 30 Total Time Saved (ms): 312 Current Active Connections Free For Fast Connection Use: 0 Total Connections Handed-off: 0 Total Connections Handed-off with Compression Policies Disabled: 0 Total Connections Handed-off to SSL: 0 Total Connection Hand-off Failures: 0 Total Fast Connection Successes: 5 <snip>
CIFS Acceleration
Verify CIFS is properly configured
WAE# WAE#
sh run | i cifs
name WAFS classifier CIFS action optimize full accelerate cifs
sh run | b CIFS
...skipping classifier CIFS match dst port eq 445 match dst port eq 139 exit
WAE# show statistics connection optimized cifs <snip> D:DRE,L:LZ,T:TCP Optimization, A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO ConnID 222 Source IP:Port 212.1.4.10:3117
BRKAPP-3006
Verify established connections
Dest IP:Port 212.1.6.11:139

PeerID 00:0f:fe:ed:9c:1f
Cisco Public
Accel TCDL
82
EPM / MAPI Acceleration

WAE# sh policy-engine application dynamic Dynamic Match Freelist Information: Allocated: 32768 In Use: 1 Max In Use: 10 Dynamic Match Type/Count Information: None 0 Clean-Up 0 Host->Host 0 Host->Local 0 Local->Host 0 Local->Any 0 Any->Host 1 Any->Local 0 Any->Any 0 Individual Dynamic Match Information: Number: 1 Src: ANY:ANY Type: Any->Host (6) User Id: EPM (3) Dst: 10.56.44.245:1248 Allocations: 4957
1 entry per server
Map Name: uuida4f1db00-ca47-1067-b31f-00dd010662da Flags: TIME_LMT REPLACE FLOW_CNT Seconds: 1200 Remaining: - NA - DM Index: 32766 Hits: 1 Flows: 2 Cookie: 0x00000000
MAPI UUID Hit and Conn count

Cisco Public 83
BRKAPP-3006
EPM / MAPI Acceleration

WAE# sh stat acc mapi < snip > Number of Synch Get Buffer: Minimum Synch Get Buffer Size (bytes): Maximum Synch Get Buffer Size (bytes): Average Synch Get Buffer Size (bytes): Number of Read Stream Requests: Minimum Read Stream Buffer Size (bytes): Maximum Read Stream Buffer Size (bytes): Average Read Stream Buffer Size (bytes): Minimum Accumulated Read Ahead Data Size (bytes): Maximum Accumulated Read Ahead Data Size (bytes): Average Accumulated Read Ahead Data Size (bytes): Local Response Count: Average Local Response Time (usec): Remote Response Count: Average Remote Response Time (usec): Current 2000 Accelerated Sessions: Current 2003 Accelerated Sessions: Current 2007 Accelerated Sessions: Secured Connections: Lower than 2000 Connections: Higher than 2007 Connections: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 58 473 0 0 1 0
Current connections by version
Not handled by MAPI AO

Cisco Public 84
BRKAPP-3006
HTTP Acceleration
Check connections statistics
WAE# sh stat conn opt http det < snip > HTTP : 142 Time Statistics were Last Reset/Cleared: Total Bytes Read: Total Bytes Written: Total Bytes Buffered: Total Internal Bytes Read: Total Internal Bytes Written: Bit Flags for I/O state: Internal object pointer: Fast connections: Mon Jun 22 23:32:11 2009 1354 1064 1354 1064 0 0 16 16 80 134931216 0
Positive value = latency mitigation

BRKAPP-3006
Cisco Public
85
HTTP Acceleration
Check accelerator statistics
WAE# sh stat acc http HTTP: Global Statistics ----------------< snip > A count of time reduction done by the AO measured in milliseconds: Number of active connections free for fast connection use: Total number of connection handoffs: Number of connection handoffs to SSL Accelerator: Number of fast connections: Maximum number of fast connections on a single connection: Percentage of connection time saved: Total round trip time for all connections in milliseconds: WAE674# 2731154 0 0 0 68356 100 98 30120
Aggregate connection reuse statistics

BRKAPP-3006
ICA Acceleration
WAE# show statistics accelerator ica ICA: Global Statistics ----------------< snip > Total Handled Connections: Total Optimized Connections: Total Connections Handed-off with Compression Policies Unchanged: Total Dropped Connections: Current Active Connections: Current Pending Connections: Maximum Active Connections: Current Active CGP Connections: Current Active ICA Connections: Total CGP Connections: Total ICA Connections: Total CGP Reconnections: Total Sessions Client Version 13_0: Total Sessions Client Version 12_1: Total Sessions Client Version 12_0:
unsupported versions or parse or decryption error, features connection continues, unsupported features detected but not accelerated 0 after point of no return
0 5 11 0 0 0 0 0 0 0 0 0 0 0
Cisco Public
87
ICA Acceleration
WAE# show statistics accelerator ica ICA: Global Statistics ----------------< snip > Conns Hand-off Because of Unrecognized Protocol: 4 Conns Hand-off Because of Unsupported Client Vers: 2 Conns Hand-off Because of Unknown CGP Session ID: 7 Conns Hand-off Because of Client on Denied List: 0 Connections Handed-off Because of Resource Limit: 0 Connections Handed-off Because of Other Reasons: 0 Breakdown of disconnects Conns Disconnect Because of Unsupport Client Vers: 0 Connections Disconnected Because of I/O Error: 0 Connections Disconnected Because of Parsing Error: 0 Connections Disconnected Because of Resource Limit:0 Connections Disconnected Because of Session In Use:0 Connections Disconnected Because of Other Reasons: 0
Breakdown of connection hand-off
BRKAPP-3006
Cisco Public
88
ICA Acceleration
WAE# show statistics accelerator ica ICA: Global Statistics ----------------< snip > Conns Hand-off Because of Unrecognized Protocol: 4 Conns Hand-off Because of Unsupported Client Vers: 2 Conns Hand-off Because of Unknown CGP Session ID: 7 Conns Hand-off Because of Client on Denied List: 0 Connections Handed-off Because of Resource Limit: 0 Connections Handed-off Because of Other Reasons: 0 Conns Disconnect Because of Unsupport Client Vers: 0 Connections Disconnected Because of I/O Error: 0 Connections Disconnected Because of Parsing Error: 0 Connections Disconnected Because of Resource Limit:0 Connections Disconnected Because of Session In Use:0 Connections Disconnected Because of Other Reasons: 0
Breakdown of disconnects
BRKAPP-3006
Cisco Public
89
SSL Acceleration
WAE#
show crypto certificates
Certificate Only Store: ----------------------<EMPTY> Managed Store: -------------File: any-ssl.p12 Format: PKCS12 EEC: Subject: emailAddress=jifrench@cisco.com/C=US/ST=California/L=San Jose/OU=W AAS/O=Cisco Systems/CN=*.domain.com Issuer: emailAddress=jifrench@cisco.com/C=US/ST=California/L=San Jose/OU=WA AS/O=Cisco Systems/CN=*.domain.com -------------------------------------------------------------------------------Local Store: -----------Machine Self signed Certificate ------------------------------Format: PKCS12 Subject: C=US/ST=California/L=San Jose/OU=CNBU/O=Cisco Systems, Inc/CN=dc1-wae1. allcisco.com/emailAddress=tac@cisco.com Issuer: C=US/ST=California/L=San Jose/OU=CNBU/O=Cisco Systems, Inc/CN=dc1-wae1.a llcisco.com/emailAddress=tac@cisco.com Management Service Certificate -----------------------------Format: PKCS12 EEC:Subject: C=US/ST=California/L=San Jose/OU=CNBU/O=Cisco Systems, Inc/CN=dc1-w ae1.allcisco.com/emailAddress=tac@cisco.com Issuer: C=US/ST=California/L=San Jose/OU=CNBU/O=Cisco Systems, Inc/CN=dc1-wa e1.allcisco.com/emailAddress=tac@cisco.com The WAAS Self Signed Certificate is being used as the Management Service Certificate
Accelerated service
BRKAPP-3006
Cisco Public
90
SSL Acceleration
WAE#
Peering service
BRKAPP-3006
Cisco Public
91
SSL Acceleration
WAE#
Management service
BRKAPP-3006
Cisco Public
92
SSL Accelerator
Unlike other AOs, SSL does not use an accelerate policy map entry Dynamic policy entries are created when SSL acceleration services are brought inservice
WAE#
sh run | b crypto
...skipping crypto ssl services global-settings version all exit ! crypto ssl services accelerated-service wx1.getthere.net description Cisco Travel Network version all server-cert-key wx1.p12
Only configured on serverside (i.e. Core) WAEs
server-ip 151.193.164.6 port 443 inservice

exit crypto ssl services accelerated-service wwwin-tools.cisco.com version all server-cert-key wwwin-tools.p12
server-ip 171.70.150.5 port 443 inservice

exit ! < snip >
SSL Accelerator
WAE# sh policy-engine application dynamic Dynamic Match Freelist Information: Allocated: 32768 In Use: 3 Max In Use: 5 Allocations: 1751 < snip > Individual Dynamic Match Information: Number: 1 Type: Any->Host (6) User Id: SSL (4) Src: ANY:ANY Dst: 171.70.150.5:443 Map Name: basic Flags: SSL Seconds: 0 Remaining: - NA - DM Index: 32764 Hits: 25 Flows: - NA - Cookie: 0x00000001 Number: 2 Type: Any->Host (6) User Id: EPM (3) Src: ANY:ANY Dst: 10.88.80.53:1026 Map Name: uuide3514235-4b06-11d1-ab04-00c04fc2dcd2 Flags: TIME_LMT REPLACE FLOW_CNT Seconds: 1200 Remaining: - NA - DM Index: 32765 Hits: 0 Flows: 1 Cookie: 0x00000000 Number: 3 Type: Any->Host (6) User Id: SSL (4) Src: ANY:ANY Dst: 151.193.164.6:443 Map Name: basic Flags: SSL Seconds: 0 Remaining: - NA - DM Index: 32766 Hits: 1 Flows: - NA - Cookie: 0x00000000
Dynamically created policy for SSL traffic
BRKAPP-3006
Cisco Public
94
SSL Acceleration
WAE#
sh stat acc ssl
< snip > Number of SSLv3 negotiated on LAN: Number of TLSv1 negotiated on LAN: Number of SSLv3 negotiated on WAN: Number of TLSv1 negotiated on WAN: Number of SSLv3 negotiated on peer: Number of TLSv1 negotiated on peer: Number of server initiated SSL renegotiations: Number of client initiated SSL renegotiations: Successful certificate verifications: Failed certificate verifications: Failed certificate verifications due to invalid certificates: Failed certificate verifications due to ocsp verification: Failed certificate verifications due to other errors: OCSP connections outstanding: OCSP requests processed since last clear/system start: Maximum number of concurrent OCSP requests ever reached: 39 1237 39 1237 0 1276 0 0 0 0 0 0 0 0 0 0
BRKAPP-3006
Cisco Public
95
SSL Acceleration
WAE#
sh stat conn
<snip> D:DRE,L:LZ,T:TCP Optimization, A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO ConnID 198 199 201 Source IP:Port 22.1.32.100:1939 22.1.32.100:1940 22.1.32.100:2046 Dest IP:Port 22.1.34.100:80 22.1.34.100:80 22.1.34.100:443 PeerID Accel 00:1a:64:c2:2b:9c THDL 00:1a:64:c2:2b:9c THDL 00:1a:64:c2:2b:9c TSDL
WAE-CORE# sh stat conn <snip> D:DRE,L:LZ,T:TCP Optimization, A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO ConnID 198 199 201 Source IP:Port 22.1.32.100:1939 22.1.32.100:1940 22.1.32.100:2046 Dest IP:Port 22.1.34.100:80 22.1.34.100:80 22.1.34.100:443 PeerID 00:1a:64:c3:08:2c 00:1a:64:c3:08:2c 00:1a:64:c3:08:2c Accel THDL THDL TSDL
BRKAPP-3006
Cisco Public
96
Video Acceleration
WAE# sho stat acc video Time elapsed since "clear statistics": 1days 0hr 50min 30sec Video Connections ================================================================== Connections handled num % -----------------------------------------------------------------Total handled 3330 100.00 Windows-media live accelerated 3329 99.97 Un-accelerated pipethru 1 0.03 Un-accelerated dropped due to config 0 0.00 Error dropped connections 0 0.00 Windows-media active sessions current max -----------------------------------------------------------------Outgoing (client) sessions 10 10 Incoming (server) sessions 1 10 Windows-media byte savings ================================================================== % Bytes saved Incoming(server) bytes Outgoing(client) bytes 56.01 2.07 GB 4.71 GB
Summary connection statistics
BRKAPP-3006
Cisco Public
97
Video Acceleration
WAE# sho stat acc video Time elapsed since "clear statistics": 1days 0hr 50min 30sec Video Connections ================================================================== Connections handled num % -----------------------------------------------------------------Total handled 3330 100.00 Windows-media live accelerated 3329 99.97 Un-accelerated pipethru 1 0.03 Un-accelerated dropped due to config 0 0.00 Error dropped connections 0 0.00 Windows-media active sessions current max -----------------------------------------------------------------Outgoing (client) sessions 10 10 Incoming (server) sessions 1 10 Windows-media byte savings ================================================================== % Bytes saved Incoming(server) bytes Outgoing(client) bytes 56.01 2.07 GB 4.71 GB
Current and maximum stream splitting activity
BRKAPP-3006
Cisco Public
98
Detailed accelerator statistics provider unaccelerated and error details

WAE#
Video Acceleration
sho stat acc video detail
< snip >
Unaccelerated Connections num -----------------------------------------------------------------Total Unaccelerated 1 100.00 Unsupported player 0 0.00 Unsupported transport 0 0.00 Unsupported protocol 0 0.00 Windows-media VoD 1 100.00 Max stream bitrate overload 0 0.00 Max aggregate bitrate overload 0 0.00 Max concurrent sessions overload 0 0.00 Other 0 0.00 Error dropped connections num % -----------------------------------------------------------------Total errors 0 0.00 Client timeouts 0 0.00 Server timeouts 0 0.00 Client stream errors 0 0.00 Server stream errors 0 0.00 Other errors 0 0.00
RTSP header in response missing x-wmsstream-type=broadcast cache control header %
Cisco Public
99
Packet Capture Debugs WAE Overview

Diagnostic Reports Physical Components Platform Transport Optimizations Application Acceleration > Packet Capture Debugs Summary
Packet Capture Debugs

Packets can be captured on all WAAS interfaces using one of the following CLI tools:
tethereal tcpdump
The Problem?
A packet capture taken on the WAE will contain packets of all TCP segments
How to differentiate between original and optimized connections?
BRKAPP-3006
Cisco Public
101
Multiple TCP Segments

header IP eth src Client Client dst Server Router header IP eth src Client Router dst Server
Client
61 in
62 in
header IP eth header IP eth header IP eth
src Server Router src Client Router src Server WAE
dst Client Client dst Server WAE dst Client Router
exclude in
header IP eth header IP
src Server
dst Client Router
src Server Router src Client WAE
dst Client WAE dst Server Router
WAE
eth header IP eth
BRKAPP-3006
Cisco Public
102
Displaying Optimized Segments

Client
61 in
62 in
exclude in
header IP eth header IP
src Server
dst Client Router
src Server Router
dst Client WAE
WAE
eth
Wireshark Display Filter: tcp && ip.src == <ServerIP> && eth.dst == <WAE MAC addr>
Displaying Original Segments

Client
61 in
62 in
header IP eth header IP eth header IP eth
src Server Router src Client Router src Server WAE
dst Client Client dst Server WAE dst Client Router
exclude in
WAE
Wireshark Display Filter: tcp && ip.src == <ServerIP> && eth.src == <WAE MAC addr>
BRKAPP-3006
WAE Overview Diagnostic Reports Physical Components Platform Transport Optimizations Application Acceleration Packet Capture Debugs > Summary
Summary
Summary
WAAS self diagnostic tool Validate configuration on interception device and WAE WCCPv2 statistics on the WAE and router Automatic discovery counters to verify traffic flow Connection statistics provides granular details TFO transaction logs provide a history AO specific statistics Packet traces
BRKAPP-3006
Cisco Public
106
Recommended Reading
BRKAAP- 3006
Check out Cisco WAAS Troubleshooting Guide at: http://docwiki.cisco.com/

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Dont forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.
Cisco Public 108
BRKAPP-3006
BRKAPP-3006
Cisco Public

Advanced Troubleshooting Cisco WAAS

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Advanced Troubleshooting Cisco WAAS

Încărcat de

Drepturi de autor:

Formate disponibile

Advanced Troubleshooting Cisco Wide Area Application Services

2012 Cisco and/or its affiliates. All rights reserved.

2012 Cisco and/or its affiliates. All rights reserved.

Wide Area Application Engine (WAE)

IOS Platform with Services and CLI

TCP Proxy with Scheduler Optimizer (SO) DRE, LZ, TFO

Virtual Blade Storage /vbspace

Ethernet Network I/O

Self Diagnostic Tool

2012 Cisco and/or its affiliates. All rights reserved.

Self Diagnostic Tool

2012 Cisco and/or its affiliates. All rights reserved.

WAAS System Report

The sysreport includes the following:

2012 Cisco and/or its affiliates. All rights reserved.

Wide Area Application Engine (WAE)

IOS Platform with Services and CLI

TCP Proxy with Scheduler Optimizer (SO) DRE, LZ, TFO

Virtual Blade Storage /vbspace

Ethernet Network I/O

Disk Health and Status

Online Defunct Missing <null> Rebuilding

2012 Cisco and/or its affiliates. All rights reserved.

Replacing Failed Disks

Administrator can also shutdown disk for scheduled replacement:

< snip >

Command completed successfully.

SIZE 991MB 991MB 5951MB 6943MB 117798MB 991MB

INUSE 912MB 879MB 196MB 978MB 1786MB 16MB

2012 Cisco and/or its affiliates. All rights reserved.

Important Directories and Log Files

File system navigation commands

2012 Cisco and/or its affiliates. All rights reserved.

Ethernet Interface Errors

Verify Interface Statistics for Errors

2012 Cisco and/or its affiliates. All rights reserved.

Ethernet Interface Speed / Duplex

Verify Interface State, Speed and Duplex

2012 Cisco and/or its affiliates. All rights reserved.

CPU Utilization (Historical)

2012 Cisco and/or its affiliates. All rights reserved.

CPU Utilization (Real Time)

2012 Cisco and/or its affiliates. All rights reserved.

Wide Area Application Engine (WAE)

IOS Platform with Services and CLI

TCP Proxy with Scheduler Optimizer (SO) DRE, LZ, TFO

Virtual Blade Storage /vbspace

Ethernet Network I/O

2012 Cisco and/or its affiliates. All rights reserved.

WCCP Configuration Recommendations

WCCP on a hardware-based platform

2012 Cisco and/or its affiliates. All rights reserved.

WCCPv2 Interception Verification

L2 Forwarding and GRE Forwarding Hash assignment

2012 Cisco and/or its affiliates. All rights reserved.

WCCPv2 Interception Verification (IOS)

2012 Cisco and/or its affiliates. All rights reserved.

WCCPv2 Interception Verification (IOS)

10.88.81.242 2.0 1 1 68755 2 68753 Open -none0 -none0 0 -none0 0 0

Verify That Counters Are Incrementing on SoftwareBased Platforms (e.g. ISR)

2012 Cisco and/or its affiliates. All rights reserved.

WCCPv2 Interception Verification (IOS)

10.88.81.242 2.0 1 1 102 1 101 Open -none0 -none0 0 -none0 0 0

2012 Cisco and/or its affiliates. All rights reserved.