ACE Overview & Troubleshooting v2.0

ACE Overview & Troubleshooting
Author: Stuart Hare

Confidential
Copyright 2009 Accenture All Rights Reserved. Accenture, its logo, and High Performance Delivered are trademarks of Accenture.
Cisco Application Control Engine

Overview
Copyright 2009 Accenture All Rights Reserved.

Product Overview
ACE Products
This Cisco Application Control Engine comes in 2 flavours:
ACE Service Module for the Catalyst 6500 series switch
Specification
Up to 16Gps Throughput ( Up to 64Gbps by clustering 4 modules); 3.3Gbps SSL Throughput;
Up to 250 Virtual Device Contexts; 4,000,000 Concurrent Connections
ACE 4700 Appliance 1RU Rack Mount
Specification
Up to 4Gbps throughput; 1Gbps SSL Throughput; Up to 20 Virtual Device Contexts; 1,000,000 Concurrent Connections
ACE Service Module Catalyst 6500

Data Sheet:
http://www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6906/product_data_sheet0900aecd80
45861b.html
ACE 4700 Appliance

Data Sheet:
http://www.cisco.com/en/US/prod/collateral/contnetw/ps5719/
ps7027/Data_Sheet_Cisco_ACE_4710.html

Product Overview
Bang 6 Data Centre

The Bang 6 Data Centre network currently only utilises the Application Control Engine Services Modules, which are located in
the Web/Internal Aggregation switches (Cat 65K).
ACE Module Licensing

ACE licensing is provided via license file similar to other cisco devices. You request the license from cisco using a purchased
PAK code, Cisco then email you the license file, which you then have to FTP to your device.
Base product License:
5 Virtual Device Contexts
4Gps Throughput
5000 SSL Transactions per second

Current Bang 6 Device License:
20 Virtual device Contexts
4Gps Throughput
5000 SSL Transactions per second

Configuration Components

Initial Configuration
The ACE Module is very similar in many cases to the Firewall Services Modules for the Cat 65K.
As the device is a line card in the 6500 catalyst switch you will need to deliver the VLANs to the module as you would for the
FWSM on the Cat65K supervisor i.e.:
svclc
svclc
svclc
svclc
multiple-vlan-interfaces
module 1 vlan-group 1,2,
vlan-group 1 10,20
vlan-group 2 30,40
Admin (System) Configuration

To configure any global device settings you will need to access the Admin partition of the device (System partition on the
FWSM). Here you will be able to define management access, virtual device contexts, cluster redundancy etc.
Cisco Application Control Software (ACSW)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.
CHP-BANG6-LB1A/Admin#

Admin (System) Configuration - continued
Fault tolerance (cluster redundancy) will be defined here in the Admin partition.
Before you commence configuration on the device ensure that you have created a FT VLAN on the 65K switch, it is allowed on
the inter-switch trunk and presented to the module as per above config.
FT requires an FT VLAN interface, an FT peer and an FT group ( 1 group needed per Virtual device context ).
This needs to be repeated on the secondary module with the peer IPs and priority reversed.
E.g.
ft interface vlan 12
ip address 192.168.252.253 255.255.255.252
peer ip address 192.168.252.254 255.255.255.252
no shutdown
ft peer 1
heartbeat interval 300
heartbeat count 10
ft-interface vlan 12
ft group 1
peer 1
priority 115
peer priority 105
associate-context Admin
inservice
!## Only 1 peer reqd. Binds the FT interface to the peer.
!## Admin context assigned to FT Group

!## Bring group into operation

In the Admin partition an interface will need to be created for management access.
To allow management access to the device you need to define an access policy and apply it to the inbound interface.
Below a policy is created to allow default management protocols to come inbound into the device.
This is required as mgmt access is denied by default.
E.g.
class-map type management match-any mgmt_class
10 match protocol https any
20 match protocol snmp any
30 match protocol ssh any
40 match protocol icmp any
50 match protocol telnet any
!## Classify traffic
policy-map type management first-match mgmt

class mgmt_class
permit
!## Apply action to matched traffic
interface vlan 903

ip address 10.200.85.123 255.255.255.192
peer ip address 10.200.85.122 255.255.255.192
service-policy input mgmt
no shutdown
!##
!##
!##
!##
!##
Create Vlan Interface

Interface Mgmt IP address
Peer mgmt IP address
Assign policy to interface
Enable interface

Context Creation / definition is also carried out in the Admin partition.
Before creating a new virtual device context you need to either confirm that suitable resource class exists or create a new
resource class. One resource class can be used for multiple contexts.
Resource classes are used to assign a portion of the devices available resources to the defined context i.e. Stickiness.
Sticky resource is not included in the All resource, hence why specified separately.
You will also need to ensure that the relevant vlans have been assigned to the module.
Below shows how to create a resource class, and a new context:
resource-class ClassA
limit-resource all minimum 10.00 equal-to-min
limit-resource sticky minimum 10.00 equal-to-min
!## Allocate 10% of resources to class

!## Allocate sticky resource
context CIO-CHP-BANG6-LB1A
!## Create VDC
description CIO LB CONTEXT
allocate-interface vlan 502
!## Allocate VLAN Interface to VDC
allocate-interface vlan 802
member ClassA
!## Assign Resource Class to Context
ft group 2 !## Create FT Group for VDC
peer 1
priority 115
peer priority 105
associate-context CIO-CHP-BANG6-LB1A
inservice
!## Assign VDC to FT Group

VD Context Configuration
You can view the created contexts in the Admin partition using the following command:
CHP-BANG6-LB1A/Admin# sh context
Number of Contexts = 8
...truncated
Name: CIO-CHP-BANG6-LB1A , Id: 1
Config count: 128
Description: CIO LB CONTEXT
Resource-class: ClassA
Vlans: Vlan502, Vlan802
FT Auto-sync running-cfg configured state: enabled
FT Auto-sync running-cfg actual state: enabled
FT Auto-sync startup-cfg configured state: enabled
FT Auto-sync startup-cfg actual state: enabled
...truncated
Once the context is created switch to the context to perform context specific configuration.
CHP-BANG6-LB1A/Admin# changeto ?
Admin
CIO-CHP-BANG6-LB1A
...truncated
CHP-BANG6-LB1A/Admin# changeto CIO-CHP-BANG6-LB1A
CHP-BANG6-LB1A/CIO-CHP-BANG6-LB1A#
10

VD Context Configuration continued

From within the context you can now continue to configure your specific virtual environment.
Here you will configure your vlan interfaces, access and load balance policies, security configurations, probes, real servers and server
farms etc.
Once your interfaces are configured, we can then proceed to allow traffic into the context and define our load balancing policies.
First we will need allow inbound traffic on each interface.
access-list PERMIT-ALL line 10 extended permit ip any any
access-list PERMIT-ALL line 20 extended permit icmp any any
interface vlan xxx
ip address 10.200.120.137 255.255.255.240
alias 10.200.120.139 255.255.255.240
peer ip address 10.200.120.138 255.255.255.240
access-group input PERMIT-ALL
access-group output PERMIT-ALL
!## Assign inbound and outbound ACLs
11


Load Balancing Policies
Load balancing policies on the ACE are made up of the following components:
Virtual IP address
Real Servers
Server Farms
Load Balancing Algorithm
Class Maps
Policy Maps
Service Policies
Persistence (Optional)
Probes
- IP address that all load balanced requests are directed to.

- Actual server definitions that will provide content
- Pools of Real Servers; Load balancing algorithms and probes are set here
- Computation that decides which server will receive the a particular request
- Used to match on interesting traffic
- Decides what to do with the traffic matched on in the class map
- Enables the LB policy by being applied to an interface
- Stickiness needed to tie a request to a server for the duration of the session
- Health checks for real servers
Akin to Modular QOS Command-line (MQC Cisco IOS) or Modular Policy Framework (MPF Cisco Firewalls).
Slightly different functionality but concepts are the same.
The following slide contains a base Load Balancing policy which demonstrates the configuration for all the above components, to a site
listening for SSL/TLS requests.
12

probe https HTTPS
port 443
expect status 200 200
!## Create Health probe
rserver host
ip address
inservice
rserver host
ip address
inservice
!## Define real webserver, assigning it IP address
SVR1
10.1.1.1
!## set the expected healthy status code from the server
!##Enable Server
SVR2
10.1.1.2
serverfarm host SFARM1

failaction purge
predictor leastconns
probe HTTPS
rserver SVR1 443
inservice
rserver SVR1 443
inservice
!##
!##
!##
!##
!##
!##
Define Server pool

Set the pool to purge conns for failed servers
Set load balancing algorithm
Assign probe
Assign Server to pool defining its service port
Enable server within the pool
sticky ip-netmask 255.255.255.255 address source APP1.accenture.com_STICKY !## Enable src ip sticky
timeout 30
!## Set sticky table timeout
replicate sticky
!## Ensure sticky table is replicated across cluster
serverfarm SFARM1
!## State which farm to apply stickiness to
13

class-map match-any APP1.accenture.com_VIP
2 match virtual-address 170.252.1.1 tcp eq 443
!## Define interesting traffic from the VIP

!## Specify VIP and service
policy-map type loadbalance first-match APP1.accenture.com_default

class class-default
!## Default catch all class
sticky-serverfarm SFARM1
!## Use sticky serverfarm for load balancing
policy-map multi-match APP1.accenture.com_LB
class APP1.accenture.com_VIP
loadbalance vip inservice
loadbalance policy APP1.accenture.com_default
loadbalance vip icmp-reply active
loadbalance vip advertise active
!##
!##
!##
!##
!##
!##
Policy that binds everything together

Bind VIP to policy
Enable VIP
Bind server farm to policy
Enable ICMP to VIP
Enable LB to advertise the VIP (Proxy-Arp)
interface vlan xxx

service-policy input APP1.accenture.com_LB
!## Enable the entire policy
Note: The above load balancing policy will not work until the service-policy is applied to the interface.
The service-policy is generally applied to the public facing vlan interface (depending on design).
14

Troubleshooting
15

Troubleshooting
General Troubleshooting
The following sections will cover general troubleshooting of the ACE module.
This will include show commands and their resulting output. Many commands are the same or similar to many other Cisco devices, so
should be familiar.
Basic Device Information
Starting with basic device information, as with other Cisco devices show version displays code version, uptime, and installed license
keys.
CHP-BANG6-LB1A/Admin# show version
Cisco Application Control Software (ACSW)
Software
loader:
Version 12.2[120]
system:
Version A2(1.0a) [build 3.0(0)A2(1.0a)
adbuild_04:14:49-2008/04/18_/auto/adburel3/rel_a2_1_throttle/REL_3_0_0_
A2_1_0A]
system image file: [LCP] disk0:c6ace-t1k9mz.A2_1_0a.bin
installed license: ACE-VIRT-020 ACE-SEC-LIC-K9
ACE-SSL-05K-K9
memory info:
total: 956180 kB, free: 245800 kB
shared: 0 kB, buffers: 5048 kB, cached 0 kB
cf info:
filesystem: /dev/cf
total: 1014624 kB, used: 418464 kB, available:
596160 kB
last boot reason: reload command by admin
configuration register: 0x1
CHP-BANG6-LB1A kernel uptime is 55 days 0 hour 36
minute(s) 8 second(s)
Hardware
Cisco ACE (slot: 1)
cpu info:
number of cpu(s): 2
cpu type: SiByte
cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz
cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz
16

Troubleshooting
General Troubleshooting - continued
To see what license files you have currently installed on the device and their status use the following commands:
Show license
Show license status
Show license usage

CHP-BANG6-LB1A/Admin# sh license
ACE20080922234710689.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT ACE-SEC-LIC-K9 cisco 1.0 permanent 1 \
VENDOR_STRING=<count>1</count> HOSTID=ANY \
NOTICE="<LicFileID>20080922234710689</LicFileID><LicLineID>1</LicLineID> \
<PAK>12110260310</PAK>" SIGN=099AECA211B2
INCREMENT ACE-SSL-05K-K9 cisco 1.0 permanent 1 \
<PAK>12110260310</PAK>" SIGN=11E1EE28D826
INCREMENT ACE-VIRT-020 cisco 1.0 permanent 1 \
<PAK>12110260310</PAK>" SIGN=BAC755969E68
CHP-BANG6-LB1A/Admin# sh license status
Licensed Feature
-----------------------------SSL transactions per second
Virtualized contexts
Module bandwidth in Gbps
Count
----5000
20
4
17

Troubleshooting
General Troubleshooting continued
CHP-BANG6-LB1A/Admin# sh license usage
License
Ins
Lic
Status
Expiry Date
Comments
Count
-------------------------------------------------------------------------------ACE-08G-LIC
No
Unused
ACE-16G-LIC
No
Unused
ACE-UPG1-LIC
No
Unused
ACE-UPG2-LIC
No
Unused
ACE-VIRT-020
Yes
1
In use
never
ACE-VIRT-050
No
Unused
ACE-VIRT-100
No
Unused
ACE-VIRT-250
No
Unused
ACE-VIRT-UP1
No
Unused
ACE-VIRT-UP2
No
Unused
ACE-VIRT-UP3
No
Unused
ACE10-16G-LIC
No
Unused
ACE-SEC-LIC-K9
Yes
1
Unused
never
ACE-SSL-05K-K9
Yes
1
In use
never
ACE-SSL-10K-K9
No
Unused
ACE-SSL-15K-K9
No
Unused
ACE-SSL-20K-K9
No
Unused
ACE-SSL-UP1-K9
No
Unused
ACE-SSL-UP2-K9
No
Unused
ACE-SSL-UP3-K9
No
Unused
-------------------------------------------------------------------------------CHP-BANG6-LB1A/Admin#
18

Troubleshooting
To display hardware information about your device, including serial number, product number etc, use the show hardware command:
CHP-BANG6-LB1A/Admin# sh hardware
Hardware
Product Number: ACE20-MOD-K9
Serial Number: SAD1211005A
Card Index:
207
Hardware Rev:
2.3
Feature Bits:
0000 0002
Slot No. :
1
Type:
ACE
To display the hardware inventory of the device , daughter cards etc use the show inventory command.
CHP-BANG6-LB1A/Admin# sh inventory
NAME: "module 1", DESCR: "Application Control Engine Service Module"
PID: ACE20-MOD-K9
, VID: V02, SN: SAD1211005A
In this case only the device hardware is shown. Below shows output for CPU utilisation using show processes cpu
CHP-BANG6-LB1A/Admin# sh processes cpu
CPU utilization for five seconds: 1%; one minute: 4%; five minutes: 4%
PID
Runtime(ms) Invoked
uSecs 1Sec
5 Sec
1 Min
5 Min Process
----- ----------- -------- ----- -------------- ----------1
76722
975171
78
0.0
0.0 %
0.0 %
0.0 % init
2
1
238
6
0.0
0.0 %
0.0 %
0.0 % keventd
19

Troubleshooting
General Troubleshooting Fault Tolerance
For identifying issues with the cluster device redundancy use the show ft group detail command.
When used in the Admin partition this command will display the ft group information for all available contexts on the device.
When used in a specific context it will only display the ft group information for that context.
CHP-BANG6-LB1A/Admin# show ft group detail
FT Group
: 1
No. of Contexts
: 1
Context Name
: Admin
Context Id
: 0
Configured Status
: in-service
Maintenance mode
: MAINT_MODE_OFF
My State
: FSM_FT_STATE_ACTIVE
My Config Priority
: 115
My Net Priority
: 115
My Preempt
: Enabled
Peer State
: FSM_FT_STATE_STANDBY_HOT
Peer Config Priority
: 105
Peer Net Priority
: 105
Peer Preempt
: Enabled
Peer Id
: 1
Last State Change time
: Wed Dec 17 10:40:28 2008
Running cfg sync enabled
: Enabled
Running cfg sync status
: Running configuration sync has completed
Startup cfg sync enabled
: Enabled
Startup cfg sync status
: Startup configuration sync has completed
Bulk sync done for ARP: 0
Bulk sync done for LB: 0
Bulk sync done for ICM: 0
20

Troubleshooting
General Troubleshooting Fault Tolerance continued

Issues with the FT group can be seen using this command. Below you can see an example where the secondary is in Cold Standby
mode due to missing certificates.
Context Name
Context Id
Configured Status
Maintenance mode
My State
My Config Priority
My Net Priority
My Preempt
Peer State
Peer Config Priority
Peer Net Priority
Peer Preempt
Peer Id
Last State Change time
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
CIO-DEV-BANG6-WEB-LB1A
2
in-service
MAINT_MODE_OFF
FSM_FT_STATE_ACTIVE
115
115
Enabled
FSM_FT_STATE_STANDBY_COLD
105
105
Enabled
1
Wed Dec 17 10:40:28 2008
Enabled
Peer in Cold State. Incremental Sync Failure: SSL Certificate does not exist
Enabled
Peer in Cold State. Incremental Sync Failure: SSL Certificate does not exist
21

Troubleshooting
General Troubleshooting Fault Tolerance continued
To rectify this issue import the same certificates that are on the primary to the secondary device and then disable/enable FT group
syncing using the following commands:
No
No
Ft
Ft
Sh
ft auto-sync running
ft auto-sync startup
auto-sync running
auto-sync startup
ft group detail
!##
!##
!##
!##
!##
disable sync for running config

disable sync for startup config
enable sync for running config
enable sync for startup config
rerun this command to confirm sync is successful
Below is a truncated version of the successful output after certificates are imported and re-syncing of both running and startup config.
Context Name
Configured Status
My State
My Preempt
Peer State
Peer Preempt
:
:
:
:
:
:
:
:
:
:
CIO-DEV-BANG6-WEB-LB1A
in-service
FSM_FT_STATE_ACTIVE
Enabled
FSM_FT_STATE_STANDBY_HOT
Enabled
Enabled
Running configuration sync has completed
Enabled
Startup configuration sync has completed
Note: During device synchronisation configuration is disabled and the following messages will be displayed.
Configuration mode is currently disabled
NOTE: Configuration mode is enabled on all sessions
22

Troubleshooting
To check the resources currently assigned to a virtual device context by a resource class, use the show resource usage command
CHP-BANG6-LB1A/Admin# sh resource usage
Allocation
Resource
Current
Peak
Min
Max
Denied
------------------------------------------------------------------------------Context: CIO-PROD-WEB-BANG6-LB1A
conc-connections
338
986
800000
2400000
0
mgmt-connections
6
22
10000
30000
0
proxy-connections
0
84
104858
314572
0
xlates
0
0
104858
314572
0
bandwidth
2441
34434453
50000000 275000000
0
throughput
2091
34429456
50000000 150000000
0
mgmt-traffic rate
350
4997
0 125000000
0
connection rate
5
507
100000
300000
0
ssl-connections rate
0
40
500
1500
0
mac-miss rate
0
476
200
600
0
inspect-conn rate
0
0
600
1800
0
acl-memory
22872
25280
7858944
23583130
0
sticky
0
0
419430
0
0
regexp
18
3755
104858
314573
0
syslog buffer
448512
448512
418816
1262592
0
syslog rate
8
989
10000
30000
0
Note that if you have a constant increase on a figure in the denied column, this means that you have breached the maximum available
resource for that item and you will need to increase the available resource in the resource class.
23

Troubleshooting
To finish off the basics commands we have three commands that will be used regularly for tracking traffic flows:
Show logging
- needs to be enabled with logging level to function
Show conn
- show state table for current connections through the device
Capture
- captures traffic thru the device based on an access-l / interface etc.
CHP-BANG6-LB1A/Admin# sh logging
Syslog logging:
enabled
Facility:
20
History logging:
disabled
Trap logging:
disabled
Timestamp logging:
enabled
Fastpath logging:
disabled
Console logging:
disabled
Monitor logging:
disabled
Logging to 10.200.86.25 udp/514
Device ID:
disabled
Reject-newconn:
rate-limit-reached:
disabled
tcp-queue-full:
disabled
cp-buffer-full:
disabled
Buffered logging:
enabled (level - warnings) maximum size 4096
Buffer info: current size - 4096 global pool - 4194304 used pool - 4194304
min - 0 max - 4096
cur ptr = 1536 wrapped - yes
This output is truncated and will be followed by the log buffer contents.
24

Troubleshooting
Show conn
Show conn detail
- show state table for current connections through the device

- more detailed output
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A# sh conn
total current connections : 218
conn-id
np dir proto vlan source
destination
state
----------+--+---+-----+----+---------------------+---------------------+------+
393
1 in UDP
309 10.200.116.7:50218
10.200.87.65:53
-753
1 out UDP
360 10.200.87.65:53
10.200.116.7:50218
-913
1 in TCP
309 10.200.116.43:3083
170.252.165.50:5615
ESTAB
408
1 out TCP
360 170.252.165.50:5615
10.200.116.43:3083
ESTAB
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A# sh conn detail
total current connections : 212
conn-id
np dir proto vlan source
destination
state
----------+--+---+-----+----+---------------------+---------------------+------+
393
1 in UDP
309 10.200.116.7:50218
10.200.87.65:53
-[ idle time
: 00:00:44,
byte count : 76
]
[ elapsed time: 00:00:44,
packet count: 1
]
913
1 in TCP
309 10.200.116.43:3083
170.252.165.50:5615
ESTAB
[ idle time
: 00:00:55,
byte count : 42753
]
[ elapsed time: 18:22:44,
packet count: 338
]
This output is truncated.
25

Troubleshooting
Capture
- captures traffic thru the device based on an access-l / interface etc.
This example will show how to capture traffic real time through the device via capture using an access-list to filter traffic.
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A(config)# access-l stu ext permit ip any any
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A# capture STUCAP int vlan 360 access-list stu
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A# capture STUCAP start
13:21:11.880025 0:1d:70:58:b6:40 0:b:fc:fe:1b:7 0800 74: 10.10.140.36 > 10.200.116.50: icmp: echo request (ttl
114, id 22207, len 60)
13:21:11.880404 0:b:fc:fe:1b:7 0:50:56:90:24:81 0800 74: 10.10.140.36 > 10.200.116.50: icmp: echo request (ttl
114, id 22207, len 60)
13:21:11.880763 0:50:56:90:24:81 0:b:fc:fe:1b:7 0800 74: 10.200.116.50 > 10.10.140.36: icmp: echo reply (ttl
128, id 6314, len 60)
13:21:11.881136 0:b:fc:fe:1b:7 0:0:c:7:ac:5a 0800 74: 10.200.116.50 > 10.10.140.36: icmp: echo reply (ttl 128,
id 6314, len 60)
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A# capture STUCAP stop
CHP-BANG6-LB1A/CIO-DEV-BANG6-WEB-LB1A# sh cap
0007: msg_type: ACE_HIT
ace_id: 1049633
0008: msg_type: CON_SETUP con_id: 1107299107
0009: msg_type: PKT_RCV
con_id: 1107299107
0010: msg_type: PKT_XMT
con_id: 16780023
0011: msg_type: PKT_RCV
con_id: 16780023
0012: msg_type: PKT_XMT
con_id: 1107299107
0013: msg_type: ACE_HIT
ace_id: 1049633
0014: msg_type: CON_CLOSE con_id: 1090521171
STUCAP
action_flag:
out_con_id:
other_con_id:
other_con_id:
other_con_id:
other_con_id:
action_flag:
reason:
0x3
16780023
0
0
0
0
0x3
0
26

Troubleshooting
Troubleshooting Load Balanced Policies

This following slides will move into troubleshooting commands for the load balancing policy configurations:
Show probe
- displays the status of probes for individual server farms / real servers.
probe
: PING
type
: ICMP
state
: ACTIVE
---------------------------------------------port
: 0
address
: 0.0.0.0
addr type : interval : 30
pass intvl : 300
pass count : 3
fail count: 3
recv timeout: 10
--------------------- probe results -------------------probe association
probed-address probes
failed
passed
health
------------------- ---------------+----------+----------+----------+------serverfarm : AbacusLite-DIME-Bang6.accenture.com
real
: TSTPVH1001-2[0]
10.200.102.36
79640
3
79637
SUCCESS
serverfarm : AbacusLite-Test-Bang6.accenture.com
real
: TSTPVH1001-1[0]
10.200.102.35
72473
3
72470
SUCCESS
27

Troubleshooting
Troubleshooting Load Balanced Policies - continued
The below show output shows a show probe detail output for a failed rserver.
CHP-BANG6-LB1A/CIO-PROD-WEB-BANG6-LB1A# sh probe https_443 det
probe
: https_443
type
: TCP
state
: ACTIVE
description :
---------------------------------------------port
: 443
address
: 0.0.0.0
addr type : interval : 30
pass intvl : 300
pass count : 3
fail count: 3
recv timeout: 10
conn termination : GRACEFUL
expect offset
: 0
, open timeout
: 10
expect regex
: send data
: --------------------- probe results -------------------probe association
probed-address probes
failed
passed
health
------------------- ---------------+----------+----------+----------+------serverfarm : ECRF-DCNBPO.ACCENTURE.COM
real
: BG6WS0007-ECRF[443]
10.200.101.21
3740
3740
0
FAILED
Socket state
: CLOSED
No. Passed states
: 0
No. Failed states : 1
No. Probes skipped : 0
Last status code : 0
No. Out of Sockets : 0
No. Internal error: 0
Last disconnect err : Server open timeout (no SYN ACK)
Last probe time
: Mon Feb 9 13:39:04 2009
Last fail time
: Tue Jan 27 14:06:53 2009
Last active time
: Never
28

Troubleshooting
Troubleshooting Load Balanced Policies continued
To view the details and operational status of a server use the show rserver command.
Below you can see output for both operational and failed servers.
CHP-BANG6-LB1A/CIO-PROD-WEB-BANG6-LB1A# sh rserver
rserver
: BG6AV1101, type: HOST
state
: OPERATIONAL (verified by arp response)
------------------------------------------connections----------real
weight state
current
total
---+---------------------+------+------------+----------+-------------------serverfarm: epo-bang6.accenture.com
10.200.102.1:0
8
OPERATIONAL 19
12394
rserver
: BG6WI1102, type: HOST
state
: ARP_FAILED
weight state
current
total
---+---------------------+------+------------+----------+-------------------serverfarm: bang6appfarm.accenture.com
10.200.101.12:0
8
ARP_FAILED
0
0
rserver
: BG6WS0008-SMP, type: HOST
state
: OPERATIONAL (verified by arp response)
weight state
current
total
---+---------------------+------+------------+----------+-------------------serverfarm: SMP-DCNBPO.ACCENTURE.COM
10.200.101.30:443
8
PROBE-FAILED 0
0
29

Troubleshooting
To view the status of a server pool use the show serverfarm command.
CHP-BANG6-LB1A/CIO-PROD-WEB-BANG6-LB1A# sh serverfarm SMP-DCNBPO.ACCENTURE.COM det
serverfarm
: SMP-DCNBPO.ACCENTURE.COM, type: HOST
total rservers : 1
active rservers: 0
description
: state
: INACTIVE
predictor
: LEASTCONNS
slowstart
: 0 secs
failaction
: REASSIGN
back-inservice
: 0
partial-threshold : 0
num times failover
: 0
num times back inservice : 0
total conn-dropcount : 0
Probe(s) :
https_443, type = TCP
weight state
current
total
failures
---+---------------------+------+------------+----------+----------+--------rserver: BG6WS0008-SMP
10.200.101.30:443
8
PROBE-FAILED 0
0
0
max-conns
: , out-of-rotation count : min-conns
: conn-rate-limit
: , out-of-rotation count : bandwidth-rate-limit : , out-of-rotation count : retcode out-of-rotation count : load value
: 0
30

Troubleshooting
The output below shows of an operational serverfarm, a small number of failures will be seen through normal operation.
serverfarm
: c2a-dvl.accenture.com, type: HOST
total rservers : 1
active rservers: 1
description
: state
: ACTIVE
predictor
: ROUNDROBIN
failaction
: REASSIGN
back-inservice
: 0
partial-threshold : 0
num times failover
: 12
num times back inservice : 12
total conn-dropcount : 0
Probe(s) :
https_443, type = TCP
weight state
current
total
failures
---+---------------------+------+------------+----------+----------+--------rserver: BG6WS0613
10.200.116.61:443
8
OPERATIONAL 0
257
10
max-conns
: , out-of-rotation count : min-conns
: conn-rate-limit
: , out-of-rotation count : bandwidth-rate-limit : , out-of-rotation count : retcode out-of-rotation count : load value
: 0
31

Troubleshooting
The status and statistics of a load balanced policy can be viewed using the show service-policy command.
CHP-BANG6-LB1A/CIO-PROD-WEB-BANG6-LB1A# sh service-policy
Policy-map : AbacusLite-DIME-Bang6.accenture.com_LB
Status
: ACTIVE
----------------------------------------Interface: vlan 358
service-policy: AbacusLite-DIME-Bang6.accenture.com_LB
class: AbacusLite-DIME-Bang6.accenture.com_VIP
loadbalance:
L7 loadbalance policy: AbacusLite-DIME-Bang6.accenture.com_default
VIP Route Metric
: 77
VIP Route Advertise : ENABLED-WHEN-ACTIVE
VIP ICMP Reply
: ENABLED-WHEN-ACTIVE
VIP State: INSERVICE
curr conns
: 0
, hit count
: 478
dropped conns
: 23
client pkt count : 1993
, client byte count: 174310
server pkt count : 1663
, server byte count: 597478
conn-rate-limit
: 0
, drop-count : 0
bandwidth-rate-limit : 0
, drop-count : 0
32

Troubleshooting

You can go into more detail, to check what classes the traffic is being matched against using the show service-policy detailcommand.
Policy-map : MTPOC_LB_POL
L7 Loadbalance policy : MTPOC_L7_POL
Status
: ACTIVE
class/match : MTPOC_APP1_L7_Class
Description: ssl-proxy client : CLIENT_SSL_INITIATION
----------------------------------------LB action :
Interface: vlan 358
primary serverfarm: MTPOC_APP1_FARM
service-policy: MTPOC_LB_POL
state: UP
class: MTPOC_L4_VIP_Class
backup serverfarm : GoFish-B6-Prod-RD-SF
ssl-proxy server: MTPOC_SSL_PS
state: UP
VIP Address:
Protocol: Port:
hit count
: 71
170.251.150.42 tcp
eq
443
dropped conns
: 7
loadbalance:
class/match : MTPOC_APP2_L7_Class
L7 loadbalance policy: MTPOC_L7_POL
LB action :
VIP Route Metric
: 77
primary serverfarm: MTPOC_APP2_RD_FARM
VIP Route Advertise : ENABLED
state: UP
VIP ICMP Reply
: ENABLED
backup serverfarm : VIP State: INSERVICE
hit count
: 31
curr conns
: 0
, hit count
: 393
dropped conns
: 0
dropped conns
: 212
Parameter-map(s):
client pkt count : 1787
, client byte count: 203127
HTTP_REBALANCE_MAP
server pkt count : 660
, server byte count: 395457
conn-rate-limit
: 0
, drop-count : 0
bandwidth-rate-limit : 0
, drop-count : 0
33

Troubleshooting Command Summary
Show version
- displays uptime code version etc
Show license [status] | [usage] - displays license files and status e.g. how many licenses you have / are in use
Show running-config
- displays the current configuration held in RAM
Show rserver detail
- displays detailed output for each configured real server
Show serverfarm detail
- displays detailed output for each configured serverfarm
Show service-policy detail
- displays detailed output for each LB service policy
Show parameter-map
- displays output & settings for each configured parameter-map
Show probe detail
- displays detailed output for server health check probes
Show stats crypto client
- displays output relating to the back end LB-server encryption including hits and ciphers
- used etc.
Show stats crypto server
- displays output relating to the front end client-LB encryption including hits and ciphers
- used etc.
Show conn
- displays active connections in state table
Show logg
- displays entries in buffered log
Show ft group detail
- displays detailed output for the fault tolerant group i.e. Active / Standby device pair
No ft auto-sync running
- disable sync for running config
No ft auto-sync startup
- disable sync for startup config
Ft auto-sync running
- enable sync for running config
Ft auto-sync startup
Show sticky database
- enable sync for startup config

- displays the contents of the sticky table for LB persistence
Note: Most of the commands can be appended with the detail sub command to provide more information.
34

ACE Overview &amp; Troubleshooting v2.0

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

ACE Overview &amp; Troubleshooting v2.0

Încărcat de

Drepturi de autor:

Formate disponibile

ACE Overview & Troubleshooting

Author: Stuart Hare

Cisco Application Control Engine

Copyright 2009 Accenture All Rights Reserved.