BRKSEC-3020 Advanced Firewalls

Advanced Firewalls
Session BRKSEC-3020
Agenda
Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best Practices
BRKSEC-3020
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Packet Flow
Understanding the Packet Flow

To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/FWSM, always
Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol Determine the interfaces through which the flow passes
Note: All Firewall Issues Can Be Simplified to Two Interfaces (Ingress and Egress) and the Rules Tied to Both
BRKSEC-3020 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public
Example Flow
Flow
SRC IP: 10.1.1.9 DST IP: 198.133.219.25 SRC Port: 11030 DST Port: 80 Destination: Outside
Servers
Protocol: TCP
Interfaces
Source: Inside
Client: 10.1.1.9
Packet Flow
Eng
Accounting Outside Server: 198.133.219.25
With the Flow Defined, Examination of Configuration Issues Boils Down to Just the Two Interfaces: Inside and Outside
BRKSEC-3020
Cisco Public
Packet Processing: Ingress Interface

Ingress Interface
Packet arrives on ingress interface Input counters incremented Software input queue is an indicator of load No buffers indicates packet drops, typically due to bursty traffic
ASA-5540# show interface gb-ethernet1 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.470d.6214 IP address 10.1.1.1, subnet mask 255.255.255.0 MTU 1500 bytes, BW 1 Gbit full duplex 5912749 packets input, 377701207 bytes, 0 no buffer Received 29519 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 286298 packets output, 18326033 bytes, 0 underruns input queue (curr/max blocks): hardware (4/25) software (0/0) output queue (curr/max blocks): hardware (0/3) software (0/0)
Packet Processing: Locate Connection

Existing Conn
Check first for existing connection If connection exists, flow is matched; bypass ACL check If no existing connection
TCP non-SYN packet, drop and log TCP SYN or UDP packet, pass to ACL checks Established Connection:
ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO
Syslog Because of No Connection, and Non-SYN Packet:

ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface inside
Packet Processing: ACL Check

ACL Permit
First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and logged
Packet Permitted by ACL:
ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1)
Syslog When Packet Is Denied by ACL:

ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside: 198.133.219.25/80 by access-group "inside"
Translation and NAT Order of Operations

Match xLate
For your reference
Translation Matching
Pre version 8.3 Version 8.3+
1. nat 0 access-list (nat-exempt) 2. Match existing xlates 3. Match static commands (Cisco ASA/PIX first match; FWSM best match) 4. Match nat commands First Match First Match
1. Manual NAT entries 2. Auto NAT entries 3. After-Auto NAT entries
BRKSEC-3020
Cisco Public
Packet Processing: Inspections/Sec Checks

Inspections Sec Checks
Inspections are applied to ensure protocol compliance (Optional) customized AIC inspections NAT-embedded IPs in payload Question! Additional security checks are applied to the packet
What command will show you if packets are being dropped (Optional) packets passed to Content Security and Control (CSC) module one of the Inspection engines? by
Syslog from Packets Denied by Security Check:
ASA-4-406002: FTP port command different address: 10.2.252.21 (192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP
10
Packet Processing: NAT IP Header

Nat IP Header
Translate the IP address in the IP header Translate the port if performing PAT Update checksums (Optional) Following the above, pass packet to IPS (AIP) module
BRKSEC-3020
Cisco Public
11
Packet Processing: Egress Interface

Egress Interface
Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet) Egress interface is determined first by translation rules If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface Example:
Inside
172.16.0.0/16
Outside DMZ
172.16.12.0/24 172.16.12.4
Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

static (inside, outside) 192.168.0.0 172.16.0.0 netmask 255.255.0.0 static (dmz, outside) 192.168.12.0 172.16.12.0 netmask 255.255.255.0
BRKSEC-3020
Cisco Public
12
Packet Processing: L3 Route Lookup

L3 Route
Once on egress interface, an interface route lookup is performed Only routes pointing out the egress interface are eligible Remember: translation rule can forward the packet to the egress interface, even though the routing table may point to a different interface
Syslog from Packet on Egress Interface with No Route Pointing Out Interface:
%ASA-6-110003: Routing failed to locate next hop for TCP from inside: 192.168.103.220/59138 to dmz:172.18.124.76/23
BRKSEC-3020
Cisco Public
13
Packet Processing: L2 Address Lookup

L2 Addr
Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed Layer 2 rewrite of MAC header If Layer 2 resolution failsno syslog show arp will not display an entry for the L3 next hop debug arp will indicate if we are not receiving an ARP reply
BRKSEC-3020
Cisco Public
14
Packet Processing: Transmit Packet

Xmit Pkt
Packet is transmitted on wire Interface counters will increment on interface

ASA-5585# show int Gig0/0 Interface GigabitEthernet0/0 "outside", is up, line protocol is up Hardware is bcm56801 rev 01, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps MAC address 5475.d05b.0fa6, MTU 1500 IP address 14.36.103.96, subnet mask 255.255.0.0 4337255 packets input, 394043049 bytes, 0 no buffer Received 1957325 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 pause/resume input 0 switch ingress policy drops 282901 packets output, 28855690 bytes, 0 underruns 0 pause/resume output
15
Agenda
BRKSEC-3020
Cisco Public
16
Cisco ASA Understanding the Architecture

ASA processes all packets in software (via the central CPU)
All packets are processed first in usually also first out
ASA platforms have software imposed connection limits Multi-CPU / Multi-Core systems hash packets in the same flow to the same CPU/core. 10 Gig interfaces hash flow to same RX ring. Architecture optimized for multi-flow traffic patterns ASASM packet processing is also done in software, unlike FWSM
BRKSEC-3020
Cisco Public
17
Maximum ACL Limits

No hard-coded limit on the number of elements (ACEs) in an ACL. Bound only by Memory. Each ACE uses a minimum of 212 bytes of RAM However, maximum performance may decrease (typically 10-15%) as you reach or exceed the Max Recommended ACEs.
5505
Max Recommended ACEs Tested ACEs Max Observed (from customers) 25k
5510
80k 80k
5520
200k 300k
5540
500k 700k
5550
700k 700k 2.74 million
5580
750k 1 million+ 2.77 million
5585
10/20/40/60 500k / 750k 1 / 2 million 500k / 750k 1 / 2 million
ASA SM
2 million 2 million
Note: Issue show access-list | include elements to see how many ACEs you have
18
Warning - ACE Explosion

Object-groups:
Sources (10 addresses) Destinations (21 addresses) Ports (33 ports) Result: 10x21x33 = 6,930 rules
Single line ACL explodes to
Nested object-groups:
Assume you add a SRC object-group to the above, which contains 25 additional sources Result: (10+25)x21x33 = 24,255 rules (ACEs) New command to reduce ACL memory impact for large ACLs. Available starting in 8.3(1)
ASA-5585(config)# object-group-search access-control
19
Global ACLs
Interface Independent Policies Global ACLs introduced in version 8.3 Best used for new installations, or migration from other vendors
access-group <access_list> global
ASA Only
Policy Ordering Interface Specific access-list Global access-list Default (implicit) deny ip any any
20
Object-NAT (Auto-NAT) (version 8.3+)

Object NAT is the simplest form of NAT, and is defined within an object
Host NAT object network obj-WebServer host 10.3.19.50 nat (inside,outside) static 198.51.100.50 Network NAT object network Servers subnet 10.0.54.0 255.255.255.0 nat (inside,outside) static 203.0.113.0 Dynamic PAT (interface overload) object network InternalUsers subnet 192.168.2.0 255.255.255.0 nat (inside,outside) dynamic interface
21
Manual NAT (Twice NAT) (version 8.3+)

Manual NAT should be used to translate the destination, or for policy NAT
object network ServerReal host 10.3.19.50 object network ServerTrans host 198.51.100.50 object network RemoteSite subnet 10.0.0.0 255.255.255.0 Static NAT nat (inside,outside) source static ServerReal ServerTrans
Static Policy NAT nat (inside,outside) source static ServerReal ServerTrans destination static RemoteSite RemoteSite
22
NAT Order of Operation version 8.3+
For your reference
The ASA configuration is built into the NAT Table (show nat) The NAT Table is based on First Match (top to bottom)
NAT Table Static NAT
Longest Prefix
Manual NAT Policies (Section 1)
First Match (in config)
Shortest Prefix
Dynamic NAT
Longest Prefix
Auto NAT Policies (Section 2)
Shortest Prefix
Manual NAT [after auto] Policies (Section 3)
First Match (in config)
BRKSEC-3020
Cisco Public
23
Real-IP (version 8.3+)

Finally, a reminder that with 8.3+ Real-IPs are used in ACLs
Real, Un-translated address object network obj-WebServer of internal Server host 10.3.19.50 nat (inside,outside) static 198.51.100.50 ! access-list allowIn permit tcp any host 10.3.19.50 eq 80 ! access-group allowIn in interface outside
BRKSEC-3020
Cisco Public
24
FWSMUnderstanding the Architecture

FWSM Process Most Packets in Hardware, with Some Packets Needing to be Processed in Software via the Control Point (CP)
Packets processed in hardware have zero impact on CPU Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware Note that FWSM packet processing is different from ASA
BRKSEC-3020
Cisco Public
25
FWSM Architectural Overview

Control Point (CP) Central CPU
Software Hardware
Session Manager NP 3
Control Point ACL Compilation, Fixups, Syslog, AAA, IPv6 in Software Session Manager Session Establishment and Teardown, AAA Cache, ACLs Fast Path Flow Identification, Security Checks and NAT in Hardware
Fast Path NP 1
Fast Path NP 2
FWSM
C6K Backplane Interface
BRKSEC-3020
Cisco Public
26
FWSMHardware Limits
See Appendix
FWSM has several hardware limits that should be considered in your network design Limits are hard set, but vary based on single or multimode Some limits include:
Increase over 2.3 Increase over 3.1
2.3 (Multimode) ACEs AAA Rules Global Statements Static NAT Statements Policy NAT ACEs NAT Translations Connections Route Table Entries Fixup/Inspect Rules Filter Statements
BRKSEC-3020
3.1/3.2 (Multimode) 72,806 (11,200) 6,451 (992) 4K (4K) 2K (2K) 1,843 (283) 256K (256K) 999,990 (999,990) 32K (32K) 4147 (1,417) 2764 (425)
Cisco Public
4.0/4.1 (Multimode) 100,567 (14,801) 8,744 (1,345) 4K (4K) 2K (2K) 2,498 (384) 256K (256K) 999,990 (999,990) 32K (32K) 5621 (1,537) 3747 (576)
3.2 / 4.0 /4.1 Configurable X X
56,627 (9,704) 3,942 (606) 1K (1K) 2K (2K) 3,942 (606) 256K (256K) 999,990 (999,990) 32K (32K) 32 (32 per) 3942 (606)
X X
27
*Complete List in FWSM Docs, Appendix A (Specifications)

FWSMACL Rule Limits

ACL rules are about the only hardware limit users encounter In multimode, ACL resources are divided in 13 equal partitions (12 active, one backup)
If you have less than 12 contexts, wasted reserved space
Single Context
Tree 0 : Active 100,567 ACEs
Multi-Context
Tree 0 : active = 14,801 ACEs Tree 1 : active = 14,801 ACEs Tree 2 : active = 14,801 ACEs Tree 3 : active = 14,801 ACEs Tree 4 : active = 14,801 ACEs Tree 5 : active = 14,801 ACEs Tree 6 : active = 14,801 ACEs Tree 7 : active = 14,801 ACEs
177612 combined total ACEs
Backup Tree: 100,567 (mirror of active tree)
Tree 8 : active = 14,801 ACEs Tree 9 : active = 14,801 ACEs Tree 10 : active = 14,801 ACEs Tree 11 : active = 14,801 ACEs Tree 12 : backup
BRKSEC-3020
Cisco Public
28
Classifier in Multimode
When the firewall receives a packet, it must classify it to determine where to send the packet (which context) Packets are classified based on the following
Unique ingress interface/VLAN Packets destination IP matches a global IP
FWSM has a single MAC address for all interfaces ASA has single MAC for shared interfaces (physical interfaces have unique MACs)
ASA Ver 7.2 introduces mac-address auto option to change this
BRKSEC-3020
Cisco Public
29
Classifier in Multimode
Example
Inbound traffic is classified to context CTX3, based on the global IP in the NAT translation
FWSM Inside 10.1.1.2 Inside VLAN 5 10.1.2.2 Inside VLAN 6 10.1.3.2 static (inside, outside) 10.14.3.89 10.1.3.2
DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89 Outside MSFC
SRC IP 192.168.5.4
VLAN 4
Inbound Packet
CTX2
.2
CTX3
.3
Shared Interface
30
Multi-Context - Common Issues on FWSM

Overlapping statics (globals) across contexts Missing statics (globals), and unable to classify packets check Admin context log
%FWSM-6-106025: Failed to determine security context for packet: vlan3 tcp src 192.168.5.4/1025 dest 72.163.4.161/80
Forgetting to monitor-interface for Failover Forgetting to assign unique IP for each Transparent mode context Transparent mode, multi-BVI, one routing table
BRKSEC-3020
Cisco Public
31
Agenda
BRKSEC-3020
Cisco Public
32
Failover Basics
Active/Standby vs. Primary/ Secondary Serial vs. LAN failover Stateful failover (optional) A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall Both firewalls swap MAC and IP addresses when a failover occurs Level 1 syslogs will give reason of failover
Internet
Stateful LAN/Serial Secondary (Standby) Primary (Active)
Corp
33
Verifying Failover Configuration

ASA# show failover Failover On Failover unit Primary Failover LAN Interface: failover Redundant5 (up) Unit Poll frequency 200 milliseconds, holdtime 1 seconds Interface Poll frequency 500 milliseconds, holdtime 5 seconds Interface Policy 1 Monitored Interfaces 2 of 250 maximum Version: Ours 8.2(2), Mate 8.2(1) Interface Last Failover at: 10:37:11 UTC May 14 2010 Monitoring This host: Primary - Active Active time: 1366024 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys) Interface outside (10.8.20.241): Normal Interface inside (10.89.8.29): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys) Interface outside (10.8.20.242): Normal Interface inside (10.89.8.30): Normal Stateful Failover Logical Update Statistics Link : stateful Redundant6 (up) Stateful Obj xmit xerr rcv rerr General 424525 0 424688 0 sys cmd 423182 0 423182 0
34
What Triggers a Failover?

Power loss/reload (this includes crashes) on the Active firewall SSM interface/module failure The Standby becoming healthier than the Active firewall In LAN based Failover, what happens if the LAN interface communication is severed?
BRKSEC-3020
Cisco Public
35
What Triggers a Failover? (Cont)

Two consecutive hello messages missed on any monitored interface forces the interface into testing mode Both units first verify the link status on the interface Next, both units execute the following tests
Network activity test ARP test Broadcast ping test
The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failed
BRKSEC-3020
Cisco Public
36
How Well do you Understand Failover? What Happens When

You disable failover? (By issuing no failover) You RMA/Replace the Primary unit? You dont define Standby IP addresses?
BRKSEC-3020
Cisco Public
37
What to Do After a Failover

Always check the syslogs to determine root cause Example: switch port failed on inside interface of active firewall
Syslogs from Primary (Active) Firewall
See Appendix
ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status Down on interface 1 ASA-1-104002: (Primary) Switching to STNDBYinterface check, mate is healthier
Syslogs from Secondary (Standby) Firewall

ASA-1-104001: (Secondary) Switching to ACTIVEmate want me Active
BRKSEC-3020
Cisco Public
38
Failover Zero Downtime Upgrades

State
Start
State Secondary
Issue failover active
Act Stb
Primary
Stb Act
Copy new image over and reboot Wait for failover to finish syncing, and to normalize approx 2 min Verify config; conns replicated
Act
Issue failover active Copy new image over and reboot Wait for failover to finish syncing, and to normalize approx 2 min Verify config; conns replicated
Stb
Upgrade Complete
39
Agenda
BRKSEC-3020
Cisco Public
40
Troubleshooting Tools
Syslogs Debug commands Show commands Packet capture Packet tracer TCP Ping
BRKSEC-3020
Cisco Public
41
Uses of Syslogs
Primary mechanism to record traffic to and through the firewall The best troubleshooting tool available
Archival Purposes Debugging Purposes
Console
Syslog Server Internet
SSH Client
Trap SNMP Server Syslog . Buffered
BRKSEC-3020
Cisco Public
42
ASA Syslog Level vs. Number of Messages

Log Description Level Emergenci es Alerts Critical Errors Warnings Notification s Information al Debugging
BRKSEC-3020
Number of Messages (SUM) Ver. 7.0 0 62 (62) 29 (91) 274 (365) 179 (544) 161 (705) 234 (939) 217 (1156) Ver. 7.2 0 77 (77) 35 (112) 334 (446) 267 (713) 206 (919) 302 (1221) 258 (1479) Ver. 8.0 0 78 (78) 49 (127) 361 (488) 280 (768) 216 (984) 335 (1319) 266 (1585) Ver. 8.1 0 87 (87) Ver. 8.2 0 87 (87) Ver. 8.3 0 95 (95) 57 (152) 408 (560) 324 (884) 246 (1130) 377 (1507) 269 (1776) Ver. 8.4 0 109 (109) 63 (172) 448 (620) 357 (997) 265 (1242) 395 (1637) 276 (1913)
43
0 1 2 3 4 5 6 7
50 (137) 56 (143) More messages 363 (500) 281 (781) 218 (999) 337 (1336) 267 (1603) 384 (527) 315 (842) 237 (1079) 368 (1447) 269 (1716)
Cisco Public
What Are Modifiable Syslog Levels?

[no] logging message <syslog_id> level <level>
Modifiable syslog levels

Allows one to move any syslog message to any level
Levels
0Emergency 1Alert 2Critical 3Errors 4Warnings 5Notifications 6Informational 7Debugging
Problem
You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level seven (debug) %ASA-7-111009: User johndoe executed cmd: show run The problem is we dont want to log all 1775 other syslogs that are generated at debug level
2011 Cisco and/or its affiliates. All rights reserved. Cisco Public
BRKSEC-3020
44
How to Create Modifiable Syslog Levels

Solution
[no] logging message <syslog_id> level <level>
Lower syslog message 111009 to level 3 (error)

ASA(config)# logging message 111009 level 3
If you were only interested in logging one syslog message, how could you Now our syslog looks as follows do it?
%ASA-3-111009: User johndoe executed cmd: show run
To restore the default syslog level

ASA(config)# no logging message 111009 level 3
Tip: Use show logging message all to see the default level for any message
45
Logging Common Issues

logging flash-bufferwrap should only be used when logging to buffer at Level 1 logging history should only be used when you really have an SNMP server that you want to receive all syslogs logging console should only be enabled while actively troubleshooting on the Console logging standby should only be used if you want to receive double the syslogs logging permit-hostdown should always be used with TCP syslogging
46
Debug Commands
1. Debugs should not be the first choice to troubleshoot a problem 2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution 3. Debugs are not conditional* 4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debug
* Crypto Conditional Debugging Was Added to Cisco ASA/PIX 8.0

47
Debug ICMP Trace

Internet
http://www.cisco.com
Valuable tool used to troubleshoot connectivity issues Provides interface and translation information to quickly determine flow Echo-replies must be explicitly permitted through ACL, or ICMP inspection must be enabled
Example debug icmp trace output
ICMP echo-request from inside:10.1.1.2 to 198.133.219.25 ID=3239 seq=4369 length=80 ICMP echo-request: translating inside:10.1.1.2 to outside:209.165.201.22 ICMP echo-reply from outside:198.133.219.25 to 209.165.201.22 ID=3239 seq=4369 length=80 ICMP echo-reply: untranslating outside:209.165.201.22 to inside:10.1.1.2
48
Show Output Filters
See Appendix
show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Use output filters to filter the output of show command to only the information you want to see To use them, at the end of show <Command>, use the pipe character | followed by
begin include exclude grep grep v
Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output Display any line that matches the RegEx Display any line that does not match the RegEx Same as include Same as exclude
BRKSEC-3020
Cisco Public
49
Show CPU Usage

Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled The show cpu usage command displays the CPU over time as a running average
ASA# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%
*First Introduced in Cisco PIX OS Version 6.0(1)/FWSM 1.1(1)

50
Show Processes cpu-usage

The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec, 1min, and 5min
ASA# # show processes cpu-usage PC Thread 5Sec 081aa124 d51ab230 0.2% 08070416 d51aa660 0.0% 081a954c d51a96a0 0.0% 08c2a91d d51a7f00 0.0% 0924fe95 d51a7528 0.0% 088a6e14 d51a7138 0.0% 08a6d7f6 d51a6f40 0.0% 08bcf736 d51a53b0 0.0% 08685627 d51a3a18 0.0% 0851ca68 d51a3820 0.0% 08b9ffab d5198ae0 0.0% 08b99aec d5195d98 3.9% ... 1Min 2.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.5% 5Min 2.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.1% Process Dispatch Unit aaa dbgtrace netfs_thread_init Chunk Manager IP Address Assign QoS Support Module Logger netfs_mount_handler arp_timer ssh/timer ssh
*First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1). Currently not Available in FWSM
51
Show Processes cpu-hog

The show processes cpu-hog command displays a list of processes, and the function stack (Traceback) which executed, and lead to a process running on the CPU longer than the minimum platform threshold
ASA# show processes cpu-hog Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10 LASTHOG At: 14:18:47 EDT May 29 2009 PC: 8b9ac8c (suspend) Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971 8ba02b4 8062413 CPU hog threshold (msec): 10.240 Last cleared: None
A corresponding syslog message is also generated Note: The Traceback syslog below does not signify a crash
May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec, Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C 0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413 *First introduced in Cisco ASA Version 7.0(1). Currently not Available in FWSM
0x08BA77ED
52
Show Traffic
The show traffic command displays the traffic received and transmitted out each interface of the firewall
ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/sec !
53
show np blocks (FWSM Only)

The show np blocks command is used to see if the FWSM is over subscribed
Data and Control packets dropped Data packets dropped
Warning
FWSM# show np blocks MAX FREE NP1 (ingress) 32768 32768 (egress) 521206 521206 NP2 (ingress) 32768 32768 (egress) 521206 521206 NP3 (ingress) 32768 32768 (egress) 521206 521206 THRESH_0 0 0 0 0 13 0 THRESH_1 0 0 0 0 460417 0 THRESH_2 550 0 92 0 4427509 0
BRKSEC-3020
Cisco Public
54
Show Xlate and Show Xlate Debug

The show xlate command displays information about the translations through the firewall You can limit the output to just the local or global IP
ASA-5585# show xlate 5014 in use, 5772 most used TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756 flags ri idle 0:00:00 timeout 0:00:30 TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464 flags ri idle 0:00:00 timeout 0:00:30 Added in version 8.3 ASA-5585# show nat pool TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1 TCP PAT pool outside, address 10.2.1.2, range 512-1023, allocated 0 TCP PAT pool outside, address 10.2.1.2, range 1024-65535, allocated 2321 ASA-5585#
BRKSEC-3020
Cisco Public
55
Show Nat Detail

The show nat command displays information about the nat table of the firewall The detail keyword will display object definitions
ASA-5585# show nat detail Manual NAT Policies (Section 1) 1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj translate_hits = 0, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16 Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24 Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static webserver-obj 14.36.103.83 translate_hits = 0, untranslate_hits = 3232 Source - Origin: 192.168.22.32/32, Translated: 14.36.103.83/32 2 (inside) to (outside) source dynamic science-obj interface translate_hits = 37723, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 14.36.103.96/16 ASA-5585/admin#
56
Show Conn and Show Conn Detail

Real Interface Names Added in 7.2(4), 8.0(4)
ASA# show conn 2 in use, 64511 most used TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags ASA# show conn detail and Timeout in 7.2(4), 2 in use, 64511 most used 8.0(4) Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN, G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i - incomplete, J - GTP, j - GTP data, K - GTP t3-response k - Skinny media, M - SMTP data, m - SIP media, n - GUP O - outbound data, P - inside back connection, q - SQL*Net data, R - outside acknowledged FIN, R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, X - inspected by service module TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431
Idle Time, Bytes Transferred
Connection Flags
detail Adds Uptime
57
ExampleConnection Build Up
Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag) The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection Finally, the inside device has sent a data packet and the connection is updated to include the O flag
1 5 3 SYN+ACK SYN Data ACK Connection Flags 42
UI UIO saA U A
Client
Inside
Outside Server
BRKSEC-3020
Cisco Public
58
ExampleConnection Teardown
Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists
3 1
FIN+ACK ACK FIN
Connection Flags 2
UfFRr UfUfFR
Inside Client Outside Server
Cisco Public
BRKSEC-3020
59
Connection FlagsQuick Reference

Outbound Connection Inbound Connection
For your reference
BRKSEC-3020
Cisco Public
60
TCP Connection Termination Reasons

If a TCP connection is built through the firewall, it will always have a teardown reason The TCP teardown syslog is logged at level six If you are having problems with connections abnormally closing, temporally increase your logging level (or move the syslog down), What does teardown reason and check the the Reset-O Termination reason mean in the
Teardown TCP connection syslog?
ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]
BRKSEC-3020
Cisco Public
61
TCP Connection Termination Reasons your For Quick Reference reference

Reason Conn-Timeout Deny Terminate Failover Primary Closed FIN Timeout Flow Closed by Inspection Flow Terminated by IPS Flow Reset by IPS Flow Terminated by TCP Intercept Invalid SYN Idle Timeout IPS Fail-Close SYN Control Description Connection Ended Because It Was Idle Longer Than the Configured Idle Timeout Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or After Half-Closed Timeout Flow Was Terminated by Inspection Feature Flow Was Terminated by IPS Flow Was Reset by IPS Flow Was Terminated by TCP Intercept SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer than the Timeout Value Flow Was Terminated Due to IPS Card Down Back Channel Initiation from Wrong Side
BRKSEC-3020
Cisco Public
62
TCP Connection Termination Reasons your For Quick Reference (Cont.) reference
Reason SYN Timeout TCP Bad Retransmission TCP Fins TCP Invalid SYN TCP Reset-I TCP Reset-O TCP Segment Partial Overlap TCP Unexpected Window Size Variation Tunnel Has Been Torn Down Unauth Deny Unknown Xlate Clear Description Force Termination After Two Minutes Awaiting Three-Way Handshake Completion Connection Terminated Because of Bad TCP Retransmission Normal Close Down Sequence Invalid TCP SYN Packet TCP Reset Was Sent From the Inside Host TCP Reset Was Sent From the Outside Host Detected a Partially Overlapping Segment Connection Terminated Due to a Variation in the TCP Window Size Flow Terminated Because Tunnel Is Down Connection Denied by URL Filtering Server Catch-All Error User Executed the Clear Xlate Command
BRKSEC-3020
Cisco Public
63
show local-host
A local-host entry is created for any IP tracked through the firewall It groups the xlates, connections, and AAA information Very useful for seeing the connections terminating on servers
ASA# show local-host ASA# show local-host detail connection tcp 50 Interface dmz: 0 active, 0 maximum active, 0 denied Interface inside: 1 active, 1 maximum active, 0 denied local host: <192.168.103.220>, TCP flow count/limit = 798/unlimited TCP embryonic count to host = 0 Add show local-host TCP intercept watermark = unlimited UDP flow count/limit = 0/unlimited detail connection Conn: arguments TCP outside:172.18.124.76/80 inside:192.168.103.220/34078, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 TCP outside:172.18.124.76/80 inside:192.168.103.220/34077, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 (output truncated)
64
show service-policy
The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them
ASA-5585/admin# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: ftp, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: h323 h225 _default_h323_map, packet 0, drop 0, reset-drop 0 tcp-proxy: bytes in buffer 0, bytes dropped 0 Inspect: h323 ras _default_h323_map, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: rtsp, packet 0, lock fail 0, drop 0, reset-drop 0 tcp-proxy: bytes in buffer 0, bytes dropped 0 Inspect: sqlnet, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: http, packet 1215927, lock fail 0, drop 0, reset-drop 0 Inspect: icmp, packet 57, lock fail 0, drop 0, reset-drop 0 ASA-5585/admin# ... Interface outside: Service-policy: VoIP Class-map: voice_marked Priority: Interface outside: aggregate drop 0, aggregate transmit 349
65
show service-policy flow

Use to determine what policies a given flow will match in the Modular Policy Framework (MPF)
ASA# show service-policy flow tcp host 10.1.9.6 host 10.8.9.3 eq 1521 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside Class-map: oracle-dcd Match: access-list oracle-traffic Access rule: permit tcp host 10.1.9.6 host 10.8.9.3 eq sqlnet Action: Input flow: set connection timeout dcd
BRKSEC-3020
Cisco Public
66
show asp drop

Packets dropped in the Accelerated Security Path (ASP) will increment a counter FWSM applies only to traffic sent to the control-point Frame drop counters are per packet, flow drops are per flow Some counters have corresponding syslogs
ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) Invalid tcp length (invalid-tcp-hdr-length) Invalid udp length (invalid-udp-length) No valid adjacency (no-adjacency) No route to host (no-route) Reverse-path verify failed (rpf-violated) Flow is denied by access rule (acl-drop) First TCP packet not SYN (tcp-not-syn) Bad TCP flags (bad-tcp-flags) TCP option list invalid (tcp-bad-option-list) TCP MSS was too large (tcp-mss-exceeded) Bad TCP Checksum (bad-tcp-cksum) *Drop Counters Are Documented in the CMD Ref, Under show asp drop
10897 9382 10 5594 1009 15 25247101 36888 67148 731 10942 893
67
Packet Capture
capture <capture-name> [access-list <acl-name>] [buffer <buf-size>] [ethernet-type <type>] [interface <if-name>] [packet-length <bytes>] [circular-buffer] [type raw-data|asp-drop|isakmp|webvpn user <username>] [match <prot> {host <sip> | <sip> <mask> | any} [eq | lt |gt <port>] {host <dip> | <dip> <mask> | any} [eq | lt | gt <port>]] [real-time [dump] [detail] [trace]] [trace [detail] [trace-count <1-1000>]]
Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later ASA 7.2(3) and 8.0(3) added a real-time option ASDM 6.0 adds a capture wizard Capture sniffs packets on an interface that match an ACL, or match line Key steps
Use the match keyword to specify what traffic to capture (implicitly bidirectional) Define the capture and bind it to an access-list and interface View the capture on the firewall, or copy it off in .pcap format
68
Packet Capture (Cont.)
See Appendix
Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface Capture buffer saved in RAM (default size 512 KB) Default is to stop capturing when buffer is full Default packet length is 1518 bytes Copy captures off via TFTP or HTTPS
Inside Capture Capture In Inside

BRKSEC-3020 2011 Cisco and/or its affiliates. All rights reserved.
Outside Capture Out Capture Outside

Cisco Public
69
Where Packets Are Captured in Packet Flow
Ingress Packets Captured
Egress Packets Captured
Packets are captured at the first and last points they can be in the flow Ingress packets are captured before any packet processing has been done on them Egress packets are captured after all processing (including L2 source MAC rewrite)
70
Capturing Packets Dropped by the ASP

Capture all packets dropped by the ASP
ASA# capture drops type asp-drop all
Capture on a specific drop reason

ASA# capture drops type asp-drop tcp-not-syn
Applies to both ASA and FWSM

ASA# capture drop type asp-drop ? acl-drop all bad-crypto bad-ipsec-natt bad-ipsec-prot bad-ipsec-udp bad-tcp-cksum bad-tcp-flags Flow is denied by configured rule All packet drop reasons Bad crypto return in packet Bad IPSEC NATT packet IPSEC not AH or ESP Bad IPSEC UDP packet Bad TCP checksum Bad TCP flags
BRKSEC-3020
Cisco Public
71
Packet Tracer: Overview

Introduced in ASA ver 7.2
ASA Only
A packet tagged with the trace option is injected into the interface, and processed in the data-plane Each action taken on the packet is recorded in the packet itself When the packet reaches the egress interface, or is dropped, it is punted to the control-plane The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configuration
BRKSEC-3020
Cisco Public
72
Packet Tracer: ASDM (Located off Tools Menu)

Define Packet
Action Matching Config Link Back to Edit Rule
Final Result
BRKSEC-3020
Cisco Public
73
Packet Tracer: Example Output

ASA# packet-tracer input inside tcp 10.1.1.2 1024 198.133.219.25 80 Phase: 1 Type: FLOW-LOOKUP Subtype: Result: ALLOW Config: Additional Information: Found no matching flow, creating a new flow Phase: 2 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group in in interface inside access-list in extended permit tcp any any eq www Additional Information: Phase: 3 Type: INSPECT Subtype: np-inspect Result: ALLOW Config: class-map match-all inspection_default match default-inspection-traffic policy-map global_policy class inspection_default inspect http service-policy global_policy global Additional Information:
74
Packet Tracer: Example Output (Cont.)

... Phase: 10 Type: NAT Subtype: Result: ALLOW Config: nat (inside) 1 10.1.1.0 255.255.255.0 Additional Information: Dynamic translate 10.1.1.2/4 to 209.165.201.3/516 using netmask 255.255.255.255 ... Phase: 15 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.165.201.1 using egress ifc outside adjacency Active next-hop mac address 000a.f331.83c0 hits 0 >>>>Packet successfully forwarded to fast path<<<<
BRKSEC-3020
Cisco Public
75
Packet Tracer: Tracing Captured Packet

Create a capture using the trace option Find the packet in the capture you want traced
ASA# 68 1: 2: 3: 4: 5: show capture inside packets captured 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: select that packet to be traced ...
Important!
.
ASA# capture inside access-list web interface inside trace
Then
S S . P .
ack ack ack Ack
ASA# show capture inside trace packet-number 4
BRKSEC-3020
Cisco Public
76
TCP Ping
New troubleshooting tool added in ASA ver 8.4.1 Why is it needed??? Consider the following
10.1.1.7
www server (209.165.200.225)

77
TCP Ping
Previously limited reachability tools: Ping and Traceroute Access to client machine? What about Attempts to validate the path but with ICMP PAT? NAT and/or
ICMP Echo Request ICMP Echo Reply ICMP Echo Request ICMP Echo Reply
10.1.1.7
www server (209.165.200.225)

78
TCP Ping
Sources TCP SYN packet with Clients IP and injects it into Clients interface of the ASA
Packet with SRC of 10.1.1.7 injected on Inside interface
Internal hosts are PATed to 198.51.100.2
TCP SYN sent to server
inside 10.1.1.7
outside www server (209.165.200.225)
ASA Datapath Validated (NAT, ACLs, etc)
Packet PATed to 198.51.100.2 on Egress
TCP SYN+ACK sent from server
BRKSEC-3020
Cisco Public
79
TCP Ping The Big Picture

Validates 2 of the 3 legs of the connection from client to server
TCP path from client side of ASA to Server through the cloud -Validated-
inside 10.1.1.7
1st Leg
2nd Leg
3rd Leg
BRKSEC-3020
Cisco Public
80
TCP Ping - Example

Specify Clients source Interface Specify Clients real IP Address
asa# ping tcp Interface: inside Target IP address: 209.165.200.225 Target IP port: 80 Specify source? [n]: y Source IP address: 10.1.1.7 Source IP port: [0] Repeat count: [5] Timeout in seconds: [2] Type escape sequence to abort. Sending 5 TCP SYN requests to 209.165.200.225 port 80 from 10.1.1.7 starting port 3465, timeout is 5 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
inside 10.1.1.7

Cisco Public
81
Agenda
BRKSEC-3020
Cisco Public
82
Case Study
Leveraging Smart Call Home
Case Study: Smart Call Home

Email CMD Output to You
Objective Send the output of a command directly to your e-mail. This is easily accomplished with SCH. Use the command:
call-home send <cmd> email <email_addr> Example: call-home send show run email userid@cisco.com
ASA Only
This will send a plain-text e-mail with the output of the command to the e-mail address specified, with the command in the subject line.
Example: Subject: CLI show run output
84

Collecting Memory Diagnostics over Time Objective Memory appears to be depleting over time on your ASA. Use SCH to collect the detailed memory output hourly, for further investigation. This is easily accomplished with SCH. Setting a snapshot alert-group to e-mail commands at a specified interval Snapshot will contain the following command:
show conn count show memory detail
BRKSEC-3020
Cisco Public
85

Example Config
service call-home call-home alert-group-config snapshot add-command show conn count add-command "show memory detail contact-email-addr user@cisco.com sender from user@cisco.com sender reply-to user@cisco.com mail-server smtp-server.cisco.com priority 1 profile SENDCMD active destination address email user@cisco.com destination preferred-msg-format long-text destination transport-method email subscribe-to-alert-group snapshot periodic hourly
BRKSEC-3020
Cisco Public
86
Case Study
Intermittent Access to Web Server
Case Study: Intermittent Access to Web Server

Problem Most external clients are not able to load companys web page
NATed to 10.1.1.50
HTTP Requests to 192.168.1.50
Internet Web Server

10.1.1.50 ASA-5510
Clients
BRKSEC-3020
Cisco Public
88
Traffic Spike
BRKSEC-3020
Cisco Public
89

show perfmon indicates high number of embryonic connections
ASA-5510# show perfmon PERFMON STATS: Xlates Connections TCP Conns UDP Conns URL Access URL Server Req TCP Fixup TCP Intercept Established Conns TCP Intercept Attempts TCP Embryonic Conns Timeout HTTP Fixup FTP Fixup AAA Authen AAA Author AAA Account VALID CONNS RATE in TCP INTERCEPT: Current 0/s 2059/s 2059/s 0/s 0/s 0/s 0/s 0/s 0/s 1092/s 0/s 0/s 0/s 0/s 0/s Current N/A Average 0/s 299/s 299/s 0/s 0/s 0/s 0/s 0/s 0/s 4/s 0/s 0/s 0/s 0/s 0/s Average 95.00%
BRKSEC-3020
Cisco Public
90

Issue show conn to see who is creating the connections
Random Sources Embryonic Conns
ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 13.50.2.216:3496 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 99.92.72.60:47733 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 30.34.246.202:20773 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 95.108.110.131:26224 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 76.181.105.229:21247 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aB TCP outside 82.210.233.230:44115 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 134.195.170.77:28138 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB TCP outside 70.133.128.41:22257 inside 10.1.1.50:80, idle 0:00:15, bytes 0, flags aB TCP outside 124.82.133.172:27391 inside 10.1.1.50:80, idle 0:00:27, bytes 0, flags aB TCP outside 26.147.236.181:37784 inside 10.1.1.50:80, idle 0:00:07, bytes 0, flags aB TCP outside 98.137.7.39:20591 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 37.27.115.122:24542 inside 10.1.1.50:80, idle 0:00:12, bytes 0, flags aB . . .
91
Traffic Permitted Connection Count Jumps
SYN Flood Detected
BRKSEC-3020
Cisco Public
92

Apply TCP Intercept to stop the SYN flood attack access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 ! service-policy interface_policy interface outside
BRKSEC-3020
Cisco Public
93
Why did the Connection countFew Clients Represent Intercept drop after TCP TCP Intercept 50+ was applied? % of Traffic Applied
BRKSEC-3020
Cisco Public
94

Apply per-client-max option to limit the number of connections any single client can establish
access-list 140 extended permit tcp any host 192.168.1.50 eq www ! class-map protect description Protect web server from attacks match access-list 140 ! policy-map interface_policy class protect set connection embryonic-conn-max 100 per-client-max 25 ! service-policy interface_policy interface outside
BRKSEC-3020
Cisco Public
95

per-client-max
TCP Intercept
BRKSEC-3020
Cisco Public
96
Attacks Being Mitigated
Attacks Still Occurring

97
Agenda
BRKSEC-3020
Cisco Public
98
Online Resources
Support Communities - Supportforums.cisco.com TAC Security Show Podcast Online learning modules (VoD Training) Security RSS Feeds
BRKSEC-3020
Cisco Public
99
Supportforums.cisco.com
Public wiki anyone can author articles Combines supportwiki and Netpro forums Sections for: ASA, FWSM and PIX Hundreds of Sample Configs Troubleshooting Docs FAQs
http://supportforums.cisco.com/
100
TAC Security Podcast

Great way to obtain valuable troubleshooting insights. Conversational shows, which focus on providing in-depth information on a given feature. New episodes posted Monthly http://www.cisco.com/go/tacsecuritypodcast/
BRKSEC-3020
Cisco Public
101
TAC Security Podcast Episodes
Search iTunes for TAC Security Podcast
BRKSEC-3020
Cisco Public
102
Online Learning Modules VoD Training

Great way to learn about new features in the ASA From www.cisco.com select: Products and Services
Security Network Security (expand) Cisco ASA 5500 Series Training resources Online learning modules
Search cisco.com for ASA Online Learning Modules Direct link

http://www.cisco.com/en/US/partner/products/ps6120/tsd_ products_support_online_learning_modules_list.html
BRKSEC-3020
Cisco Public
103
Security Hot Issues RSS Feeds

Subscribe with an RSS reader Receive weekly updates on the Hot Issues customers are facing Separate feeds for: ASA, FWSM, ASDM
https://supportforums.cisco.com/docs/DOC-5727
BRKSEC-3020
Cisco Public
104
Agenda
BRKSEC-3020
Cisco Public
105
Cisco ASA/FWSM Best Practices

Enable ip verify reverse-path on all interfaces Set embryonic and maximum connection counts on static and nat statements; for 7.2.1+ use per-client-max Configure logging to syslog server Move messages you want to see to lower levels, instead of raising logging levels and capturing messages you dont want to see Disable telnet access! Use SSH for management access Enable authentication for management access (console/ SSH/telnet/enable); use TACACS+ or RADIUS with LOCAL as the fallback
BRKSEC-3020
Cisco Public
106
Cisco ASA/FWSM Best Practices

Restrict DMZ access inbound to your internal networks Baseline CPU load, connection counts, xlate counts, and traffic (per interface) Monitor stats using MRTG or other snmp graphing tools. Keep config archives (and show tech ouputs) (smart call home) Run the latest maintenance release in your train Upgrade major feature trains only when you need new features, or after train has matured
BRKSEC-3020
Cisco Public
107
ASA Software Trains

7.0 7.0.2 7.0.1 7.0.4 7.0.5 7.0.6 7.0.7 7.0.8 7.1.1 7.1.2 7.1 7.2.1 7.2.2 7.2 8.0 8.1 8.2 8.3 8.4 8.5 8.0.2 8.0.3 8.0.4 8.1.1 8.1.2 7.2.3 7.2.4
Bug Fixes Waterfall Down
EOL EOL
7.2.5
8.0.5
ASA-5580 only
EOL
8.2.1 8.2.2 8.2.3 8.2.4 8.2.5
8.3.1 8.3.2 8.4.1 8.4.2 8.5.1
ASA-SM only
BRKSEC-3020
Cisco Public
108
Q&A
Visit the Cisco Store for Related Titles http://theciscostores.com
BRKSEC-3020
Cisco Public
110
1 1
Complete Your Online Session Evaluation

Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Dont forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and ondemand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
BRKSEC-3020
Cisco Public
111
1 1
Thank you.
BRKSEC-3020
Cisco Public
112
Appendix
Lucky You This appendix contains extra information which you may find useful, but I just didnt have enough time to cover in the lecture or which was covered in previous years. Enjoy :-)
BRKSEC-3020
Cisco Public
113
Appendix
ASA 8.3 Memory Requirements SNMP OIDs to Monitor Example: Show Output Filters Code Base History Case studies
Poor Voice Quality Out-of-order packet buffering TCP MSS issue Out of memory High CPU Capture Example
FWSM Additional Architecture Slides Failover Extras Packet Capture Example Online Tools ASDM Information to include when opening a TAC case
2011 Cisco and/or its affiliates. All rights reserved. Cisco Public
BRKSEC-3020
114
Redirecting Debugs to Syslog

Problem
Log only debug output to syslog
Solution
Create a logging list with only syslog ID 711001
ASA(config)# logging list Networkers message 711001 .
Enable debug output to syslogs

ASA(config)# logging debug-trace INFO: 'logging debug-trace' is enabled. All debug messages are currently being redirected to syslog:711001 and will not appear in any monitor session Log on the logging list ASA(config)# logging trap Networkers
.
115
ASA 8.3 Memory Requirements

ASA Models 5505 5540 Require Memory Upgrades before upgrading to ASA version 8.3 New ASAs ship with the upgraded RAM installed
ASA Model 5505 * 5510 5520 5540
Original Default RAM 256 MB 256 MB 512 MB 1024 MB
Required RAM for version 8.3 512 MB 1024 MB 2048 MB 2048 MB
Upgrade Kit Part Number ASA5505-MEM-512= ASA5510-MEM-1GB= ASA5520-MEM-2GB= ASA5540-MEM-2GB=
* For the 5505, only the Security Plus or Unlimited licenses require the memory upgrade
116
SNMP OIDs
CPU usage 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 (5 sec) 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 (1 min) 1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 (5 min) Connections 1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.6 (Current total) 1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.7 (Max total) Traffic 1.3.6.1.2.1.2.2.1.{10|16}.n (in/out octets) Use SNMPwalk to verify the interfaces!
For your reference
BRKSEC-3020
Cisco Public
117
Example: Show Output Filters

show <cmd> | begin|include|exclude|grep [-v] <regular_exp>
Examples Display the interface stats starting with the inside interface
show interface | begin inside
Display the access-list entries that contain address 10.1.1.5

show access-list | grep 10.1.1.5
Display the config, except for the access-lists

show run | exclude access-list
Display only access-list entries that have non-zero hitcounts

show access-list | grep v hitcnt=0
Display a count of the number of connections each host has

show local-host | include host|count/limit
Note: You must Include a Space on Either Side of the Pipe for the Command to Be Accepted; Also, Trailing Spaces Are Counted
118
Cisco PIX/ASA/FWSM Code Base History
PIX
6.0(1)
Feature Releases 6.1(1) 6.2(1) 6.3(1)
In Sync
7.0(1) 7.1(1)
PIX/ASA
7.2(1) 8.0(2) 8.3(1)
Port Features Bug Fixes
FWSM
1.1(1) 2.2(1) Feature Releases 1.1(2) 1.1(3) 2.3(1) 2.3(2) 3.1(1) 3.2(1) 4.0(1) 4.0(2) 4.1(1)
SafeHarbor
4.0(4)
4.0(11)
Maintenance Releases
SafeHarbor
3.2(2)
SafeHarbor
3.2(4)
GD
3.2(17)
3.1(2) Time
3.1(6)
3.1(10)
3.1(17)
119
Case Study
Poor Voice Quality
Presentation_ID
Cisco Public
120
Case Study: Poor Voice Quality

Problem Poor outbound voice quality at SOHO sites
Outbound RTP Stream
100 Mbps ASA-5505
100 Mbps
Cable Modem
2 Mbps
WAN
BRKSEC-3020
Cisco Public
121

Solution: Traffic Shaping What is traffic shaping, and why is it needed here? Why wont policing work? Why wont priority queuing alone work?
Shape to 2 Mbps
Cable Modem 100 Mbps ASA-5505 100 Mbps
WAN
2 Mbps
BRKSEC-3020
Cisco Public
122

Configuration Example (Traffic Shaping)
Solution Prioritize voice traffic and shape all traffic down to 2 Mbps on the outside interface.
class-map voice-traffic ! match dscp af13 ef! !! policy-map qos_class_policy ! class voice-traffic ! priority! !! policy-map qos_outside_policy ! class class-default ! shape average 2000000 ! service-policy qos_class_policy! !! service-policy qos_outside_policy interface outside!
To view statistics on the operation of the shaper, use the command show service-policy shape
123

Things to Keep in Mind: Shaping can only be applied to the class class-default Shaping only works in the outbound direction on an interface The shaping value is in bits per second, and must be a multiple of 8000 The shaping policy is applied to all sub-interfaces on a physical interface Not supported on the ASA-5580 platform Not supported in Transparent or Multi-context mode
BRKSEC-3020
Cisco Public
124
Case Study
Out-of-Order Packet Buffering
Presentation_ID
Cisco Public
125
Case Study: Out-of-Order Packets

Inspections require ordered packets Packets sent to the SSM (AIP and CSC) require ordered packets Cisco ASA/PIX will buffer up to three packets by default Buffering can be increased on ASA by using the queue-limit option under the tcp-map
BRKSEC-3020
Cisco Public
126
Case Study: Out-of-Order Packets

Problem
Some networks have high numbers of out-of-order packets; often caused by asymmetric traffic flows If the out-of-order packet buffer isnt large enough, traffic is dropped and packets must be retransmitted
192.168.1.30 Client
Inside
Outside Server
10.16.9.2
Dropped on Network Buffer Dropped by Firewall
Packet 10 Packet 11 Packet 12 Packet 13 Packet 14 Packet 15
BRKSEC-3020
Cisco Public
127
Case Study: Out-of-Order Packet Buffering Example

How to detect?
ASA# show asp drop Frame drop: ... TCP packet SEQ past window TCP packet buffer full ...
46331 90943
How to fix?
access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0 ! tcp-map OOO-Buffer queue-limit 6 ! class-map tcp-options match access-list OOB-nets ! policy-map global_policy class tcp-options set connection advanced-options OOO-Buffer ! service-policy global_policy global
128
Case Study: Out-of-Order Packet Buffering Example

How to verify?
ASA# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default ... Class-map: tcp-options Set connection policy: Set connection advanced-options: OOB-Buffer Retransmission drops: 0 TCP checksum drops : 0 Exceeded MSS drops : 0 SYN with data drops: 0 Out-of-order packets: 2340 No buffer drops : 0
BRKSEC-3020
Cisco Public
129
Case Study
TCP MSS (Maximum Segment Size)
Presentation_ID
Cisco Public
130
Case Study: TCP MSS

MSS is the Maximum Segment Sizeor the maximum amount of data that can be sent in a single packet The MSS is set in the SYN packets The device that receives the MSS advertisement cannot send more data in a single packet to the peer than specified by the MSS
BRKSEC-3020
Cisco Public
131
Case Study: TCP MSS

Problem
Some servers have broken TCP stacks and ignore the MSS advertised by the Client The firewall will drop packets that exceed the advertised MSS
192.168.1.30 Client
Inside
Outside Server
10.16.9.2
SYN
MSS=1380
SYN+ACK MSS=1400
DATA=1390
132
Case Study: TCP MSS Example

How to detect?
ASA# show asp drop Frame drop: TCP MSS was too large 943
%ASA-4-419001: Dropping TCP packet from outside:10.16.9.2/80 to inside:192.168.1.30/1025, reason: MSS exceeded, MSS 1380, data 1390
How to fix?
access-list MSS-hosts permit tcp any host 10.16.9.2 ! tcp-map mss-map exceed-mss allow ! class-map mss match access-list MSS-hosts ! policy-map global_policy class mss set connection advanced-options mss-map ! service-policy global_policy global
133
Case Study: TCP MSS Example

How to verify?
ASA# capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518 ASA# show capture mss-capture 0 packets captured 0 packets shown
How else could you verify?
BRKSEC-3020
Cisco Public
134
Case Study
Out of Memory
Presentation_ID
Cisco Public
135
Case Study: Out of Memory

Problem
Users are unable to access the Internet No new connections are working All old (long lived) connections continue to work Step 1: Check the Syslogs
%PIX-3-211001: Memory allocation Error %PIX-3-211001: Memory allocation Error
Step 2: Check the Amount of Free Memory Available

Hardware: PIX-515E, 64 MB RAM pixfirewall# show memory Free memory: 714696 bytes Used memory: 66394168 bytes ---------------------------Total memory: 67108864 bytes
136

Step 3: What Eats Up Memory (RAM) on the Cisco PIX?
Cisco PIX image (run from RAM) Configuration IPSec database Xlates (translations) Connections
What Can Eat Up 64 MB on a Cisco PIX-515E?
Step 4: Lets Check the Translations
pixfirewall# show xlate 251 in use, 258 most used PAT Global 209.165.201.26(2379) Local 10.1.1.132(52716) PAT Global 209.165.201.26(2378) Local 10.1.1.227(20276) Global 209.165.201.25 Local 10.1.1.102 PAT Global 209.165.201.26(2255) Local 10.1.1.125(12783) PAT Global 209.165.201.26(2382) Local 10.1.1.175(39197) PAT Global 209.165.201.26(2254) Local 10.1.1.34(43543)
A Small Global Pool Is Used, Overloading to a PAT Address
Varied Source IPs

137

Step 5: Check the Connections
pixfirewall# show conn 147456 in use, 147456 most used TCP out 64.102.144.194:80 in 10.1.1.38:26749 idle 0:00:19 Bytes 312 flags OIU TCP out 64.101.22.236:80 in 10.1.1.74:32209 idle 0:00:14 Bytes 239 flags OIU TCP out 64.102.147.77:21 in 10.1.1.48:32893 idle 0:00:48 Bytes 0 flags saA TCP out 64.103.31.215:80 in 10.1.1.136:18664 idle 0:00:46 Bytes 934 flags OIU TCP out 64.101.19.69:80 in 10.1.1.235:46712 idle 0:00:17 Bytes 8394 flags OIU TCP out 64.101.205.10:135 in 10.1.1.139:62296 idle 0:00:15 Bytes 0 flags saA TCP out 64.101.200.200:80 in 10.1.1.83:51864 idle 0:00:32 Bytes 902 flags OIU TCP out 64.102.80.27:80 in 10.1.1.66:52301 idle 0:00:03 Bytes 7813 flags OIU TCP out 64.103.95.35:80 in 10.1.1.231:51532 idle 0:00:24 Bytes 3891 flags OIU TCP out 64.102.206.172:80 in 10.1.1.223:28585 idle 0:00:28 Bytes 239 flags OIU TCP out 64.102.57.106:80 in 10.1.1.135:44945 idle 0:00:48 Bytes 9717 flags OIU TCP out 64.102.21.85:80 in 10.1.1.20:19578 idle 0:00:06 Bytes 2348 flags OIU TCP out 64.101.25.203:80 in 10.1.1.170:28149 idle 0:00:47 Bytes 419 flags OIU TCP out 64.101.86.97:135 in 10.1.1.54:43703 idle 0:00:12 Bytes 0 flags saA . . Q: Why is the connection count so high? .
BRKSEC-3020
Cisco Public
138

Take a Look at the Traffic Load
pixfirewall# show traffic outside: received (in 25.000 secs): 1475 packets 469050 bytes 59 pkts/sec 18762 bytes/sec transmitted (in 25.000 secs): 167619 packets 9654480 bytes 6704 pkts/sec 386179 bytes/sec inside: received (in 25.000 secs): 180224 packets 10410480 bytes 7208 pkts/sec 416419 bytes/sec transmitted (in 25.000 secs): 1050 packets 118650 bytes 42 pkts/sec 4746 bytes/sec
Traffic Flow
Vast majority of traffic is coming in the inside interface and going out the outside interface
Inside
Outside
139

Step 6: Review What We Know and Take Action
pixfirewall# show conn count 147456 in use, 147456 most used pixfirewall# show xlate count 251 in use, 258 most used
Conn Count Is Very High, but xlate Count Is Low Many connections per xlate Probably one, or a few hosts, are generating the vast majority of connections Most likely due to a virus on the host(s)
BRKSEC-3020
Cisco Public
140

Step 7: Find the Host(s) Generating All the Connections
pixfirewall# show local-host | local host: <10.1.1.131>, TCP connection count/limit UDP connection count/limit local host: <10.1.1.51>, TCP connection count/limit UDP connection count/limit local host: <10.1.1.236>, TCP connection count/limit UDP connection count/limit . . . local host: <10.1.1.99>, TCP connection count/limit UDP connection count/limit include host|count/limit = 0/unlimited = 0/unlimited = 2/unlimited = 0/unlimited = 0/unlimited = 0/unlimited
Only Show Lines That Have the Word host or count/limit in Them
= 146608/unlimited = 0/unlimited
Host 10.1.1.99 is eating up all the connections, and they are TCP-based connections
BRKSEC-3020
Cisco Public
141

Step 8: Now that We Found the Host, Lets Look at the Connections It Is Generating
pixfirewall# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, TCP connection count/limit = 146608/unlimited TCP embryonic count = 146606 Note: All Connections UDP connection count/limit = 0/unlimited Are Embryonic Xlate(s): Global 209.165.201.21 Local 10.1.1.99 Conn(s): TCP out 64.101.32.157:135 in 10.1.1.99:34580 idle 0:01:43 Bytes 0 flags TCP out 64.103.108.191:135 in 10.1.1.99:8688 idle 0:01:43 Bytes 0 flags TCP out 64.100.205.160:135 in 10.1.1.99:7774 idle 0:01:43 Bytes 0 flags TCP out 64.101.182.19:135 in 10.1.1.99:39193 idle 0:01:43 Bytes 0 flags Connections to Random TCP out 64.102.218.45:135 in 10.1.1.99:16462 idle 0:01:43 Bytes 0 flags Destinations on TCP/135 TCP out 64.100.21.120:135 in 10.1.1.99:30322 idle 0:01:43 Bytes 0 flags MS Blaster TCP out 64.101.25.195:135 in 10.1.1.99:41116 idle 0:01:43 Bytes 0 flags TCP out 64.103.17.219:135 in 10.1.1.99:59163 idle 0:01:43 Bytes 0 flags TCP out 64.102.201.141:135 in 10.1.1.99:2978 idle 0:01:43 Bytes 0 flags TCP out 64.103.176.75:135 in 10.1.1.99:41589 idle 0:01:43 Bytes 0 flags . . .
saA saA saA saA saA saA saA saA saA saA
142

Cisco PIX provides two methods to limit the number of connections per host
TCP intercept Max connections
Question: Which help because the source address is valid TCP intercept wont One Can Be Used Here? Limiting the maximum number of connections each internal host can have is the only option
BRKSEC-3020
Cisco Public
143

Step 9: Limit Infected Host(s) Impact on Network Configure the MAX TCP connections for NATed hosts to be 50
pixfirewall(config)# nat (inside) 1 0.0.0.0 0.0.0.0 50 0
Note: the local-host must be cleared before the new connection limits are applied
pixfirewall(config)# clear local-host 10.1.1.99 pixfirewall(config)# show local-host 10.1.1.99 Interface inside: 250 active, 250 maximum active, 0 denied local host: <10.1.1.99>, The Infected Host TCP connection count/limit = 50/50 Is Limited to 50 TCP embryonic count = 50 TCP intercept watermark = unlimited TCP Connections UDP connection count/limit = 0/unlimited . . .
BRKSEC-3020
Cisco Public
144

Take One Last Look at the Memory and Connection Counts After Applying the TCP Connection Limit
pixfirewall# show conn count 126 in use, 147456 most used pixfirewall# show memory Free memory: 47716152 bytes Used memory: 19392712 bytes ---------------------------Total memory: 67108864 bytes
Things look much better now Question: How could we configure the Cisco PIX so the connection limit was only applied to the one host (10.1.1.99) which was infected with the virus?
nat (inside) 1 10.1.1.99 255.255.255.255 50 0 nat (inside) 1 0.0.0.0 0.0.0.0 0 0
BRKSEC-3020
Cisco Public
145
Case Study
High CPU Usage
Presentation_ID
Cisco Public
146
High CPU Usage on the Cisco PIX

Problem: Cisco PIX CPU Running Very High
A quick overview of the show processes command

Number of msec This Process Has Been on the CPU pixfirewall(config)# show processes Hsi Lsi Lwe Lwe Crd Lsi PC 001eab19 001f00bd 00119abf 003e4425 001e26fb 00300a29 SP 008a5a74 00a28dbc 02d280dc 02d2a26c 0533940c 04c0f504 STATE 00557910 00557910 0055b070 00557dd8 00557d88 00557910 Runtime 0 0 0 74440 6070290 0 SBASE 008a4aec 00a27e44 02d27274 02d28324 05338484 04c0e57c Stack 3628/4096 3832/4096 3688/4096 6936/8192 3684/4096 3944/4096 Process arp_timer FragDBGC dbgtrace Logger 557poll xlate clean The Name of the Process
For more Information on the Output of the show processes Command, See
http://www.cisco.com/en/US/partner/products/hw/vpndevc/ps2030/products_tech_note09186a008009456c.shtml
147

Step 1: Determine What Process Is Eating the CPU Take the difference in output of two show processes over a period of time The following output was a diff of the processes taken one minute apart
Process_Name Logger pix/intf3 557poll i82543_timer i82542_timer
Runtime (msec) 25940 18410 9250 4180 2230
In One Minute, These Processes Account for 44 Seconds of CPU Time ~ 73% The Interface Polling Processes Always Run, and Are not Counted in the CPU Usage
BRKSEC-3020
Cisco Public
148

Step 2: Focus on the Processes with High CPU Time Logging is taking up much of the CPU; lets review what we have configured to log
This Is Cumulative pixfirewall(config)# show log Since the Cisco PIX Syslog logging: enabled Standby logging: disabled Was Last Rebooted Console logging: disabled Monitor logging: disabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 5919412 messages logged Logging to lab 172.18.173.123 History logging: disabled Notice the Change . . . Over a Few Minutes pixfirewall(config)# show log Syslog logging: enabled Buffer logging: level alerts, 0 messages logged Trap logging: level warnings, 6172472 messages logged Logging to lab 172.18.173.123
149

Syslog Server Is Controlled by a Different Group
Enable buffered logging to same level as syslog server, and examine the buffered messages
pixfirewall(config)# show log Buffer logging: level warnings, 31527 messages logged Trap logging: level warnings, 6453127 messages logged Logging to lab 172.18.173.123
. . . 400011: 400011: 400011: 400011: 400011: 400011: IDS:2001 IDS:2001 IDS:2001 IDS:2001 IDS:2001 IDS:2001 ICMP ICMP ICMP ICMP ICMP ICMP unreachable unreachable unreachable unreachable unreachable unreachable from from from from from from 172.18.173.123 172.18.173.123 172.18.173.123 172.18.173.123 172.18.173.123 172.18.173.123 to to to to to to
Cisco PIXs Interface Address
14.36.1.88 14.36.1.88 14.36.1.88 14.36.1.88 14.36.1.88 14.36.1.88
on on on on on on
interface interface interface interface interface interface
lab lab lab lab lab lab
BRKSEC-3020
Cisco Public
150

Examine IDS Configuration
pixfirewall(config)# show run | grep audit ip audit name IDS info action alarm ip audit interface lab IDS
Syslog service was down on the syslog server ICMP unreachable was generated by syslog server for each syslog message the Cisco PIX sent it Cisco PIXs IDS configuration also logged every ICMP unreachable message, creating the exponentially increasing problem
Syslog Server
Lab
Outside Syslog Message
ICMP Unreachable IDS Syslog Message
BRKSEC-3020
Cisco Public
151

Solution
Bring back up syslog service on server Take server offline Configure Cisco PIX to not log IDS ICMP unreachable messages
ip audit signature 2001 disable or no logging message 400011
pixfirewall# show run | grep signature ip audit signature 2001 disable pixfirewall# show cpu usage CPU utilization for 5 seconds = 2%; 1 minute: 50%; 5 minutes: 99%
BRKSEC-3020
Cisco Public
152

Summary
Examine the DIFF of two show processes taken over a one minute interval Find the process taking up the highest amount of CPU (excluding the polling processes) Take actions to lower that processs CPU time Reexamine the CPU output, and repeat as necessary
BRKSEC-3020
Cisco Public
153
FWSM
Additional architecture information
BRKSEC-3020
Cisco Public
154
FWSM Syslog Level vs. Number of Messages
Log Level 0 1 2 3 4 5 6 7
Number of Messages (SUM) Description Ver. 2.3 Emergencies Alerts Critical Errors Warnings Notifications 0 58 (58) 21 (79) 94 (173) 131 (304) 26 (330) Ver. 3.1 0 67 (67) 29 (96) 305 (401) 194 (595) 167 (762) 245 (1007) 225 (1232) Ver. 3.2 0 67 (67) 29 (96) 306 (402) 196 (598) 169 (767) 248 (1015) 225 (1240) Ver. 4.0 0 67 (67) 29 (96) 318 (414) 199 (613) 178 (791) 255 (1046) 226 (1272) Ver. 4.1 0 67 (67) 29 (96) 318 (414) 199 (613) 178 (791) 259 (1050) 231 (1281)
Informational 116 (446) Debugging 23 (469)
BRKSEC-3020
Cisco Public
155
FWSM and ACLs

ACLs on the FWSM are compiled on the control point and pushed down into hardware (NP 3) During compile time, CPU should stay at ~ 99%
ACL compile uses all free CPU cycles Allows compile to complete in shortest time possible
Once compile is complete, rules are attempted to be pushed into hardware

Successful download
Access Rules Download Complete: Memory Utilization: 49%
Failed download (exceeded HW memory)

ERROR: Unable to add, access-list config limit reached
BRKSEC-3020
Cisco Public
156
FWSM and ACLs (Multimode)

Use show np 3 acl stats to see the current ACL resource utilization in that context
FWSM/admin(config)# show np 3 acl stats ---------------------------ACL Tree Statistics ---------------------------Rule count : 9584 Bit nodes (PSCB's): 8760 Leaf nodes : 8761 Total nodes : 17521 (max 24260) Leaf chains : 6912 Total stored rules: 15673 Max rules in leaf : 3 Node depth : 32 ---------------------------Note: One ACE Does not Equal One Node
Total Number of ACEs This Is the Hardware Limit
157

Use show np 3 acl tree to see which ACL tree a context is mapped to
ACL Tree Number
FWSM# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map -------------------------------------------Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 3 Context (004) Accounting -------------------------------------------Context Name
BRKSEC-3020
Cisco Public
158
FWSMACL Rule Limits

FWSM 2.3 introduced
resource acl-partitionset the number of ACL partitions allocate-acl-partitionassigns a context to a specific partition
FWSM 3.2 introduced

resource-ruleallows further customization of a partition
FWSM 4.0 introduced

resource partitioncustomize the size of individual partitions access-list optimization enablemerges and/or deletes redundant and conflicting ACEs without affecting the policy
BRKSEC-3020
Cisco Public
159

Use the command resource acl-partition <num-ofpartitions> to reduce the number of active partitions created; default is 12 Use the command allocate-acl-partition <num> to assign a context to a specific ACL tree
FWSM(config)# context Accounting FWSM(config-context)# allocate-acl-partition 0 FWSM(config-context)# show np 3 acl tree -------------------------------------------ACL Tree Instance <-> Context Name (ID) Map Both Use -------------------------------------------- Tree 0 Tree Instance 0 Context (001) admin Tree Instance 1 Context (002) core Tree Instance 2 Context (003) Engineering Tree Instance 0 Context (004) Accounting -------------------------------------------BRKSEC-3020 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public
160
FWSMResource Rule
FWSM 3.2 introduced
resource-ruleallows further customization of a partition
resource rule nat 10000 acl 2200 filter 400 fixup 595 est 70 aaa 555 console 283
show resource-ruledisplays information about the current rule allocation

FWSM# show resource rule Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 1843 1843 10000 ACL 74188 74188 74188 Filter 2764 2764 5528 Fixup 4147 4147 10000 Est Ctl 460 460 460 Est Data 460 460 460 AAA 6451 6451 10000 Console 1843 1843 3686 -----------+---------+----------+--------Total 92156 92156 Partition Limit - Configured Limit = Available to allocate 92156 92156 = 0
161
FWSMResource Partition
FWSM 4.0 introduced
resource partitionallows customization of the size of individual partitions (multi-context mode)
FWSM(config)# resource partition 10 FWSM(config-partition)# size 1000 WARNING: The rule max has been reset based on partition size 1000. The <size> command leads to re-partitioning of ACL Memory. It will not take effect until you save the configuration and reboot.
Before
FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 384 384 833 ACL 14801 14801 14801 Filter 576 576 1152 Fixup 1537 1537 3074 Est Ctl 96 96 96 Est Data 96 96 96 AAA 1345 1345 2690 Console 384 384 768 -----------+---------+----------+--------Total 19219 19219 Partition Limit - Configured Limit = Available to allocate 19219 19219 = 0
After
FWSM# show resource rule partition 10 Default Configured Absolute CLS Rule Limit Limit Max -----------+---------+----------+--------Policy NAT 20 20 43 ACL 770 770 770 Filter 30 30 60 Fixup 80 80 160 Est Ctl 5 5 5 Est Data 5 5 5 AAA 70 70 140 Console 20 20 40 -----------+---------+----------+--------Total 1000 1000 Partition Limit - Configured Limit = Available to allocate 1000 1000 = 0
Cisco Public
162
FWSM and Control Point

The traffic that makes it to the control point is traffic that requires Layer 7 fixup (embedded NAT, or cmd inspection)
FTP VoIP (SIP/SKINNY/H.323/RTSP) DNS XDMCP, etc. Session Manager NP 3
Traffic sourced from, or destined to, the FWSM also goes through the control point
Syslogs AAA (RADIUS/TACACS+) Management traffic (telnet/SSH/ HTTPS/SNMP) Failover communications Routing protocols (OSPF/ RIP) etc.
Fast Path Fast Path NP 1 NP 2 FWSM
URL filtering (WebSense/N2H2)
163
FWSM and Network Processors

The session managerNP 3
Processes first packet in a flow ACL checks Translation creation Embryonic/established connection counts TCP/UDP checksums Sequence number randomization TCP intercept etc.
Control Point (CP) Central CPU Session Manager NP 3 Fast Path NP 1 FWSM Fast Path NP 2
The fast pathNP 1 and 2

Performs per packet session lookup Maintains connection table Performs NAT/PAT TCP checks Fragmentation reassembly etc.
Session Manager NP 3 Fast Path NP 1 FWSM Fast Path NP 2

164
Cisco Public
FWSMEnabling the Completion Unit

Due to the FWSMs NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the firewall
4 3 2 1 4 2 3 1
This issue might be encountered when performing TCP throughput testing, or passing high speed TCP flows through the FWSM
Examples: CIFS, FTP, AFP, backups
FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order (by enabling a hardware knob on the NPs called the completion unit) In multiple mode enter this command in the admin context configuration; It will then be enabled for all contexts on the firewall
165
Case Study
Advanced Syslog Analysis
Case Study: Advanced Syslog Analysis

Problem Find Services which are permitted through the firewall, yet the servers no longer exist
Get a fast Linux/Solaris machine with a decent amount of memory Learn to use the following commands: cat grep, egrep, fgrep cut awk (basic) sort uniq Perl (advanced manipulation) Pipe the commands to construct the necessary outputs!
167

Interesting syslogs appear as follows:
Syslog ID Destination
May 24 2010 23:19:53: %ASA-6-302014: Teardown TCP connection 1019934 for outside:203.0.113.126/6243 to inside:10.100.19.190/21 duration 0:00:30 bytes 0 SYN Timeout
Reason
BRKSEC-3020
Cisco Public
168

Results:
syslogserver-sun% grep 302014 syslog.txt | grep "SYN Timeout" | awk '{print $13}' | uniq -c | sort -r -n 673 451 392 358 119 inside:10.100.19.190/21 dmz:192.168.5.13/80 dmz:192.168.5.11/443 inside:10.0.0.67/1521 inside:10.0.1.142/80
grep used to find the syslogs we want awk used to print the destination column (IP/port) uniq used to print only unique entries, with a count sort used to display ordered list, highest count first
169
Case Study
FWSM Slow Single-Flow TCP Throughput
Move this case-study to the appendix
Presentation_ID
Cisco Public
170
Case Study: FWSM Slow TCP Throughput

Problem TCP based backups are taking longer than expected through the FWSM iPerf performance testing is only showing ~450 Mbps through FWSM
BRKSEC-3020
Cisco Public
171
Case Study: FWSM Slow TCP Throughput
FWSM Only
Due to the FWSMs NP architecture, there exists a possibility that packets arriving with a low inter-packet gap might be re-ordered by the FWSM. 4 3 2 1 4 2 3 1
TCP Flow
FWSM version 3.1(10) and 3.2(5) introduce a new command sysopt np completion-unit to ensure the firewall maintains the packet order
Enable np completion-unit
TCP Flow Note: In multi-mode add command to admin context, and it will be applied globally
172
Failover
What to Do After a Failover Additional Failover Commands
BRKSEC-3020
Cisco Public
173

show failover state will provide specific details about the failure reason. This information is not saved across reboots
ASA# show failover state This host State Primary Failed Last Failure Reason Ifc Failure Inside: Failed None Date/Time 12:56:00 UTC May 6 2010
Other host -
Secondary Active
====Configuration State=== Sync Done ====Communication State=== Mac set

174

Starting with FWSM 2.3 and Cisco ASA/PIX 7.0, the reason for failover is saved in the failover history This information is not saved across reboots
ASA# show failover history ========================================================================== From State To State Reason ========================================================================== Disabled Negotiation Set by the CI config cmd Negotiation Just Active No Active unit found Just Active Active Drain No Active unit found Active Drain Active Applying Config No Active unit found Active Applying Config Active Config Applied No Active unit found Active Config Applied Active No Active unit found Active Failed Interface check ==========================================================================
BRKSEC-3020
Cisco Public
175
Other Useful Failover Commands

failover exec mate allows you to execute commands on the peer and receive the response back. failover reload-standby only valid on Active unit prompt changes the prompt to display failover priority and state.
ASA(config)# prompt hostname priority state ASA/sec/act(config)#
BRKSEC-3020
Cisco Public
176
Failover Prompt Display Configuration

The firewalls prompt maybe changed to display certain keyword Usage
prompt <keyword> [<keyword> ...]
Syntax
keywords: Hostname Configures the prompt to display the hostname Domain Configures the prompt to display the domain Context Configures the prompt to display the current context (multi-mode only) Priority Configures the prompt to display the failover lan unit setting State Configures the prompt to display the current traffic handling state Slot Configures the prompt to display the slot location (when applicable)
Example
FWSM(config)# prompt hostname domain priority state slot FWSM/cisco.com/sec/actNoFailover/4(config)#
BRKSEC-3020
Cisco Public
177
Capture Example
BRKSEC-3020
Cisco Public
178
Capture Command: Example

Problem: user on the inside with an IP of 10.1.3.2 is having a problem accessing www.cisco.com (198.133.219.25); the user is getting PATed to 192.168.2.2
Capture In Inside 10.1.3.2 Capture Out www.cisco.com Outside
Internet
198.133.219.25
10.1.3.2
192.168.2.2
Step 1: Create ACL for Both Inside and Outside Interface Step 2: Create Captures on Both Inside and Outside Interface Step 3: Have Inside User Access www.cisco.com Step 4: Copy the Captures Off to a TFTP Server Step 5: Analyze Captures with Sniffer Program
179
Capture Command: Example

Step 1: create ACL for both inside and outside interface
! Outside Capture ACL Access-list 100 permit tcp host 192.168.2.2 host 198.133.219.25 eq 80 Access-list 100 permit tcp host 198.133.219.25 eq 80 host 192.168.2.2 ! Inside Capture ACL Access-list 101 permit tcp host 10.1.3.2 host 198.133.219.25 eq 80 Access-list 101 permit tcp host 198.133.219.25 eq 80 host 10.1.3.2
Step 2: create captures on both inside and outside interface

capture out access-list 100 interface outside packet-length 1518 capture in access-list 101 interface inside packet-length 1518
Step 3: have inside user access www.cisco.com Step 4: copy the captures off to a TFTP server
! ASA ver 7.0+ / FWSM 3.0+ copy capture copy /pcap capture:out tftp://10.1.3.5/out.pcap copy /pcap capture:in tftp://10.1.3.5/in.pcap ! PIX ver 6.x / FWSM 2.3 copy capture copy capture:out tftp://10.1.3.5/out.pcap pcap copy capture:in tftp://10.1.3.5/in.pcap pcap
Or copy using https:
https://<FW_IP>/capture/out/pcap
BRKSEC-3020
Cisco Public
180
Packet Capture: Example

Step 5: analyze captures with sniffer program
Outside CAP
Inside CAP
Outbound SYN, No SYN+ACK
BRKSEC-3020
Cisco Public
181
Packet Capture: Limitations on FWSM

Capture functionality is available on the FWSM starting in 2.3
However, only packets processed by the control point could be captured Control Point (CP) Central CPU
FWSM 3.1(1) added support to capture packets in hardware

Only ingress packets were captured
Session Manager NP 3
FWSM 3.1(5) both ingress and egress transient packets can be captured which flow through hardware
Capture requires an ACL to be applied Capture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much traffic FWSM
Fast Path Fast Path NP 1 NP 1
BRKSEC-3020
Cisco Public
182
Online Tools
Networking professionals connection Bug toolkit Output Interpreter
BRKSEC-3020
Cisco Public
183
Networking Professionals Connection

Online Open Forum to Ask Questions Anyone Can Ask a Question, and Anyone Can Answer
Regular Ask the Expert Events on Certain Topics
BRKSEC-3020
Cisco Public
184
Networking Professionals Connection

Online Open Forum to Ask Questions Anyone Can Ask a Question, and Anyone Can Answer
Regular Ask the Expert Events on Certain Topics
http://www.cisco.com/go/netpro
185
Bug Toolkit
On the Support Tools and Resources Page
BRKSEC-3020
Cisco Public
186
Bug ToolkitProduct Selection
Select Security, then Cisco ASA 5500 Series
BRKSEC-3020
Cisco Public
187
Bug ToolkitAdvanced Search
Version
Search Keywords
Severity
Status
BRKSEC-3020
Cisco Public
188
Bug ToolkitSearch Results
Select Link to View Details of Bug
BRKSEC-3020
Cisco Public
189
Bug ToolkitBug Details
First Fixed-In Releases
BRKSEC-3020
Cisco Public
190
Output Interpreter
Linked off the Technical Support and Documentation Tools and Resources Section on CCO
Great Tool for Catching Configuration Errors
Paste in the show run Output and Hit submit
BRKSEC-3020
Cisco Public
191
Output Interpreter: Example Output
Warning: Unused Statics
Warning: Unapplied Crypto Map Warning: Invalid Crypto Map
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
192
ASDM
BRKSEC-3020
Cisco Public
193
ASDM
Run as a standalone application using the ADSM Launcher This allows for one-stop access to multiple firewalls ASDM 6.0 adds Upgrade Wizard to upgrade ASA and ASDM software direct from cisco.com ASDM 6.2 works with ASA 8.2, ASA 8.1 and 8.0 releases ASDM 6.1F works with FWSM 4.0, 3.2, and 3.1 releases
BRKSEC-3020
Cisco Public
194
ASDM Home Page
Device Information
CPU, Memory, Conns/Sec, Interface Traffic
Real-Time Syslogs
195
Using ASDM for Monitoring

Up to Four Different Graphs Can Be Displayed
Great for Monitoring Trends
BRKSEC-3020
Cisco Public
196
ASDM: Editing Rules from the Log Viewer
Select Log Entry from Viewer
Right-Click on Message to View or Edit Associated Rule
BRKSEC-3020
Cisco Public
197
ASDM: Syslogs Explained
BRKSEC-3020
Cisco Public
198
Opening a TAC Case

If after using all your troubleshooting tools you still cannot resolve the problem, please open a TAC case
http://www.cisco.com/techsupport/servicerequest/
At a minimum include:
Detailed problem description Output from show tech
Optionally include:
Syslogs captured during time of problem Sniffer traces from both interfaces using the capture command (capturing only the relevant packets, and saved in pcap format)
BRKSEC-3020
Cisco Public
199

BRKSEC-3020 Advanced Firewalls

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

BRKSEC-3020 Advanced Firewalls

Încărcat de

Drepturi de autor:

Formate disponibile

Advanced Firewalls

2011 Cisco and/or its affiliates. All rights reserved.

Understanding the Packet Flow

Accounting Outside Server: 198.133.219.25

2011 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Ingress Interface

Packet Processing: Locate Connection

Syslog Because of No Connection, and Non-SYN Packet:

Packet Processing: ACL Check

Syslog When Packet Is Denied by ACL:

Translation and NAT Order of Operations

For your reference

1. Manual NAT entries 2. Auto NAT entries 3. After-Auto NAT entries

2011 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Inspections/Sec Checks

Packet Processing: NAT IP Header

2011 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Egress Interface

Inbound Packets to 192.168.12.4 Get Routed to Inside Based on Order of Statics

2011 Cisco and/or its affiliates. All rights reserved.

Packet Processing: L3 Route Lookup

2011 Cisco and/or its affiliates. All rights reserved.

Packet Processing: L2 Address Lookup

2011 Cisco and/or its affiliates. All rights reserved.

Packet Processing: Transmit Packet

Packet is transmitted on wire Interface counters will increment on interface

2011 Cisco and/or its affiliates. All rights reserved.

Cisco ASA Understanding the Architecture

2011 Cisco and/or its affiliates. All rights reserved.

Maximum ACL Limits

Warning - ACE Explosion

Object-NAT (Auto-NAT) (version 8.3+)

Manual NAT (Twice NAT) (version 8.3+)

NAT Order of Operation version 8.3+

For your reference

Manual NAT Policies (Section 1)

First Match (in config)

Auto NAT Policies (Section 2)

Manual NAT [after auto] Policies (Section 3)

First Match (in config)

2011 Cisco and/or its affiliates. All rights reserved.

Real-IP (version 8.3+)

2011 Cisco and/or its affiliates. All rights reserved.

FWSMUnderstanding the Architecture

2011 Cisco and/or its affiliates. All rights reserved.

FWSM Architectural Overview

C6K Backplane Interface

2011 Cisco and/or its affiliates. All rights reserved.

3.2 / 4.0 /4.1 Configurable X X

*Complete List in FWSM Docs, Appendix A (Specifications)

FWSMACL Rule Limits

177612 combined total ACEs

Backup Tree: 100,567 (mirror of active tree)

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

DST IP VLAN 310.14.3.x CTX1 .1 10.14.3.89 Outside MSFC

Multi-Context - Common Issues on FWSM

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

Stateful LAN/Serial Secondary (Standby) Primary (Active)

Verifying Failover Configuration

What Triggers a Failover?

2011 Cisco and/or its affiliates. All rights reserved.