Check Point Troubleshooting

“Oops! It’s not working!”

Introduction
Troubleshooting is more or less the same since
years

The great „How to use fw monitor” document is from

2003 – still valid!

Some minor changes to buffer size, command line

options

New kernel modules introduced with R70 and R71,

but no information officially available

Æ We have to stick with the old stuff

How to approach troubleshooting
Collect information

What is the problem? What are the symptoms?

Can the problem be replicated?
Random occurence?
Anything changed in the setup?
User-related or machine-related?
List systems that are part of the conversation
How to approach troubleshooting
Bug or configuration problem?

Common configuration problems:

Firewall rule prevents traffic

SmartDefense / IPS blade prevents traffic
Antispoofing
misconfigured routing
wrong encryption domain
wrong username / password
How to approach troubleshooting
Any reference for problem or error message?

official documentation
SecureKnowledge
CPUG forum
Check Point forum
Google
fw ctl zdebug drop
Replicate the problem and have a look at the gateway:

fw ctl zdebug drop

lists all dropped packets in realtime

gives an explanation why the packet is dropped

fw_log_drop: Packet proto=6 81.63.88.122:2720 ->

212.1.52.64:445 dropped by
fw_handle_first_packet Reason: Rulebase drop -
rule 12;

Why is it called zdebug? Developed by Tamir Zegman.

Firewall status
Current connections?

fw tab –t connections –s

[Expert@firewallr70]# fw tab -t connections -s

HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 1 1 1

fw ctl pstat | grep Connections

[Expert@firewallr70]# fw ctl pstat | grep Connections

Concurrent Connections: 0% (1 out of 24900) - below
low watermark
ClusterXL
Status information

fw hastat
HOST NUMBER HIGH AVAILABILITY STATE MACHINE STATUS
localhost 2 stand-by OK

cphaprob state
Cluster Mode: New High Availability (Primary Up)
Number Unique Address Assigned Load State
1 192.168.55.202 100% Active
2 (local) 192.168.55.201 0% Standby
ClusterXL
Displays ClusterXL Devices

cphaprob –ia list

Displays physical and cluster interfaces

cphaprob –a if

Statistics of ClusterXL sync

fw ctl pstat
cphaprob syncstat

Reset statistics of ClusterXL sync

cphaprob –reset syncstat

Licenses
Limited number of hosts?

fw lichosts

Count of used hosts

fw lichosts | wc –l

SecureClient licenses used

dtps lic
Licenses
Show license

cplic print

Compare to SmartUpdate / SmartView Monitor output

Especially UTM products sometimes tend to mess up

with licenses which can cause Antivirus, Antispam
or URL filtering to stop working

You need to keep contracts updated!

Use evaluation licenses for testing!

Content scanning
Verify update process of Antivirus or URL filtering
using avsu_client command

avsu_client –app „URL Filtering“

fetch_remote –fi

for fetching the index file (signatures up-to-date?)

avsu_client –app „URL Filtering“

fetch_remote –fe <email@domain.tld>
<password>

for fetching entitlement / signatures

fw monitor
What is it?

fw monitor command triggers a Check Point

kernel module that is used to capture packets.

What makes it different?

Packet capture at multiple positions within the kernel

module chain, both for inbound and outbound
packets. It doesn‘t work on Layer-2, so no MAC
addresses are shown in the output.

fw monitor is available on all platforms.

fw monitor
What makes it different?

filters packets using INSPECT code

sees packets „with the eyes of the gateway“

Shows flow of packets through the gateway

No Layer-2 information in capture files

 fw monitor

App. App.

TCP TCP

IP Routing IP

post-inbound (I) pre-outbound (o)

VM VM
pre-inbound (i) post-outbound (O)

NIC NIC
 fw monitor
[Expert@fw1]# fw monitor -e "accept (src=212.1.52.68 or
dst=212.1.52.68);"
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
eth3.7:i[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406
TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000
eth3.7:I[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406
TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000
eth0:o[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406
TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000
eth0:O[52]: 212.1.56.233 -> 212.1.52.68 (TCP) len=52 id=18406
TCP: 56661 -> 22 .S.... seq=b2f3509d ack=00000000
eth0:i[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0
TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e
eth0:I[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0
TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e
eth3.7:o[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0
TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e
eth3.7:O[52]: 212.1.52.68 -> 212.1.56.233 (TCP) len=52 id=0
TCP: 22 -> 56661 .S..A. seq=68a919c9 ack=b2f3509e
fw monitor
eth3.7:O[52]: 212.1.52.68 -> 212.1.56.233
(TCP) len=52 id=0
TCP: 22 -> 56661 .S..A. seq=68a919c9
ack=b2f3509e
fw monitor
fw monitor options overview
-u | s Shows UUID or SUUID for every packet
-i write data to STDOUT
-d | D debug / more debug output
-e <expr> filter for expression (CLI mode)
-f <file> read filter expression from file
-l <len> limit length of captured packet
-m <mask> which positions should be shown
-x print raw packet data
-o <file> write packet into file
-p|x| <pos> insert fw monitor at specific chain position
-p all insert fwmonitor between all kernel modules
-ci <count> stop capture after count incoming packets
-co <count> stop capture after count outgoing packets
fw monitor
fw monitor
Capture only ICMP packets

fw monitor -e "accept [9:1]=1;“

 fw monitor
Capture only packets from a special host

fw monitor -e "accept [12,b]=192.168.1.1;“

fw monitor
Filtering will be easier for you if you use macros.

Macros for fw monitor are defined in

$FWDIR/lib/fwmonitor.def which references
$FWDIR/lib/tcpip.def, where the actual
expression is located.

Example: filter for source IP

fwmonitor.def macro = src

tcpip.def macro = ip_src
expression = [12,b]
fw monitor
Use macros together with operators to add
complexity:

accept (src=x.x.x.x or dst=x.x.x.x)

accept ((src=x.x.x.x, dst=y.y.y.y) or

(src=y.y.y.y, dst=x.x.x.x));

accept not (sport=22 or dport=22);

accept sport=21 and not (src=x.x.x.x);

fw monitor
Use fw monitor to see if packets are translated

fw monitor -e „accept (src=212.1.56.151 or dst=212.1.56.151);“

eth0:i[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053

eth0:I[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053
eth1:o[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053
eth1:O[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=27053

fw monitor -e „accept (src=212.1.56.151 or dst=212.1.56.151);“

eth0:i[48]: 212.1.56.151 -> 195.244.116.166 (TCP) len=48 id=31171

eth0:I[48]: 212.1.56.151 -> 192.168.199.2 (TCP) len=48 id=31171
eth1:o[48]: 212.1.56.151 -> 192.168.199.2 (TCP) len=48 id=31171
eth1:O[48]: 212.1.56.151 -> 192.168.199.2 (TCP) len=48 id=31171
fw monitor
Common expressions for fw monitor

fw monitor –e „accept (src=x.x.x.x or

dst=x.x.x.x);“

fw monitor –m iO –e „accept host(x.x.x.x);“

fw monitor –e „accept ((src=x.x.x.x, dst=y.y.y.y)

or (src=y.y.y.y, dst=x.x.x.x));“

fw monitor –e „accept (ip_p=x);“

Combine with –o <file> for output into a file.

Inspect Code Generator: http://decock.org/ginspect/

fw monitor
Read complex expressions from a filter file:

fw monitor –f <filename>

If you use macros in a filter file, make sure to

include the appropriate definition file.

#include „fwmonitor.def“
accept ((sport=22 or dport=22) and not
(host(x.x.x.x));
 fw monitor

Use for better analysis of capture files.

Preferences Æ Protocols Æ Ethernet Æ Check box

Attempt to interpret as Firewall-1 monitor file
Preferences Æ Protocols Æ FW-1 Æ Activate UUID,
chain position, summary in protocol tree
Add column fw1 chain of format FW-1 monitor if/direction
Add coloring rules

preIn Æ Filter String fw1.direction == i

postIn Æ Filter String fw1.direction == I
preOut Æ Filter String fw1.direction == o
postOut Æ Filter String fw1.direction == O
fw monitor
On UTM-1 Edge

Æ Setup Æ Tools Æ Packet Sniffer

Æ two modes: normal sniffer or fw monitor

On SecuRemote/SecureClient

Æ srfw monitor –o <filename>

Troubleshooting UTM-1 Edge
Analyse local policy

Run info fw rules on command line

or WebUI Æ Setup Æ Tools Æ Command Line

Analyse NAT policy

Run info nat on command line

or WebUI Æ Setup Æ Tools Æ Command line
Troubleshooting UTM-1 Edge
Create diagnostics file

Log into WebUI

Æ Setup Æ Tools Æ Diagnostics

Troubleshooting UTM-1 Edge
Is the SMS process running on SmartCenter?
ps –aux | grep sms

Is traffic reaching the SmartCenter?

fw monitor

libsw must be current, at least same version as

latest firmware installed on a Edge.

Check /opt/CPEdgecmp-R71/libsw/version.txt

[Expert@fwm]# head -n1 version.txt

libsw built with version 8.1.21
Troubleshooting UTM-1 Edge
Sofaware Management Server Console

http://<ip SmartCenter>:9283/

- restart SMS
- reload SMS settings
- force policy update
- reboot
- reset local (Edge) password
- view status information
Troubleshooting UTM-1 Edge
Troubleshooting UTM-1 Edge
Debugging Sofaware Management Server

Edit $FWDIR/conf/sofaware/SWManagement.ini
Change in line containing LogPolicy1 the value
Info to Debug
Smsstop
sms –confdir $FWDIR/conf/sofaware

Replicate the problem and watch for console output.

Terminate programm and restart SMS afterwards

smsstart
Troubleshooting UTM-1 Edge
Configuration for Edge Devices on SPLAT under
/opt/CPEdgecmp-R71/tmp

<name of Edge object>.pf Æ ruleset

<name of Edge object>.pfz Æ compressed ruleset
<name of Edge object>.topo Æ topology for VPN
<name of Edge object>.tpz Æ compressed topology
<name of Edge object>.p12 Æ PKCS#12 certificate

Delete files. Install policy again to re-generate them.

Make sure, that the files are compiled and the Edge
gets the latest version.
Opening a service request
Submit info to Check Point TAC or your CCSP/CSP

provide contact info

describe Check Point environment
list used gateway hardware
provide info about network topology and hardware
describe the problem / the symptoms in detail
what kind of business impact has this problem

recommendation: get your supporter on the phone

and be available for remote sessions
use chat tool!
Opening a service request
Create compressed CPInfo diagnostic file

/opt/CPinfo-10/bin/cpinfo –z <filename>

Create compressed CPInfo diagnostic file including logs

/opt/CPinfo-10/bin/cpinfo –l –z <filename>

CPInfo files can be viewed using InfoView

Make sure to have the latest CPinfo build installed!

Check sk30567 for instructions!
 TAC organisation

Customer Focus
Director TAC
Programmers

Data Security
INTL Support Escalations Diamond Services Knowledge Center
Escalation

3 Product Teams 3 Product Teams

High end High end Technical
Secure Knowledge
Core Core Publications
VPN VPN
TAC escalation

Support desk

Product team

Escalations

Customer focus programmer

 TAC escalation path

{
http://www.checkpoint.com/services/contact/escalation.html
General debugging
kernel mode user mode
usbcore security server
… sms
cpd
rtmmod
fwd
simmod
fwm
vpntmod
…
vpnmod
fwmod

fw, VPN, FG-1, H323,

BOA, WS, CPAS, CLUSTER,
RTM, kiss, kissflow, multik,
SFT, CI
kernel mode debug
View kernel modules with fw ctl debug –h

kiss ??????
kissflow ???????
fw "Firewall Module"
h323 "VoIP H.323 Module"
multik "related to CoreXL"
BOA "Malicious Code Protection Module"
WS "SmartDefense Web Intelligence Module"
CI „Content Inspection“
CPAS "Active Streaming Module"
VPN "VPN Module"
RTM "SmartView Monitor Module"
SFT ???????
Cluster "ClusterXL Module"
FG-1 "Floodgate-1 QoS Module"
kernel mode debug
Some examples for modules and options:
Module: fw
Options: error warning cookie crypt domain ex
driver filter hold if install ioctl kbuf
ld log machine memory misc packet q xlate
xltrc conn synatk media align balance
chain bridge tcpstr scv ndis packval sync
ipopt link nat cifs drop

Module: vpn
Options: driver err packet policy sas rdp
clear cipher init sr comp xl counters mspi
cphwd ref vin cluster nat l2tp warn
kernel mode debug
fw ctl debug

Allocation of a buffer for the debug logs

fw ctl debug –buf [size in kb]

The main debug command

fw ctl debug –m <module> <option>

Writing the debug logs into a file

fw ctl kdebug –T –f –o <filename>

Stop debugging

fw ctl debug 0
kernel mode debug
Filter debug, only lines with <strings> in it are
written to the output (best practice: error, failed)

fw ctl debug –d <strings>

Filter debug, only lines that don‘t contain <string>

in it are written to the output

fw ctl debug –d ^<strings>

Can be combined

fw ctl debug –d error,failed,^packet

kernel mode debug
Stop debug messages when a certain string is
issued.

fw ctl debug –s <string>

Example:

fw ctl debug –s error

kernel mode debug
Example: debugging ClusterXL

fw ctl debug –buf 32000

fw ctl debug –m fw + conn drop packet if
sync
fw ctl debug –m cluster all
fw ctl kdebug –T –f –o <filename>

Example: debugging Site to Site VPN

fw ctl debug -buf 32000

fw ctl debug -m VPN all
fw ctl debug -m fw + conn drop ld xlate
xltrc nat
fw ctl kdebug –T –f –o <filename>
kernel mode debug
Example: debugging SIP

fw ctl debug –buf 32000

fw ctl debug –m fw + conn drop vm sip
fw ctl kdebug –T –f –o <filename>

Example: debugging VoIP

fw ctl debug -buf 32000

fw ctl debug -m fw + conn drop vm
fw ctl debug –m h323 all
fw ctl kdebug –T –f –o <filename>
kernel mode debug
Example: debugging SmartDefense

fw ctl debug –buf 32000

fw ctl debug –m fw + conn drop vm tcp-str
spii
fw ctl kdebug –T –f –o <filename>

Example: debugging NAT

fw ctl debug -buf 32000

fw ctl debug -m fw + xlate xltrc
fw ctl kdebug –T –f –o <filename>
kernel mode debug
Example: debugging QoS

fw ctl debug –buf 32000

fw ctl debug –m FG-1 all
fw ctl kdebug –T –f –o <filename>

Example: debugging SmartView Monitor

fw ctl debug -buf 32000

fw ctl debug -m RTM all
fw ctl kdebug –T –f –o <filename>
VPN debug
Best practice before starting debug

Compare configuration on both ends

often Phase I / Phase II parameters are not equal
which causes the VPN to fail
take special notice of networks and subnet masks
carefully compare Pre-Shared-Secrets

Have a close look at the logs in SmartView Tracker

Most informations can be found in the logs

VPN debug
To determine status of VPN tunnels, use menu based

vpn tunnelutil Æ vpn tu

or SmartView Monitor

To shutdown all VPN operation, use

vpn drv off

To enable VPN again, use

vpn drv on
install policy
VPN debug
VPN debugging events can be logged on the
gateway
vpn debug on

Debug output is written to $FWDIR/log/vpnd.elg

More details can be logged using the command

vpn debug on TDERROR_ALL_ALL=5

Turn off debugging with

vpn debug off

VPN debug
IKE negotiations during VPN tunnel establishment
can be logged in ike.elg

On the gateway:
vpn debug ikeon / vpn debug ikeoff
Debug output is written to $FWDIR/log/ike.elg
VPN debug
Initiate VPN and IKE debug together

vpn debug trunc

Disable VPN and IKE debug

vpn debug off

vpn debug ikeoff
VPN debug
Capture traffic using fw monitor

fw monitor –e „accept port(500) or

port(4500);“ –o monitor.out

Output file is monitor.out, IKE payloads are

encrypted.

Capture traffic using vpn debug

vpn debug mon

Output file is ikemonitor.snoop, IKE payloads are in

clear.
Turn off with vpn debug moff.
VPN debug
On UTM-1 Edge appliance:
WebUI -> Reports -> Tunnels -> save IKE trace
Click Save IKE Trace, which creates ike.elg
 user mode debug
General syntax

fw debug <process> <on|off> TDERROR_ALL_ALL=<value>

fw debug <process> <on|off> OPSEC_DEBUG_LEVEL=<value>

Exception: cpd
fwm debug
FWM controls connections from the SmartConsole to
the SmartCenter server and is responsible for
policy related functions

To debug fwm do the following

fw debug fwm on TDERROR_ALL_ALL=5

fw debug fwm on OPSEC_DEBUG_LEVEL=9

To stop debug run

fw debug fwm off TDERROR_ALL_ALL=0

fw debug fwm off OPSEC_DEBUG_LEVEL=0

Logs are written to $FWDIR/log/fwm.elg

fwm debug
[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]
fwnetobj_getbysicname:
table_chosen_get_with_param(eTABLE_NETWORK_OBJECTS,
is_obj_SIC_name,
IP=212.1.56.233,CN=Gui_Client) returned NULL.
Login failed: 212.1.56.233 is not allowed for remote
login
[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]
fwm_log: Login failed from
IP=212.1.56.233,CN=Gui_Client: Unauthorized client
Wed Sep 8 18:46:32 2010 (GMT): reject client
IP=212.1.56.233,CN=Gui_Client
[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]
PM_policy_query: rule not found.
[FWM 11927 1981476992]@firewallr70[8 Sep 18:46:32]
PM_policy_query:
finished successfully. 1st method = deny

IP not defined in $FWDIR/conf/gui-clients

fwm debug
[FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07]
fwm_cpmi_auth_handler: authenticating admin admin by
Name and Password
[FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07]
Administrator admin found in fwm database
[FWM 11927 1981476992]@firewallr70[8 Sep 18:48:07]
CBinObjCommon::PackLogData: Field number:12, Data
offset:34, Type:eFtCstring, Value:Administrator
failed to log in: Wrong Password
fwd debug
FWD daemon controls logging, alerts,
communication with the kernel, OPSEC
communication, invokes child processes (security
servers, ICA)

To debug fwd do the following

fw debug fwd on TDERROR_ALL_ALL=5

To stop debug run

fw debug fwd off TDERROR_ALL_ALL=0

Logs are written to $FWDIR/log/fwd.elg

Desktop log server debug
To debug dtls do the following

fw debug dtls on

To stop debug run

fw debug dtls off

Logs are written to $FWDIR/log/dtlsd.elg

Security servers debug
Some examples for security servers:

FTP security server – in.aftpd

Telnet security server – in.atelnetd
HTTP security server – in.ahttpd
SMTP security server – in.asmtpd
ClientAuth (900) – in.ahclientd
ClientAuth (259) – in.aclientd
AntiSpam security server – in.msd
URL filtering security server – in.aufpd
Security servers debug
Verify that security server process exists. Check
$FWDIR/tmp for existing PID files.

Start debugging (example for FTP security server)

fw debug in.aftpd on FWAFTPD_LEVEL=3

Stop debugging

fw debug in.aftpd off FWAFTPD_LEVEL=3

cpd debug
CPD controls SIC, Policy install

To debug cpd do the following

cpd_admin debug on TDERROR_ALL_ALL=5

To stop debug run

cpd_admin debug off TDERROR_ALL_ALL=0

Logs are written to $CPDIR/log/cpd.elg

Secure Platform debug
Sometimes it is useful to verify file integrity and
version against a test environment, for example
after installation of ad-hoc fixes or HFA.

Use md5sum for creating hashes.

[Expert@fwm]# md5sum upgrade_import
e6c6417cca9db098b94673dd420a4903 upgrade_import

Use cpvinfo for displaying version information.

[Expert@fwm]# cpvinfo upgrade_import

Build Number = 730080036

Major Release = NGX
Minor Release = fli_up_ga
Release Number = 5.0.5
Version Name = NGX
Secure Platform debug
For some problems with processes a core dump can
be usefull.

A core dump is a disk file that contains an image of

the process‘s memory at the time of termination.

Core dumps are mainly used by Check Point R&D for

fixing a specific problem.

Handling Core Files

http://downloads.checkpoint.com/dc/download.htm?ID=10479
Secure Platform debug
To enable core dumps do the following

ulimit –c unlimited

um_core enable

Reboot

Check that /etc/sysconfig/enable_cores exist

after Reboot.

Dumps will be in /var/log/dump/usermode

Debugging GUI clients
Debug GUI clients

Dashboard Æ fwpolicy.exe –d –o fwp_debug.txt

Tracker Æ cplgv.exe –d –o cplgv_debug.txt
Monitor Æ smartcons.exe –d –o smartcons_debug.txt

general syntax: <executable> -d –o <file_name>

Output is in specified directory or in

C:\Programme\CheckPoint\SmartConsole\R70\PROGRAM\data

if directory is omitted.
Resources
SmartSPLAT from Çağdaş Ulucan

www.smartsplat.com
Resources
Resources
Resources
Resources
Resources
Resources
fw monitor
http://www.checkpoint.com/techsupport/downloads/html/
ethereal/fw_monitor_rev1_01.pdf

The CPinfo utility

https://supportcenter.checkpoint.com/supportcenter/porta
l?eventSubmit_doGoviewsolutiondetails=&solutionid=sk30
567

Documents related to troubleshooting

http://blog.lachmann.org/2010/09/documents-related-
to-troubleshooting/
Questions?
Still got a question?

Tobias Lachmann

tobias@lachmann.org

http://blog.lachmann.org