Debugging with R65 CoreXL

Potential R65 CoreXL Difficulties

The main development of R65 CoreXL is the ability to load several firewall instances and
attach them so they can run concurrently on different cores. A new component known as ‘the
Dispatcher' receives the traffic from the NICs and load balances it across the instances.

In CoreXL, the firewall instances do not synchronize any data, thus it is crucial to maintain
full stickiness for the connections. Every packet of a specific session should always be
directed by the Dispatcher to the same instance that started handling the first packet of the
session.

R65 CoreXL and Complex Connections

When you are using complex connections, the scenario where the Dispatcher must maintain
full stickiness is more complicated. If a control connection was directed to instance X, then
all of the data connections created from this control connection should also be directed to
instance X (in FTP – both control and data connection).

You should check all the connections that have been opened on a specific instance. You can
then verify that the control and data connection were directed to the same instance.

To verify that control and data connections of a session are redirected to the same
instance:
1. Run the command: fw –i <instance_number> tab –t connections –u.
All the connections that have been opened on the instance are displayed.

2. Run the grep command on the displayed connections.

3. Verify that the control connection and data connection were directed to the same
instance.

Here is a list of protocols which create data connections:

• ftp • realaudio

• h323 • rsh

• iiop • rtsp

• netshow • sqlnet

• pptp

Introducing a New I/S

R65 CoreXL allows each firewall instance to be autonomous and they do not interfere with
each other. There is a full parallel use of the multiple cores and utilization can reach 100% of
the capabilities of the firewall.

©2009 Check Point Software Technologies Ltd. All rights reserved. 1

Classification: [Unrestricted]—For everyone

However, there are some cases where the instances need to share information and
therefore a new I/S was developed – local sync. The local sync I/S is similar to the ClusterXL
sync mechanism. A table can now be defined to have local sync enabled, which means that
each update on this table triggers a message that is sent to all the instances. This message
is handled and the update is applied to the local table of the instance.

To confirm that the local sync mechanism is working, you should verify that the table has the
same data on all the instances. For example, you can run fw –i <instance_num> tab
–t <table_with_local_sync> -u on all instances and verify that the output is the
same for all instances.

Here is a list of tables that have that use the local sync mechanism:

• dynobj_cache • exchange_notifies

• userc_users • rpc_serv

• DAG_IP_to_ID • rpc_serv_hosts

• dcerpc_maps • pmap_not_responding

• dcerpc_rmaps • freetel_connections

• dcerpc_udp_maps • p2p_logged

• dcerpc_udp_rmaps • sip_dynamic_port

• dcerpc_udp_hpov_maps • mgcp_dynamic_port

• dcom_objects • domain_cache

• dcom_high_port • arp_table

• dcom_remote_activations

R65 CoreXL Commands

Command Explanation
fw ctl multik stat Gives statistics about:
• Number of instances that were loaded on boot.
• Which instances are active and which are inactive.
• Number of concurrent connections handled by each instance.
• The peak number of concurrent connections of each instance.
This command can be used to verify that the load is distributed
evenly across the instances. (Relevant for ISI, COST and
performance teams.)
fw ctl multik Used for activating/deactivating instances.
stop/start An instance that is inactive is still loaded, but the Dispatcher does
not forward traffic to this instance.
fw ctl zdebug + Uses the multik debug flag to debug multik specific debug
multik logs.

©2009 Check Point Software Technologies Ltd. All rights reserved. 2

Classification: [Unrestricted]—For everyone

Command Explanation
fw –i Debugs a specific instance.
<instance_number> Alternatively, fw ctl zdebug can in combination with grep on
zdebug fw_<instance_number>. For example: fw ctl zdebug +
drop | grep fw_3
fw –i Retrieves fw monitor output of a specific instance.
<instance_number>
monitor
fw ctl set int VoIP traffic in CoreXL is handled by instance 0 to overcome
voip_multik_enable_ stickiness problems. This command changes the handling to
forwarding 1 instance 1, unless only 1 instance is loaded.
(default 0)
fw ctl set int Trigger to shift the map of the Dispatcher and make the
fwmultik_shift_hash connections distribute differently. Use cpstop before triggering
1 this command and cpstart afterwards. Dynamically shifting the
map is not supported.
fw ctl set int Helps you to "silently" drop certain services (destination ports).
fw_drop_service_ There is no log or debugging for services dropped with this
early 0/1/2/3/4/5 command (only Performance Pack and ClusterXL see them). You
(default 0)
can drop the five services from fw_drop_dport1 through
fw_drop_dport5.
• Each service can be set with the fw ctl set command.
• Each fw_drop_dport should equal a TCP/UDP port.
• fw_drop_service_early should equal the number of
fw_drop_dport's that you wish to consider.
For example: fw_drop_service_early=3 with
fw_drop_dport1=8116, fw_drop_dport2=137,
fw_drop_dport3=138 - causes a silent drop of all CCP (8116)
and NETBIOS (137, 138).
The default values are: fw_drop_dport1=8116 (CCP),
fw_drop_dport2=137 (NETBIOS), fw_drop_dport3=138
(NETBIOS), fw_drop_dport4=67 (DHCP),
fw_drop_dport5=68 (DHCP).
NOTE: It is recommended that you should use
fw_drop_service_early=3. Be careful not to drop DHCP if
you are using it (Dynamically assigned Gateway, DHCP office
mode) and not to drop 8116 if you are using Windows.
fw ctl set int Trigger to dump the dispatcher global connection table (to
fwmultik_dump_gconn dmesg).
_tab 1
fw ctl set int In ClusterXL, multicast causes the Dispatcher to drop the packets
fwha_multik_ which the member should not handle. The purpose of this flag is
multicast_early_ for optimization - the traffic that should not be processed by the
drop 0/1 (default
1) member is dropped at an early stage.
NOTE: This does not apply if sxl or sdf is on.
fw ctl set int In ClusterXL, pivot causes the Dispatcher on the pivot member to
fwha_multik_pivot_ forward packets which the member should not handle. The
early_forwarding 1 purpose of this flag is for optimization - the traffic that should not
be processed by the member is dropped at an early stage.
NOTE: This does not apply if sxl or sdf is on.

©2009 Check Point Software Technologies Ltd. All rights reserved. 3

Classification: [Unrestricted]—For everyone

Command Explanation
fw ctl set int If fwha_multik_multicast_early_drop or
fwha_multik_test_ fwha_multik_pivot_early_forwarding are active, this
cpls_dist -1 command dumps information regarding:
• Number of packets that reached the Dispatcher.
• Number of packets that belonged to this member.
fw ctl set int Dumps statistics regarding an instance.
fwmultik_get_stats • Number of packets that the instance handled.
1
• The CPU that handled the instance.
Most statistics are counted only in the debug kernels.
NOTE: Use -1 to reset.
fw ctl set int Enables lock verification and if there is a lock violation, then the
cplock_enable_ system crashes.
verification 1 NOTE: Do not enable this command when you are doing
(default 0)
performance tests.
fw ctl set int Enables gathering statistics on all locks.
cplock_enable_ NOTE: Do not enable this command when you are doing
statistics 1 performance tests.
(default 0)
fw ctl set int Dumps locks information (to dmesg). This flag works only if
cplock_test_locks 1 cplock_enable_statistics is on.
NOTE: Use -1 to reset.
fw ctl set int Enables specific clustering lock statistics.
fwha_multik_enable_ NOTE: Do not enable this command when you are doing
lock_statistics 1 performance tests.
(default 0)
fw ctl set int Dumps cluster lock statistics. This flag only works if
fwha_multik_test_lo fwha_multik_enable_lock_statistics is on.
cks 1 NOTE: Use -1 to reset.
fw ctl set int This command selects whether debug prints formatted IP
fw_print_conn_forma addresses or in hexadecimal.
t 0/1 (default 1) • 1 makes the debug print formatted IPs. (172.23.10.92)
• 0 makes the debug print in hexadecimal. (ac170a5c).

Debugging R65 CoreXL

To debug R65 CoreXL:

1. Load only 1 instance and try to reproduce the problem.

2. If the problem does not reproduce, then there is most likely a problem with CoreXL.

3. If the problem reproduces – try to reproduce the same problem on R65. It is likely
that there is a problem with R65 and not with CoreXL.

©2009 Check Point Software Technologies Ltd. All rights reserved. 4

Classification: [Unrestricted]—For everyone