Ipso3.6 Fp3 v1.9 Clustring

Nokia Clustering IPS0 3.
6FCS4 / Check Point NG FP3

Load Balancing
Configuration Document ©
This document describes how to configure load balancing on Nokia IPSO 3.6 FCS4 with
clustering, and Check Point NG FP3 in a step-by-step process. Any statements contained herein
are those of the author and in no way reflect the views of Nokia Internet Communications or
Check Point Software Technologies.
Questions should be directed to the author.
This and other configuration documents can be found at:

http://www.digitalmigrations.com
By:
Jeff Mousseau
Tel. +1-416-829-5753
Fax +1-416-829-5553
Email: jeff@digitalmigrations.com
© Copyright North America All Rights Reserved, July 2002

NG is a registered trademark of Check Point Software Technologies.
IPSO is a registered trademark of Nokia.
Jeff Mousseau 2/6/2003 1

Disclaimer
This paper details how to configure two Nokia IP Security gateways (GWs) in a load
balancing configuration using Nokia Clustering IPSO 3.6 FCS4, and Check Point NG
Feature Pack 3 (FP3), in a step-by-step manner.
It does not focus on how to configure a Nokia GW for optimum security. It is

recommended that anyone implementing this configuration into a production
environment, or one where security is necessary, that the device(s) be configured
securely.
The author is not responsible loss of data, damage or monetary loss.
Environment
- 2 x Nokia IP330 (256 MB RAM recommended)
- 1 P4 PC 1.4GHz with 256 MB RAM running the NG Management Server and GUI
Client
Gotcha(s)
Please note that additional hotfix(es) are required for the cluster to function properly.
They are not listed here since they are released periodically. Hotfixes and their
installation are detailed on the software downloads page at https://support.nokia.com/.
Details
Clustering in IPSO 3.6 is already present and is free to use – there is no license
required.
Supported platforms: IP330, 440, 530, 650, 710, and 740. Should one be tempted to
install clustering on IP110s, in addition to not being supported, some clustering functions
will not operate properly.
Supported interfaces: Currently only 10/100 ethernet interfaces and GigE interfaces
are supported. A design issue is that the switch and cluster sync interface(s) need to be
able to handle multicast traffic at the rate presented to the cluster. Thus, if one is using
GigE interfaces, the switch will need to be high-performance and the sync interface(s)
should be 100 Mbit full-duplex (flow control turned on). The traffic send is a mix of multi
and unicast traffic.
On the configuration issues static ARP entries may be required on some routers should
they have difficulty with ARP replies with multicast addresses. Please see the section on
routers at the end of this paper.
Supported applications: Check Point NG FP3.
Number of supported nodes: 2 to 4 with an unlimited number of interfaces.
Unofficial performance scaling (IP740 sited. More modest scaling on smaller platforms):
Total Nodes Performance Scaling
FW 2 1.5
3 1.7
4 1.8

VPN 2 1.9
3 2.7
4 3.5
As you see from the chart, VPN scaling is excellent at approximately 80 – 90 per cent.
Firewalling is much less perfect. This is due to NG’s robust state sync protocol, which
requires far more overhead. Firewall performance scaling will be addressed in the next
version of IPSO.
Failover time: 4 seconds.
The following dynamic and multicast routing protocols are not supported when clustering
is enabled: BGP, OSPF, RIP, IGRP, IGMP, PIM, DVMRP. This will be addressed in a
later version with unicast cluster.
State versus clustering synchronization
The Nokia clustering synchronization is configured in Voyager on the Cluster web page.
Check Point NG synchronization is configured in the cluster object. In this document we
will configure both on the same network (interface pair). It is highly recommended that
they be configured on separate networks. (Please see ‘What Happens…’ on page 34 for
more information).
Note: all configuration details for NAT have been omitted from this document. Nokia was
completing a hotfix for NATing with multiple internal networks. Please contact Nokia TAC
to determine if this hotfix is now available. For details on how to configure NAT, please
refer to the online documentation. The documentation (nic-doc3.6.tgz) can be
downloaded from https://support.nokia.com.

Final Note From The Author
This document is not meant to be the definitive document on configuring IPSO and NG
clustering. More information is available in the GW online documentation.

What’s New?
Version 1.6 (ipso3.6_fp3_v1.6.doc) 10.12.02

- Contains additional information on clustering protocols (pg. 34).
- Revision of the number of IP octets from which the virtual MAC is calculated (pg. 39).
Courtesy Morten Bonde, S.E., Nokia, Copenhagen, morten.bonde@nokia.com.
- Statement regarding the recommendation of VRRP over clustering when in a NAT’ed
environment with multiple internal networks was added (pg. 3).
- minor formatting changes.
- Statement regarding the issue of switch failure and catastrophic failure (pg. 41).

- Update regarding hotfix for clustering and NAT with multiple internal networks. (pg. 3).

- Update on the source port of TCP objects 11003 & 11004. (pg. 31)

- Update on the configuration of anti-spoofing.

Proposed Network Architecture
Consider the following diagram as an example of a Nokia Clustering / FP3
implementation.
Diagram 1
IPSO 3.6 FCS4 & CP NG FP3

Cluster Router
INTERNET
64.231.169.249
jeff
10.250.135.11/24 Intel FW / VPN GW
FTP Client 10.250.135.99/24
FTP FTP
10/100 Hub
VRID
10.250.135.1/24 10.250.135.3/24 10.250.135.2/24
dm1
VRID dm2
- NG enforcement 172.16.31.1/24 172.16.31.2/24
172.16.31.3/24 - NG enforcement
module 10.1.1.1/24 10.1.1.2/24 module
10/100 Hub
VRID W2KPro SP 2
blanc 10.1.1.3/24 10.1.1.11/24
Win2KPro SP2
FTP Server
NG Management Console & GUI Client
10.1.1.11 / 24

From here I will consider that we have established a console or telnet connection to the
Nokia Gateway and have Voyager access.
IPSO Operating System
We will ensure that we are running IPSO 3.6 or later using a shell connection or
Voyager.
N.B. Be sure to check both GWs.
Using Voyager to verify the version of IPSO that is running:
Ensure that IPSO 3.6 or a later version that supports CP NG FP3 is installed. It can be
downloaded from http://support.nokia.com. A support account is required. Please refer to
the release notes for installation instructions.
dm1 Setup
10.1.1.1/24 int1 (eth-s1p1) - internal segment
172.16.31.1/24 ss1 (eth-s2p1) - secure segment (a.k.a. DMZ)
10.250.135.1/24 ext1 (eth-s3p1) - external or Internet segment
Note that 24 bits are the net mask or class ‘C’. This is equivalent to 255.255.255.0. A
class B net mask is 16 bits or 255.255.0.0. If one is having difficulty, click on the
following link to download and then install a subnet calculator.
http://www.digitalmigrations.com/IPSub2.exe
Interfaces
From the Voyager GUI ensure proper interface configuration:

Hosts Address Assignment
Gotcha(s)
1 Although it is not as critical as it once was, it is still recommended by the author

that host naming be completed. The best rule of thumb is the following:
hostid = the hostname of the external interface = the NG Workstation name
If one does not follow this advice, the following error may occur:
Jun 26 10:38:11 dm2 [LOG_CRIT] kernel: FW-1: fw_attach: cannot find name of
interface 3
Ideally, all of the interfaces of the firewalls should have unique names. If any or all of the
interfaces are named the same on one or both gateways, it is undefined which interfaces
would be returned in a name lookup. This can make policy pushing and logging difficult
(read: it won’t work!) or logs confusing to read.
Local Time Setup
Time is used for both NG state table synchronization and encryption. Devices should be
set to the second. To ensure this it is strongly recommended (read imperative) to use
NTP. This will guard against click drift and ensure valid synchronization of the NG state
table, and thus proper failover.

Following the order of the steps below is recommended. There is a sanity check on the
NTP, thus if the clock is too far out of time sync with the NTP server, it will not update.
So, set the timezone and time first to an appropriate local time, then set the NTP.
Time and NG
Gotcha(s)
1 Clocks on the Nokia GWs must be set to within the second in order to ensure
successful NG state synchronization. It is recommended that one perform the following:
a set the proper timezone under ‘Local Time Setup’. In this instance it will be set to
Canada/Quebec/Montreal. (The reason for the odd timezone locations is a holdover from
BSD).
b configure NTP.
Local Time Setup
NTP
To setup NTP go to the NTP page in Voyager and enable it. There are several ways to
configure NTP. Should one’s corporate security policy disallow outgoing NTP from the
Nokia GWs, one GW can be configured as NTP master using its own clock and the other
GW can point to it. The drawback to this that should a GW ever be breached, a company
will need valid timestamps on logs in a court of law.
In the above example, the GW was configured to point to tock.utoronto.ca.
tick.utoronto.ca 128.100.103.17
tock.utoronto.ca 128.100.100.128

A function of the NTP protocol is that it factors in the travel time of the time request to
response so it does not matter how far a GW is away from the NTP master – only that
the server is reliably available.
We will make sure that we have applied, and saved the changes. Then reboot the device
by typing ‘reboot’ at the prompt.
By clicking on the apply button, we will be taken to the configuration page.
Be sure to reboot the GW(s).
dm1[admin]#reboot
To ensure that the NTP cron is operating one can type the following:
dm1[admin]# tail /var/log/messages

Jan 29 13:17:12 dm1 [LOG_NOTICE] xntpd[199]: restarting
Jan 29 13:17:12 dm1 [LOG_NOTICE] pm[71]: Reaped: xntpd[199]
Jan 29 13:17:12 dm1 [LOG_NOTICE] pm[71]: Scheduled xntpd for +2 secs
Jan 29 13:17:14 dm1 [LOG_NOTICE] xntpd[200]: xntpd version=3.4e (beta
multicast); Dec 15 2001 00:34:26
Jan 29 13:17:14 dm1 [LOG_NOTICE] pm[71]: Restarted /bin/xntpd[200],
count=2
Jan 29 13:17:14 dm1 [LOG_NOTICE] xntpd[200]: tickadj = 5, tick = 10000,
tvu_maxslew = 495
Jan 29 13:17:46 dm1 [LOG_NOTICE] xntpd[200]: time reset (step) -
225.360385 s
The output marked in bold confirms that the time update occurred.

dm2 Setup
10.1.1.2/24 int1 (eth-s1p1) - internal segment

172.16.31.2/24 ss1 (eth-s2p1) - secure segment (a.k.a. DMZ)
10.250.135.2/24 ext1 (eth-s3p1) - external or Internet segment
Interfaces
Hosts

NTP
Static Routes
Management Server
At this point we will want to ensure our static routing is properly configured. In this
configuration, the default route on the Management Server will be the internal cluster IP
address or 10.1.1.103.
Gotcha(s)
1 Should we ever want to perform NAT (hide), a route is required to send any
policy traffic destined to the external interface of dm2 through dm2’s internal interface. In
short, add the following two routes to the Management Server.
c:\>route add –p 10.250.135.1 mask 255.255.255.255 10.1.1.1

c:\>route add –p 10.250.135.2 mask 255.255.255.255 10.1.1.2
If the second route is not added the traffic will flow from to the default gateway of the
Management Server (the internal cluster IP). From there is will travel to the cluster
Master and out the cluster master’s external interface, and will be denied at the external
interface of the cluster slave. (See the NAT section for more information)
dm1 and dm2
Both of the enforcement modules have the same simple routing configuration.

Clustering Configuration
Cluster Status Table:
Cluster ID: Id of the cluster.

Cluster Uptime: Time since the cluster was formed.
Number of Members: Current number of members in the cluster.
Number Of Interfaces: Number of interfaces on which clustering is enabled.
Network: Networks on which clustering is enabled.
Cluster IP Address: Cluster IP Address on each network.
Cluster Members Table:

Member Id: Node id in the cluster (1).
IP Addr: Primary IP address of the member.
Hostname: Hostname of the node.
Platform: Type of platform.
OS Release: Operating system version node is running.
Rating : Node performance rating.
Time since join: Time since node joined the cluster.
Work Assigned(%): Percentage of work load assigned to this node.

dm1
N.B. In this configuration we will not be performing NAT. Should you wish to after
you have completed this exercise, you will need to return to this page and
change the ‘Hash Selection’ from ‘default’ to either NAT-INT for internal
interfaces or NAT-EXT hash for external interfaces.

dm2

How to install CP NG-FP3
Installation of NG-FP3 is becoming more simplified by the introduction of Checkpoint’s

Comprehensive Bundle for IPSO. This bundle automates the process of installing
Firewall-1 & SVN Foundation FP1, then upgrades to Firewall-1 & SVN Foundation to
FP3. It is approximately 31 megabytes.
Gotcha(s)
1 Depending on the FTP daemon, I have seen inconsistencies when attempting to

install using anonymous ftp. Hence, it is recommended to use a username and
password on the FTP account. You can take your chances, but be warned.
The transfer of the FP3 tar file (CP_FP3_IPSO.tgz) will take place during the running of
the newpkg script using FTP. We will run the following command:
dm1[admin]# newpkg -i
Load new package from:

1. Install from CD-ROM.
2. Install from anonymous FTP server.
3. Install from FTP server with user and password.
4. Install from local filesystem.
5. Exit new package installation.
Choose an installation method (1-5): 3

Enter IP address of FTP server (0.0.0.0): 10.1.1.10
Enter user name: cp
Enter password for cp :
Enter pathname to the packages [ or 'exit' to exit ]: . Å Be sure to use a dot here to indicate
that the package is in the root directory.
Loading Package List
Processing package CP_FP3_IPSO.tgz ...

Package Description: Check Point NG Feature Pack 3 wrapper package
Would you like to :
1. Install this as a new package

2. Upgrade from an old package
3. Skip this package
4. Exit new package installation
Choose (1-4): 1
Installing CP_FP3_IPSO.tgz
CP_FP3_IPSO does not exist previously. Proceeding with Installation.
Running Pre-install script

Running Post-install script
Oct 4 10:49:55 dm1 [LOG_CRIT] PKG_INSTALL:
*********************************************************************
PKG_INSTALL: INSTALL STARTED at Fri Oct 4 10:49:55 EDT 2002
PKG_INSTALL: Trying to install CPshrd-50/cpshared_ipso.tgz
PKG_INSTALL: Trying to install CPfw1-50/fw1_ipso.tgz
PKG_INSTALL: Running /tmp/pkg/CP_FP3_IPSO/CPfw1-50/POST_INSTALL
PKG_INSTALL: Running /tmp/pkg/CP_FP3_IPSO/CPdtps-50/PRE_INSTALL
PKG_INSTALL: Running /tmp/pkg/CP_FP3_IPSO/CPuag-50/PRE_INSTALL
*******************************************************
PKG_INSTALL: /etc/newpkg -S -m LOCAL -i -n CPfwbc-41/fw-1_ipso.tgz
*******************************************************
PKG_INSTALL: /etc/newpkg -S -m LOCAL -i -n CPdtps-50/polsrv_ipso.tgz
*******************************************************

PKG_INSTALL: /etc/newpkg -S -m LOCAL -i -n CPfg1-50/fg1_ipso.tgz
*******************************************************
PKG_INSTALL: /etc/newpkg -S -m LOCAL -i -n CPrtm-50/rtm_ipso.tgz
*******************************************************
PKG_INSTALL: /etc/newpkg -S -m LOCAL -i -n CPuag-50/uag_ipso.tgz
PKG_INSTALL:
****************************************************************
PKG_INSTALL: Running /tmp/pkg/CP_FP3_IPSO/CPdtps-50/POST_INSTALL
****************************************************************
*******************INSTALL/UPGRADE PROCESS COMPLETED************
Please do the following if the INSTALL/UPGRADE is Successful:

1) Logout and re-login.

2) Run 'cpconfig' and configure the firewall.
3) Install the new License if required.
4) Reboot the box.
PKG_INSTALL: *******************INSTALL/UPGRADE PROCESS COMPLETED

PKG_INSTALL: ****************************************************
Done installing CP_FP3_IPSO
End of new package installation

cleaning up ..done
Use Voyager to activate packages
dm1[admin]#
N.B. Be sure to install it on both GWs.
Logout and log back in via the console. You can also go into Voyager and click on
‘Manage Packages’. The installed packages should look like this.

If you upgraded from FP2 to FP3 using the ‘newpkg –i –k’ command, the installed
packages will look like this. The –k switch will keep the firewall active after installation.

Windows 2000 Server Install
We are now ready to run cpconfig, but before we do, we will install FP3 on the Windows
management station. This will be demonstrated using a series of ‘selected’ self-
explanatory screen shots.

You have now completed installing the FP3 for managing the cluster.

cponfig on dm1 and dm2
Gotcha(s)
1 Be sure to use VT100 terminal emulation when running cpconfig. The NG

cpconfig script appears to have problems with any other type…trust me!
2 If the following error appears: “fw_ipaddr: cannot get my ipaddr” that is because
NG is unable to resolve the host name from the hosts file. Never use special characters.
Not even a ‘-‘ as in ‘dm-ms’.
3 If the following error appears: cannot open /opt/CPfw1-50-01/state/default.bin this
is also a result of not having configured the hosts file. Allow the script to exit, configure
the hosts file and then type the following:
dm1[admin]#touch /opt/CPfw1-50-01/state/default.bin. Now rerun cpconfig.
dm1[admin]# cpconfig
Welcome to Check Point Configuration Program

=================================================
Please read the following license agreement.
Hit 'ENTER' to continue...
^C
Å You can type ‘control c’ to skip the
license agreement.
Select installation type:
-------------------------
(1) Enforcement Module.

(2) Enterprise Management.
(3) Enterprise Management and Enforcement Module.
(4) Enterprise Log Server.
(5) Enforcement Module and Enterprise Log Server.
Enter your selection (1-5/a-abort) [1]: 1
Would you like to install a Check Point clustering product (CPHA, CPLS
or State Synchronization)? (y/n) [n] ? y
IP forwarding disabled
Hardening OS Security: IP forwarding will be disabled during boot.
Generating default filter
Default Filter installed
Hardening OS Security: Default Filter will be applied during boot.
This program will guide you through several steps where you
will define your Check Point products configuration.
At any later time, you can reconfigure these parameters by
running cpconfig
Configuring Licenses...
=======================
Host Expiration Signature
Features
Note: The recommended way of managing licenses is using SmartUpdate.

cpconfig can be used to manage local licenses only on this machine.

Do you want to add licenses (y/n) [y] ? y
Do you want to add licenses [M]anually or [F]etch from file: M

IP Address: eval
Expiration Date: 01Nov2002
Signature Key: dD2zRuxtJ-uytGvkMEn-6RHstqpWQ-mcFptBSKv
SKU/Features: CPMP-EVAL-1-3DES-NG CK-CP
License was added successfully
Configuring Random Pool...

==========================
You are now asked to perform a short random keystroke session.
The random data collected in this session will be used in
various cryptographic operations.
Please enter random text containing at least six different

characters. You will see the '*' symbol after keystrokes that
are too fast or too similar to preceding keystrokes. These
keystrokes will be ignored.
Please keep typing until you hear the beep and the bar is full.
[....................]
Thank you.
Configuring Secure Internal Communication...

============================================
The Secure Internal Communication is used for authentication between
Check Point components
Trust State: Uninitialized

Enter Activation Key: xxxxxxxx
Again Activation Key: xxxxxxxx
The Secure Internal Communication was successfully initialized
initial_module:
Compiled OK.
Hardening OS Security: Initial policy will be applied

until the first policy is installed
In order to complete the installation

you must reboot the machine.
Do you want to reboot? (y/n) [y] ? y
************* Installation completed successfully *************
dm1[admin]#
N.B. Don’t forget to run cpconfig on both GWs.

NG Configuration
Gotchas
The default policy filter in NG has now changed to drop all except internal
communications. Although unnecessary under most conditions, to ease any issues we will
uninstall the default filter on both GWs during this phase of the configuration.
dm1[admin]# fw unloadlocal
Create a workstation object defining the Management Server (dm). When we are finished
and click ‘OK’ it will automatically generate a certificate to be used for secure
communications (SIC). This replaces ‘putkeys’ which in the past were occasionally
problematic.
Management Server

dm1 and dm2
Create ‘Host – Gateway’ objects for dm1

& dm2.
Å Click on the Communications tab. This is

used to establish secure connectivity to the
enforcement modules.
Gotcha
If you receive the error after you have
typed in your one time password, “Error:
Failed to connect the module" perform
the following:
1. At the NG Module machine, use
cpconfig to re-initialize SIC by typing a
new one-time password (OTP)
2 In the Policy Editor, in the
Communications window of the Module
object, reset SIC communication
3 Reinitialize SIC communication by
typing the same OTP as used at the
Module machine
4 Verify SIC Communication, by pressing
"Test SIC Status".

Click Get Topology to obtain the
interface configuration. Configure anti-
spoofing as described in the next
section.
Although not shown here, this is where

you would configure anti-spoofing.
If you are using Cisco switches in high

availability mode you may experience
anomalies and as a result, may want to
omit this step.
Click on the Communications tab. This is

used to establish secure connectivity to the
enforcement modules
N.B. Perform the same procedures for the second enforcement point (dm2).

Setting Up a Cluster Object and State-sync
Once that has been successfully achieved, from the file menu click ‘Manage’ and then
select ‘Network Objects’. Click ‘New’ and select ‘Gateway Cluster’. Define the gateway
cluster using the external cluster address i.e. 10.250.135.103.
It is important to know that the individual GW objects will disappear after the cluster
object is created. They can be accessed through the ‘cluster members’ tab of the cluster
object should you need to edit them.
N.B. Do not select ‘Cluster XL’.

Add the two enforcement GWs to the
cluster members.
New to FP3 is the ‘Availability Mode’ on the third tab. Be sure to specify ‘Load Sharing’.
Select ‘Load Sharing’.
SIC is used for authentication and as we will see below. State table information is
exchanged via a network broadcast with the state table in the NG module itself.

Add the network to be used for state
sync. This will also create an implied rule
that allows traffic to and from the
interfaces on this network.
When we click on ‘okay’ you will see the following message:

Clustering Synchronization of the Cluster
In order for clustering synchronization to occur, we must add a rule that allows for cluster
sync (or xpand) traffic. This has changed in FP3. You will need to create two ‘Service’
objects. The first is for TCP port 11003. These two ports are used to pass cluster sync
traffic over.
The second ‘Service’ object is for TCP port 11004.
Now create a ‘Services’ group and place both object into that group.

We are now ready to create our clustering and state sync policy rule. Ensure that this is
the first rule in the policy.
N.B. State synchronization will occur via SIC and as a result, an implicit rule is not
required.
Default Enforcement Policy
We can now add the rest of our rules, create, and push a simple policy rule base like the
one below.
Rule Number 1: cluster sync

Rule Number 2: NTP time for GWs
Rule Number 3: management of GWs
Rule Number 4: internal networks can go anywhere
Rule Number 5: world can access FTP server
Rule Number 6: do not allow or log Microsoft NBT (clean log rule).
Rule Number 7: clean up rule

Note: if you are having difficulty pushing policy to the cluster object this is probably due
to the default enforcement policy in NG which is ‘drop all’ except for implied rules. It is
much easier to proceed if we uninstall the policy as follows.
dm1[admin]# fw stat
HOST POLICY DATE
localhost defaultfilter 23Feb2002 9:46:22 : [>eth-s5p1c0] [<eth-s5p1c0] [>eth-s4p1c0]
[>eth-s3p1c0]
dm1[admin]# fw unloadlocal
Uninstalling Security Policy from all.all@dm1

Done.
dm1[admin]#
N.B. Be sure to complete this command on both GWs.

What Happens When Everything Is Working?
The IP Clustering implementation in IPSO balances IP packets among members of the

cluster by referring to IP source and destination address. This may change in the future
using balancing based on IPSec Security Associations.
The gateway cluster uses a keepalive mechanism to monitor the health of all its
members. The master member sends multicast keepalive messages to the other cluster
members. The members send keepalive messages to the master member. If a gateway
should become unavailable for any reason, planned or unplanned, TCP / UDP sessions
and IPSec SA sets are assigned to other gateways immediately. The keepalive
mechanism is a proactive method for allowing members to confirm presence of other
gateways and have work allocated to the other gateways when needed.
Dynamic Load Balancing is implemented using the same process described above.
However, the IPSO implementation has added an aging algorithm to the process to
assist load re-balancing.
Clustering Protocols
Nokia and Check Point use the following protocols for clustering:
Protocol 144 - Nokia

This is the identifier protocol. It monitors the cluster(s), and assigns work. Essentially it
runs the inner workings of the cluster. This protocol is used for the discovery of new
cluster members, and handles failover.
Here is an example of a tcpdump on the cluster master interface:

dm3[admin]#tcpdump -i eth-s4p1c0 -n -s 500 -v ip and proto 144
09:28:25.713209 O 172.16.31.101 > 224.0.1.144: "2" KEEPALIVE 250mSec, Members=1,

Seq=200319 [tos 0xc0] (ttl 255, id 15987)
Seq=200320 [tos 0xc0] (ttl 255, id 15988)
Seq=200321 [tos 0xc0] (ttl 255, id 15991)
09:28:26.430566 I 172.16.31.102 > 224.0.1.144: "2" JOIN Perf=200 [tos 0xc0] (ttl 255, id
1127)
09:28:26.430672 O 172.16.31.101 > 172.16.31.102: "2" JOIN_NAK (Operation now in
progress) [tos 0xc0] (ttl 255, id 15993)
Seq=200322 [tos 0xc0] (ttl 255, id 15994)
09:28:26.431132 O 172.16.31.101 > 172.16.31.102: "2" JOIN_ACK Node 2, M=172.16.31.101,
172.16.31.103(1:50:5a:10:1f:67) M=172.16.31.101 10.1.1.103(1:50:5a:1:1:67) M=10.1.1.101
10.250.135.103(1:50:5a:fa:87:67) M=10.250.135.101 [tos 0xc0] (ttl 255, id 15995)
09:28:26.431761 I 172.16.31.102 > 172.16.31.101: "2" CLIENT_KEEPALIVE Member=2, Seq=0
[tos 0xc0] (ttl 255, id 1128)
Many thanks to Morten Bonde (Nokia, S.E., Copenhagen) for providing insight on this.
Here is how Morten reads the traffic.
The cluster ID is ‘2’. The current master of the Cluster with the ID 2 multicasts
KEEPALIVE packets to the multicast address 224.0.1.144 every 200 - 250 msec. The
master sends this traffic out to announce that there already exist a master for cluster
number 2 - even though it knows that the only node in the cluster is the master itself.

The local node (a cluster prospect) with the IP address 172.16.31.102 sends a JOIN
packet to the multicast address 224.0.1.144. After sending the JOIN_NACK to the
cluster prospect, the master informs the existing members about the member.
The master responds directly back to the prospect's unicast IP-address with a
JOIN_NACK, which basically means "Hang on ... I'll prepare the cluster for accepting a
new member". The master then sends a KEEPALIVE packet to the existing members of
the cluster, informing them that the number of nodes in the cluster has increased
(indicated by the "Members=2").
The master now sends a BUCKET_ASSIGN packet to the prospect's unicast address
saying that whenever the hashing function returns "59" that means that the new node
(as soon as it has become a full member of the cluster) should handle that traffic. At
some point the master assigns more work with a BUCKET_ASSIGN unicast packet,
which is acknowledged by the member with a BUCKET_ASSIGN_ACK multicast packet.
09:42:05.577419 O 172.16.31.102 > 172.16.31.101: "2" CLIENT_KEEPALIVE Member=2, Seq=751
[tos 0xc0] (ttl 255, id 3475)
09:42:06.499296 I 172.16.31.101 > 172.16.31.102: "2" BUCKET_ASSIGN Bucket 59 -> Node 2
[tos 0xc0] (ttl 255, id 16850)
09:42:06.499390 O 172.16.31.102 > 224.0.1.144: "2" BUCKET_ASSIGN_ACK (Node 2) Bucket 59
[tos 0xc0] (ttl 255, id 3481)
09:42:07.275009 I 172.16.31.101 > 172.16.31.102: "2" BUCKET_ASSIGN Bucket 97 -> Node 2
[tos 0xc0] (ttl 255, id 16851)
TCP 11003 & TCP 11004 - Nokia

These two ports are essential used for cluster configuration and cluster member
monitoring. If you perform a ‘get info’ from the Voyager monitoring page, this is the
protocol that is being used. It also is used for what version of IPSO is running, and the
current load of each machine. Another function is to monitor the configuration of each
cluster member i.e. weighting. Furthermore, it is used for exchanging information such
as what IP addresses cluster members have. In IPSO 3.7, both those ports will be
replaced with TCP port 1111.
An example of this follows. Here you can see update traffic on port 11003 being
received.
dm2[admin]# tcpdump -i eth-s2p1c0
tcpdump: listening on eth-s2p1c0
23:37:05.605716 O 0:a0:8e:10:5c:69 0:a0:8e:20:1e:14 0800 82: 172.16.31.3.10004 >
172.16.31.2.11003: S 810354388:810354388(0) win 65535 <mss 512,nop,wscale
1,nop,nop,timestamp[|tcp]>
UDP 8116 – Check Point

This is used for Check Point’s state synchronization. It is also capable of monitoring the
status of members of a cluster at the Check Point level.
10:03:42.147120 O 0.0.0.0.8116 > 172.16.31.0.8116: udp 24
10:03:42.147144 O 0.0.0.0.8116 > 172.16.31.0.8116: udp 28

Troubleshooting
Troubleshooting the cluster can be done through Voyager. A functioning cluster should
look like these when you go to ‘Monitor’ and ‘Cluster’ in Voyager. The page will refresh
every 30 seconds.
dm1
dm2

During the boot phase of Nokia GW we should see the following output:
dm1[admin]# CPHA : Getting into preconfigured mode...

Feb 9 15:29:00 dm2 [LOG_CRIT] kernel: CPHA : Getting into preconfigured
mode...
This will tell us that the CPHA is loading.
If we are testing load balancing and failover with an FTP session such as getting a file
from an internal FTP server, and the session is stopping, but not resuming once the
cluster slave takes over in a failover, ensure that the state tables on the master and
slave are synchronizing by using the following command: fw tab -t connections –s
Wait a moment and perform the following operation. They should have the same number
of connections indicated by #VALS, if the state is being synchronized.
dm1[admin]# fw tab -t connections -s

HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 16 18 18
dm2[admin]# fw tab -t connections -s

HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 16 26 18
More On State Tables
Here is the output from two state tables. The command ‘fw tab -t connections’ was
performed after logging into the FTP server, but before the data transfer was initiated.
Although in hex, we see the FTP connections, which I have underlined as 00000015.
dm1[admin]#fw tab -t connections

<cff5ddc2, 00000469, 0afa8764, 00000015, 00000006; 00000000, 00004001,
ffffff10;
3594/3600>

<cff5ddc2, 00000469, 0afa8764, 00000015, 00000006; 00000000, 00004001,
ffffff10;
3594/3600>
Here is the full dump. The FTP connection (port 21) is 00000015 while the FTP data
connection is 00000014.

localhost:
-------- connections --------
attributes: refresh, sync, expires 60, free function 4074551916 4, kbuf
1, hashs
ize 16384
<0afa8764, 00000464, 0afa87c9, 00000017, 00000006; 00000000, 00004001,

ffffff00; 3596/3600>

<0afa8764, 00000465, 0afa87ca, 00000017, 00000006; 00000000, 00004001,
ffffff00; 3599/3600>
<ac101f01, 00000418, ac101f02, 00000100, 00000006; 00000000, 00004001,
ffff0800; 3600/3600>
<ac101f02, 00000424, ac101f01, 00000100, 00000006; 00000000, 00004001,
ffff0800; 3600/3600>
<cff5ddc4, 0000046f, cff5ddce, 00000101, 00000006; 00000000, 00004001,
ffff0800; 16/50>
<cff5ddc3, 00000463, cff5ddce, 00000101, 00000006; 00000000, 00004001,
ffff0800; 5/50>
<cff5ddc3, 00000464, cff5ddce, 00000101, 00000006; 00000000, 00004001,
ffff0800; 35/50>
<cff5ddc4, 00000470, cff5ddce, 00000101, 00000006; 00000000, 00004001,
ffff0800; 46/50>
<cff5ddc2, 0000046a, 0afa8764, 00000014, 00000006; 00000000, 00004001,
ffff8202; 3599/3600>
<cff5ddc2, 00000469, 0afa8764, 00000015, 00000006; 00000000, 00004001,
ffffff10; 3594/3600>
Useful Commands
1 Although sometimes inconsistent the ‘cphaprob state’ command is useful for

determining if both GWs are now active.
dm1[admin]# cphaprob state
Working mode: Service
Number Unique Address State
1 (local) 172.16.32.1 active

2 172.16.32.2 active
2 A tcpdump on the cluster sync interface will display the cluster sync information.
This is an example of the TCP 11003 and 11004 services that we created earlier.

tcpdump: listening on eth-s2p1c0
23:37:05.605716 O 0:a0:8e:10:5c:69 0:a0:8e:20:1e:14 0800 82:
172.16.31.3.10004 > 172.16.31.2.11003: S 810354388:810354388(0) win
65535 <mss 512,nop,wscale 1,nop,nop,timestamp[|tcp]>
3 The ‘clish’ and then ‘show clusters’ commands are useful in determining cluster
status.
dm1[admin]# clish
Nokia> show clusters
CID 1
Cluster State up
Member ID 2
Protocol State member
System Uptime At Join 0:00:03:32
Performance Rating 85
Failure Interval 4000
Cold Start Delay 30

Number of Interfaces 3
Primary Interface eth-s2p1c0
Interface eth-s1p1c0
IP Address 10.1.1.2/24
Cluster IP Address 10.1.1.3
Hash default
IP Address 172.16.31.2/24
Hash default
IP Address 10.250.135.2/24
Hash default
Member(s) information
Number of Member(s) 2
4 Check Point NG’s Smart View Tracker is also useful in monitoring the state
synchronization.

Switches and Routers
When a cluster is running in Multicast mode, a virtual Multicast MAC address is mapped
to the cluster IP address and all nodes in the cluster expect to receive packets sent to
this address. Catalyst 5000 switches do not advertise this address automatically. In
order for this mode to work off a Catalyst 5000, you must create a static CAM entry for
each port that an inside interface of your cluster is attached to.
The format of the Multicast MAC address is the following:
01:50:5A:00:<X>:<Y>
...where X and Y are the last 2 numbers in the IP address of the cluster inside interface
(in hex), so:
10.0.32.4 is 01.50.5A.00.20.04
If you are using three octets, i.e. 10.1.32.4, then you would use the last 3 numbers in the
IP address.
If you have a 2 node cluster, with the inside interfaces attached to Catalyst ports 10 and
11, where 10 and 11 are in vlan3, then to set the static CAM entry for those ports, issue
the command:
Console> (enable) set cam static 01-50-5a-0-20-4 3/10,3/11

Static multicast entry added to CAM table.
Console> (enable) sho cam static
VLAN Destination MAC Destination Ports or VCs
---- ------------------ ------------------------
4 01-50-5a-00-20-04 3/10-11
Matching CAM Entries = 1
CISCO ROUTERS AND CLUSTER MULTICAST MODE
Cisco routers do not automatically discover multicast addresses. In order for the router
to see it, you need to create a static ARP entry on the router for the interface on the
same LAN.
If you have a cluster with the cluster address of 10.0.32.4, then on the router, issue the
command:
router(config)#arp 10.0.32.4 0150.5a00.2004 arpa
Note, in Cisco terminology, 3/10 and 3/11 indicate module 3 ports 10 and 11, not the
VLAN number 3 ports 10 and 11. So, in a smaller chassis, such as the 1900, 2900, and
3500, the module number is always 0, because these are not modular switches.

A Caution On Catastrophic Failure
Consider if you will that cluster configuration has three nodes. There may be several
internal networks that these nodes are passing traffic for. The each node of the cluster is
then plugged into a single switch that leads to the Internet.
Should the sole Internet switch die, you will experience catastrophic failure of the cluster.
This is a result of the cluster using multicasts for passing traffic. In the next release, IP
forwarding (although slower in terms of performance) will overcome this.
It is possible to overcome this, and since we are implementing a cluster design for high
availability, one would use multiple switches and trunk the switches.

Ipso3.6 Fp3 v1.9 Clustring

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Ipso3.6 Fp3 v1.9 Clustring

Încărcat de

Drepturi de autor:

Formate disponibile

Nokia Clustering IPS0 3.

6FCS4 / Check Point NG FP3

Questions should be directed to the author.

This and other configuration documents can be found at:

© Copyright North America All Rights Reserved, July 2002

Jeff Mousseau 2/6/2003 1

It does not focus on how to configure a Nokia GW for optimum security. It is

The author is not responsible loss of data, damage or monetary loss.

Supported applications: Check Point NG FP3.

Number of supported nodes: 2 to 4 with an unlimited number of interfaces.

Jeff Mousseau 2/6/2003 2

Failover time: 4 seconds.

State versus clustering synchronization

Jeff Mousseau 2/6/2003 3

Jeff Mousseau 2/6/2003 4

Version 1.6 (ipso3.6_fp3_v1.6.doc) 10.12.02

Version 1.7 (ipso3.6_fp3_v1.7.doc) 11.10.02

Version 1.8 (ipso3.6_fp3_v1.8.doc) 11.13.02

Version 1.9 (ipso3.6_fp3_v1.8.doc) 02.06.03

Jeff Mousseau 2/6/2003 5

IPSO 3.6 FCS4 & CP NG FP3

Jeff Mousseau 2/6/2003 6

IPSO Operating System

N.B. Be sure to check both GWs.

Using Voyager to verify the version of IPSO that is running:

From the Voyager GUI ensure proper interface configuration:

Jeff Mousseau 2/6/2003 7

1 Although it is not as critical as it once was, it is still recommended by the author

hostid = the hostname of the external interface = the NG Workstation name

Local Time Setup

Jeff Mousseau 2/6/2003 8

Local Time Setup

In the above example, the GW was configured to point to tock.utoronto.ca.

Jeff Mousseau 2/6/2003 9

By clicking on the apply button, we will be taken to the configuration page.

Be sure to reboot the GW(s).

dm1[admin]# tail /var/log/messages

Jeff Mousseau 2/6/2003 10

Jeff Mousseau 2/6/2003 11

10.1.1.2/24 int1 (eth-s1p1) - internal segment

Jeff Mousseau 2/6/2003 12

c:\>route add –p 10.250.135.1 mask 255.255.255.255 10.1.1.1

dm1 and dm2

Jeff Mousseau 2/6/2003 13

Cluster Status Table:

Cluster ID: Id of the cluster.

Cluster Members Table:

Jeff Mousseau 2/6/2003 14

Jeff Mousseau 2/6/2003 15

Jeff Mousseau 2/6/2003 16

Installation of NG-FP3 is becoming more simplified by the introduction of Checkpoint’s

1 Depending on the FTP daemon, I have seen inconsistencies when attempting to

Load new package from:

Choose an installation method (1-5): 3

Processing package CP_FP3_IPSO.tgz ...

Would you like to :

1. Install this as a new package

Running Pre-install script

Jeff Mousseau 2/6/2003 17

Please do the following if the INSTALL/UPGRADE is Successful:

1) Logout and re-login.

PKG_INSTALL: *******************INSTALL/UPGRADE PROCESS COMPLETED

End of new package installation

* Installation completed successfully *