CommuniGate Pro Best Practices

Benchmarking

June 11, 2007

Best Practices: Benchmarking

Introduction
This document is provided by CommuniGate Systems Services as a helpful Best
Practices guide to benchmarking a CommuniGate Pro Dynamic Cluster.
Following is a simplified overview diagram of a CommuniGate Pro Dynamic
Cluster:

Testing Environment
Prior to installing and configuring CommuniGate Pro, it is recommended to setup
an environment for testing. This would include the following:

1. DNS environment: a good, fast DNS server (such as BIND 9) providing

internal-only, “fake” domains for use in testing:

Page 2 of 12
Best Practices: Benchmarking
• Example.lan: your CGP “local domain”
• Remote.lan: a remote domain, for relaying outbound messages to.
It is critical that the DNS zones used in these tests include proper A-
records and PTR-records for all systems in the benchmark environment,
including the load generators. Improper DNS setup will greatly decrease
your test results.

2. Load generators: the test environment will include, at a minimum, the

following systems for the benchmark load generators:

a. Benchmark manager – the main testing box, could also function as

other roles

b. Load generator – SPECmail or mstone, or others

c. SMTP sink – the postfix smtp-sink is a good, free tool

The benchmark manager, load generators, and sinks can run on one box
or many. Generally speaking, 2-4 is sufficient.

3. Tuning: Depending on your test platform, some tuning may be required.

Please reference the following “Scalability” page of the CommuniGate Pro
Guide for recommendations on tuning:

http://www.communigate.com/CommuniGatePro/Scalability.html

Dynamic Cluster Overview

The foundation building block of the CommuniGate Pro Dynamic Cluster is the
“Shared File System” mounted on all CommuniGate Pro Backend Servers. The
Shared File System represents one or more file systems which are
simultaneously accessible by all Backend Servers. It is critical to note that this file
system must be shared (read and written) by multiple servers concurrently.
The Shared File System can be of one of two types:
1. A Network File System (NFS) on a Network-Attached Storage (NAS)
2. A Cluster File System (CFS) on a Storage-Attached Network (SAN)

System Layout

Every CommuniGate Pro Frontend server performs all of its high-I/O operations
in /var/CommuniGate, which is called the “BASE” directory. This directory may be
placed on a high-performance disk system for best performance. If located

Page 3 of 12
Best Practices: Benchmarking
anywhere except /var/CommuniGate, then /var/CommuniGate should be a
symbolic link to the real location.
Every CommuniGate Pro Backend server uses two main directory locations:
/var/CommuniGate
/var/CommuniGate/SharedDomains
/var/CommuniGate provides the configuration functions for each system, and is
generally stored on the local disk (although, can be placed on the NAS or SAN
for some highest-level performance requirements). However, /var/CommuniGate
CANNOT be shared across multiple systems – if locating /var/CommuniGate on
the SAN, each system must have its own separate directory.
/var/CommuniGate/SharedDomains must always be located on the shared
filesystem, whether on NFS or when using a Cluster File System on a SAN. All
Backend servers must have access to this shared filesystem. A symbolic link
should be used to link this location to other real directory locations.
Here are two example layouts then:
Layout #1: Configuration directory on local disk, SharedDomains on NAS/SAN:
System1: /var/CommuniGate
System1: /data (Shared File System mount)
System1: /var/CommuniGate/SharedDomains ->
/data/SharedDomains (symbolic link)

System2: /var/CommuniGate
System2: /data (Shared File System mount)
System2: /var/CommuniGate/SharedDomains ->
/data/SharedDomains (symbolic link)

Layout #2: Both the configuration directory and the SharedDomains on

NAS/SAN:
System1: /data (Shared File System mount)
System1: /var/CommuniGate -> /data/CommuniGate-system1
(symbolic link)
System1: /var/CommuniGate/SharedDomains ->
/data/SharedDomains (symbolic link)

System2: /data (Shared File System mount)

System2: /var/CommuniGate -> /data/CommuniGate-system2
(symbo2ic link)
System1: /var/CommuniGate/SharedDomains ->
/data/SharedDomains (symbolic link)

Page 4 of 12
Best Practices: Benchmarking
Installation

Please reference these basic installation instructions here for specifics on

installing on your specific platform:
http://www.communigate.com/CommuniGatePro/Install.html
Once installed, you log into the WebAdmin Interface (GUI) on each system at this
location:
http://<hostname>:8010
or
https://<hostname>:9010
Upon initial startup of CommuniGate Pro, and once you’ve connected to the
WebAdmin Interface, you will be prompted to enter a new “postmaster” password.
This password has complete administration rights to each system, and should be
created securely.

CommuniGate Pro Cluster Configuration

Once you've gotten this far, you want to configure your system something like the
following steps, for best performance. All paths below are “clicks” within the
CommuniGate Pro WebAdmin GUI.
Under “Settings” section:

4. Domain Name
Settings->General->Domain Name
Set to “<hostname>.example.com” on each host (or similar, if using a different
domain). Depending on your systems’ true host and domain names, this may be
filled in correctly by default. This domain name entry MUST match the Dynamic
Cluster license, such as “*.example.com”.

5. Licenses
Settings->General->Licenses
Install all license keys on all CommuniGate Pro servers. This is important, please
remember to install all licenses on all servers.

6. Cluster IP Addresses
Settings->General->Cluster
Add all Frontend and Backend IPs on all hosts:
Example:
Cluster Members

Page 5 of 12
Best Practices: Benchmarking
This Server Cluster Address: On restart Active
[first IP_____] [192.168.1.3]
Backend Server Addresses:
192.168.1.3_____________________________
192.168.1.4_____________________________
________________________________________
________________________________________
Frontend Server Addresses:
192.168.1.1_____________________________
192.168.1.2_____________________________
________________________________________
________________________________________

7. Stop the CommuniGate Pro servers on each host.

8. Add a “Startup” file on each server. Additional tuning recommendations for

various operating systems for CommuniGate Pro are always available in the
“Scalability” section of the CommuniGate Pro manual, available by default
here:
http://www.communigate.com/CGatePro/Scalability.html
For every server you are testing, you will need to add this file (default location):
/var/CommuniGate/Startup.sh
For Frontends:
#### Startup.sh
SUPPLPARAMS=”—-ClusterFrontend --DefaultStackSize 131072 \
--useNonBlockingSockets --closeStuckSockets \
--CreateTempFilesDirectly 10”
ulimit –n 16384
#### end Startup.sh
For Backends:
#### Startup.sh
SUPPLPARAMS=”—ClusterBackend --DefaultStackSize 131072 \
--useNonBlockingSockets --closeStuckSockets \
--CreateTempFilesDirectly 10”
ulimit –n 16384
#### end Startup.sh

9. Start all servers

Page 6 of 12
Best Practices: Benchmarking
Be sure to start at least one Backend server first, as the first Backend server
automatically becomes the “Cluster Controller”. A Cluster Controller is required
before any other servers can join the cluster:
# /etc/init.d/CommuniGate start

10. If the restart was successful, the Cluster Controller will show in the log the
rest of the service:
http://<controller hostname>:8010
Monitor -> Logs -> Display

Confirm that all systems join the cluster. This page will display the entire
cluster:
MONITORS->Cluster
(Note: The Cluster monitoring page is also a great place to manage all of your
systems from. If you open the WebAdmin Interface on your Cluster Controller,
then use the system links on the MONITORS->CLUSTER page, you can then
easily login to the WebAdmin Interface for all your systems).

Tuning optimizations
You must set these settings on all servers for most of these options. Cluster
settings (as part of any “Cluster-Wide” setting) only need to be set on one server,
and will take affect for all servers within the cluster.

1. Server-Wide and Cluster-Wide Client IPs

Be sure to add your “local” IP ranges to the Cluster-Wide Client IPs
Settings->Network->LANIPs->Cluster-Wide

2. Queue
Settings->Queue->Settings->Message Queue->Queue Foldering =
100

Settings->Queue->Settings->Message Enqueuer->Processors =
50

Settings->Queue->Settings->Message Dequeuer->Processors =
10

3. DNS Resolver

Page 7 of 12
Best Practices: Benchmarking
Settings->Obscure->Domain Name Resolver->Concurrent
Requests=25

4. SMTP Sending
Settings->SMTP->Sending->Channels=1000

Settings->SMTP->Sending->Channels/Host->Other=50

5. SMTP Receiving
Settings->SMTP->Receiving->Channels=1000

Settings->SMTP->Receiving->Disconnect after: [5] errors and

Deny Access for: [15 minutes]

6. LOCAL
Settings->LOCAL->other=50

7. Access
Settings->Access->POP=1000

Settings->Access->IMAP=1000

Settings->Access->PWD clients=50
# (set PWD channels to at least 5 * <number of systems in
cluster>)
Also note here, if testing the WebUser Interface, it would probably be best to set
the HTTPU->Listener ports to listen on 80 for HTTP and 443 for HTTPS, rather
than the default 8100/9100.

8. Cluster-Wide Domain Defaults

Users->Domain Defaults->Cluster-Wide->”Account Storage”-
>Foldering Method = Hashed 2 Levels

Users->Domain Defaults->Cluster-Wide->”Account Storage”-

>Rename in Place =Yes

Users->Domain Defaults->Cluster-Wide->”Account Storage”-

>Generate Index =Yes

9. TextMailbox or MaildirMailbox
One can use with two options - different filesystems perform different with each.
MaildirMailbox format uses more small files and many more filesystem metadata
operations, and additionally can take advantage of single-instance message
store (which especially benefits Enterprises, Educational facilities, and ASPs

Page 8 of 12
Best Practices: Benchmarking
where one message may be sent to thousands of recipients). TextMailbox format
uses bigger mailbox files but these files are indexed for high performance. A
combination using MaildirMailbox in the INBOX and TextMailbox is also
possible – this method can be easily scripted in provisioning scripts.
For most Cluster File Systems and NFS, TextMailbox format may perform better
due to less overall metadata file operations and good indexing. The default is
TextMailbox, and in most cases it is recommended.
When using the “TextMailbox” (default) format, it is recommended to reduce the
default mailbox index size and increase the concurrent large buffers, which are
configured in the “Settings->General->Other” settings page:
Settings->General->Other->”TextMailbox Manager”-
>Concurrently used large buffers=”50”

Settings->General->Other->”TextMailbox Manager”->Index
Mailboxes larger than=”1024K” (1 Mbyte)
Optionally:
Objects->Domains->Account Defaults->Server-Wide->New
Mailboxes=TextMailbox (or MailDirMailbox)

Objects->Domains->Account Defaults->Cluster-Wide->New
Mailboxes=TextMailbox (or MailDirMailbox)

Domain Setup
Finally, add your clustered domain(s), and proceed with adding accounts.
Domain setup is managed in the WebAdmin Interface at the location:
Users->Domains
Be sure to create all your domains as “Clustered Domains”.

Benchmark Tools
There are many tools available for benchmarking and testing e-mail, calendaring,
and voice services. We provide some details of one of these, SPECmail2001,
here.

SPECmail2001
SPECmail2001 is a standardized benchmark developed by messaging vendors
and research organizations that measures mail server performance using a real-
world workload. CommuniGate Systems is a member of the SPEC organization
"The Standard Performance Evaluation Corporation, at http://www.spec.org) and
a voting member of the SPECmail, SPECimap, and SPECsip committees. As

Page 9 of 12
Best Practices: Benchmarking
well, CommuniGate Pro is the current world-record holder in the SPECmail2001
benchmark, beating all previous records by 67% or more at 2.5-million users.
These results provide a good overview of what a large SPECmail test looks like:
http://www.spec.org/mail2001/results/mail2001.html

The CommuniGate Systems Services department has used the SPECmail2001

application to test CommuniGate Pro implementations prior to going live in
production. The purpose of using this application in validation testing is that the
initialization and benchmark test stages generate significant mail delivery and
access load on the mail system.
Since 2004, CommuniGate Systems Professional Services has regularly tested
most new implementations with SPECmail2001 to ensure that the system is

Page 10 of 12
Best Practices: Benchmarking
prepared for deployment. While SPECmail2001 is not able to completely
duplicate the 'real world' environment, it is capable of demonstrating that the
initial design of the system is fundamentally solid, and that the hardware used is
stable and capable of providing sufficient performance.
The types of problems which SPECmail2001 can discover includes the following:
• Faulty or insufficient hardware
• DNS server errors or configuration problems
• Load Balancer mis-configuration
• Network performance, routing, or stability issues
• Shared storage performance
• External authentication or performance problems with integrated
applications
• Content filtering (anti-spam, anti-virus) and plugin performance

SPECmail Preparation
SPECmail2001 is a Java-based application, and runs best on a relatively recent
version of Sun Java (1.4.2 or later). Other java-like implements (such as GNU
Java) are not sufficient - we highly recommend you download the latest version
of Java from http://java.sun.com.
The SPECmail2001 application should be capable of running on any Sun Java-
supported platform, including but not limited to Linux, Solaris, Windows, and Mac
OSX. Generally speaking, at least two "load-generator" systems are required for
SPECmail testing, one acting as the SMTP/POP load generator, and the other
acting as the SMTP sink for outbound delivery. These (minimum) two systems
cannot be part of the CommuniGate Pro Dynamic Cluster, and for best results,
should be placed in a location to route connections through any relevant load-
balancers, routers, and firewalls in order to verify not just the CommuniGate Pro
system(s), but also the entire production environment.

SPECmail2001 Overview from SPEC.org

SPECmail2001 is a standardized mail server benchmark designed to measure a
system's ability to act as a mail server servicing email requests, based on the
Internet standard protocols SMTP and POP3. The benchmark characterizes
throughput and response time of a mailserver system under test with realistic
network connections, disk storage, and client workloads. The benchmark focuses
on the ISP as opposed to Enterprise class of mail servers, with an overall user
count in the range of approximately 10,000 to 1,000,000 users. The goal is to
enable objective comparisons of mail server products.

Page 11 of 12
Best Practices: Benchmarking

SPEC® and the benchmark name SPECmail® are registered trademarks of the
Standard Performance Evaluation Corporation. Competitive benchmark results
stated above reflect results published on www.spec.org <http://www.spec.org>
as of August 15, 2005. The comparison presented above is based on Standard
Performance Evaluation Corp.'s benchmark that measures a system's ability to
act as a mail server servicing email requests, based on the Internet standard
protocols SMTP and POP3. For the latest SPECmail® benchmark results, visit
http://www.spec.org/mail2001.

Page 12 of 12