Network Monitoring Using Nagios

LiveSecurity Editorial,
Watchguard “Smarter Together” Contest 2003

Daniel Heinze
network4you GmbH
Albert-Rosshaupter-Straße 35

D-81369 Munich
Germany
d.heinze@network4you.com

05/26/2003
Imagine a server crash ...

and you as the systems administrator are the last one to know. A central network
monitoring unit may help you avoid those unpleasant occasions when your boss bursts
into your office, to let you know that the web server is down, or an annoyed customer
calls you on the phone to tell you that the system crashed overnight because of a hard
disc overflow. Having all the latest information about your networks status at hand and
being immediately informed if any incidents occur, makes life easier for you and saves
the nerves of all other parties concerned.

Nagios (former NetSaint) is one example of an Open Source network monitoring tool,
published under the GNU Public License GPL, which can help you to accomplish your
administration tasks more efficiently. It offers customizable monitoring mechanisms for
various components of your network, a notification system and an intuitive graphical
representation.

The Nagios process is the core of the monitoring system, collecting all the data and
writing it to log-files. After detecting a problem within the network, the Nagios process

26.05.2003 2 Daniel Heinze

sends a notification to the responsible administrator. Using a named pipe called External
Command File you can send additional commands to the Nagios process. The Nagios Web
Interface also utilizes this method to read all the data from the log-files and display it in
your web browser.

Installation of Nagios as a central monitoring unit

The installation of the Nagios demon is fairly easy. For a test setup, it is enough to have
a default server installation of a common Linux distribution with a running web server.
After a Nagios user and an appropriate group have been set up, you can compile and
install the Nagios sources (http://www.nagios.org/download) with the following
commands:

configure --with-cgiurl=/nagios/cgi-bin --with-hdmurl=/nagios/ \

--with-nagios-user=nagios –with-nagios-grp=nagios \
–-with-template-extinfo
make all
make install

In addition you might find it useful to install some example configuration files:

make install-config

and some starting script examples:

make install-init.

As soon as the installation is complete, text files are used to configure the Nagios
process. The file >>nagios.cfg<< contains the basic configuration settings and includes
further files, such as >>hostgroups.cfg<<, >>hosts.cfg<<, >>contactgroups.cfg<< and
>>contacts.cfg<<. These files primarily serve the purpose of defining hosts, services and
contact persons. They are well commented and easy to understand.

Plugin Architecture

In the next step we download and install the available Nagios plugins. In contrast to
other commercial monitoring products, Nagios doesn't provide any built in mechanisms
to generate network status requests but utilizes other programs for this task, the so-
called plugins (compiled executables or scripts like Perl, shell, etc.). Nagios uses the
plugins to determine the status of hosts, network components or services. The plugin
return-code represents the status information (0 = OK, 1 = warning, 2 = critical, 3=
unknown). Additionally, Nagios reads the first line of the plugin output (“stdout”), saves
the information in one of the log-files and, should an incident occur, sends it as a
notification to the administrators.

For detailed installation instructions and hints concerning the various plugins see
http://www.nagios.org/docs.

Monitoring Windows Server Using the NS Client Plugin

An administrator of any large network will be dealing with one or more Windows Servers.
Monitoring these servers requires an OS-specfic agent like NS Client (Netsaint Windows
Client, http://nsclient.ready2run.nl/download.htm). The NS Client uses the default
Windows API to retrieve performance data from the OS, such as CPU and memory load,
disc space, service state or system uptime. Integration into the NS Client is possible for
any other performance counter, which works together with the MS Windows Performance
Tool. This means that other Windows-specific information can also be processed, for
examp le Exchange Server, SQL Server or RAS data. As soon as the NS Client is installed
on the Windows Server, the “check_nt” plugin is able to read the performance counter

26.05.2003 3 Daniel Heinze

data and transfer it to the Nagios Central.

Described below is the configuration of the NS Client, necessary for the monitoring of a
Microsoft DNS Server.

Copy the Nagios Client into any directory on the Windows Server and install it as service
using

pNSClient /install

Afterwards, start the service “NetSaint NT Agent” manually using the Windows Services
Configuration.

On the Linux computer modify the file >>checkcommand.cfg<<, to provide the

parameters required to properly run the “check_nt” Plugin. To view the status of the
configuration of the DNS Service, the entry will be as follows…

define command{
command_name check_nt_service
command_line $USER1$/check_nt -H $HOSTADDRESS$ -p 1248 \
-v SERVICESTATE -d SHOWALL -l $ARG1$
}

Additionally specify the allocation of host and plugin to be executed in the file
>>services.cfg<< .

define service{
use generic-service
host_name dc2
service_description DNS-Server Services
is_volatile 0

26.05.2003 4 Daniel Heinze

 check_period 24x7
max_check_attempts 3
normal_check_interval 1
retry_check_interval 1
contact_groups nt-admins
notification_interval 300
notification_period 24x7
notification_options w,u,c,r
check_command check_nt_service!DNS
}

If required, change any settings for the graphical output of the individual elements on the
Web Server in the file >>hostextinfo.cfg<< .

Shorty after re-starting the Nagios demon, the first status report on the Windows Server
is available via the Nagios Web Interface.

Using the SNMP protocol for communication

The Simple Network Management Protocol (SNMP) makes it possible to retrieve more
detailed information, than shown before. Using SNMP for network monitoring requires
having a Management Station, an SNMP Agent and an Information Basis (Management
Information Base, MIB). Nagios acts as the Management Station. Using the SNMP plugin,
Nagios accesses the SNMP agents. Today, many network components, like hosts, routers,
switches or firewalls support communication via SNMP. There are two ways, in which the
SNMP agents running on these components communicate with the Nagios process. First,
an agent answers a query sent by Nagios. Second an agent sends information to the
Management Station on it’s own initiative. The SNMP protocol provides the following core
functionalities: GET (to query data from an agent) and TRAP (enables an agent to send
information to the Management Station). The MIB specifies the objects which are
monitored on a certain component. For example on an SQL Server this might be the
number of performed transactions per second.

Watchguard Firebox Vclass SNMP Trap Handling

Nagios cannot completely replace large SNMP Management applications like HP OpenView
or OpenNMS. However, small manual changes in the Nagios configuration allow the
Nagios process to analyze the SNMP-Traps of the Linux USC-SNMP demon (snmptrapd).
Compared to active queries, initiated by Nagios itself, this kind of event handling is called
Passive Service Check. This method allows detailed monitoring of Windows 2000 Servers
(using additional MIBs of third parties), which goes beyond the possibilities mentioned so
far (for example monitoring of SQL Server replications or retrieving detailed information
about Exchange Storages). Also the Watchguard Firebox Vclass series supports SNMP
and can therefore be part of an ambitious monitoring scenario. The following steps
describe how to make your Firebox part of the monitoring system.

Using the Vcontroller Software on your Firebox, specify the IP address of the Nagios
Server under System Configuration - > SNMP. Select Enable SNMP Traps to activate
support for SNMP Traps.

On the Nagios Server install the SNMP Trap demon's RPM packages (ucd-snmp and ucd-
snmp-utils). Afterwards create several scripts to specify the handling of the incoming
Traps.

FIREBOX VCLASS – trap à SNMPTRAPD (SNMPTRAPD.CONF) à HANDLE-

RAPID-TRAP à SUBMIT_CHECK_RESULT à NAGIOS.CMD à NAGIOS-
PROCESS

Additionally, the SNMP Trap demon's configuration file >>/etc/snmp/snmptrapd.conf<<

26.05.2003 5 Daniel Heinze

contains information about the handling of incoming traps.

traphandle RAPID-SYSTEM-CONFIG-MIB::rsAlarmTrap
/usr/local/nagios/libexec/eventhandlers/handle-rapid-trap

This entry specifies that all incoming Traps, marked as RAPID-SYSTEM-CONFIG-

MIB::rsAlarmTrap (Vclass MIB) are passed to the handle-rapid-trap script.

#!/bin/sh
# handle-rapid-trap
read host
read ip
read Trap
read Uptime
read rsAlarmId
read rsAlarmLabel
read rsAlarmTime
read rsAlarmLevel
read rsAlarmHostname
read rsAlarmMsg

case $host in 192.168.3.11)

hostname="V60"
;;
esac

# CITICAL WARNING
state=2

/usr/local/nagios/libexec/eventhandlers/submit_check_result \
$hostname "SNMP-Trap" $state "$rsAlarmLabel Error on $host: \
$rsAlarmMsg"

exit 0

This script extracts all relevant information from the incoming traps, allocates the
respective environment variables (for example $host, $rsAlarmLabel, $rsAlarmMsg) and
passes them to the submit_check_result script.

#!/bin/sh
# submit_check_result

echocmd="/bin/echo"
CommandFile="/usr/local/nagios/var/rw/nagios.cmd"
datetime=`date +%s`
cmdline="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$1;$2;$3;$4"
`$echocmd $cmdline >> $CommandFile`

Because the Trap messages cannot be directly processed by Nagios, this script acts as a
parser, translates the messages into a suitable syntax and writes them down into the file
nagios.cmd. Plugins gather information from the nagios.cmd file about their respective
return values.

As a next step, insert the following lines into the >>nagios.cfg<< file, which tells the
Nagios Demon to process “passive checks” (SNMP Traps).

check_external_commands=1
command_check_interval=1

Finally specify the handling of captured SNMP Traps in >>services.cfg<<. The entry

26.05.2003 6 Daniel Heinze

 define service{
host_name V60
service_description SNMP-Trap
is_volatile 1
check_period 24x7
normal_check_interval 5
retry_check_interval 1
active_checks_enabled 0
passive_checks_enabled 1
max_check_attempts 1
contact_groups firewall-admins
notification_interval 30
notification_period 24x7
notification_options c,r
check_command check_none
}

enables the virtual entity “V60” to receive SNMP Traps.

Any events reported by the Firebox Vclass Traps are managed accordingly. Also crucial
security events are logged (for instance the discovery of a spoofing attack). Customizing
similar parameters, Nagios also acts as a very small intrusion detection system.

Note that by default any MIBs should be installed into the directory
/usr/share/snmp/mibs. Run snmptrapd with the options -m All and -O v.

The Nagios configuration settings and applications presented in this article cover only a
small range of the program's entire potential. Yet, all of the examples given, show that a
non-commercial product, free of charge, can easily be turned into a basic network and
host monitoring tool. Certainly some hands-on experience is required to handle the SNMP
Traps but in the end there is a reliable realtime notification system, which can be used,
not only to guarantee the availability of Watchguard Vclass Fireboxes, but also that of
any other network device supporting SNMP Traps.

26.05.2003 7 Daniel Heinze