Sunteți pe pagina 1din 100

Rescue CD System Management


Smart tools for better networks

Four Live systems on one disc
✚ SystemRescueCd ✚ Parted Magic ✚
Network & Security

✚ Clonezilla Live ✚ Redo Backup ✚


Network & Security

System 5Free
System Management

Back-up Tools

MySQL Forks

Spacewalk: Corral your Red Hat Servers Teamviewer

Easy remote control
Optimizing with SystemTap
Exchange 2010

Mastering Microsoft’s virtual ModSecurity

machine manager Protect your Apache
web server

Free Backup Tools

Is this nascent network

monitor nattier than

Oracle Clusters EXCHANGE 2010

Efficient data access What’s new with
with OCFS2 Microsoft messaging?

MySQL Forks
A perfect tool for

every task
ADMIN Magazine
Issue 01 £7.99

Windows • Linux • Unix • OpenSolaris


9 772045 070003
“You even get six-star customer support thrown in."
- PC Pro magazine



Scalable, affordable computing instances

from virtual machines to performance dedicated servers

We’ve been voted the UK’s best hosting company for 4 years running.
Isn’t it time you found out why?

ISO 9001: Quality ISO 14001: Environmental ISO 27001: Security
0800 634 9270 hosting
Welcome to Admin W E LCO M E


Our Admin special edition was so popular we’re back, with a new quarterly magazine that is all for
admins. Welcome to the first issue of Admin: Network and Security – a magazine for administrators
of heterogeneous networks.
In these pages, you’ll learn about tools for configuring, managing, and troubleshooting your net-
works. You’ll hear about cloud infrastructures, database systems, server plugins, and enterprise man-
agement applications. We’ll tune in to security and network protocols, and we’ll show you the latest
interoperability techniques.
This issue is packed with practical information for real networks. Red Hat published the source code
for their popular Network Satellite management tool in 2008, and a community version known as
Spacewalk soon followed. Our first article takes you on a walk with Spacewalk. Other special fea-
tures include a study of Icinga – a GPLed fork of the popular Nagios network monitoring
system, as well as an article on the forks and patches of MySQL and a roundup of
open source backup tools.
Farther in, we’ll show you the OCFS2 filesystem for Oracle clusters.
We’ll also look at some virtualization tools, including Micro-
soft’s vm2008 and OpenVZ – a container-based virtual-
ization alternative built on Linux. We’ll investi-
gate some security and system monitoring
tools, and we’ll cook up some recipe’s
for fast client configuration with the
Chef configuration management
If you’re an IT professional,
and you’re looking for a
magazine with detailed,
technical articles that
are relevant to the
real-life world you live
and work in, read on.
And if you would like
to receive future issues
of Admin delivered to
your door, see page 58
for information on sub-

W W W. A D M I N - M AGA Z I N E .CO M ADMIN 01 3

S E RV I C E Table of Contents

ADMIN Network & Security

Features Tools Virtualization

Management, monitoring, database Save time and simplify your workday Still finding your way through the
forks, and other pertinent glimpses at with these useful tools for real-world cloud? Keep on course with these cool
innovations across the IT landscape. networks. tools for virtual environments.

8 Spacewalk 36 OCFS2 48 Package Your Scripts

Walk on air with this free version of Build a database cluster with Oracle’s We show you how to use Debian’s
Red Hat's enterprise-ready Satellite Cluster Filesystem. packaging tools to deploy and manage
management server. scripts in the cloud.

14 Icinga
This is not your father's Nagios fork.

52 OpenVZ
42 Synergy Container-based virtualization tools like
20 MySQL Forks Too many monitors on your desk? OpenVZ are sometimes more efficient
We investigate some invaluable This handy tool lets you control your than hypervisor systems.
variations on the MySQL theme. servers from a single desktop.
Container Context

25 Exchange 2010 Resource Resource Resource

Container 1 Container 2 Container 3
What's new in Microsoft's email and Applications

messaging system? OpenVZ

Host Context

Abstraction Layer

28 Backup Tools Host System Kernel

We round up some of the best open

source backup utilities. 44 SystemTap
Optimize and troubleshoot your home- 60 SCVMM 2008
grown apps with this powerful profiling Exploring Microsoft's Service Center
tool. Virtual Machine Manager 2008.

34 BlackHat USA 2010

Learn the latest tricks of network
intruders at the BlackHat conference.

4 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

Table of Contents S E RV I C E

86 ModSecurity

Rescue CD
8 Spacewalk Careful admins know new exploits
Red Hat released the source code appear every day. Keep the intruders
for the Satellite Server management off your pages with this web

tool in 2008. Spacewalk is a new free application firewall for Apache web
version. servers.


Management Nuts and Bolts

Use these practical apps to extend, Timely tutorials on fundamental ✚ Four fabulous Live
simplify, and automate routine admin techniques for system Linux systems
tasks. administrators.
✚ SystemRescueCd –
64 Teamviewer 78 PAM
This popular remote control tool isn't The powerful Pluggable Authentication
A handy collec-
just for Mac and Windows anymore. System offers centralized authentication tion of advanced
for Unix and Linux systems.
rescue utilities
68 Chef
This snappy configuration manager lets ✚ Parted Magic –
you roll out Linux systems with a couple
of mouse clicks.
Partition your hard disk
✚ Clonezilla Live – Clone and
restore Linux systems
✚ Redo Backup – World‘s
easiest backup distro
86 ModSecurity
How safe is your web server? This
powerful Apache extension will help
d etails
74 Sysinternals
Get the pulse of your Windows network
keep intruders from getting control.
See p 6 for full
with this convenient collection of 92 Monitoring Daemons
management tools. Why write a custom script? A few simple
shell commands might be all you need
to monitor system daemons.

94 VPNs with SSTP

Build a virtual private network with
the Secure Sockets Tunneling Protocol


3 Welcome
4 Table of Contents
6 On the CD
98 Call for Papers

W W W. A D M I N - M AGA Z I N E .CO M ADMIN 01 5

S E RV I C E On the CD

Rescue CD Toolbox

On the CD
The CD included with this issue lets you boot to four
special-purpose Live Linux systems:
■ SystemRescueCd – This versatile rescue distro in-
cludes a copious collection of networking and trou-
bleshooting tools, including utilities for accessing and
repairing Windows systems.
■ Parted Magic – A little Linux that specializes in for-
matting, resizing, and recovering partitions.
■ Clonezilla Live – Clone systems for fast and easy
backup and restore. The bare-metal backup and re-
covery approach will bring back your system in a
fraction of the time.
■ Redo Backup – Another bare-metal backup tool with
an emphasis on simplicity.
Place the CD in the drive and reboot your system. (Make
sure your computer is configured to boot to the CD
DEFECTIVE ill be replaced. drive.) Choose a distro in the handy boot menu. See the
ective C w
Def Ds box titled “Resources” for links to more information on
email to
Please send an m.
the tools in our Rescue CD Toolbox. ■

[1] SystemRescueCd: []
[2] Parted Magic: []
[3] Clonzilla: []
[4] Redo Backup: []

6 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

11 cool projects!

■ Math Tricks: Solve math problems with Perl

■ Daily Tip: Perl with an SQLite database
■ AJAX: Add dynamic updates to web pages
■ isp-switch: Switch your computer to
another ISP if your connection goes down
■ MAC Addresses: Monitor your network for
unfamiliar MAC addresses
■ Multimeter: Read and interpret data from
an external device
■ Google Chart: Create custom graphs
■ Twitter: Help your scripts send Tweets
■ Webcam: Control a web camera with Perl
■ Perl Gardening: Water your house plants
with a little help from Perl
■ GPS: Extract data from a GPS device and
plot it on a Yahoo! Map
FREE DVD inside:
openSUSE 11.2
Linux Mint 8.0

New to Perl? Perl expert Randal L. Schwartz provides an in-depth

introduction to the principles of the versatile Perl language.
Then Perlmeister Mike Schilli explains how to speed up and debug your
scripts. Also inside: Get hands-on with a collection of some of the
Perlmeister’s best columns!

H O US E P E azin e . c o m /
g copy today
W E - m a
IND PO x order your
F AT lin u m it e d so
Supplies are
F e at u r e s Spacewalk
© patrimonio designs,

Managing Linux systems with Spacewalk

Moon Landing
As your network grows, managing Linux systems manually becomes time consuming and impractical.
Enter Spacewalk: an open source tool that takes the footwork out of network management.
By Thorsten Scherf

Spacewalk [1] is the open source first has to register with the server. nels. The base channel contains the
derivative of the popular Red Hat Net- Registration can be based either on RPM-based operating system, such
work Satellite Server. Red Hat pub- a username/​password combination as Red Hat Enterprise Linux, Fedora,
lished the source code for the server or an activation key that is pregener- or CentOS. The subchannels contain
in the summer of 2008, and the com- ated by the Spacewalk server. After additional software packages that are
munity has now released version 1.0. registration, the system appears in the independent of the operating system,
The application’s core tasks include server’s web GUI. such as the Red Hat Cluster Suite or
RPM package software provisioning, If the server has more resources, you the 389 Directory Server.
managing configuration files, and can assign them to the system at this Spacewalk can clone existing chan-
kickstart trees, thus supporting the point. Resources include software nels and create new channels from
installation of bare-metal systems. packages or configuration files that scratch. This feature gives you full
The approach that Spacewalk uses are normally organized in channels. control of the software stack that you
is quite simple: Before a system can A system always has exactly one provide via Spacewalk. Configura-
access Spacewalk’s resources, it base channel with optional subchan- tion channels help you distribute the

8 Admin 01 w w w. a d m i n - m aga z i n e .co m

Spacewalk F e at u r e s

configuration files for the software command centrally on the Spacewalk [4], or CentOS [3] Linux. Note that
packages. Spacewalk also keeps older server. Spacewalk does need a current Java
versions of the files to let you roll Installing new systems is also quite Runtime Version 1.6.0 or newer. You
back to a previous version at any time simple. Spacewalk has the installation can use the Open JDK for this; Fedora
if the need arises. files you need in the form of kickstart includes it out of the box. Admins
The software packages or configura- trees. The installation candidate uses on RHEL or CentOS can retrieve the
tion files can be installed either via a boot medium such as a CD, a USB package via the additional EPEL (Ex-
the target system or centrally in the stick, or a PXE-capable network card tra Packages for Enterprise Linux)
Spacewalk web front end. To avoid to contact the server. The First-Stage software repository.
spending too much time on the instal- Installer, which is part of the instal- Besides the Java package, an Oracle
lation of a large number of systems, lation medium, defines which server 10g database is also required for
you can assign systems to logical will handle the installation. installing Spacewalk. Oracle XE pro-
groups and apply the installation of The remaining installation steps are vides a free version of the database.
a resource to a group. For example, it handled by the Second-Stage Installer, The developers are currently working
might make sense to assign all your located on the Spacewalk server and hard on implementing support for an
web servers to a WWW-Server group transferred to the client system when open source database after identify-
in Spacewalk. When a new version of the installation starts. If you want to ing PostgreSQL as the best alterna-
the web server software is released, automate the installation fully, define tive to Oracle. As of this writing it is
you would simply tell Spacewalk to the kickstart file location in the boot hard to say when official support for
apply the update to the group, au- medium. The kickstart file is a kind of PostgreSQL will be available, but it
tomatically updating all the systems answer file that describes the proper- makes sense to check the roadmap
belonging to the group. ties of the installation candidate, such [5] or the mailing lists [6] at regular
The installation uses polling by as partitioning, software, language, intervals.
default; in other words, the client and firewall settings. Of course, you
systems query the server at a pre- can create a kickstart file on the Oracle XE
defined interval (which defaults to Spacewalk server and just include a
four hours) to see if new actions have link to the file on the boot medium. After installing the repository RPM
been defined since the last poll. If so, Spacewalk can manage any RPM- for your distribution, the first step is
Spacewalk then runs these actions. based distribution. You even have to install Oracle Express, which you
As an alternative, you can trigger the the option of operating client systems can download for free [7]. You will
installation of software packages and across multiple organizations. Using need version Besides the
other actions using a push approach. the web interface, the administrator database, you also need the oracle‑in-
The client system and the Spacewalk creates various organizations and as- stantclient‑basic and oracle‑instant-
server talk to each other constantly signs a certain number of system en- client‑sqlplus, which you can then
using the Jabber protocol. Any new titlements to them. Entitlements are install with Yum:
actions you define are immediately linked to certificates that Spacewalk
yum localinstall ‑‑nogpgcheck U
run on the client by Spacewalk. automatically generates during the in- oracle‑xe‑univ*.rpm
stallation. You can then add users to oracle‑instantclient‑basic*.rpm

Ground Control the organizations. oracle‑instantclient‑sqlplus*.rpm

If a client is registered with a user

Communications are always from the account from a specific organiza- Before configuring the database, you
client to the server; this is important tion, the system is assigned to this should make sure that your hostname
with respect to firewall rules. A list organization. When users from the points to the correct IP address in
of the network ports you need to en- organization logs into the Spacewalk your /etc/hosts to avoid problems
able can be found online [2]. Besides server, they will only see the systems
software package or configuration in their own organization. This fea- Listing 1: Oracle Listener Configuration
file installation, actions can also run ture is useful if you manage multiple cat >> /etc/tnsnames.ora << 'EOF'

arbitrary commands on the individual departments and prefer to manage the XE =

systems via the Spacewalk server. systems in the individual departments
For example, after creating a new separately. You just assign them to
ADDRESS = (PROTOCOL = TCP)(HOST = localhost)
configuration file for your web serv- different organizations, which, of (PORT = 1521))
ers and distributing it to the systems, course, you need to create up front. )
you need to restart the web server (CONNECT_DATA =
process to parse the new configura- Installation (SERVICE_NAME = xe)

tion instructions. Instead of logging )

in to each individual system or using Spacewalk can be installed on Red
a for loop, simply issue the restart Hat Enterprise (RHEL) [3], Fedora

w w w. a d m i n - m aga z i n e .co m Admin 01 9

F e at u r e s Spacewalk

with the Oracle Listener configuration the appropriate repository in /etc/ you can set up subchannels for the
later on. Use the following parameters yum.repos.d/. The following com- base channel and assign the subchan-
for the configuration: mand starts the installation: nels to clients as needed. After doing
so, you can use the subchannels to
HTTP port for Oracle Application U yum install spacewalk‑oracle
Express: 9055
distribute more RPM packages to the
Database listener port: 1521 Because this package depends on all systems. The packages can be your
Password for SYS/SYSTEM: Password the other Spacewalk packages, the own creations or RPMs from other
Start at boot: y package manager will automatically repositories.
The default HTTP port for the Oracle download and install the dependen- The easiest approach to setting up a
Express application (8080) is already cies in the next step. Then you can software channel is to use the web in-
occupied by the Tomcat application configure the application interactively terface (Channels | Manage Software
server, so you need to choose an al- with the setup tool or with the use of Channels | Create; Figure 1). Thanks
ternative port to avoid conflicts. an answer file (Listing 4). to the Spacewalk API, you can also
To talk to the database, you need to Pass the file in to the setup tool as script this process [8]. Call the script
configure the listener in the /etc/ follows: as follows:
tnsnames.ora file (Listing 1).
spacewalk‑setup ‑‑disconnected U ‑‑label=fedora‑12‑i386 U
Now you just need to make a few ‑‑answer‑file=answerfile ‑‑name "Fedora 12 32‑bit" U
changes to the database. To do this, ‑‑summary "32‑bit Fedora 12 channel"
log in to the database with sqlplus The configuration can take some time
and create a spacewalk user, to which to complete as the process sets up In the script, you need to provide
you could assign a password of the database tables. The setup tool the Fully Qualified Domain Name
spacewalk (Listing 2). then launches all the required ser- (FQDN) for the Spacewalk server
The standard configuration of Oracle vices. You can manually restart using and the user account for creating
Express supports a maximum of 40 the /usr/sbin/rhn‑satellite tool. the channels, such as the Spacewalk
simultaneous connections, which is To configure the system, launch the administrator account created previ-
not enough for Spacewalk operations. Spacewalk web interface via its URL ously. The Users tab also gives you
The instructions in Listing 3 change (http://​spacewalk.server.tld). Besides the option of creating more users with
the limit to a maximum of 400 con- contact information, you can also set specific privileges (Figure 2).
nections. the password for the Spacewalk ad- The channel you set up should now
Now you need to restart the database ministrator here. be visible in the Channels tab of the
by giving the /sbin/service oracle‑ web interface but will not contain
xe command. Software Channels any software packages. Although you
can upload software packages to the
Spacewalk Setup The next step is to set up an initial server in several ways, the method
software channel for the client sys- you choose will depend on whether
The next step is to install the Space- tems. When you register a client, you the packages are available locally
walk server. To do so, you need to must specify exactly one base channel (e.g., DVD) or you want to synchro-
include the Spacewalk repository as for the client; it will use this channel nize a remote Yum repository with
described previously. You should have to retrieve its operating system pack- the Spacewalk server. If you choose
a spacewalk.repo file that points to ages and their updates. Of course, the local upload, you can use the

Listing 2: Creating the Spacewalk User

sqlplus 'sys@xe as sysdba'
create user spacewalk identified by spacewalk
default tablespace users;
SQL> grant dba to spacewalk;
SQL> quit

Listing 3: Oracle Tuning

sqlplus spacewalk/spacewalk@xe

SQL> alter system set processes = 400 scope=spfile;

SQL> alter system set "_optimizer_filter_pred_pullup"

=false scope=spfile;

alter system set "_optimizer_cost_based_

transformation"=off scope=spfile;

SQL> quit
Figure 1: The easiest approach to setting up a software channel is to use the web graphical interface.

10 Admin 01 w w w. a d m i n - m aga z i n e .co m

Spacewalk F e at u r e s

them with the Spacewalk server.

Start by installing the Spacewalk Cli-
ent Repository RPM on the clients.
Fedora 12 systems have a matching
RPM [10], as do RHEL5 and CentOS5
[11]. On RHEL and CentOS, you also
need to install the RPM for the EPEL
repository [12] because the client tool
dependencies might not resolve cor-
rectly otherwise. The following com-
mand installs the Yum file on a 32-bit
Figure 2: Assigning individual users different privileges on the Spacewalk server. Fedora 12 system:

rpm ‑Uvh U

rhnpush tool, which you launch as too. Note that any RPM packages yum/1.0/Fedora/12/i386/spacewalk‑ U
follows: you build yourself must be digitally client‑repo‑1.0‑2.fc12.noarch.rpm
signed. Both the Spacewalk server
rhnpush ‑v ‑‑channel=fedora‑13‑i386 U
‑‑server=http://localhost/APP U
and the Yum client application will Then, use Yum to install the client
‑‑dir=/path/to/the/packages reject unsigned packages by default. tools:
Although you can disable this feature,
yum install rhn‑client‑tools U
To synchronize with a remote soft- it makes more sense to work with rhn‑check rhn‑setup rhnsd m2crypto U
ware repository, you simply need to digital signatures for security reasons. yum‑rhn‑plugin
specify the URL for the remote reposi- The rpm ‑‑resign RPM package com-
tory in the software channel proper- mand will sign the package for you; The easiest approach to registering
ties in the web interface (Channels | you must have GPG keys in place for a system on the server is to run the
Manage Software Channels | Fedora the RPM tool. The ~/.rpmmacros file rhnreg_ks tool, which expects a reg-
12 32-bit). Synchronization can take tells you the name and location of the istration key. You need to create the
a while to happen. Your other op- key (Listing 5). key up front on the Spacewalk server
tion here is the spacewalk‑repo‑sync To allow client systems to verify pack- (Systems | Activation Key | Create
command-line tool that downloads ages signed with this key, you need to Key). When you create a key, you can
software packages from a Yum reposi- deposit the public key on the Space- bind various resources to it, such as
tory to your own Spacewalk server. walk server, preferably in /var/www/ the Fedora 12 software channel just
To keep the server up to date, you can html/pub, which any client can ac- created here, or various configuration
use cron to run a script [9] at regular cess. The following command exports channels, if you have created some
intervals. This script will check your the public key from the GPG keyring: (Figure 3). Also, you can assign sys-
configured software sources and au- tem groups to the key. Systems that
tomatically download any new pack- gpg ‑‑armor ‑‑export > U use this key to register are granted
ages. This approach removes the need /var/www/html/pub/rpm‑gpg‑key access to the associated resources.
for manual synchronization. To allow the existing client systems To do so, specify the key you created
Incidentally, you can use the method to access the software packages you during the registration process: E
discussed here to set up subchannels, just uploaded, you need to register
Listing 4: Answer File
admin‑email = root@localhost
ssl‑set‑org = Tuxgeek Org
ssl‑set‑org‑unit = Tuxgeek OU
ssl‑set‑city = Essen
ssl‑set‑state = NRW
ssl‑set‑country = DE
ssl‑password = spacewalk
ssl‑set‑email = root@localhost
ssl‑config‑sslvhost = Y
Figure 3: Various resources can be bound to the registration key. Systems that use the key are given access
to the associated resources.

w w w. a d m i n - m aga z i n e .co m Admin 01 11

F e at u r e s Spacewalk

rhnreg_ks ‑‑serverUrl=U

http://spacewalk.server.tld/XMLRPC U


If all of this worked correctly, you will

see the system in the Systems tab of
the server web interface. Viewing the
system’s properties should also show
you the configured software chan-
nel. The easiest approach to check-
ing whether access to the channel is
working is to install a package from
the channel. If this doesn’t work, one
possible issue could be that the client
system is not using the Spacewalk
server’s CA certificate. The certificate
is stored in http://spacewalk.server. Figure 4: After completing the registration, the system appears in the Spacewalk server’s web interface.
tld/pub/ on the server and must be
stored in /usr/share/rhn on the cli- you want to install, such as Fedora distro‑trees/Fedora‑12 directory. If
ent side. The /etc/sysconfig/rhn/ 12, but the basic installation files, like all of this works out, just point to the
up2date file needs a reference to the the Anaconda tool. distribution you created when you
certificate. The software repositories you syn- made the kickstart file. When a cli-
As before, you need to enter the chronized earlier will not normally ent system is installed from scratch,
name of the Spacewalk server. You provide a kickstart distribution, and it will automatically pick up the right
only need to perform these steps on this means creating the distribution files from this source.
systems you have already installed. on the Spacewalk server. Again, just Although there are a number of ways
Any that you install from scratch via navigate to Systems | Kickstart | Dis- to install a Fedora 12 system from
the Spacewalk server are automati- tributions in the web interface and scratch, the easiest approach is to
cally registered with the server as part point to the required files. The easiest point any client PXE requests by your
of the installation process and can way to provide the files is to mount clients to the Spacewalk server with
thus access the server immediately an installation CD/​DVD for your pre- the next‑server command. Thanks to
(Figure 4). ferred distribution via the loopback Cobbler [13] integration, the Space-
device: walk server has a TFTP server and
Kickstart Installation mount ‑o loop U
any kickstart profiles that you have
/var/iso‑images/Fedora‑23‑i386‑DVD.iso U
set up. To confirm this, you can type
To automate the installation of new /var/distro‑trees/Fedora‑12 cobbler profile list at the com-
client systems, you need two pieces mand line.
of information on the Spacewalk When you create a Fedora 12 kick- When you boot a client system via a
server. One of them is a kickstart file start distribution, you simply point PXE-capable network card, you will
with details of how to install the new the Spacewalk server to the /var/ automatically see a list of the existing
system, including partitioning, the
software selection, and other settings
that you would need to provide for
a manual install. The easiest way to
create a kickstart file is to select Sys-
tems | Kickstart | Profiles in the web
front end.
After checking out the overview of
existing profiles, you can also create a
new profile. The kickstart distribution
must be specified as part of the pro-
file file. This does not mean the RPM
files that belong to the distribution

Listing 5: GPG Configuration for RPM

cat .rpmmacros
%_signature gpg
Figure 5: The system properties give you a neat option for handling a variety of administrative tasks for a
%_gpg_name Thorsten Scherf <>
system via the Spacewalk server.

12 Admin 01 w w w. a d m i n - m aga z i n e .co m

Spacewalk F e at u r e s

kickstart profiles. To install the cli- number of systems. The rhnsd service before rolling it out to your produc-
ent, simply select the required profile on the systems queries the Spacewalk tion systems. Thanks to the compre-
from the list. The client is then auto- server at predefined intervals to check hensive API, many tasks can also be
matically registered on the Spacewalk for new actions, such as software in- scripted. n
server. Existing systems can easily be stallations.
reinstalled using: When a system finds an action, it
then executes it. If the osad service Info
koan ‑‑replace‑self U
‑‑server=Spacewalk‑Server U
is enabled on the system, you can [1] Spacewalk project homepage:
‑‑profile=Kickstart‑Profile even run actions immediately with- [https://​­fedorahosted.​­org/​­spacewalk]
out waiting for the polling interval to [2] Spacewalk network ports:
This creates an entry in the system’s elapse. The client and the server then [http://​­magazine.​­redhat.​­com/​­2008/​­09/​
bootloader menu and automatically use the Jabber protocol for a continu- ­30/​­tips‑and‑tricks‑what‑tcpip‑ports‑are‑r
selects the entry when the system ous exchange. equired‑to‑be‑open‑on‑an‑rhn‑satellite‑pr
reboots. Finally, don’t forget the feature-rich oxy‑or‑client‑system/]
Spacewalk API, which is accessible at [3] RHEL5, CentOS5 Spacewalk Server Repos‑
System Management http://Servername/rhn/apidoc/index. itory RPM: [http://​­spacewalk.​­redhat.​­com/​
jsp on the installed server. This tool ­yum/​­1.​­0/​­RHEL/​­5/​­i386/​­spacewalk‑repo‑1.​
All of the systems registered on the gives you access to a plethora of func- ­0‑2.​­el5.​­noarch.​­rpm]
Spacewalk server retrieve their soft- tions that are not available in the web [4] Fedora12 Spacewalk Server Repository
ware packages from this source, with interface. RPM: [http://​­spacewalk.​­redhat.​­com/​­yum/​
no need to access external reposito- The API can be accessed with XML- ­1.​­0/​­Fedora/​­12/​­i386/​­spacewalk‑repo‑1.​­0‑2.​
ries. This method not only improves RPC, which makes it perfect for your ­fc12.​­noarch.​­rpm]
your security posture but also saves own Perl or Python scripts. A Python [5] Spacewalk Roadmap: [http://​
network bandwidth. With a registered script [8] for creating a software ­fedorahosted.​­org/​­spacewalk/​­roadmap]
system, you can customize various channel is just one example of access- [6] Spacewalk mailing list:
settings in the System Properties sec- ing the Spacewalk server via the API [http://​­www.​­redhat.​­com/​­spacewalk/​
tion (Figure 5). (Figure 6). ­communicate.​­html#​­lists]
For example, you can assign new [7] Oracle XE: [http://​­www.​­oracle.​­com/​
software or configuration channels, Conclusions ­technology/​­software/​­products/​­database/​
compare the installed software with ­xe/​­htdocs/​­102xelinsoft.​­html]
profiles on other systems, or create Spacewalk gives administrators a very [8] Spacewalk API script for creating a
snapshots as a backup that you can powerful tool for managing large- software channel: [http://​­fedorahosted.​
roll back later. Additionally, you can scale Linux landscapes. It facilitates ­org/​­spacewalk/​­attachment/​­wiki/​
install new software or distribute many daily tasks, such as the instal- ­UploadFedoraContent/​­create_channel.​­py]
configuration files from a centralized lation of software updates or upload- [9] Repository sync: [http://​­fedorahosted.​
location. ing of configuration files. Advanced ­org/​­spacewalk/​­attachment/​­wiki/​
Thanks to the ability to assign reg- features, such as channel cloning, ­UploadFedoraContent/​­sync_repos.​­py]
istered systems to groups, you can make it possible to put any software [10] Fedora12 Spacewalk Client Reposi‑
point and click to do this for a large through a quality assurance process tory RPM: [http://​­spacewalk.​­redhat.​
c­ om/​­yum/​­1.​­0/​­Fedora/​­12/​­i386/​
[11] RHEL5 and CentOS5 Client Repository
RPM: [http://​­spacewalk.​­redhat.​­com/​­yum/​
[12] EPEL Repository: [http://​­download.​
[13] Cobbler:

The Author
Thorsten Scherf is a Senior Consultant for Red
Hat EMEA. You can meet him as a speaker at
Figure 6: An XML-RPC interface opens up a huge selection of Spacewalk server functions via the conferences. He is also a keen marathon runner
programmable API. whenever time permits.

w w w. a d m i n - m aga z i n e .co m Admin 01 13

F E AT U R E S Icinga

Monitoring network computers with the Icinga Nagios fork

Server Observer
© Alterfalter,

Icinga’s developers grew weary of waiting for updates to the popular Nagios monitoring tool, so they
started their own project. By Falko Benthin

A server can struggle for many server and take action immediately. Icinga delivers improved database
reasons: System resources like the Of course, you could check every connectors (for MySQL, Oracle, and
CPU, RAM, or hard disk space could server and service individually, but it PostgreSQL), a more user-friendly
be overloaded, or network services is far more convenient to use a moni- web interface, and an API that lets
might have crashed. Depending on toring tool like Icinga. administrators integrate numerous
the applications that run on a server, extensions without complicated
consequences can be dire – from Nagios Fork modification of the Icinga core. The
irked users to massive financial impli- Icinga developers also seek to reflect
cations. Icinga [1] is a relatively young project community needs more closely and to
Therefore, it is more important than that was forked from Nagios [2] be- integrate patches more quickly. The
ever in a highly networked world to cause of disagreements regarding the first stable version, 1.0, was released
be able to monitor the state of your pace and direction of development. in December 2009, and the version

Listing 1: my_hosts.cfg
01 # Webserver 19 # Fileserver
02 define host{ 20 define host{
03 host_name webserver 21 host_name fileserver
04 alias languagecenter 22 alias Fileserver
05 display_name Server at language center 23 display_name Fileserver
06 address
24 address
07 active_checks_enabled 1
25 active_checks_enabled 1
08 passive_checks_enabled 0
26 passive_checks_enabled 0
09 max_check_attempts 3
27 max_check_attempts 3
10 check_command check-host-alive
28 check_command check-host-alive
11 check_interval 5
29 check_interval 5
12 retry_interval 1
30 retry_interval 1
13 contacts spz_admin
14 notification_period 24x7 31 contacts admin

15 notification_interval 60 32 notification_period 24x7

16 notification_options d 33 notification_interval 60
17 } 34 notification_options d,u,r
18 35 }

14 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

Icinga F e at u r e s

Table 1: States
Option Status
o OK
d Down
u Unreachable
r Recovered
o OK
w Warning
c Critical
r Recovered
u Unknown

counter has risen every couple of

months ever since.
Icinga comprises three components:
the core, the API, and the optional
web interface. The core collects sys-
tem health information generated by
plugins and passes it via the IDOMOD
interface to the Icinga Data Out Data-
base (IDODB) or the IDO2DB service
daemon. The PHP-based API accepts
information from the IDODB and
displays it in a web-based interface. Figure 1: If the hosts are healthy, the admin is happy.
Additionally, the API facilitates the
development of add-ons and plugins. distributions offer binaries in their re- Icinga can monitor the private ser-
Icinga Web is designed to be a state- positories, but if not, or if you prefer vices on a computer, including CPU
of-the-art web interface that is easily to use the latest version, the easy-to- load, RAM, and disk usage, as well as
customized for administrators to keep understand documentation includes public services like web, SSH, mail,
an eye on the state of the systems a quick-start guide (for the database and so on. The lab network environ-
they manage. At the time of writing, via libdbi with IDOUtils), which can ment consists of three computers, one
Icinga Web was in beta, and it has a help you set up the network monitor of which acts as the Icinga server; the
couple of bugs that make it difficult in next to no time for access at http://​ other two are a web server and a file
to recommend for production use. Server/​icinga. The challenges come server that send information to the
If you only need to monitor a single when you want to monitor a larger monitoring server. Because no native
host, Icinga is installed easily. Some number of computers. approach lets you request information

Listing 2: my_services.cfg (Excerpt)

01 #
 SERVICE DEFINITIONS 15  notification_interval 60
02 d
efine service{ 16  notification_options w,c,u,r
03  host_name webserver 17  }
04  service_description HTTP 18 d
efine service{
05  active_checks_enabled 1 19  host_name fileserver, webserver
06  passive_checks_enabled 0 20  service_description SSH
07  check_command check_http 21  active_checks_enabled 1
08  max_check_attempts 3 ;
how often to perform 22  passive_checks_enabled 0
the check before 23  check_command check_ssh
Icinga notifies 24  max_check_attempts 3
09  check_interval 5 25  check_interval 15
10  retry_interval 1 26  retry_interval 1
11  check_period 24x7 27  check_period 24x7
12  contacts spz_admin 28  contacts admin
13  notifications_enabled 1 29  notifications_enabled 0
14  notification_period weekdays 30  }

w w w. a d m i n - m aga z i n e .co m Admin 01 15

F e at u r e s Icinga

monitoring environments, notification

escalation, and check schedules.
Icinga differentiates between active
and passive checks. Active checks are
initiated by the Icinga service and run
at times specified by the administra-
tor. For a passive check, an external
application does the work and for-
wards the results to the Icinga server,
which is useful if you can’t actively
check the computer (e.g., it resides
behind a firewall). A large number of
plugins [4] already exist for various
styles in Nagios and Icinga. But be-
fore the first check, the administrator
needs to configure the computers and
the services to monitor in Icinga.
The individual elements involved in
a check are referred to as objects in
Icinga. Objects include hosts, ser-
Figure 2: Everything is working, but the NRPE plugin is causing problems. vices, contacts, commands, and time
slots. To facilitate daily work, you can
externally about CPU load, RAM, or transmit the required information. group hosts, services, and contacts.
disk space usage, you need to install Icinga sends the system administrator The individual objects are defined in
a verbose add-on, such as NRPE all the information needed and alerts CFG files, which reside below Icinga’s
[3], on each machine. The remote the admin of emergencies. Advanced etc/objects directory. The network
Icinga server will tell it to execute features that are a genuine help in monitor includes a number of sample
the plugins on the local machine and daily work include groups, redundant definitions of various objects that ad-

Listing 3: commands.cfg (Excerpt)

# 'notify‑service‑by‑email' command definition
define command{
03  command_name notify‑service‑by‑email
04  command_line /
usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService:
$LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" |
05 T
06  }
08 #
 'check‑host‑alive' command definition
09 d
efine command{
10  command_name check‑host‑alive
11  command_line $USER1$/check_ping ‑H $HOSTADDRESS$ ‑w 3000.0,80% ‑c 5000.0,100% ‑p
12 5
13  }

Listing 4: timeperiods.cfg (Excerpt)

define timeperiod{ 12 
02  timeperiod_name 24x7 13 d
efine timeperiod{
03  alias 24 Hours A Day, 7 Days A Week 14  timeperiod_name wochentags
04  sunday 00:00‑24:00
15  alias Robot Robot
05  monday 00:00‑24:00
16  monday 07:00‑17:00
06  tuesday 00:00‑24:00
17  tuesday 07:00‑17:00
07  wednesday 00:00‑24:00
18  wednesday 07:00‑17:00
08  thursday 00:00‑24:00
09  friday 00:00‑24:00 19  thursday 07:00‑17:00

10  saturday 00:00‑24:00 20  friday 07:00‑17:00

11  } 21  }

16 Admin 01 w w w. a d m i n - m aga z i n e .co m

Icinga F e at u r e s

ministrators only need to customize. a language center (display_name) and a host is unreachable, you have to de-
In principle, you can define multiple is displayed accordingly in the web fine the nodes passed along the route
objects in a CFG file, but you can just interface. to the host as parents – and this will
as easily create separate files for each To inform the administrator (con‑ only work if the routes for outgoing
object in a directory below /path‑​ tacts) when the server goes down packets are known. The file server
to‑Icinga/etc/objects. Lines that (notification_options), I want Icinga definition looks similar.
start with a hash mark within an to ping (check_command) the server ev- Once the servers are defined, the
object definition are regarded as com- ery 5 minutes (check_interval). If the administrator configures the respec-
ments, as is everything within a line server is still down 60 minutes (noti‑ tive services that Icinga will monitor
to the right of a semicolon. fication_interval) after notifying the (Listing 2), along with the matching
administrator, I want to send another commands (Listing 3), the intervals
Defining Hosts and Services message. (Listing 4), and the stakeholding
Icinga is capable of deciding whether administrators (Listing 5). The indi-
Listing 1 provides a sample host defi- a host is down or unreachable (see vidual configuration files have a simi-
nition. The host is the web server at Table 1). However, to determine that lar structure. For each service, you

Listing 5: contacts.cfg (Excerpt)

01 d
efine contact{ 08  host_notification_options d,u,r

02  contact_name icingaadmin 09  service_notification_options w,u,c,r

03  alias Falko Benthin
10  host_notification_commands notify‑host‑by‑email
04  host_notifications_enabled 1
11  service_notification_commands notify‑service‑by‑email
05  service_notifications_enabled 1

06  host_notification_period 24x7 12  email root@localhost

07  service_notification_period 24x7 13  }

Open Source Monitoring Conference 2010

Live at your PC
You can watch high quality speeches about monitoring topics from a Live-Stream on the
6th and 7th of October at your PC. You will be able to see all the slides as well.

Conference Topics:
• NSClient++ (Michael Medin)
• Clientless Windows Monitoring about
WMI with Samba4 (Thomas Sesselmann)
• The social seismograph at XING
(Dr Johannes Mainusch)
• RRDCacheD - how to escape the I/O hell
(Sebastian Harl)
• Monitoring at Thales Hengelo
using Nagios (Pieter van Emmerik)

Register at: ss to the including acce
video archive
CONTACT: Phone: +49 (0) 89 / 99 34 11 - 0 • Fax: +49 (0) 89 / 99 34 11 - 99 • E-Mail:
F e at u r e s Icinga

Figure 3: A manual check of commands in commands.cfg reveals the culprit. Figure 4: Mail dispatched by Icinga is short and to the point.

need to consider the interval between Icinga with /etc/init.d/icinga re‑ are typically linked, so that clicking
checks. One useful feature is the abil- start. one takes you to more detailed infor-
ity to define time slots, within which mation.
Icinga will perform checks and, if GUI and Messages If something is so drastically wrong
necessary, notify the administrator. that a message is necessary, Icinga
Here, time limitations or holidays can Icinga works without a graphical will check its complex ruleset to see
be defined. interface, but it’s much nicer to have whether it should send a message
The contact configuration can include one. The standard interface can’t and, if so, to whom (Figure 4). The
email addresses or cell phone num- deny its Nagios ancestry, but it is filters through which the message
bers, but to integrate each contact clear-cut and intuitive. passes check the following: whether
with, for example, an Email2SMS If everything is working, you’ll see a notifications are required, if the
gateway or a Text2Speech system lot of green in the user interface (Fig- problem occurred at a time when the
(e.g., Festival), you need a matching ure 1), but if something goes wrong host and service should be running,
command. somewhere, the color will change and if messages should be sent for this
Icinga can use macros, which notice- move closer and closer to red to re- service in the current time slot, and
ably simplifies and accelerates many flect the status of the hosts or services what the contacts linked to the ser-
tasks because you can use a single (Figures 2 and 3). Status messages vice actually want. Each contact can
command for multiple hosts and ser-
vices. Listings 2 and 3 give examples
of macros.
All services defined for monitoring
the file server include a check_nrpe
instruction with an exclamation
mark. Each exclamation mark can
be followed by an argument, which
in turn is evaluated by the macros in
other definitions. Macros are nested
in $ signs.
After creating the configuration files
and storing them in etc/objects, you
still need to tell Icinga by adding a


to the main configuration file, /etc/

icinga.cfg. After doing so, you
should verify the configuration,
/path‑to‑Icinga/bin/icinga ‑v
assuming there are no errors, restart Figure 5: Icinga Web beta was not entirely convincing. Version 1.0.3 is out now.

18 Admin 01 w w w. a d m i n - m aga z i n e .co m

Icinga F e at u r e s

Figure 6: Network overview. If you need to monitor a large number of machines Figure 7: The alert histogram, another useful gadget Icinga offers, shows peak
and have defined “parents,” you can also visualize the intermediate nodes. trouble times.

define its own rules to stipulate when ready for action 24/​7. If the contact and the final results were disappoint-
it wants to receive messages and for that Icinga notifies does not respond ing. The interface was buggy and
what status. If multiple administrators within a defined period, Icinga can at- very slow under my, admittedly, not
exist and belong to a single group, tempt to establish contact on another very powerful Icinga test server (Via
Icinga will notify all of them. Again, channel (e.g., a cell phone instead of C3, 800MHz, 256MB RAM). As a de-
you can define individual notification email). If this notification fails as well, fault, you need a new username and
periods so that each admin will be the case can be escalated to someone password for Icinga Web. That said,
responsible for one period. higher up the chain of responsibility – however, the current status does re-
the team leader, for example. veal some potential; it makes sense to
Interesting Features check how the new interface is devel-
Conclusions oping from time to time.
Icinga contains several interesting The Icinga kernel is well and com-
features that allow administrators to Icinga is a complex tool that provides prehensively documented and leaves
customize the network monitor to valuable services whenever an ad- no questions unanswered. Icinga also
reflect their needs and system envi- ministrator needs to monitor comput- offers a plethora of useful gadgets,
ronment. For example, you can define ers on a network. But don’t expect to such as the status map (Figure 6) or
distributed monitoring environments. be able to set up the network monitor the alert histogram (Figure 7), mak-
If you need to monitor several hun- in a couple of minutes of spare time; ing the job of monitoring hosts less
dred or thousand hosts, the Icinga if all goes well, the installation and boring – at least initially. The depth
server might conceivably run out of configuration will take at least a cou- of information that Icinga provides is
resources because every active check ple of hours. Once you have battled impressive and promises an escape
requires system resources. To take through the extensive configuration, route for avoiding calls from end us-
some of the load off the main server, you can reward yourself with an ers. In short, Icinga is a useful tool
Icinga can delegate individual tasks extended lunch break: If something that makes the administrator’s life
to auxiliary servers which, in turn, happens that requires your attention, more pleasant. n
forward the results to a central server. Icinga will tell you all about it.
Scheduling the checks can also help The traditional web interface is clear
reduce this load. Instead of running cut and packed with information; Info
all your active checks in parallel, you when this article went to print, how- [1] Icinga: [http://​­www.​­icinga.​­org/]
can let Icinga stagger them. ever, the new interface wasn’t entirely [2] Nagios: [http://​­www.​­nagios.​­org/]
Another interesting feature is the abil- convincing (Figure 5). The installa- [3] NRPE: [https://​­git.​­icinga.​­org/]
ity to escalate notifications. Not every tion was tricky, the documentation [4] Nagios plugins: [http://​­sourceforge.​­net/​
administrator can be available and required some imagination at times, ­projects/​­nagiosplug/]

w w w. a d m i n - m aga z i n e .co m Admin 01 19

F e At u r e s mysQL Forks and Patches

complete list of patches and notes on

how to use them.

Extended reporting allows adminis-
Modern MySQL Forks and Patches trators to collect more granular in-
formation about the MySQL server’s

Spoiled for
behavior under load. Thus far, slow.
log, which offers very little in the
line of configuration options, might
be your first port of call. However, its

utility value is restricted to identifying
individual, computationally intensive
queries on the basis of the time they
use – and non-used indexes. The
MicroSlow patch offers new filters
for a more targeted search for poorly
formulated queries. Thus, it logs que-
ries that are responsible for writing
temporary tables to disk, performing
complete table scans, or reading a
freely defined minimum number of
© Dmitry Karasew,

lines in a table. The mysqldumpslow

Slow.log statistics tool, which is not
very well known but is part of the
MySQL standard distribution, has
been modified to be able to read and
evaluate the extended entries.
mysQL is the standard solution for free relational database systems in web applica- Aggregated run-time statistics on us-
tions, but new forks, more storage engines, and patched versions muddy the water. age behavior are equally as useful.
The UserStats patch extends MySQL
now’s the time to take stock of the most important offerings. By caspar clemens mierau
by adding statistics for users, clients,
tables, and indexes. After enabling
If your MySQL server is too slow, cona [3], whose claim to fame is data collection in my.cnf or issuing
you have various approaches to solv- the “MySQL Performance Blog” [4] the SQL SET GLOBAL userstat_running
ing the problem. Besides optimizing (standard reading for anyone inter- = 1 command at run time, four tables
queries and indexes, reworking the ested in MySQL). The patches can be in the information_schema, USER_STA‑
configuration, and upgrading your grouped into three categories: (1) re- TISTICS, CLIENT_STATISTICS, INDEX_
hardware, moving to a customized porting enhancements, (2) functional STATISTICS, and TABLE_STATISTICS,
version of the MySQL server can be enhancements of the MySQL kernel are continually populated with data.
a good idea. In recent years, so many and database engines, and (3) perfor- The statistics can be accessed via the
patches, forks, and new storage en- mance optimizations. In most cases, a SHOW command. For example, SHOW
gines have been released that it is combination of patches from all three TABLE_STATISTICS will give you a
hard to keep track of them. For hard- categories will make the most sense. table-by-table evaluation of lines read
working developers and database Moving to a database server that you and modified and indexes updated.
administrators, this means a change patched and compiled yourself can Direct access to the statistics tables in
from the simple choice of a standard be a daunting prospect. Thankfully, information_schema is useful because
MySQL distribution. projects such as OurDelta [5] offer re- they are accessed as normal tables,
positories with meaningfully patched and you can target the results to
Little Band-aids and prebuilt MySQL packages for manipulate. Listing 1 shows a query
popular distributions like Debian, for the five tables with the most
Many enhancements for MySQL come Ubuntu, and CentOS/RHEL. The frequently read lines. This example
from major corporations like Face- patches I will be looking at in the rest taken from Live operations of the
book [1] and Google [2], who run of this article are a cross-section of Rails-based Moviepilot movie com-
their ad services on top of MySQL the current OurDelta versions of the munity and the underlying movie da-
or from MySQL specialists like Per- MySQL server; see their website for a tabase OMDB (both anonymized for

20 Admin 01 w w w. A d m i n - m AgA z i n e .co m

MySQL Forks and Patches F e at u r e s

this evaluation) shows that read ac- nodb_disallow_writes = 0 disables database engine holds more ­promise.
cess to the images table is particularly the freeze. The InnoDB plug­in and Percona
frequent. The next step would be for XtraDB engines are becoming increas-
the database user to experiment with Performance Enhancements ingly widespread.
code optimization or other changes to
the table format to reduce access in- Thanks to its support for transactions The InnoDB Plugin
cidence and save time. The relatively and line-based locking, InnoDB has
write-intensive movies table has a developed into a modern alternative The InnoBase InnoDB plugin is an
suspiciously high write-access count for the now fairly ancient MyISAM ongoing development of the InnoDB
and, at the same time, a large number engine. Despite performance gains engine that ships with MySQL [6].
of index updates. in write access thanks to line-based Improvements include general optimi-
Because the statistics can be reset locking, the overhead for support- zation of CPU load and I/​O access, a
easily with FLUSH TABLE_STATISTICS, ing transactions, foreign keys, and faster locking mechanism, extended
interval-based evaluation by means of other functions (even if you don’t configuration and reporting options,
a Munin plugin that you write your- use them) costs valuable CPU cycles and optional table compression.
self, or some similar method, would and hardware I/​O resources. MySQL MySQL 5.1 introduced the option of
be your best bet. Retrospectively, you systems under heavy load thus need unloading the standard engine and
could investigate load peaks in rela- a perfectly configured and powerful replacing it with a different version.
tion to table access and modification. InnoDB engine. Starting with MySQL 5.1.38, MySQL
A variety of performance-boosting additionally supplies the InnoDB
New Functions patches are available for the version plugin. As MySQL describes this as
of InnoDB that ships with MySQLM; a release candidate, administrators
New functions in the MySQL kernel some of them are included in the do need to enable it manually. The
give administrators additional options OurDelta version. One that is worthy official MySQL documentation de-
so that maintenance of the MySQL of mention is a reworked RW lock scribes the steps required to do so
server is more secure and conve- that improves locking behavior on [7]. To benefit from the combination
nient. A typical task is to stop MySQL multi-processor systems in particular. of distribution updates for the MySQL
processes with the KILL command. A description of all the improvements kernel and the latest functions and
Under load, you might see a process is beyond the scope of this article, but optimizations of the InnoDB plugin, it
listed as Idle by SHOW PROCESSLIST one thing is clear; patches for InnoDB is a good idea to install the latest In-
start to handle a new query just at the exist that retain compatibility and noDB plugin version from the InnoDB
moment you kill it. The “Kill if Idle” offer transparent optimization of the website.
patch adds an option to kill a process engine. Typically, it is difficult to mea- Ubuntu 10.04 comes with MySQL
only if it is doing nothing: KILL IF_ sure the performance gain in an ob- server version 5.1.41. The /usr/lib/
IDLE Process_Id. This saves you the jective way because the performance mysql/plugin/ houses an InnoDB pl-
embarrassment of accidentally killing of the MySQL server will depend to ugin version 1.0.4 that is disabled by
a process while it is handling a query. a great extent on the hardware, con- default. The InnoDB website has the
LVM and ZFS snapshots are com- figuration, and data it uses. The most current version, 1.0.6, which you can
monly regarded as the simplest exhaustive and reliable source is the download and unpack. Then copy
methods for backing up an InnoDB MySQL Performance Blog [4], which to the /usr/lib/mysql/
database on the fly without inter- regularly publishes test results. plugin/ directory. Because Ubuntu
rupting operations. If this method is Improvements by selectively installing uses AppArmor to protect services by
not an option for you and you are patches for the legacy InnoDB engine default, you need to disable or modify
forced to rely on a legacy dump file, are regarded as a fairly conservative AppArmor to let you load content
you need to make sure that the data approach. The use of an alternative from the plugin directory by adding
on your MySQL server do not change
while a file is being dumped. The Listing 1: Userstats Patch in Action
legacy approach to doing this is FLUSH mysql> select * from information_schema.TABLE_STATISTICS ORDER BY ROWS_READ DESC LIMIT 0,5;

TABLES WITH READ LOCK. However, this +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+


case because background processes +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑+‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

also write to the database. The In- | moviepilot | images | 13138219791 | 14778 | 118224 |
| moviepilot | events | 3957858216 | 59964 | 359784 |
noDB Freeze patch executes SET
| moviepilot | comments | 2650553183 | 3408 | 20448 |
GLOBAL innodb_disallow_writes = 1
| moviepilot | movies | 2013076357 | 598505 | 7780565 |
and then freezes all processes that
| omdb | log_entries | 1106683022 | 2737 | 5474 |
write InnoDB data so you can create
a backup. Afterward, SET GLOBAL in‑

w w w. a d m i n - m aga z i n e .co m Admin 01 21

F e at u r e s MySQL Forks and Patches

these lines to the /etc/apparmor.d/ racuda” file format. In contrast to tive to MyISAM – the new Maria en-
usr.sbin.mysqld ruleset: the standard “Antelope” format, it gine. At the same time, the MariaDB
stores the InnoDB tables in a com- fork happily integrates XtraDB as a
/usr/lib/mysql/plugin/ r,
/usr/lib/mysql/plugin/* mr,
pressed format. Although this could high-performance update to InnoDB.
cost additional CPU cycles, it will For the administrator, this means a
Then restart Apparmor with the ser‑ give you huge I/​O performance sav- whole lot more optimizations: a state-
vice apparmor restart command ings – depending on your data struc- of-the-art MySQL version reworked
to enable the new rules. In my.cnf, ture – because far fewer operations by the MariaDB project and extended
you then need to enable the InnoDB are required on disk/​SSD. To discover to include XtraDB (and Maria), along
engine and load the InnoDB plugin whether it is worthwhile changing to with additional patches courtesy of
as shown in Listing 2. The InnoDB Barracuda, you will need to measure the OurDelta project.
plugin documentation contains de- performance. Tables with large text A conversion from InnoDB to XtraDB
tailed information on this procedure. and blob fields in particular will ben- tables is not needed because XtraDB
The MySQL server error log contains efit from compression. Incidentally, replaces the standard InnoDB engine,
the message shown in Listing 3 after MyISAM has supported “compressed” just as the InnoDB plugin does. Exist-
loading the InnoDB plugin. tables for some time; however, you ing or new tables are automatically
The documentation provides detailed cannot modify compressed MyISAM managed by XtraDB. A downgrade to
information on the optimizations tables in ongoing operations. the InnoDB plugin and the standard
and new features offered in the In- InnoDB engine is also possible. To
noDB plugin. One thing that stands Percona XtraDB see that XtraDB still refers to itself as
out against the rest is the new “Bar- “InnoDB,” you can call SHOW ENGINES
Percona XtraDB [8] takes things one – also, you will see the other modern
Listing 2: Loading the InnoDB Plugin into my.cnf step further than the InnoDB plugin. engines, such as Maria and PBXT,
[mysql] This storage engine is a merge of the here. Listing 4 shows the engines on
ignore_builtin_innodb current InnoDB plugin version with a current MariaDB server.; additional performance and feature Tests on Live systems demonstrates;;
patches. From a codebase point of that migrating to the Maria­DB pack-;
view, XtraDB is thus the most innova- age is unproblematic for the most;;
tive version of InnoDB. But don’t let part. MyISAM tables are left un-;
the name worry you: XtraDB is an In- changed; InnoDB tables continue to
noDB engine. The new name simply work. However, MySQL-specific con-
serves to underline the major differ- figurations in my.cnf are interpreted
Listing 3: With and Without the InnoDB Plugin ences between it and the version of in a fairly strict manner. sql‑mode=NO_
MySQL server without InnoDB plugin: InnoDB that ships with MySQL. ENGINE_SUBSTITUTION,TRADITIONAL
InnoDB: Started; log sequence number 0 44233 Your easiest approach to installing will put MySQL in traditional mode,
XtraDB is to resort to the MariaDB which handles many warnings as er-
MySQL server with current InnoDB plugin: packages created by OurDelta. rors. For example, a typical Rails-style
InnoDB: The InnoDB memory heap is disabled
Maria­DB [9] itself is a MySQL fork database migration failed because
InnoDB: Mutexes and rw_locks use GCC atomic builtins
by the well-known MySQL developer it did not use completely standards-
InnoDB: highest supported file format is Barracuda.
Michael “Monty” Widenius. Widenius compliant queries, such as setting de-
InnoDB Plugin 1.0.6 started; log sequence number 44233
is working on a transactional alterna- fault values for text and blob fields.

Listing 4: Engines on a MariaDB Server

MariaDB [(none)]> show engines;
| Engine | Support | Comment | Transactions | XA | Savepoints |
| BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| FEDERATED | YES | FederatedX pluggable storage engine | YES | NO | YES |
| MARIA | YES | Crash‑safe tables with MyISAM heritage | YES | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
| MyISAM | YES | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
| InnoDB | DEFAULT | Supports transactions, row‑level locking, and foreign keys | YES | YES | YES |
| PBXT | YES | High performance, multi‑versioning transactional engine | YES | YES | NO |

22 Admin 01 w w w. a d m i n - m aga z i n e .co m

MySQL Forks and Patches F e at u r e s

In non-traditional mode, MySQL ig- write out the InnoDB buffer to disk. on the code of the not-yet-released
nores the default use and still runs If you do need to restart the MySQL MySQL 6.0 and mainly pursues the
the queries; in traditional mode, the or MariaDB server, the InnoDB engine goals of removing unnecessary func-
query quits with an error. Thus, it is loses its valuable buffer pool in RAM. tions and reducing complexity.
a good idea in many cases to change Depending on your configuration and The developers really have been radi-
the line to sql‑mode=NO_ENGINE_SUB‑ the application, the pool can be sev- cal in the features they eradicated: E
STITUTION and then restart the eral gigabytes and might be filled in
server. This problem is not specific the course of hours. Storing the buffer Listing 5: Storing and Loading the Buffer Pool
to MariaDB, simply a very restrictive pool before quitting and loading it // storing the buffer pool

configuration in line with the MySQL again after restarting will save valu- MariaDB [(none)]> select * from

standard. Additionally, it makes sense able warmup time, which you would information_schema.XTRADB_ADMIN_COMMAND
to check whether programs compiled notice as slow response on the part of
against Libmysqlclient-dev need to be the database server. Listing 5 shows
| result_message |
recompiled against the current Lib- the commands and returns for storing
mariadbclient-device. In a Rails envi- and loading the buffer pool. | XTRA_LRU_DUMP was succeeded. |
ronment, this will affect Mysql-Gem. +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
Besides the benefits of the InnoDB Drizzle // loading buffer pool and
plugin described thus far, XtraDB also MariaDB [(none)]> select * from information_schema.
offers a considerable performance At this point, I’ll take a quick look XTRADB_ADMIN_COMMAND /*!XTRA_LRU_RESTORE*/;

boost, as a variety of benchmarks at a new development, Drizzle [11]. +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+

with various setups proves [10]. Nu- According to the project, the fork is a | result_message |

merous additional functions make life return to the original MySQL values: +‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑+
| XTRA_LRU_RESTORE was succeeded. |
easier for developers and administra- simplicity, reliability, and perfor-
tors – and don’t forget the ability to mance. Drizzle was originally based

Make the most of your Data Center

Watch our video archiv of the Open Source Data Center
Conference – each session complete with slides.

Hot Topics
• High Availability
• Computer Clusters
• Load Balancing
• Configuration Management
• Security Management

Register here:

w w w. a d m i n - m aga z i n e .co m Admin 01 23

Contact: Phone: +49 (0) 89 / 99 34 11 - 0 • Fax: +49 (0) 89 / 99 34 11 - 99 • E-Mail:
F e at u r e s MySQL Forks and Patches

perfect solution for their require-

MySQL Community Edition ments profile, this does involve a
considerable amount of overhead. At
MySQL 5.0 MySQL 5.1 plugin MySQL 6.0 the same time, the installation of the
server is no more complex than using
Percona, Google, Percona distribution packages thanks to pre-
and other patches MariaDB 5.1 XtraDB Drizzle
packetized software.
Work is already in progress on inte-
grating modern forks into distribution
Percona, Google, PBXT
repositories [14]. The reward for all
and other patches
this effort will ideally be a notice-
ably faster database server that helps
OurDelta MySQL 5.0 OurDelta MariaDB 5.1 reduce hardware and development
costs. n
Figure 1: The MySQL development community now has many forks.

storage engines such as Federated and Community Edition of MySQL server Info
Merged have been removed; others, is to launch new and active projects [1] MySQL on Facebook:
such as CSV and MyISAM, have been (Figure 1). Existing MySQL 5.0 instal- [https://​­launchpad.​­net/​­mysqlatfacebook]
demoted to temporary engines. Mod- lations can be replaced easily by the [2] Google patches: [http://​­code.​­google.​
ern engines such as XtraDB are main- OurDelta MySQL 5.0 build, which ac- ­com/​­p/​­google‑mysql‑tools/​­wiki/​
tained in a separate branch. The stan- celerates the server, thanks to perfor- ­Mysql5Patches]
dard engine for Drizzle is InnoDB. mance patches, and offers advanced [3] Percona patches: [http://​­www.​­percona.​
However, this does not mean that reporting functionality so the admin- ­com/​­docs/​­wiki/​­patches:start]
data dumped from a classical MySQL istrator can plan further steps on the [4] MySQL performance blog: [http://​­www.​
server with InnoDB tables can be basis of run-time statistics. Installa- ­mysqlperformanceblog.​­com/]
integrated without problems, because tions of version 5.1 can benefit from [5] Patches from OurDelta:
Drizzle has also eradicated many field the latest optimizations by installing [http://​­ourdelta.​­org/​­patches]
types, such as TINYINT, TINYTEXT, and the current InnoDB plugin – ideally [6] InnoDB plugin: [http://​­www.​­innodb.​­com/​
YEAR. Migrating to Drizzle thus means without needing to rebuilding. Migrat- ­products/​­innodb_plugin/​­features/]
architectural changes to your data- ing to the state-of-the-art MariaDB, [7] Installing the InnoDB plugin:
base design. Although a change from which outperforms the InnoDB plugin [http://​­dev.​­mysql.​­com/​­doc/​­refman/​­5.​­1/​­en/​
TINYINT to INT could simply mean in performance tests, turns out to be ­innodb.​­html]
searching and replacing occurrences more effective. Luckily, MariaDB is [8] Percona XtraDB: [http://​­www.​­percona.​
in a dump file, the lack of a YEAR field packetized by the OurDelta project, ­com/​­docs/​­wiki/​­percona‑xtradb:start]
can have a more serious effect on ex- which also adds a number of addi- [9] MariaDB:
isting applications. A generic solution tional patches. [http://​­askmonty.​­org/​­wiki/​­MariaDB]
for the migration does not exist. The Drizzle database, with its simpli- [10] Benchmarks: InnoDB plugin and XtraDB vs.
On a more positive note, Drizzle of- fied variant of InnoDB, is still at a InnoDB:
fers totally new replication mecha- very early stage. The Maria engine [http://​­www.​­mysqlperformanceblog.​­com/​
nisms. One feature that stands out also represents a possible fast future ­2010/​­01/​­13/​­innodb‑innodb‑plugin‑vs‑xtrad
is the ability to perform rabbit repli- alternative to the classical combina- b‑on‑fast‑storage/]
cation to NoSQL databases such as tion of MyISAM/​InnoDB; however, [11] Drizzle: [http://​­drizzle.​­org/]
Voldemort [12] or services such as you need to perform extensive checks [12] Project Voldemort:
Memcached [13]; thus, you would before using it. Both projects’ engines [http://​­project‑voldemort.​­com]
be able to provision a variety of back require architectural changes to the [13] “Memcached” by Tim Schürmann, Linux
ends automatically from a central lo- database system and the program Magazine, November 2009, pg. 28
cation. As a state-of-the-art, high-per- code that accesses it, in contrast to [14] MariaDB in Ubuntu: [https://​­wiki.​­ubuntu.​
formance, non-transactional database the InnoDB plugin, XtraDB, and pop- ­com/​­Lucid‑MariaDB‑Inclusion]
engine, the Drizzle project is working ular MySQL patches.
on BlitzDB, which will be positioned Administrators and developers are The Author
as an alternative to MyISAM. put in an ambivalent situation: Al- Caspar Clemens Mierau’s “Screenage” project
though it has become inevitable for provides consultancy services to Rails and PHP
Conclusions administrators to concern themselves portals such as,, and
with alternative MySQL patches, Artfacts.Net. Caspar Clemens works as a free‑
The community’s response to the le- engines, and forks and, ideally, to lance author and is collecting literature for his
thargic integration of patches into the deploy benchmarks to discover the thesis on development environments.

24 Admin 01 w w w. a d m i n - m aga z i n e .co m

Exchange 2010 F e at u r e s

© peapop,
Microsoft Exchange 2010: The Highlights

Message Exchange
For years, Exchange has been the standard in-house server solution for all messaging tasks on Windows. This
article introduces the highlights of the new Exchange version 2010. By Björn Bürstinghaus

Exchange 2010 [1] is the latest gen- pany is using more licenses than it again or the transport rules are modi-
eration of the Microsoft email server, has purchased. fied.
and it comes with a whole bunch Giving users permission to create dis-
of new and useful functions for tribution groups and their members is New Features from the
administrators and users alike [2]. also new; this task can be handled in User’s Point of View
The version is even interesting for the new, web-based Exchange Control
companies currently using Exchange Panel (ECP). In ECP, users can modify Exchange 2010 lets end users search
2007: Besides an archiving function, their own Active Directory informa- multiple mailboxes and send the re-
the new Exchange also integrates an tion, such as cellphone numbers or sults as a PST export to another per-
intelligent SMS gateway architecture, addresses, without needing to contact son. The Unified Messaging function
thus removing the need for expensive IT to do so. In this way, ECP will integrates a voicemail function for
third-party add-ons. probably make a major contribution each user in Exchange and supports
to reducing IT costs in the enterprise. speech-to-text conversion, making it
Improved Management for Whereas Exchange 2007 restricted possible to display a voicemail as well
Admins bulk changes to the PowerShell, Ex- as listen to the attached audio file.
change 2010 finally lets administrators The ability to display messages as
Many functions that posed tedious make bulk changes in the manage- conversations both in Microsoft Of-
Exchange Management Shell tasks for ment console, so you can run a task fice Outlook 2010 and in the Outlook
administrators in Exchange 2007 have simultaneously against multiple mail- Web App gives users a clearer view of
now been integrated into the console. boxes, which wasn’t possible in previ- their email folders. Users do not need
For example, you can create new ous versions (Figure 2). to check the sent items for email they
certificates or view the current crop The new transport cache prevents sent to the same person on the same
of certificates at the console, without email messages transmitted via SMTP subject, which can save time.
needing to compose your own Power- from being deleted until the down- The Outlook Wep App’s premium
Shell request (Figure 1). stream node confirms that they have functions were previously restricted
This version of Exchange also of- been forwarded successfully. In other to Internet Explorer. Exchange 2010
fers a console view of the number of words, if a hub transport server in lifts this restriction. The new Outlook
licenses used in the company. This your company goes down, the trans- Web App now supports premium
new transparency allows administra- port cache will retain the messages functions in Mozilla Firefox and Ap-
tors to identify cases in which a com- until the server becomes available ple Safari (Figure 3). E

w w w. a d m i n - m aga z i n e .co m Admin 01 25

F e at u r e s Exchange 2010

G Figure 2: Bulk Edit lets admins make changes to multiple recipients.

F Figure 1: Wizard for creating new certificates

Another new feature is the approach box, the administrator can create an ing or moving email from the mailbox
to migrating mailboxes. Previously, a archive for each member of staff, al- to the archive mailbox, and a match-
user’s mailbox was switched offline though this is a premium feature and ing retention policy that prevents
during migration to another mailbox does require an Exchange Premium deletion.
database, which meant delays of a Client Access License for each user Exchange 2010 currently does not of-
few minutes to several hours. Now, (Figure 4). Once enabled, the archive fer the same feature set as archiving
while the mailbox is in transit, the is automatically displayed in the Out- solutions by third parties such as
user can stay online. Exchange copies look 2010 and the Outlook Web App MailStore Server or GFI Mail Ar-
the total content of the mailbox and folder navigation, in addition to the chiver; however, the functionality that
then synchronizes any changes that normal mailbox. is currently available does relieve the
occurred during the migration. This solution allows administrators to burden on mailboxes, and it puts an
create an Exchange solution quickly end to the “PST Hell” that some ad-
Archive Mailbox that lets end users restore messages ministrators rightly complained of.
they deleted from their mailboxes
Centralized archiving of email previ- without having to call the help desk. Text Messages
ously relied on third-party add-ons. All you need for this is the journaling
Again, Exchange 2010 puts an end function, the enterprise-wide policies If a company’s employees use smart-
to this. In addition to a normal mail- to use for automatically synchroniz- phones with Windows Mobile 6.5,

Figure 4: The mailbox configuration now supports

Figure 3: The Outlook Web App implements a mail client in the web browser. Archiving for email.

26 Admin 01 w w w. a d m i n - m aga z i n e .co m

they can now synchronize Also, it provides an option of
text messages with their Ex- integrating the logfiles for a
change mailbox, along with database at a later stage. You
email messages, contacts, can refer to the help [3] or
appointments, and tasks. forum [4] websites for addi-
Exchange previously did tional information.
not offer an option for send-
ing or receiving text mes- Is It Worthwhile?
sages (unless you relied on
a third-party product), and If you are planning to invest
SMS gateways were the only in an email archiving solu-
option for integrating text tion or an SMS gateway,
message functionality. Un- you might consider moving
fortunately, this meant that to Exchange 2010, instead,
the SMS gateway number because it already integrates
was the same for the whole many of these functions.
company, rather than having The new version also im-
extensions provided for indi- proves and facilitates the
vidual users. experience of integrating
Exchange 2010, in combi- subsidiaries, so a change
nation with the Windows could also help reduce your
Mobile 6.5 platform, uses company’s IT costs.
the smartphone as the SMS The test version of Exchange
gateway and thus supports supports virtualization, thus
composing or reading per- giving those who are inter-
sonalized text messages with ested in migrating a chance
Outlook 2010 or Outlook Web to familiarize themselves
App. with the new functions and
When you send a text mes- options before they invest in
sage via Exchange, the mes- new hardware. ■
sage is forwarded by the
server via an ActiveSync
interface to the Smartphone, Info
which then uses the em- [1] Exchange Server 2010 product
ployee’s cellphone number page: [
to send the message; this com/en-us/exchange/dd203064.
approach is far more flex- aspx]
ible than using a centralized [2] Exchange Server 2010 new
gateway. features: [http://technet.
Improved Availability dd298136.aspx]
[3] Exchange Server 2010 help:
The 24/7 availability of ser- [
vices plays an important role en-us/library/bb124558.aspx]
in many large corporations [4] Exchange Server 2010 Fo-
that need permanent avail- rum: [http://social.technet.
ability across multiple time
zones because of globaliza- exchange2010/threads]
Exchange 2010 takes this one The Author
step further than the previ- Björn Bürstinghaus is a systems
ous versions: Windows Clus- administrator with simyo GmbH in
ter lets you group up to 16 Düsseldorf, Germany. In his leisure
mailbox servers in a cluster, time, he runs Björn’s Windows blog,
which means databases can a blog on Microsoft Windows topics.
be replicated on up to 15 ad- You can find his blog at [http://blog.
ditional servers.] (in German).

w w w. A d m i n - m AgA z i n e .co m Admin 01 27

F e at u r e s Backup Software

files still need to be backed up.

Sometimes, reinstalling is not prac-
tical if you have to install a lot of
patches, or it might be impossible if
you have software that is licensed for
a particular machine and you cannot
simply reinstall the OS without ob-
taining a new license key.
How often you should back up your
files is the next thing to determine.
For example, database files could be
backed up to remote machines every
Backup tools to save you time and money 15 minutes, even if the files are on

Safety Net
cluster machines with a hardware
RAID. Local machines could be
backed up twice a day: The first time
to copy the backup from the previ-
© Melissa King,

ous day to an external hard disk, and

One small oversight can cost you hours of extra work and your the second time to create a new full
company thousands of dollars. Here are a few backup tools to backup. Then, once a month, the
most important data could be burned
help you recover gracefully. By James Mohr
to DVD (see also the “Incremental vs.
Differential” box).
Computers are of little value if they addressed in the software are key is-
are not doing what they’re supposed sues when deciding which product What Color Is your
to, whether they’ve stopped work- to implement (see also the “Support” Parachute?
ing or they’re not configured cor- box).
rectly. Redundant systems or standby Individuals and many businesses are
machines are common methods of How Strong Is Your not concerned with storing various
quickly getting back to business, but Parachute? versions of files over long periods
they are of little help if the problem is of time. However, in some cases, it
caused by incorrect configuration and A simple copy of your data to a lo- might be necessary to store months
those configuration files get copied cal machine with rsync started from or even years worth of backups, and
to the standby machines. Sometimes, a cron job once a day is an effective the only effective means to store them
the only solution is to restore from way to back up data when everything is on removable media (e.g., a tape
backup. comes from a single directory. This drive). Further requirements to store
Backing up all of your data every day method is too simple for businesses backups off-site often compound
is not always the best approach. The that have much more and many dif- the problem. Although archives can
amount of time and the amount of ferent kinds of data – sometimes be kept on external hard disks, it
storage space required can be limiting so much data that it is impractical becomes cumbersome when you get
factors. On workstations with only a to back it all up every day. In such into terabytes of data.
handful of configuration files and few cases, you need to make decisions In deciding which type of backup
data files, it might be enough to store about what to back up and when (see medium you need to use, you have to
these files on external media (such the “Backup Alternatives” box). consider many things. For example,
as USB drives). Then, if the system In terms of what to back up and how you need to consider not only the
should crash, it could be simpler to often, files increase in importance total amount of data but also how
reinstall and restore the little data you on the basis of how difficult they are many machines you need to back
have. Doing a full system restore of- to recreate. The most important files up. Part of this involves the ability to
ten takes longer. are your “data,” such as database distinguish quickly between backups
Because each system is different, no files, word-processing documents, from different machines. Moving an
single solution is ideal. Even if you spreadsheets, media files, email, and external hard disk between two ma-
find a product that has all the fea- so forth. Such files would be very chines might be easiest, but with 20
tures you could imagine, the time and difficult to recreate from scratch, so machines, you should definitely con-
effort needed to administer the soft- they must be protected. Configuration sider a centralized system.
ware might be restrictive. Therefore, information for system software, such Here, too, you must consider the
knowing the most significant aspects as Apache or your email server, typi- speed at which you can make a
of backups and how these can be cally changes frequently, but these backup and possibly recover your

28 Admin 01 w w w. a d m i n - m aga z i n e .co m

Backup Software F e at u r e s

data. One company I worked for had Backup Alternatives

so much data it took more than a
If you are running Linux and your software repositories are configured properly, a number of
day to back up all of the machines.
backup applications are available through your respective installation tool (e.g., YaST, Synaptic).
Thus, full backups were done over
In fact, I found more than two dozen products that have defined themselves in one way or an-
the weekend, with incrementals in other as a backup tool (not counting those explicitly for backing up databases).
between. Also, in a business envi- Here are a few important questions to ask about your backup software:
ronment with dozens of machines, n Is your hardware supported?
trying to figure out exactly where the n How does the software deal with database backups?
specific version of the data resides n Can you do a directed recovery (i.e., to a different directory)?
increases the recovery time consider- n Can the software verify the data after a backup and restore?
ably. n Can the software write to multiple volumes?
Finally, you must also consider the n Do you really need all of the features?
cost. Although you might be tempted n Can the software do a backup of a remote system?
to get a larger single drive because
it is less expensive than two drives which file is stored where (i.e., which Many of the products I looked at have
that are only half as big, being able to tape or disk). the ability to define “profiles” (or
switch between two drives (or more) Once a software product has reached use a similar term). For example, you
adds an extra level of safety if one this level, it will typically also be define a Linux MySQL profile, assign
fails. Furthermore, you could poten- able to manage multiple versions of it to a subset of your machines, and
tially take one home every night. If a given file. Sometimes you will need the backup software automatically
you are writing to tape, an extra tape to make monthly or even yearly back- knows which directories to include
drive also increases safety; it can also ups, which are then stored for longer and which to ignore. The Apache pro-
speed up backups and recovery. periods of time. (This setup is com- file, for example, has a different set
mon when you have sensitive data of directories. This might also include
Which Tape? like credit card or bank information.) a pre-command that is run immedi-
To prevent the software from over- ately before the backup, then a post-
Some companies remove all of the writing tapes that it shouldn’t, you command that is run immediately
tapes after the backup is completed should be able to define a “recycle afterward.
and store them in a fireproof safe or time” that specifies the minimum
somewhere off-site. This means that amount of time before the media can Storage
when doing incremental backups, be reused.
the most recent copy of a specific Because not all backups are the same How is the backup information
file might be on any one of a dozen and not all companies are the same, stored? Does the backup software
tapes. Naturally, the question be- you should consider the ability of have its own internal format or does
comes, “Which tape?” (see also the the software to be configured to your it use a database such as my SQL?
“Whose Data” box). To solve this needs. If you have enough time and The more systems you back up, the
problem, the backup software must space, software that can only do a more you need a product that indexes
be able keep track of which version of full backup might be sufficient. On which files are saved and where they
the other hand, you might want to be are saved as well. Unless you are
Incremental vs. Differential able to pick and choose just specific simply doing a complete backup ev-
Because of the amount of data, businesses directories, even when doing a “full” ery night to one destination for one
frequently have a two-tiered backup scheme. backup. machine (i.e., one tape or remote di-
Once a week, a full backup is done (of every
single file); on subsequent days, backups are Support Whose Data?
done of only those files that have changed. One consideration that is often overlooked One important aspect is the ability to write
This approach is referred to as an incre- is the amount of support available for your data from different sources to specific
mental backup. Although it saves media, it product. Commercial support might be nec- media. For example, where I work, each
potentially takes more time to recover. With essary if implementing the backup solution customer is assigned specific tapes (often
this method, you first need to restore the for a company. However, the amount of free referred to as a “pool”). With the use of
full backup and, depending on which files support (forums, mailing lists) can be an is-
have changed, you might need to access labels written to the tape, the software can
sue. When considering open source software
every single incremental backup. tell which tape belongs to which pool, so
of any kind for a business, I always suggest
One alternative is a differential backup, that data from different environments is
taking a good look at the product’s website.
which stores only files that have been If the product has not been updated in three not mixed. This scheme is very useful if, for
changed since the last full backup. This has years, you might want to look elsewhere. If example, one customer wants weekly back-
the advantage of saving time compared with forums have few posts and most are unan- ups stored off-site and another customer
an incremental backup, because you need to swered, you likely won’t get your questions frequently requests the backup tapes to load
restore from, at most, two backups. answered either. them into a local test system.

w w w. a d m i n - m aga z i n e .co m Admin 01 29

F e at u r e s Backup Software

rectory), finding the right location for OS X, and Windows. With a handful The repository I used had version
a given file can be a nightmare. Even of machines, configuring rsync by 0.33, so I downloaded and installed
if you are dealing with just a few sys- hand might be a viable solution. If that (although v0.4.1 is current). The
tems, administration of the backups you prefer a graphical interface, sev- source code is available, but various
can become a burden. eral different graphical interfaces are Linux distributions have compiled
This leads into the question of how available. In fact, many different ap- packages.
easy it is to recover your data. Can plications rely on it to do the backup. Describing itself as a backup and
you easily find files from a specific The rsync tool can be used to copy synchronization tool, luckyBackup
date if there are multiple copies? How files either from a local machine to uses rsync, to which it passes various
easy is it to restore individual files? a remote machine or the other way configuration options. It provides the
What about all files changed on a around. A number of features also ability to pass any option to rsync, if
specific date? make rsync a useful tool for synchro- necessary. Although it’s not a client-
Depending on your business, you nizing directories (which is part of its server application, all it needs is an
might have legal obligations in terms name). For example, rsync can ignore rsync connection to back up data
of how long you are required to keep files that have not been changed since from a remote system.
certain kinds of data. In some cases, the last backup, and it can delete files When you define which files and
it might be a matter of weeks; in on the target system that no longer directories to back up, you create
other cases, it can be 10 years or lon- exist on the source. If you don’t want a “profile” that is stored under the
ger. Can you recover data from that existing files to be overwritten but user’s home directory. Profiles can be
long ago? Even if it’s not required by still want all of the files to be copied, imported and exported, so it is pos-
law, having long-term backups is a you can tell rsync to add a suffix to sible to create backup templates that
good idea. If you accidentally delete files that already exist on the target. are copied to remote machines. (You
something and don’t notice it has The ability to specify files and direc- still need the luckyBackup binary to
happened for a period longer than tories to include or exclude is very run the commands.)
your backup cycle, you will probably useful when doing backups. This can Each profile contains one or more
never get your data back. How easy is be done by full name or with wild- tasks, each with a specific source
it for your backup software to make cards, and rsync allows you to specify and target directory, and includes the
full backups at the end of each month a file that it reads to determine what configuration options you select [Fig-
– for example, to ensure that the me- to include or exclude. When deter- ure 2]. Thus, it is possible to have
dia does not get overwritten? mining whether a file is a new ver- different options for different directo-
sion or not, rsync can look at the size ries (tasks), all for a single machine
Scheduling and modification date, but it can also (profile).
compare a checksum of the files. Within a profile, the tool makes it
If your situation prevents you from A “batch mode” can be used to up- easy to define a restore task on the
doing complete backups all the time, date multiple destinations from a basis of a given backup task. Essen-
consider how easy it is to schedule single source machine. For example, tially, this is the reverse of what you
them. Can you ensure that a complete changes to the configuration files can defined for the backup task, but it is
backup is done every weekend, for be propagated to all of your machines very straightforward to change op-
example? without having to specify the change tions for the restores, such as restor-
Also, you need to consider the sched- files for each target. Rsync also has a ing to a different directory.
uling options for the respective tool. GUI, Grsync [Figure 1]. Scheduling of the backup profiles is
Can it start backups automatically? Is done by cron, but the tool provides a
it dependent on some command? Is it luckyBackup simple interface. The cron parameters
simply a GUI for an existing tool, and are selected in the GUI; you click a
all the operations need to be started At first, I was hesitant to go into de- button, and the job is submitted to
manually? Just because a particular tails about luckyBackup [1], because cron.
operating system has no client does it is still a 0.X version and it has a A console, or command-line mode,
not mean you are out of luck: You somewhat “amateurish” appearance. allows you to manage and configure
can mount filesystems using Samba However, my skepticism quickly your backups, even when a GUI is
or NFS and then back up the files. faded as I began working with it. not available, such as when connect-
luckyBackup is very easy to use and ing via ssh. Because the profiles are
rsync provides a surprising number of op- stored in the user’s home directory, it
tions. Despite its simplicity, lucky- would be possible for users to create
Sometimes you do not need to look Backup had the distinction of winning their own profile and make their own
farther than your own backyard. third place in the 2009 SourceForge backups.
Rsync is available for all Linux distri- Community Choice Awards as a “Best Although I would not recommend
butions, all major Unix versions, Mac New Project.” it for large companies (no insult in-

30 Admin 01 w w w. a d m i n - m aga z i n e .co m

Backup Software F e at u r e s

tended), luckyBackup does provide a down searches when you need to of “optimization” in your backups.
basic set of features that can satisfy recover specific files. However, the Although optimization can be useful
home users and small companies. commercial version uses MySQL to in many situations, the explanation
store the information. is somewhat vague about how this is
Amanda Backups from multiple machines accomplished – and vague descrip-
can be configured to run in parallel, tions of how a system makes deci-
Initially developed internally at the even if you only have one tape drive. sions on its own always annoy me.
University of Maryland, the Advanced Data are written to a “holding disk” One important caveat is that Amanda
Maryland Automatic Network Disk and from there go onto tape. Data was developed with a particular envi-
Archiver (Amanda) [2] is one of the are written with the use of standard ronment in mind, and it is possible (if
most widely used open source backup (“built-in”) tools like tar, which not likely) that you will need to jump
tools. The software development is means data can be recovered inde- through hoops to get it to behave
“sponsored” by the company Zmanda pendently from Amanda. Proprietary the way you want it to. The default
[3], which provides an “enterprise” tools typically have a proprietary should always be to trust the admin-
version of Amanda that you can pur- format, which often means you can- istrator, in my opinion. If the admin
chase from the Zmanda website. The not access your data if the server is wants to configure it a certain way,
server only runs on Linux and Solaris down. the product shouldn’t think it knows
(including OpenSolaris), but Mac OS Scheduling is also done with a local better.
X and various Windows versions also tool: cron. Commands are simply For example, you should determine
have clients. started at the desired time with the whether the scheduling mechanism is
The documentation describes respective configuration file as an doing full backups at times other than
Amanda has having been designed to argument. when you expect or even want. In
work in “moderately sized computer Amanda supports the concept of “vir- many cases, large data centers do full
centers.” This and other parts of the tual tapes,” which are stored on your backups on the weekend when there
product description seem to indicate disk. These can be of any size smaller is less traffic and not simply “every
the free, community version might than the physical hard disk. This ap- five days.” If your installation has
have problems with larger computer proach is useful for splitting up your sudden spikes in data, Amanda might
centers. Perhaps this is one reason for files into small enough chunks to be think it knows better and change the
selling an “enterprise” version. The written to DVD, or even CD. schedule.
latest version is 3.1.1, which came Backups are defined by “levels,” with Although such situations can be ad-
out in June 2010, but it just provided 0 level indicating a full backup; the dressed by tweaking the system, I
bug fixes. Version 3.1.0 was released subsequent levels are backups of the have a bad feeling when software
in May 2010. changes made since the last n - 1 or has the potential for doing something
Amanda stores the index of the files less. The wiki indicates that Aman- unexpected. After all, as a sys admin,
and their locations in a text file. This da’s scheduling mechanism uses I was hired to think, not simply to
naturally has the potential to slow these levels to implement a strategy push buttons. To make things easier

Figure 1: Grsync – A simple front end to rsync. Figure 2: luckyBackup profile configuration.

w w w. a d m i n - m aga z i n e .co m Admin 01 31

F e at u r e s Backup Software

in this regard, Zmanda recommends states that it cannot do filesystem or The logical view is a consolidated
their commercial enterprise product. disk images, nor can it write to CDs view of the files and directories in the
Although Amanda has been around or DVDs. Backups can be stored on archive.
for years and is used by many organi- remote machines with FTP or FTPS, Areca is able to trigger pre- and post-
zations, I was left with a bad taste in and you can back up from remotely actions, like sending a backup report
my mouth. Much of the information mounted filesystems, but with no re- by email, launching shell scripts
on their website was outdated, and mote agent. Areca provides no sched- before or after your backup, and so
many links went to the commercial uler, so it expects you to use some forth. It also provides a number of
Zmanda website, where you could other “task-scheduling software” variables, such as the archive and
purchase their products. Additionally, (e.g., cron) to start your backup auto- computer name, which you can pass
a page with the wish list and planned matically. to a script. Additionally, you can de-
features is as old as 2004. Although a In my opinion, the interface is not fine multiple scripts and specify when
note states that the page is old, there as intuitive as others, and it uses ter- they should run. For example, you
is no mention of why the page is still minology that is different from other can start one script when the backup
online or any explanation of what backup tools, making for slower prog- is successful but start a different one
items are still valid. Half of the pages ress at the beginning. For example, if it returns errors.
on the administration table of con- the configuration directory is called Areca provides a number of interest-
tents (last updated in 2007) simply a “workspace” and a collection of ing options when creating backups. It
list the title with no link to another configurations (which can be started allows you to compress the individual
page. at once) is a “group,” as opposed to a files as well as create a single com-
Also, I must admit I was shocked collection of machines. pressed file. To avoid problems with
when I read the “Zmanda Contribu- Areca provides three “modes,” which very large files, you can configure
tor License Agreement.” Amanda is determine how the files are saved: the backup to split the compressed
an open source tool, which is freely standard, delta, and image. The archive into files of a given size. Also,
available to everyone. However, in the standard mode is more or less an you can encrypt the archives with ei-
agreement “you assign and transfer incremental backup, storing all new ther AES 128 or AES 256. One aspect
the copyrights of your contribution files and those modified since the I liked was the ability to drop directo-
to Zmanda.” In return, you receive last backup. The delta mode stores ries from the file explorer directly into
a broad license to use and distribute the modified parts of files. The image Areca.
your contribution. Translated, this mode is explicitly not a disk image; The Areca forum has relatively low
means you give up your copyright basically, it is a snapshot that stores traffic, but posts are fairly current.
and not simply give Zmanda the right a unique archive of all your files with However, I did see a number of recent
to use it, which also means Zmanda each backup. The standard backups posts remain unanswered for a month
is free to add your changes to their (differential, incremental, or full) de- or longer. The wiki is pretty limited,
commercial product and make money termine which files to include. so you should probably look through
off of it – and all you get is a T-shirt! The GUI provides two views of your the user documentation, which I
backups. The physical view lists the found to be very extensive and easy
Areca Backup archives created by a given target. to understand.

Sitting in the middle of the features

spectrum and somewhat less well
known is Areca Backup [4]. Run-
ning from either a GUI [Figure 3]
or a command-line interface, Areca
provides a simple design and a wide
range of features. The documentation
says it runs on all operating systems
with Java 1.4.2 or later, but only Li-
nux and Windows packages are avail-
able for download. Installing it on
my Ubuntu systems was no problem,
and I could not find any references to
limitations with specific distributions
or other operating systems.
Areca is not a client-server applica-
tion, but rather a local filesystem
backup. The Areca website explicitly Figure 4: Bacula admin tool.

32 Admin 01 w w w. a d m i n - m aga z i n e .co m

Backup Software F e at u r e s

Two “wizards” also ease the creation sis of the function of the respective One interesting aspect of Bacula is the
of backups. The Backup Shortcut machine. built-in Python interpreter for script-
wizard simplifies the process of creat- The Director supervises all backup, ing that can be used, for example,
ing the necessary Areca commands, restore, and other operations, includ- before starting a job, on errors, when
which are then stored in a script that ing scheduling backup jobs. Backup the job ends, and so on. Addition-
you can execute from the command jobs can start simultaneously as well ally, you can create a rescue CD for a
line or with cron. as on a priority basis. The Director “bare metal” recovery, which avoids
The Backup Strategy wizard generates also provides the centralized control the necessity of reinstalling your sys-
a script containing a set of backup and administration and is responsible tem manually and then recovering
commands to implement a specific for maintaining the file catalog. The your data. This process is supported
strategy for the given target. For ex- Console is used for interaction with by a “bootstrap file” that contains a
ample, you can create a backup every the Bacula director and is available as compact form of Bacula commands,
day for a week, a weekly backup for a GUI or command-line tool. thus allowing you to store your sys-
three weeks, and a monthly backup The File component is also referred tem without having access to the
for six months. to as the client program, which is the Catalog.
software that is installed on the ma- The basic unit is called a “job,”
Bacula chines to be backed up. As its name which consists of one client and one
implies, the Storage component is set of files, the level of backup, what
The Backup Dracula “comes by responsible for the storage and recov- is being done (backing up, migrating,
night and sucks the vital essence ery of data to and from the physical restoring), and so forth.
from your computers.” Despite this media. It receives instructions from Bacula supports the concept of a “me-
somewhat cheesy tag line, Bacula the Director and then transfers data to dia pool,” which is a set of volumes
[5] is an amazing product. Although or from a file daemon as appropriate. (i.e., disk, tape). With labeled vol-
it’s a newer product than Amanda, I It then updates the catalog by send- umes, it can easily match the external
definitely think it surpasses Amanda ing file location information to the labels on the medium (e.g., tape) as
in both features and quality. To be Director. well as prevent accidental overwrit-
honest, the setup is not the point- The Catalog is responsible for main- ing of that medium. It also supports
and-click type that you get with other taining the file indexes and volume backing up to a single medium from
products, but that is not really to be database, allowing the user to locate multiple clients, even if they are on
expected considering the range of fea- and restore files quickly. The Catalog different operating systems.
tures Bacula offers. maintains a record of not only the The Bacula website is not as fancy as
Although Bacula uses local tools to files but also the jobs run. Currently, Amanda, but I found it more useful
do the backup, it is a true client- Bacula supports MySQL, PostgreSQL, because the details about how the
server product with five major com- and SQLite. As of this writing, the program works are much more acces-
ponents that use authenticated com- Directory and Storage daemons on sible, and the information is more up
munication: Director, Console, File, Windows are not directly supported to date.
Storage, and Catalog. These elements by Bacula, although they are reported
are deployed individually on the ba- to work. The Right Fit
Although I only skimmed the surfaces
of these products, this article should
give you a good idea what is possible
in a backup application. Naturally,
each product has many more features
than I looked at, so if any of these
products piqued your interest, take a
look at the website to see everything
that product has to offer. n

[1] luckyBackup:
[2] Amanda: [http://​­www.​­amanda.​­org]
[3] Zmanda: [http://​­www.​­zmanda.​­com]
[4] Areca Backup: [http://​­www.​­areca‑backup.​­org]
Figure 3: Areca backup. [5] Bacula: [http://​­www.​­bacula.​­org]

w w w. a d m i n - m aga z i n e .co m Admin 01 33

F e at u r e s BlackHat 2010

BlackHat USA 2010 tion called “Lifting the Fog” [5], in

which Marco Slaviero scanned for

memcached (a memory-caching
program widely used to speed up
web-based applications). He found
many memcached servers open to

from the Best

the world, and by using two poorly
documented commands, stat detail
on (which enables debugging) and
stats cachedump (which lists all the
key names), he was able to retrieve
all the items stored in the memcached
The latest and greatest security issues By Kurt Seifried
server. And by “all” I mean every-
thing; according to his presentation,
I’ve been to BlackHat twice now, ing in personal information, so almost he retrieved 136TB of data from 229
and both times I have taken the same all browsers support “auto-complete,” memcached servers.
lesson home: If you think things are which automatically fills out form The good news is that securing your
getting better in the field of computer fields (e.g., name, address, and credit memcached is simple: Firewall it so
security, you’re probably wrong. Over card number). Unfortunately, this that only local trusted systems can
the years, progress has been made feature can be abused by attackers connect to it (and if you must use
identifying bug types – currently the (imagine that), allowing them to steal it over the Internet, set up a VPN to
CWE lists 668 weaknesses in 120 personal information saved within connect systems to it). This solution
categories – and some progress has your web browser if you visit a web is not magical, but it drives home the
been made with projects to identify page. Using JavaScript, they can set point that you need to test and verify
and remove them systematically (e.g., it up so you don’t even have to type security measures using tools like
OpenBSD has had remarkable suc- anything in – combined with a hid- Nmap [6] (which is how he found all
cess). However, you then come to den IFRAME, you might never realize the memcached instances).
the BlackHat conference and see a that it happened. So, if you need an excuse (well, a
presentation like “HTTPS Can Byte The security talks are especially wor- work-related excuse) to go to Las Ve-
Me,” in which Robert Hansen and rying – the ones in which research- gas, BlackHat, and Defcon afterward,
Josh Sokol disclosed 24 vulnerabili- ers don’t find new vulnerabilities they’re not only a lot of fun, but very
ties (Figure 1) that can compromise but simply quantify existing ones. educational. My only complaint is
the integrity and security of SSL- In the case of SSL certificates, they that with 10 tracks, chances are you’ll
encrypted web traffic [2]. scanned the Internet and found 1.2 have to choose between two or more
The problem is not so much a failing million SSL-enabled websites [3] [4]. interesting talks, which is definitely a
within SSL, but unless you’re taking Among the problems found were cer- glass half full type of problem. n
extreme measures to protect network tificates for reserved addresses (e.g.,
traffic against analysis (e.g., padding, a reserved IP address
traffic out, introducing time delays, used by multiple sites) that never Info
etc.), chances are, attackers will be should have been allowed. Also, they [1] Common Weakness Enumeration:
able to glean information even if they found 50 percent of servers config- [http://​­cwe.​­mitre.​­org/]
can’t read the traffic directly. ured to allow SSLv2 (known to be [2] HTTPS Can Byte Me:
Also, consider the case of the well- insecure for 14 years). [https://​­media.​­blackhat.​­com/​­bh‑us‑10/​
meaning web browser that attempts Now, I’m not a glass half empty kind ­whitepapers/​­Hansen_Sokol/​­Blackhat‑USA
to be helpful. I guess people hate typ- of guy, but seeing 50 percent of serv- ‑2010‑Hansen‑Sokol‑HTTPS‑Can‑Byte‑Me
ers configured insecurely is ‑wp.​­pdf]
a bit depressing (which is [3] SSL Observatory:
probably why most security [http://​­www.​­eff.​­org/​­observatory]
people buy beer in pitchers, [4] Internet SSL Survey 2010:
not glasses). [http://​­blog.​­ivanristic.​­com/​­Qualys_SSL_
BlackHat isn’t an unend- Labs‑State_of_SSL_2010‑v1.​­6.​­pdf]
ing stream of bad news, [5] Lifting the Fog:
however. Many of the pre- [https://​­media.​­blackhat.​­com/​­bh‑us‑10/​
sentations not only present ­presentations/​­Slaviero/​­BlackHat‑USA‑2010
problems but also discuss ‑Slaviero‑Lifting‑the‑Fog‑slides.​­pdf]
the solutions. The perfect [6] “Nmap scripting” by Eric Amberg, Linux
Figure 1: Final HTTPS slide – 24 issues in all. example was a presenta- Magazine, February 2008, pg. 68

34 Admin 01 w w w. a d m i n - m aga z i n e .co m

Backup & Disaster Recovery
Georgetown University Chooses SEP sesam Backup Solution
The Solution - SEP sesam
Mike Yandrischovitz, Data Systems and Security Manager for the business
school, consulted with other members of the user community and
discovered SEP sesam. After contacting SEP, he downloaded and installed
the software. In less than two hours, Mike, along with SEP assistance,
was able to get backups for McDonough’s most critical applications.

The decision to move from the old vendor was still difficult. Business
school staff had invested a great deal financially and even more in time
and lost productivity. Nevertheless it was decided to make a change to
SEP sesam.

“James Delmonico, at SEP, had us up and running in hours. I was

getting substantial ‘heat’ from our user community because our backup
The Situation solution was unstable. We were not able to get reliable backups using
the old backup software. Thanks to the SEP sesam solution we had
Georgetown University’s McDonough School of Business (MSB), reliable backups almost immediately and restores of critical data for
one of the most renowned business schools in the United States, our customers’ everyday requirements were fast, easy and accurate.
called SEP sesam to replace their under-performing backup We are now a great fan of the software and the team at SEP,” said
software. MSB is experiencing a period of strong program growth. Yandrishovitz.
The MSB Technology Canter has been tasked with keeping their IT
systems up to date and required a state of the art backup system “SEP engineers were even instrumental in using the SEP software to
to ensure continuity of operations. After a brief discussion, help diagnose a configuration problem with our new SAN. Isolating the
SEP was able to analyze current problems and provide reliable problem, we were able to pinpoint an issue with the SAN Switch. The
backups for critical data. vendor reconfigured the switch and now, with the new software and new
hardware, our backups are completed within time windows previously
During the course of using the old solution, administrators were considered unattainable.”
continually asked to reconfigure and restart their backup systems
even though changes were not being made to the environment or According to John Carpenter, McDonough Chief Technology Officer, “SEP
the network infrastructure. sesam and their helpful engineers took a major worry off our plates. The
new implementation has performed better than we expected. Our staff
The Challenge can now go home on time and I’vesaved the cost of acquisition of SEP
MSB’s new data and applications services requirements had sesam by returning scheduled overtime back to the operating budget.
outstripped its legacy backup software. The old solution was Backups that used to take a whole weekend are now complete in under
not flexible enough to meet new demands without continuous eight hours.”
monitoring. The old system continually failed during overnight
backup tasks. Each error and failed backup required a lengthy Results
call to vendor tech support and often required custom code “Implementing SEP sesam has been truly instrumental in easing our
changes and hot patches. The situation finally became untenable workload and providing a quality backup solution for all of our customers.
when the software could not work with a newly purchased EMC The implementation has allowed us to use other equipment including our
DL3D1500 Disk Library. This final straw initiated an active search hundred-slot ADIC tape library, which was not available to us when using
for a better and more effective backup solution. the previous solution. The time we spend working on backup related
issues has been reduced by a factor of 90%. The acquisition cost for SEP
sesam was less than our annual maintenance fee for the old back up
solution. Call us one satisfied customer,” stated Carpenter.
Backup & Disaster Recovery
For more information visit: |
To o l s OCFS2
© Kheng Ho Toh,

A simple approach to the OCFS2 cluster filesystem

Divide and Conquer

The vanilla kernel includes two cluster filesystems: OCFS2 has been around since 2.6.16 and is thus senior to
GFS2. Although OCFS2 is non-trivial under the hood, it is fairly simple to deploy. By Udo Seidel

Wherever two or more computers Before you can set up a cluster file- orderly access to the data with the
need to access the same set of data, system based on shared disks, you use of file locking to avoid conflict sit-
Linux and Unix systems will have need to look out for a couple of uations. In OCFS2’s case, the Distrib-
multiple competing approaches. (For things. First, the administrator needs uted Lock Manager (DLM) prevents
an overview of the various technolo- to establish the basic framework of a filesystem inconsistencies. Initializing
gies, see the “Shared Filesystems” cluster, including stipulating the com- the OCFS2 cluster automatically
box.) In this article, I take a close puters that belong to the cluster, how launches DLM, so you don’t need to
look at OCFS2, the Oracle Cluster File to access it via TCP/​IP, and the clus- configure this separately. However,
System shared disk filesystem [1]. As ter name. In OCFS2’s case, a single the best file locking is worthless if the
the name suggests, this filesystem ASCII file is all it takes (Listing 1). computer writing to the filesystem
is mainly suitable for cluster setups The second task to tackle with a clus- goes haywire. The only way to pre-
with multiple servers. ter filesystem is that of controlled and vent computers from writing is fenc-

36 Admin 01 w w w. a d m i n - m aga z i n e .co m

OCFS2 To o l s

ing. OCFS2 is fairly simplistic in its

approach and only uses self-fencing.
If a node notices that it is no longer
cleanly integrated with the cluster, it
throws a kernel panic and locks itself
out. Just like the DLM, self-fencing
in OCFS2 does not require a separate
configuration. Once the cluster con-
figuration is complete and has been
distributed to all the nodes, the brunt
of the work has been done for a func-
tional OCFS2.
Things are seemingly quite simple
at this point: Start the cluster, create
OCFS2 if needed, mount the filesys-
tem, and you’re done.

Getting Started
Figure 1: Cluster configuration with the ocfs2console GUI tool.
As I mentioned earlier, OCFS2 is a
cluster filesystem based on shared man page for mkfs.ocfs provides a OCFS2 volumes, you can expect a
disks. The range of technologies full list of options, the most important short delay: During mounting, the
that can provide a shared disk spans of which are covered by Table 2. executing machine needs to register
expensive SAN over Fibre Channel, Once you have created the filesys- with the DLM. In a similar fashion,
from iSCSI to low-budget DRBD [7]. tem, you need to mount it. The mount the DLM resolves any existing locks
In this article, I will use iSCSI and command works much like that on or manages them on the remaining
NDAS (Network Direct Attached Stor- unclustered filesystems (Figure 2). systems in case of a umount. The doc-
age). The second ingredient in the When mounting and unmounting umentation points to various options
OCFS2 setup is computers with an
OCFS2-capable operating system. The Table 1: Directories in a Repository
best choices here are Oracle’s Enter- Filesystem Function GUI Menu CLI Tool
prise Linux, SUSE Linux Enterprise Mount Mount mount.ocfs2
Server, openSUSE, Red Hat Enterprise Unmount Unmount umount
Linux, and Fedora. Create Format mkfs.ocfs2
The software suite for OCFS2 com- Repair Check fsck.ocfs2
prises the ocfs2-tools and ocfs2con- Repair Repair fsck.ocfs2
sole packages and the ocfs2‑`uname Change name Change Label tunefs.ocfs2
‑r` kernel modules. Typing ocfs2con‑ Maximum number of nodes Edit Node Slot Count tunefs.ocfs2
sole launches a graphical interface in
which you can create the cluster con- Table 2: Important Options for mkfs.ocfs2
figuration and distribute it over the Option Purpose
nodes involved (Figure 1). However, b Block size
you can just as easily do this with vi C Cluster size
and scp. Table 1 lists the actions the L Label
graphical front end supports and the
N Maximum number of computers with simultaneous access
equivalent command-line tools.
J Journal options
After creating the cluster configu-
T Filesystem type (optimization for many small files or a few large ones)
ration, /etc/init.d/o2cb online
launches the subsystem (Listing 2).
The init script loads the kernel mod- Listing 1: /etc/​o cfs2/​c luster.conf
ules and sets a couple of defaults for node: ip_port = 7777

the heartbeat and fencing. ip_port = 7777 i

p_address =

Once the OCFS2 framework is run- ip_address =

 number = 1
number = 0 name = node1
ning, the administrator can create
name = node0 cluster = ocfs2
the cluster filesystem. In the sim-
cluster = ocfs2 cluster:
plest case, you can use mkfs.ocfs2
node: node_count = 2
devicefile (Listing 3) to do this. The

w w w. a d m i n - m aga z i n e .co m Admin 01 37

To o l s OCFS2

you can set for the mount operation. for journaling is also important here. value for the N option in mkfs.ocfs2
If OCFS2 detects an error in the data The latest version lets OCFS2 write all can become a problem. The tunefs.
structure, it will default to read- the data out to disk before updating ocfs2 tool lets you change this in next
only. In certain situations, a reboot the journal. date=writeback forces the to no time. The same thing applies to
can clear this up. The errors=panic predecessor’s mode. the journal size (Listing 4).
mount option handles this. Another Inexperienced OCFS2 admins might Also, you can use this tool to modify
interesting option is commit=seconds. wonder why the OCFS2 volume is the filesystem label and enable or
The default value is 5, which means not available after a reboot despite an disable certain features (see also List-
that OCFS2 writes the data out to disk entry in /etc/fstab. The init script ing 8). Unfortunately, the man page
every five seconds. If a crash occurs, that comes with the distribution, doesn’t tell you which changes are
a consistent filesystem can be guaran- /etc/init.d/ocfs2 makes the OCFS2 permitted on the fly and which aren’t.
teed – thanks to journaling – and only mount resistant to reboots. Once en- Thus, you could experience a tunefs.
the work from the last five seconds abled, this script scans /etc/fstab for ocfs2: Trylock failed while opening de-
will be lost. The mount option that OCFS2 entries and integrates these vice "/dev/sda1" message when you
specifies the way data are handled filesystems. try to run some commands on OCFS2.
Just as with ext3/​4, the administra-
Listing 2: Starting the OCFS2 Subsystem tor can modify a couple of filesystem More Detail
# /etc/init.d/o2cb online properties after mounting the file-
Loading filesystem "configfs": OK system without destroying data. The As I mentioned earlier, you do not
Mounting configfs filesystem at /sys/kernel/config: OK tunefs.ocfs2 tool helps with this. If need to preconfigure the cluster
Loading filesystem "ocfs2_dlmfs": OK
the cluster grows unexpectedly and heartbeat or fencing. When the
Mounting ocfs2_dlmfs filesystem at /dlm: OK
you want more computers to access cluster stack is initialized, default
Starting O2CB cluster ocfs2: OK
OCFS2 at the same time, too small a values are set for both. However, you
# /etc/init.d/o2cb status
Shared Filesystems
Driver for "configfs": Loaded
Filesystem "configfs": Mounted The shared filesystem family is a fairly colorful they first need to enter the cluster. The clus-
Driver for "ocfs2_dlmfs": Loaded bunch. By definition, they all share the ability ter setup requires additional infrastructure,
Filesystem "ocfs2_dlmfs": Mounted to grant multiple computers simultaneous ac- such as additional I/​O cards, cluster software,
Checking O2CB cluster ocfs2: Online cess to certain data. The differences are in the and, of course, a configuration. Cluster file-
Heartbeat dead threshold = 31 way they implement these requirements. systems are also categorized by the way they
Network idle timeout: 30000 On the one hand are network filesystems, in store data. Those based on shared disks allow
Network keepalive delay: 2000 which the most popular representative in the multiple computers to read and write to the
Network reconnect delay: 2000 Unix/​Linux camp is Network Filesystem (NFS) same medium. I/​O is handled via Fibre Channel
Checking O2CB heartbeat: Not active [2]. NFS is available for more or less any op- (“classical SAN”) or TCP/​IP (iSCSI). The most
erating system and to all intents and purposes popular representatives in the Linux camp
only asks the operating system to provide a here are OCFS2 and the Global Filesystem
Listing 3: OCFS2 Optimized for Mail Server TCP/​IP stack. The setup is also fairly simple. (GFS2) [4].
# mkfs.ocfs2 ‑T mail ‑L data /dev/sda1
The Andrew filesystem (AFS) is another Parallel cluster filesystems are a more recent
mkfs.ocfs2 1.4.2 network filesystem that is available in a free invention. They distribute data over computers
Cluster stack: classic o2cb implementation, OpenAFS [3]. in the cluster by striping single files across
Filesystem Type of mail On the other hand are cluster filesystems. Be- multiple storage nodes. Lustre [5] and Ceph
Filesystem label=data fore computers can access “distributed” data, [6] are popular examples of this technology.
Block size=2048 (bits=11)
Cluster size=4096 (bits=12)
Volume size=1011675136 (246991 clusters) (493982 blocks)
16 cluster groups (tail covers 8911 clusters, rest cover
OCFS2 is a fairly young filesystem. As the “2” just a year later. Version 1.2 became more
15872 clusters)
in the name suggests, the current version widespread, with a great deal of support from
Journal size=67108864
is an enhancement. Oracle developed the various Enterprise Linux distributions.
Initial number of node slots: 2
predecessor, OCFS, for use in Real Application OCFS2 has been available for the major play-
Creating bitmaps: done Cluster for Oracle databases. The new OCFS2 ers in the Linux world for some time. This
Initializing superblock: done is designed to fulfill the requirements placed applies to commercial variants, such as SLES,
Writing system files: done on a mature filesystem capable of storing ar- RHEL, or Oracle EL, and to the free Debian,
Writing superblock: done bitrary data. POSIX compatibility and the typi- Fedora, and openSUSE systems. Depending on
Writing backup superblock: 0 block(s) cal – and necessary – performance required the kernel version, users either get version
Formatting Journals: done for databases were further criteria. 1.4, which was released in 2008, or version 1.2,
Formatting slot map: done After two years of development, the program- which is two years older. The “OCFS2 Choices”
Writing lost+found: done mers released version 1.0 of OCFS2, and it box and Table 3 show you what you need to
mkfs.ocfs2 successful made its way into the vanilla kernel (2.6.16) watch out for.

38 Admin 01 w w w. a d m i n - m aga z i n e .co m

OCFS2 To o l s

F Figure 2: Unspecta­
cular: the OCFS2 mount

E Figure 3: Automatic
reboot after 20 sec­
onds on OCFS2 cluster

can modify the defaults to suit your In production use, you will probably An active OCFS2 cluster uses a
needs. The easiest approach here is want to remedy this state without handful of processes to handle
via the /etc/init.d/o2cb configure in-depth error analysis (i.e., reboot its work (Listing 5). DLM-related
script, which prompts you for the the cluster node). For this to happen, tasks are handled by dlm_thread,
required values – for example, when you need to modify the underlying dlm_reco_thread, and dlm_wq. The
the OCFS2 cluster should regard a operating system so that it automati- ocfs2dc, ocfs2cmt, ocfs2_wq, and
node or network connection as down. cally reboots in case of a kernel oops ocfs2rec processes are responsible
At the same time, you can specify or panic (Figure 3). Your best bet for for access to the filesystem. o2net
when the cluster stack should try to this on Linux is the /proc filesystem and o2hb‑XXXXXXXXXX handle cluster
reconnect and when it should send a for temporary changes, or sysctl if communications and the heartbeat.
keep-alive packet. you want the change to survive a All of these processes are started and
Apart from the heartbeat timeout, reboot. stopped by init scripts for the cluster
all of these values are given in mil- Just like any other filesystem, OCFS2 framework and OCFS2.
liseconds. However, for the heartbeat has a couple of internal limits you OCFS2 stores its management files
timeout, you need a little bit of math need to take into account when de- in the filesystem’s system directory,
to determine when the cluster should signing your storage. The number which is invisible to normal com-
consider that a computer is down. of subdirectories in a directory is mands such as ls. The debugfs.ocfs2
The value represents the number of restricted to 32,000. OCFS2 stores command lets you make the system
two-second iterations plus one for data in clusters of between 4 and directory visible (Figure 4). The
the heartbeat. The default value of 31 1,024Kb. Because the number of objects in the system directory are
is thus equivalent to 60 seconds. On cluster addresses is restricted to 232, divided into two groups: global and
larger networks, you might need to the maximum file size is 4PB. This
increase all these values to avoid false limit is more or less irrelevant be- Listing 4: Maintenance with tunefs.ocfs2
alarms. cause another restriction – the use # tunefs.ocfs2 ‑Q "NumSlots = %N\n" /dev/sda1

If OCFS2 stumbles across a criti- of JBD journaling – limits the maxi- NumSlots = 2
# tunefs.ocfs2 ‑N 4 /dev/sda1
cal error, it switches the filesystem mum OCFS2 filesystem size to 16TB,
# tunefs.ocfs2 ‑Q "NumSlots = %N\n" /dev/sda1
to read-only mode and generates a which can address a maximum of 232
NumSlots = 4
kernel oops or even a kernel panic. blocks of 4KB. #
# tunefs.ocfs2 ‑Q "Label = %V\n" /dev/sda1
Label = data
# tunefs.ocfs2 ‑L oldata /dev/sda1
# tunefs.ocfs2 ‑Q "Label = %V\n" /dev/sda1
Label = oldata

Listing 5: OCFS2 Processes

# ps ‑ef|egrep '[d]lm|[o]cf|[o]2'
root 3460 7 0 20:07 ? 00:00:00 [user_dlm]
root 3467 7 0 20:07 ? 00:00:00 [o2net]
root 3965 7 0 20:24 ? 00:00:00 [ocfs2_wq]
root 7921 7 0 22:40 ? 
root 7935 7 0 22:40 ? 00:00:00 [ocfs2dc]
root 7936 7 0 22:40 ? 00:00:00 [dlm_thread]
root 7937 7 0 22:40 ? 
00:00:00 [dlm_reco_
root 7938 7 0 22:40 ? 00:00:00 [dlm_wq]

Figure 4: The metadata for OCSFS2 are stored in files that are invisible to the ls command. They can be listed root 7940 7 0 22:40 ? 00:00:00 [ocfs2cmt]
with the debugfs.ocfs2 command.

w w w. a d m i n - m aga z i n e .co m Admin 01 39

To o l s OCFS2

local (i.e., node-specific) files. The is necessary to create further node- complexity in a cluster filesystem
first of these groups includes global_ specific system files. doesn’t help. From the viewpoint of
inode_alloc, slot_map, heartbeat, OCFS2, things can go wrong in three
and global_bitmap. They have access What Else? different layers – the filesystem struc-
to each node on the cluster; incon- ture on the disk, the cluster configu-
sistencies are prevented by a locking When you make plans to install ration, or the cluster infrastructure –
mechanism. The only programs that OCFS2, you need to know which ver- or even a combination of the three.
access global_inode_alloc are those sion you will putting on the new ma- The cluster infrastructure includes
for creating and tuning the filesystem. chine. Although the filesystem itself – the network stack for the heartbeat,
To increase the number of slots, it that is, the structure on the medium cluster communications, and possibly
– is downward compatible, mixed op- media access. Problems with Fibre
Listing 6: Debugging with mounted.ocfs2 erations with OCFS2 v1.2 and OCFS2 Channel (FC) and iSCSI also belong
# grep ‑i ocfs /proc/mounts |grep ‑v dlm v1.4 are not supported. The network to this group.
# hostname protocol is to blame for this. The de- For problems with the cluster infra-
velopers enabled a tag in the active structure, you can troubleshoot just
# tunefs.ocfs2 ‑L olddata /dev/sda1
tunefs.ocfs2: Trylock failed while opening device "/dev/
protocol version so that future OCFS2 as you would for a normal network,
sda1" versions would be downward compat- FC, or iSCSI problems. Problems can
# mounted.ocfs2 ‑f ible through the network stack. This also occur if the cluster configuration
Device FS Nodes comes at the price of incompatibility is not identical on all nodes. Armed
/dev/sda1 ocfs2 testvm
with v1.2. Otherwise, administrators with vi, scp, and md5sum, you can
have a certain degree of flexibility check this and resolve the problem.
when mounting OCFS2 media. OCFS2 The alternative – assuming the cluster
Listing 7: Restoring Corrupted OCFS2 Superblocks v1.4 computers will understand the infrastructure is up and running – is
# mount /dev/sda1 /cluster/ data structure of v1.2 and mount to synchronize the cluster configu-
mount: you must specify the filesystem type them without any trouble. This even ration on all of your computers by
# fsck.ocfs2 /dev/sda1
works the other way around: If the updating the configuration with ocfs­
fsck.ocfs2: Bad magic number in superblock while opening
OCFS2 v1.4 volume does not use the 2console.
# fsck.ocfs2 ‑r1 /dev/sda1
newer features in this version, you It can be useful to take the problem-
[RECOVER_BACKUP_SUPERBLOCK] Recover superblock can use an OCFS2 v1.2 computer to atic OCFS2 volume offline – that is,
information from backup block#262144? <n> y access the data. to unmount it and restart the cluster
Checking OCFS2 filesystem in /dev/sda1: service on all of your computers by
31 18 de 29 69 f3 4d 95 a0 99 a7
Debugging giving the /etc/init.d/o2cb restart
command. You can even switch the
23 ab 27 f5 04
A filesystem has a number of poten- filesystem to a kind of single-user
number of blocks: 367486
bytes per block: 4096
tial issues, and the added degree of mode with tunefs.ocfs2.
number of clusters: 367486
bytes per cluster: 4096
Table 3: New Features in OCFS2 v1.4
max slots: 2 Feature Description
Ordered journal mode OCFS2 writes data before metadata.
/dev/sda1 is clean. It will be checked after 20
additional mounts. Flexible allocation OCFS2 now supports sparse files – that is, gaps in files. Additionally,
# preallocation of extents is possible.
Inline data OCFS2 stores the data from small files directly in the inode and not in
Listing 8: Enabling/​Disabling OCFS2 v1.4 Features
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1
Clustered flock() The flock() system call is cluster capable.
Incompatibel: sparse inline‑data
# tunefs.ocfs2 ‑‑fs‑features=nosparse /dev/sda1 OCFS2 Choices
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1
Administrators will basically come across two The mkfs.ocfs2 supplied with version 1.4
Incompatibel: inline‑data
versions of OCFS2: version 1.2 or 1.4. As re- automatically enables all the new features,
# tunefs.ocfs2 ‑‑fs‑features=noinline‑data /dev/sda1
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1
gards the data structure on disk, the two ver- thus effectively preventing OCFS v1.2 ma-
Incompatibel: None
sions are compatible; however, this does mean chines from accessing the filesystem. To
# doing without the newer features in v1.4. The change this, use tunefs.ocfs2 to disable
# tunefs.ocfs2 ‑‑fs‑features=sparse,inline‑data /dev/ documentation lists 10 significant differences the new functions (Listing 8). An easier ap-
sda1 between versions 1.2 and 1.4. Table 3 lists the proach is to create the filesystem with the
# tunefs.ocfs2 ‑Q "Incompatible: %H\n" /dev/sda1 most interesting of these. No matter which ‑‑fs‑feature‑level=max‑compat
Incompatibel: sparse inline‑data version you decide on, you should always option set. tunefs.ocfs2 will help you mi-
# watch for a couple of things. grate from version 1.2 to 1.4.

40 Admin 01 w w w. a d m i n - m aga z i n e .co m

To do this, you need to working. The first backup
change the mount type from makes it possible to restore.
cluster to local. After doing
so, only a single computer Basically, Yes, but …
can mount the filesystem,
and it doesn’t need the clus- On the whole, it is easy to
ter stack to do so. set up an OCFS2 cluster. The
In all of these actions, you software is available for a
need to be aware that the number of Linux distribu-
filesystem can be mounted tions. Because OCFS2 works
by more than one computer. just as well with iSCSI and
Certain actions that involve, Fibre Channel, the hardware
say, tunefs.ocfs2, will not side is not too difficult either.
work if another computer ac- Setting up the cluster frame-
cesses the filesystem at the work is a fairly simple task
same time. that you can handle with
The example in Listing 6 simple tools like Vi.
shows the user attempting Although OCFS2 doesn’t
to modify the label. This include sophisticated fenc-
process fails, although the ing technologies, in contrast
filesystem is offline (on this to other cluster filesystems,
computer). In this case, fencing is not necessary in
mounted.ocfs2 will help: It many areas. The lack of a
checks the OCFS2 header to cluster-capable volume man-
identify the computer that is ager makes it easier for the

Linux Pros read

online with the filesystem. user to become immersed in
The most important filesys- the world of OCFS2. Because

tem structure data are con- OCFS2 is simpler and less
tained in the superblock. Just complex than other cluster
like other Linux filesystems, filesystems, it is well worth
OCFS2 creates backup copies investigating. n
of the superblock; however,
the approach the OCFS2
developers took is slightly Info Enjoy a rich blend of tutorials, reviews,
unusual. [1] OCFS2: [http://​­oss.​­oracle.​­com/​
OCFS2 creates a maximum ­projects/​­ocfs2/] international news, and practical
of six copies at non-config- [2] First NFS RFC: [http://​­tools.​­ietf.​ solutions for the technical reader.
urable offsets: 1, 4, 16, 64, ­org/​­html/​­rfc1094]
and 256GB and 1TB. Need- [3] OpenAFS:
less to say, OCFS2 volumes [http://​­www.​­openafs.​­org/]
smaller than 1GB (!) don’t
have a copy of the super-
[4] GFS: [http://​­sources.​­redhat.​
Subscribe now
block. To be fair, mkfs.ocfs2
does tell you this when you
[5] Lustre: [http://​­wiki.​­lustre.​­org]
[6] Ceph:
to receive:
generate the filesystem. You
3 issues
need to watch out for the [7] DRBD: [http://​­www.​­drbd.​­org/]

+ 3 DVDs
Writing backup superblock:
... line. The Author

for only
A neat side effect of these Udo Seidel is a teacher of math and
static backup superblocks is physics and has been an avid sup-
that you can reference them porter of Linux since 1996. After
by number during a filesys-
tem check. The example in
Listing 7 shows a damaged
completing his PhD, he worked as a
Linux/​Unix trainer, system adminis-
trator, and senior solutions engineer.
primary superblock that is He now works as the head of a
preventing mounting and Linux/​Unix team for Amadeus Data
a simple fsck.ocfs2 from Processing GmbH in Erding, Germany.

w w w. a d m i n - m aga z i n e .co m Admin 01 41

To o l s Synergy
©Kheng Ho Toh,

Controlling multiple systems simultaneously with Synergy

Side Effect
The many approaches to managing remote computers include VNC, No- vices – in both directions. Listing 1
contains an example with comments
machine, and SSH. Synergy is a clever tool that does a bit of lateral think- for the test case. The Synergy homep-
ing and connects multiple PCs to create a virtual desktop. By Florian Effenberger age documents many additional op-
tions [3].
To operate Synergy, you need at least the administrator has decided to use All options in the configuration file
two PCs, each with its own operat- Synergy. The admin will work mainly should be in lowercase. Also, make
ing system, monitor, and functional on the large PC, which is the Ubuntu sure you use the correct line breaks,
network card. The software supports system. In Synergy-speak, this is re- because Synergy is fussy about them
Windows 95 through Windows 7, ferred to as the control system; the and will not use the file if they are
Mac OS X as of version 10.2, and administrator will use the keyboard wrong. After completing all this work,
Linux with the current X server. Pre- and mouse on this server. The other you can launch the Synergy server on
built packages for Windows and Mac devices are clients. Ubuntu as a normal user by typing
OS X are available from the Synergy synergys. The ‑f parameter will pre-
homepage [1]. An RPM file is avail- Configuration vent Synergy from disappearing into
able for Linux and can be installed on the background.
most popular distributions with tools Before you start using Synergy, you QuickSynergy gives you an even more
such as Alien [2], if needed. Some need to configure it by editing the convenient approach to configuration.
distributions also offer prebuilt pack- /etc/synergy.conf or ~/.synergy. On Ubuntu, you can download the
ages; for example, Ubuntu Universe conf text file. The elementary unit is package from the Universe reposi-
contains a package called Synergy. a screen: Each computer belonging to tory and launch it with Applications |
a group, whether server or client, is a Tools | QuickSynergy after the install.
Test Case screen with a precisely defined posi- Unfortunately, the program failed to
tion – just like the display arrange- launch a working server during my
The administrator’s workplace com- ment in a configuration with multiple test.
prises a large desktop system running monitors. For each computer, you The Vista client, which I want to con-
Ubuntu and a small notebook run- need to enter into the configuration trol remotely with the Ubuntu system,
ning Vista on the right. To avoid con- file the name of the screen, its aliases, is even easier to configure. After the
stantly switching between keyboards, and its position relative to other de- installation, you can launch Synergy

42 Admin 01 w w w. a d m i n - m aga z i n e .co m

Synergy To o l s

ent while the PCs. And the cross-operating system

focus is on its clipboard, which removes the need to
screen (i.e., the copy text files, is really convenient.
mouse cursor is One item on my Synergy wish list,
displayed there). however, is easier configuration on
But, that’s not Linux. n
all – Synergy
also coordinates
the clipboard Info
between the two [1] Synergy homepage:
systems. Accord- [http://​­]
ing to the devel- [2] Alien:
opers, Synergy [http://​­kitenet.​­net/​­~joey/​­code/​­alien/]
automatically [3] Configuration options: [http://synergy-
identifies the cor-]
Figure 1: Synergy as a client on Windows Vista. rect character set [4] Security notes: [
and converts line pm/projects/synergy/wiki/UserFAQ#Q-
directly via the Start menu, and it will breaks between operating systems, How-secure-is-your-application]
come up with a neat graphical inter- which is perfect for centralized copy- [5] Tunneling with SSH: [http://www.revsys.
face (Figure 1). To open a connection ing of long text blocks and configura- com/writings/quicktips/ssh-tunnel.html]
to the server, select the Use another tion files. Pressing the Scroll key dis-
computer’s shared keyboard and ables Synergy temporarily if needed. The Author
mouse (client) option and enter the In the configuration file, you can set a Florian Effenberger has been a free software
name of the computer you require. number of additional options. Among evangelist for many years. He is the Lead of the
Power users might also want to con- other things, Synergy can map keys Marketing Project for interna-
figure extras like the Logging Level, between the server and the client, tional and a member of the Ger-
AutoStart, and network details. configure screen areas when you do many board. Other work involves designing and
not want to be able to toggle between implementing enterprise and school networks,
Connecting the Screens screens, and perform certain actions including software distribution solutions based
at a key press. Synergy messes up on free software. He also writes for numerous
After configuring all the clients, you some functions, however: In our German and English language publications, in
can simply run synergyc server IP lab, the tool failed to synchronize which he mainly focuses on legal issues.
(or click Start on Windows) to com- screen savers and failed to lock all
bine the screens. The whole thing is the screens centrally. According to Listing 1: Configuration
unspectacular at first, and you can the homepage, the Mac OS X variant # Define screens

continue to work on each system in of the program, in particular, is not section: screens

the normal way. as mature as the Linux and Windows ubuntu:

However, if you move the mouse on versions.
the server beyond the right edge of
the screen, the mouse pointer moves Conclusions # Alternative names
onto the Vista desktop just as on a section: aliases
single computer with multiple dis- Synergy offers an interesting ap- # ubuntu ‑> desktop
plays, but here, it moves between two proach to controlling multiple com- ubuntu:
computers with different platforms. puters centrally without investing in desktop

Keyboard input also reaches the cli- additional hardware. In contrast to # vista ‑> notebook

legacy approaches, each system keeps vista:

Enabling Universe its own display. The program is defi- notebook

Ubuntu organizes its software packages nitely useful for owners of multiple
into a number of repositories. The Ubuntu
# Screen arrangement
Universe repository contains packages that Security Note
section: left
have less comprehensive support and main- The authors of Synergy point out on their # vista: right of ubuntu
tenance than others. To use Universe, you homepage that Synergy does not provide ubuntu:
first need to enable the corresponding line right = vista
anything in the line of authentication or
in the /etc/apt/sources.list file by # ubuntu: left of vista
removing the hash sign. After an apt‑get encryption [4]. To be on the safe side, you
update, you can install the new packages, might want to set up an SSH tunnel to en- left = ubuntu
including Synergy. crypt all your data [5]. end

w w w. a d m i n - m aga z i n e .co m Admin 01 43

TO O L S SystemTap

Tracing applications with OProfile and SystemTap

Data on Tap
© boing,

Does your application data take ages to creep off your disk or your network card, even if no noticeable activity
is taking place? Tools such as OProfile and SystemTap help you find out why. By Thorsten Scherf

Experienced administrators tend to popular strace traces applications. This command line sends all open sys-
use tools such as ps, vmstat, or the In the simplest cases, the tool lists calls for the Mutt application to the
like when they need statistics for in- all the system calls (syscalls) with /tmp/mutt.trace output file. Then,
dividual subsystems such as the net- their arguments and return codes for you can easily grep the configuration
work, memory, or block I/O. These a specific application. Setting options file from the results.
tools can help identify hardware or allows for highly selective Strace Profiling applications, including the
software bottlenecks, and they are output. For example, if you need to popular OProfile tool [1], take this a
indisputably useful for a general investigate whether an application is step further by giving you details of
appraisal, but if you want to delve parsing the configuration file that you the performance of individual appli-
deeper, you need something with painstakingly put together, you can cations, the kernel, or the complete
more punch. call Strace: system (see Figure 1). For this to hap-
Again, the standard toolbox offers a pen, OProfile accesses the CPU per-
couple of utilities. For example, the strace -e trace=open -o mutt.trace mutt formance counters on state-of-the-art

Listing 1: opcontrol Events

opcontrol --list-events 0x30: prefetch: all inclusive
oprofile: available events for CPU type "Core 2" 0x10: prefetch: Hardware prefetch only
0x00: prefetch: exclude hardware prefetch
See Intel Architecture Developer's Manual Volume 3B, Appendix A and 0x08: (M)ESI: Modified
Intel Architecture Optimization Reference Manual (730795-001) 0x04: M(E)SI: Exclusive
0x02: ME(S)I: Shared
INST_RETIRED_ANY_P: (counter: all)) 0x01: MES(I): Invalid
number of instructions retired (min count: 6000) LLC_MISSES: (counter: all))
L2_RQSTS: (counter: all)) L2 cache demand requests from this core that missed the
number of L2 cache requests (min count: 500) L2 (min count:6000)
Unit masks (default 0x7f) Unit masks (default 0x41)
---------- ----------
0xc0: core: all cores 0x41: No unit mask
0x40: core: this core [...]

44 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

SystemTap To o l s

hardware. The counters have informa-

tion on how often a specific event has
occurred. In this context, an event
can be RAM access or the number of
interrupts. This information is very
useful for identifying bottlenecks or
debugging the system.
To install OProfile, you need the
kernel-debuginfo [2] package, which
provides the symbol to machine
code mappings. Note that the kernel-
debuginfo version must match your
kernel version. To install the pack-
age easily, use your distribution’s Figure 1: OProfile architecture.
standard repository. On Fedora, you
would do this with Yum: CPUs; if you set 0x40, you get the ample, for kernel functions (kernel.
value only for the CPU actually run- function("function")) inside of
yum install oprofile kernel‑debuginfo
ning the OProfile process. The com- kernel modules (module("module").
A call to RPM should confirm that the mand in Listing 2 monitors a specific function("function")) or system calls
kernel and kernel-debuginfo packages event. (syscall.system_call). The injected
have the same version number: The --event option can occur mul- program monitors the event and
tiple times to monitor more than collects information in the process.
rpm ‑q k
ernel‑PAE kernel‑PAE‑debuginfo U
one event. To make sure that the Thus, far more precise results can
kernel‑PAE‑‑90.fc12.i686 results are not distorted by histori- be achieved here than with OProfile,
kernel‑PAE‑debuginfo‑‑90.fc12.i686 cal data, the --reset option deletes which only queries an event periodi-
oprofile‑0.9.5‑4.fc12.i686 them before collecting fresh data with cally.
--start. After a while, the --stop SystemTap has also supported query-
To profile the kernel, you need to tell option stops monitoring the system. ing static tracepoints in the kernel
OProfile where the kernel image is lo- The data collected in this way are (kernel.trace("tracepoint")) and,
cated with the --vmlinux option: now available in the /​var/​lib/​opro- more recently, in userspace ap-
file/​ samples directory. For a general plications. Developers build static
opcontrol ‑‑setup U
‑‑vmlinux=/usr/lib/debug/lib/ U
overview, you can access the data tracepoints into the program code at
modules/`uname ‑r`/vmlinux with opreport – either the data for
the complete system or the data for a Listing 2: opcontrol Events
In normal use, you can omit the im- specific application (Listing 3). opcontrol ‑
‑r`/vmlinux ‑‑event L2_RQSTS:500
age details. The following command Depending on the event selected, this
gives you an overview of the events information will give you a pretty
# opcontrol ‑‑reset
that OProfile can enumerate (see clear picture of what is happening # opcontrol ‑‑start
Listing 1): on your system. For more details of Using 2.6+ OProfile kernel interface.
OProfile, check out the highly infor- Reading module info.
opcontrol ‑‑list‑events
mative website for the tool [1]. Using log file /var/lib/oprofile/samples/oprofiled.log

The events will differ depending on Daemon started.

Profiler running.
the CPU you use. The /​usr/​share/​ SystemTap
oprofile directory has lists for the
various architectures. An event com- The SystemTap [3] tool aims to com- Listing 3: opreport for Mutt
prises a symbolic name (L2_RQSTS), a bine the functionality of classical trac- opreport ‑l /usr/bin/mutt

counter (500), and an optional mask ing and profile tools such as Strace CPU: Core 2, speed 2401 MHz (estimated)
nted L2_RQSTS events (number of L2 cache requests)
(0xc0). The counter defines the accu- and OProfile while providing a simple
with a unit mask of 0x7f
racy of a profile. The lower the value, but powerful interface for the user.
(multiple flags) count 500
the more often the event will be que- SystemTap was originally developed samples % image name symbol name
ried. Special properties of an event for monitoring the Linux kernel, al- 414 10.8377 mutt imap_exec_msgset
are available by query with the mask. though more recent versions also let 162 4.2408 mutt parse_set
For example, the L2_RQSTS event tells you monitor userspace applications. 161 4.2147 mutt mutt_buffer_add
you how many requests have been SystemTap builds on the kprobes 145 3.7958 mutt mutt_extract_token

made to the CPU’s L2 cache. When kernel subsystem. It lets the user 126 3.2984 mutt ascii_strncasecmp

called with a mask of 0xc0, OProfile insert arbitrary program code before 124 3.2461 mutt imap_read_headers
returns the value for all the available any event in kernel space – for ex-

w w w. a d m i n - m aga z i n e .co m Admin 01 45

To o l s SystemTap

important locations. Because develop- the module on another system that Tapsets are easy to integrate with
ers know their program better than doesn’t have a compiler is not a prob- your own scripts. The templates are
anybody else, this kind of information lem, though. SystemTap lets you build typically located below /​usr/​share/​
is a big help. modules for kernel versions besides systemtap/​ tapsets. Besides these syn-
SystemTap programs are written in the kernel on your own system. Then chronous events, other asynchronous
a language that is similar to Awk. A you can copy the module to the target events are not bound to a specific
parser checks the script for syntax er- system and run it with staprun – event in the kernel or program code,
rors before converting it to the faster more on this subject later. and typically they are used when you
C language, which is then loaded as Because all the major Linux distribu- need to create a header or footer for
a kernel module (Figure 2). Using tions support SystemTap, you can eas- your script. They are also suitable
ily install from the standard software for running specific events multiple
Listing 4: “Hello world” in SystemTap repository. The important thing is to times.
#!/usr/bin/stap install the kernel-debuginfo package Listing 5 shows a simple example
probe begin {printf("Hello, world!\n");} along with kernel-devel: with two probes, each with an asyn-
probe timer.sec(5) {exit();}
chronous and a synchronous event.
probe end {printf("Good‑bye, world!\n");} yum install kernel‑debuginfo U
kernel‑devel systemtap U
The first outputs a header at one-
systemtap‑runtime second intervals, the second calls the
# stap helloword.stp
prebuilt tcp.receive tapset, which is
Hello, world!
The latest versions are available from defined in Listing 6. This example
<5 seconds later>
the project’s Git repository: shows the extent to which the use
Good‑bye, world!
of tapsets reduces the complexity of
git clone U
your own scripting. When you launch
Listing 5: tcpdump via SystemTap systemtap.git systemtap the script from Listing 1, typing stap
#!/usr/bin/stap tcpdump.stp lets you see the network
Assuming the installation is success- packets arriving at one-second in-
// A TCP dump like example
ful, you can use the following one- tervals with various other pieces of
liner to check that SystemTap is work- information. If you omit timer.s(1)
probe begin, timer.s(1) {
ing properly: in the first event, the header is only
06  printf("‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑
output before outputting the first net-
stap ‑v ‑e 'probe U
07  printf(
" Source IP Dest IP SPort {printf("Reading data from U
work packet.
DPort U A P R S F\n") disk.\n"); exit()}' The handler, also known as a body,
08  printf("‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑ supports instructions that will be
‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑\n") If this accesses the kernel’s VFS sub- familiar from various programming
} system, stap will send a message to languages. For example, you can
10  standard output and terminate. The initialize variables and arrays, call
11 p
robe tcp.receive {
ubiquitous “Hello World” program for functions, and query positioning pa-
12  printf(" %15s %15s %5d %5d %d %d %d %d %d
SystemTap is shown in Listing 4. rameters $ (integer) or @ (string).
13  %d\n", saddr, daddr, sport, dport, urg,
The “Hello World” example is a great Of course, you wouldn’t want to do
14 ack, psh, rst, syn, fin)
demonstration of the generic struc- without loops (while, until, for, if/​
ture of a SystemTap script. A script else), which give you useful flow
always comprises two parts: an event control options for the script.
Listing 6: Tapset tcp.stp and a handler – typically
probe tcp.receive = kernel.function("tcp_v4_rcv") { preceded by a probe in-
02  iphdr = __get_skb_iphdr($skb)
struction. In this example,
03  saddr = ip_ntop(__ip_skb_saddr(iphdr))
the event is the read.vfs
04  daddr = ip_ntop(__ip_skb_daddr(iphdr))
function and the handler
05  protocol = __ip_skb_proto(iphdr)
is the printf command
07  tcphdr = __get_skb_tcphdr($skb)
that outputs text to stdout.
08  dport = __tcp_skb_dport(tcphdr)
The handler is always ex-
09  sport = __tcp_skb_sport(tcphdr) ecuted when the specified
10  urg = __tcp_skb_urg(tcphdr) event occurs. Events can
11  ack = __tcp_skb_ack(tcphdr) be kernel functions, sys-
12  psh = __tcp_skb_psh(tcphdr) calls, or, as in this exam-
13  rst = __tcp_skb_rst(tcphdr) ple, prebuilt tapsets – that
14  syn = __tcp_skb_syn(tcphdr)
is, prebuilt code blocks for
15  fin = __tcp_skb_fin(tcphdr)
specific kernel functions Figure 2: After the syntax of the SystemTap script is checked, the
and system calls. script is converted to C and loaded as a kernel module.

46 Admin 01 w w w. a d m i n - m aga z i n e .co m

SystemTap To o l s

Instead of searching through a mass build system. The target systems only comprehensive tapset library, this can
of data for the required information, require the systemtap-runtime RPM be done without serious programming
which is the case with Strace, System- and the staprun program it contains. skills. Advanced users will enjoy the
Tap lets you output the information The following command creates a flexible, Awk-like scripting language
after exceeding a specific threshold prebuilt binary kernel module for the that gives them the freedom to create
value, or when a specific event oc- target system: highly complex tracing and profiling
curs. Thanks to the functional scope scripts. The SystemTap FAQ [4] and
stap ‑r kernel‑PAE‑‑174.2.22 U
of the language, the choice of lan- capt‑io.stp ‑m read‑io
the language reference [5] are useful
guage constructs is more than ad- ports of call for more help. n
equate. The build system also needs the
Listing 7 shows another example that kernel-debuginfo package to match
uses the tapset. The global the target system version, and you The Author
variable totals is an associative array must ensure that the build and tar- Thorsten Scherf is a Senior Consultant for Red
in this case. It contains the process get systems have the same hardware Hat EMEA. You can meet him as a speaker at
names and process IDs for all the architecture. After creating a kernel conferences. He is also a keen marathon runner
applications that access the VFS sub- module, copy it to the target system whenever time permits.
system to read data from disk. The and launch it with staprun:
counter is incremented each time it’s Listing 7: Finding I/​O-Intensive Apps
staprun capt‑io.ko
accessed. 01 
If you are interested in a specific user- If you want non-root users to load
global totals;
space program, you’ll need to install this kernel module, they need to be
the matching debuginfo package for members of the stapusr group; mem- 05 
the application. To make things easy, bers of the stapdev group can addi- 06 
I will look at the ls tool as an ex- tionally compile their own scripts. 07  totals[execname(), pid()]++

ample. To perform a trace here, you’ll 08 

need the coreutils-debuginfo package. Conclusions 10 
probe end
Calling stap as in Listing 8 gives you
an overview of the functions in a spe- The SystemTap tracing and profil- 12  printf("** Summary ** \n")
cific process. ing tool lets regular users perform 13  foreach ([name,pid] in totals‑)
If the parameters of a specific func- detailed analyses of kernel and us- 14  printf(
"%s (%d): %d \n", name, pid,

tion are also of interest, you can erspace programs without rebooting totals[name,pid])
change the call to stap as shown in the whole system. Thanks to the
Listing 9.
Listing 8: SystemTap Tracing a Userspace App
Cross-Compiling stap ‑e 'probe process("ls").function("*").call {log (pp())}' ‑c 'ls ‑l'
total 20

If you want to run a SystemTap script ‑rw‑rw‑r‑‑. 1 tscherf tscherf 17347 2010‑04‑12 08:43 systemtap.txt
on multiple systems, you will prob-
ably prefer not to have to install the
compiler and the kernel debug in-
formation on all of these machines. process("/bin/ls").function("xmemdup@/usr/src/debug/coreutils‑7.6/lib/xmalloc.c:107").call
In fact, you only need to do so on a process("/bin/ls").function("xmalloc@/usr/src/debug/coreutils‑7.6/lib/xmalloc.c:43").call
Info process("/bin/ls").function("free_pending_ent@/usr/src/debug/coreutils‑7.6/src/ls.c:1132").call

[1] OProfile project homepage: [...]

[http://​­oprofile.​­sourceforge.​­net/​­news] process("/bin/ls").function("close_stdout@/usr/src/debug/coreutils‑7.6/lib/closeout.c:107").call
[2] Kernel debuginfo information:
­2C_and_how_do_I_get_them.​­3F] Listing 9: Parameter Tracing
[3] SystemTap project homepage: stap 
‑e 'probe process("ls").function("clone_quoting_options").call {log (probefunc() . " " . $$parms) }'

[http://​­sourceware.​­org/​­systemtap/] ‑c '/bin/ls ‑l'

[4] SystemTap FAQ: [http://​­sourceware.​­org/​ total 20

‑rw‑rw‑r‑‑. 1 tscherf tscherf 18216 2010‑04‑12 09:02 systemtap.txt
clone_quoting_options o=0x0
[5] SystemTap language reference: [http://​
clone_quoting_options o=0x0

w w w. a d m i n - m aga z i n e .co m Admin 01 47

© Cathy Yeulet, V i rt ua l i z at i o n Package Your Scripts

Bundle your custom apps in a Debian package one line. If you’ve ever worked with
packages in Ubuntu or any other De-

It’s a Wrap
bian-related Linux, you’ve probably
needed to download a package from
an online source and install it:

apt‑get install blarblar.deb

Get standard scripts and custom applications into the cloud with the On the inside, a Debian package is
­Debian packaging system. By Dan Frost an archive of binary files, scripts, and
any other resources an application
You’ve got a cloud. It’s great. You ing and releasing your scripts. needs, plus a handful of control files
can scale, and you’ve got redundancy. In this article, I’ll show how to create that the various command-line tools
But you have about 20 scripts for a Debian packages and how to install use to install the package.
bunch of tasks (e.g., one for when them (which you probably already Because this standard package format
an instance is booted up and another know). And, I’ll explain how the is so easy to install on any Debian-
for when its IP changes) and these process will make you feel more com- based Linux, it’s a great way to get
scripts aren’t getting any shorter, fortable about pushing changes live standard scripts and custom applica-
they’re getting better and longer. If across your cloud. I talked to cloudy tions into the cloud. Often you need a
you want to manage them in your people about how to get code onto few lines to configure a new instance
favorite versioning software (which I new instances, and I tried lots of dif- or to connect the instance to the rest
hope is Git, but might be something ferent things, but the Debian package of your cloud, and storing all those
else), how do you get that onto the is such a solid, reliable format that I scripts in a package keeps things very
new instances simply? just had to share it. neat.
Enter the not-new-at-all technology of The parts of a Debian package I’m
Debian packages. They are straight- Debian Packages most interested in are the control
forward to use across any Debian- files, which live in a directory called
based Linux and simple to create, and Debian packages are simply archives DEBIAN. Control files tell Debian all
they provide an ideal way of contain- that are very easy to install, usually in about what the package contains,

48 Admin 01 w w w. a d m i n - m aga z i n e .co m

Package Your Scripts V i rt ua l i z at i o n

what it’s called, the version, and ing and development servers all sit interaction. Everything has to run
so on. The second part is the code, inside one Debian package. automatically when new instances
which I’ll look at in detail later. start up, and you really don’t want
A Cloudy Package your script waiting for a human that
Creating the Package doesn’t exist.
A tiny HTML file isn’t all that useful Next, package your project into a .deb
To show this process at work, I’ll in the cloud, so I’ll look at something file and place it somewhere public
create a simple package. Suppose I a bit more useful. Server configura- from which you can download. This
need a simple page that can sit on tion can be set from the Debian pack- might be where you host, but it is
your server until an application is age simply by placing your preferred much better to put it somewhere re-
installed. Good practice dictates that configuration file in the package: silient, like S3 [2].
your instances should fail nicely, so Next, log in to Scalr and add the fol-
./etc/apache2/conf.d/ our‑config.conf
if you start 10 instances when your lowing lines to a new script that will
app gets 6 million tweets, you at least As long as Apache is configured to run on the event OnHostUp (Figure 1).
want them to deliver a nice page be- include this file (which, in Ubuntu, it
# install the package
fore they’re ready to do business. often is), it will take effect right away. wget ‑O myserver.deb http://mybucket. U
To begin, Although you might want to control
n Create a directory called myserver this with tools such as Puppet after deb ‑i myserver.deb

and the directories and files inside deployment, starting the instance # run the script
it: with a good configuration will help
keep the environment sane from the Save the Scalr script under the name
outset. of the event that you want to trigger
Cloud hosting becomes difficult when it and go to the farm configuration.
n Put the code shown in Listing 1 in you use strange configurations – cre- Now you can add your neatly orga-
./DEBIAN/control. ating exceptions for some apps or nized scripts without having to edit
n Put the file shown in Listing 2 in generally working against the grain
./var/www/index.html. (e.g., using Tomcat’s configuration Listing 1: Control Files
If the path to the index.html file looks style and Apache’s config directories). Package: myserver
familiar, it’s because the file structure Avoid customizing the environment Version: 0.0.1

inside your package mirrors exactly too much because it will mean extra Section: server

the structure in the target instance. If maintenance in the future and could Priority: optional

you want a file in /var/some/where/ limit how you can scale. Architecture: all
Essential: no
here, just create that path inside your Another common script for cloud
Installed‑Size: 1024
package project. servers will run tasks at certain points
Maintainer: Dan Frost []
Once you’ve created this amazing in the instance’s life cycle. For in-
Description: My scripts for running stuff in the cloud
page, package it up with: stance, a service such as Scalr will
run scripts on various events, such
$ dpkg‑deb ‑‑build myserver
dpkg‑deb: building package U
as OnHostUp, OnHostInit, and OnIPAd‑
`myserver' in `myserver.deb' dressChanged. You can create some Listing 2: Message Page
scripts for these events in your De- 01  <html>
When you look in the directory, you’ll bian package: 02  <head>
see a file called myserver.deb. Now 03  <title>We're getting there...</title>
that your project is all packaged up, ./usr/local/myserver/bin/on‑ip‑ U
04  </head>

you can install it. But, before you do, address‑

05  <body>
06  <h1>Give us a moment.</h1>
take a look at what’s inside:
07  <p>We're just getting some more machines
The first script should download an
$ dpkg‑deb ‑‑contents myserver.deb plugged in ...</p>
HTML file or a PHP file from either S3
08  </body>
then install: or your existing repository and place
09  </html>
it in the document root:
$ sudo dpkg‑deb ‑i myproject.deb

cd /var/www/
After running this command, you’ll Listing 3: HTML File
wget ‑O tmp.tgz http://mybucket.s3. U
find the HTML file sitting there for 01 
$ cat /var/www/index.html

when Apache starts (Listing 3): tar xzf tmp.tgz 02 


The next step is to create a skeleton service apache2 restart 03  <head>

04  <title>We're getting there ...</title>
set of scripts for a cloud provisioning These scripts will be like any other
05  </head>
service, such as Scalr or RightScale script you would write, with one im-
06  ...
[1]. Indeed, the scripts I use for host- portant difference: no more human

w w w. a d m i n - m aga z i n e .co m Admin 01 49

V i rt ua l i z at i o n Package Your Scripts

deb, so you can install, run arbitrary

tests, and repeatedly verify that your
cloud scripts are stable before they
hit the live environment. And you
When everything is stable, upload to
S3, which you might want to script
as well:

s3cmd put myserver.deb U


Then, all you need is a corresponding

script to run on your instances. Cre-
ate a new script in Scalr or RightScale
that downloads and installs the latest

wget ‑O myserver.deb U U
Figure 1: Creating a new script. myserver.deb
deb ‑i myserver.deb

scripts via a web interface (Figure 2). you can test them more easily outside
If you have a team working on your the cloud. The ability to test server configuration
cloud hosting, you can even start us- in the cloud, for the cloud, is really
ing standard code management, such The Package in Production important.
as Git or SVN, to version your cloud If you’ve been running nice chunky
environment’s bootup and configura- Everything thus far might feel a bit servers for years, you wouldn’t make
tion scripts. heavy-handed. I put a lot of effort changes to them unless you were 100
A second event script, which is called into getting a short script up onto a percent sure, but with cloud comput-
each time the IP address changes, cloud instance. But suppose you have ing, you can prototype your configu-
would typically update Dynamic DNS a running server farm, and you need rations and settle your nerves before
with a one-liner (you’ll need to set up to update some scripts across the putting things live.
your Dynamic DNS account first): farm. When your cloud is running, you will
Several cloud services let you edit want every opportunity to test the
curl ' U
servlet/updateip?username= U
scripts via a web interface, which is scripts, so being able to install, run,
myuser&password=mypassword&id= U fine up to a point, but beyond a few and test them on any instances is
99999999&ip=' lines, you will start pining for Emacs valuable.
or your favorite editor. A custom .deb
Once you’ve placed this code in the package makes it easy to create and After Installation: Uninstall
script on‑ip‑address‑, sim- test the script on local machines or a
ply package it up into your .deb file, development cloud before uploading So far, Debian packages might just
upload it to S3 again, and start a new the final version to the production look like glorified tarballs, so why
instance. With this approach, testing environment. not just tar up your scripts? Well …
small changes takes a little longer, but Installing the script on instances is they’re better than that. Two hooks
because the scripts are all in a .deb, simply a matter of deb ‑i myserver. are provided: post-install and pre-
uninstall. Once your Debian pack-
age’s files have been copied to the
filesystem, the post-install script,
./DEBIAN/postinst, is run, and when
you uninstall, Debian removes your
files before running ./DEBIAN/prerm.
With these scripts, you can install
software, start services, and call a
monitoring system to tell it exactly
what’s going on with the new in-
For example, open ./DEBIAN/postinst
Figure 2: Adding a script. and add something like:

50 Admin 01 w w w. a d m i n - m aga z i n e .co m

Linux Magazine
curl U

script that installs and con-
figures HAProxy, but you do
Online Training with

this on your local machine.
When you’re happy, you
Linux Magazine Academy
How you keep your moni-
toring systems informed
commit this into your De-
bian package and install it Preparing for the LPIC
depends on what you’re
running, but you can add
on some cloud instances for
exam - the easy way:
arbitrary commands here to When your HAProxy scripts

keep yourself happy. A more all work fine, simply push

to Com pT IA
typical post-install task is your Debian package, along

powered by L
sym-linking your scripts into with the new script, up to
the standard path: S3. Next, just terminate your
HAProxy instance and wait
ln ‑s /usr/local/myserver/bin/ U
for a new one to replace it
on‑host‑ that will run the new scripts,
installing and running
Anything that gets your
scripts working, such as
HAProxy instead of whatever
you had before.
starting any services that are To get extra code onto an CERTIFIED WITH LPIC:
provided or used by your instance, just pull it by us-
scripts, should be done in ing SVN, Git, or Wget; put it Professional training
the post-install script: into place; and the work is
done. So, if you have a huge
for the exams
service apache2 start
service my‑monitor start
repository of PDFs that never LPI 101 and 102
change or a massive archive
However, this is not where database, your scripts can
you install your web app’s copy this down to instances
code, nor is it where you so that each runs indepen-
grab the latest data. Stick dently.
to getting the helper scripts Anything you can do on
running in the Debian hooks the command line can be
and installing your site from
the scripts inside your pack-
scripted, and packing up
your common tasks into a
❚ hardware settings
Remember that the key to
Debian package means your
best scripts and best config
❚ package management
cloud computing is scaling
without friction. Your scripts
will be used on all of your
❚ partitioning and file systems
must install themselves with-
out the need for checking
Finally, remember that be-
ing scalable means being
❚ shell environment
the OS afterward, so use the
hooks to leave everything
friction-free. The people I
work with use Debian pack-
❚ automate system administration
ready to go live. ages, because if the package
installs, we’ve won half the
❚ network configuration
Why It All Makes battle: Our scripts are on the
❚ security administration tasks
Sense instance. It’s a standard and
convenient way of deploying,
To finish, I’ll look at a real- and it works every time. n ❚ troubleshooting

world example. Suppose you
want to change your proxy
from Apache to HAProxy, Info

and you want your web serv- [1] “Scaling the Cloud with Scalr”
ers to host some extra code by Dan Frost, Linux Magazine,
o r c u
because this makes your August 2010, pg. 20 off f srcibers
app more scalable. Instead [2] Amazon Simple Storage sub
of changing to HAProxy on Service (S3):
the instance, you create the [http://​­aws.​­amazon.​­com/​­s3/]

For more information and to order:

w w w. a d m i n - m aga z i n e .co m
V i rt uA l i z At i o n openVz

Operating system virtualization with OpenVZ

Container Service
© sculpies,

the virtualization technology market is currently concentrating on hypervisor-based systems, but hosting pro-
viders often use an alternative technology. container-based solutions such as openVz/Virtuozzo are the most
efficient way to go if the guest and host systems are both linux. By thomas drilling

Hypervisor-based virtualization of the CPU, chipset, and peripherals. accessing devices. For the application,
solutions are all the rage. Many com- If you have state-of-the-art hardware all of this appears to be a separate
panies use Xen, KVM, or VMware (a CPU with a virtualization extension universe. A container has to be de-
to gradually abstract their hardware – VT), the performance is good. signed so that the application thinks
landscape from its physical underpin- However, hypervisor-based systems it has access to a complete operating
nings. The situation is different if do have some disadvantages. Because system with a run-time environment.
you look at leased servers, however. each guest installs its own operating From the host’s point of view, con-
People who decide to lease a virtual system, it will perform many tasks in tainers are simply directories. Because
server are not typically given a fully its own context just like the host sys- all the guests share the same kernel,
virtualized system based on Xen or tem does, meaning that some services they can only be of the same type as
ESXi, and definitely not a root server. might run multiple times. This can the host operating system or its ker-
Instead, they might be given a re- affect performance because of over- nel. This means a Linux-based con-
source container, which is several lapping – one example of this being tainer solution like OpenVZ can only
magnitudes more efficient for Linux cache strategies for the hard disk sub- host Linux guests. From a technical
guest systems and also easier to set system. Caching the emulated disks point of view, resource containers ex-
up and manage. A resource container on the guest system is a waste of time tend the host system’s kernel. Adding
can be implemented with the use of because the host system already does an abstraction layer then isolates the
Linux VServer [1], OpenVZ [2], or this, and emulated hard disks are ac- containers from one another and pro-
Virtuozzo [3]. tually just files on the filesystem. vides resources, such as CPU cycles,
memory, and disk capacity (Figure 1).
Benefits Parallel Universes Installing a container means creating
a sub-filesystem in a directory on the
Hypervisor-based virtualization solu- Resource containers use a different host system, such as /var/lib/vz/
tions emulate a complete hardware principle on the basis that – from the gast1; this is the root directory for the
layer for the guest system. Ideally, application’s point of view – every guest. Below /var/lib/vz/gast1 is a
any operating system including appli- operating system comprises a filesys- regular Linux filesystem hierarchy but
cations can be installed on the guest, tem with installed software, space for without a kernel, just as in a normal
which will seem to have total control data, and a number of functions for chroot environment.

52 Admin 01 w w w. A d m i n - m AgA z i n e .co m

openVz V i rt uA l i z At i o n

live migration, checkpointing and restoring (see the box “Live Migration, Check-
pointing and Restoring”). Incidentally,
OpenVZ containers can be shifted from one physical host to another during operations (Live
the host is referred to as the hardware
migration). Ideally, the user will not even notice this process. However, the host environment
must be configured to support live migration from a technical point of view. In other words, both
node in OpenVZ-speak.
virtual environments must reside on the same subnet, and data transmission rate must be high To be able to use OpenVZ, you will
enough. Additionally, the target virtual environment (VE) must have sufficient hard disk space. If need a kernel with OpenVZ patches.
these conditions are fulfilled, the following command starts the migration: One problem is that the current stable
release of OpenVZ is still based on
vzmigrate ‑online target IP VEID
kernel 2.6.18, and what is known as
Target IP is the network address of the VE into which you want to migrate to the VE with the ID the super stable version is based on
of VEID. Of course, the vzmigrate tool supports a plethora of different options (e.g., for migrat- 2.6.9. It looks like the OpenVZ de-
ing over secure connections). The exact syntax and other examples of applications are discussed velopers can’t keep pace with official
[12]. Additionally, OpenVZ can create what it refers to as checkpoints (snapshots) of VEs: A kernel development. Various distribu-
checkpoint freezes the current state of the VE and saves it in a file. The checkpoint can be cre- tions have had an OpenVZ kernel,
ated from within the host context with the vzctl chkpnt VEID command. The checkpoint file such as the last LTS release (v8.04) of
can be used later to restore the VE on another OpenVZ host with vzctlrestore VEID.
Ubuntu, on which this article is based
(Figure 2).
The container abstraction layer makes ever, you can’t load any drivers or Ubuntu 9.04 and 9.10 no longer fea-
sure that the guest system sees its kernels from within a container. The ture OpenVZ, apart from the VZ tools;
own process namespace with separate predecessors of container virtualiza- this also applies to Ubuntu 10.04. If
process IDs. On top of this, the kernel tion in the Unix world are technolo- you really need a current kernel on
extension that provides the interface gies that have been used for decades, your host system, your only option is
is required to create, delete, shut such as chroot (Linux), jails (BSD), or to download the beta release, which
down and assign resources to con- Zones (Solaris). With the exception uses kernel 2.6.32. The option of us-
tainers. Because the container data of (container) virtualization in User- ing OpenVZ and KVM on the same
are extensible on the host file system, Mode Linux [4], only a single host host system opens up interesting pos-
resource containers are easy to man- kernel runs with resource containers. sibilities for a free super virtualization
age from within the host context. solution with which administrators
OpenVZ can experiment.
Efficiency If you are planning to deploy OpenVZ
OpenVZ is the free variant of a com- in a production environment, I sug-
Resource containers are magnitudes mercial product called Parallels gest you keep to the following rec-
more efficient than hypervisor sys- Virtuozzo. The kernel component is ommendations: You must disable
tems because each container uses available under the GPL; the source SELinux because OpenVZ will not
only as many CPU cycles and as code for the matching tools under the work correctly otherwise. Addition-
much memory as its active applica- QPL. OpenVZ runs on any CPU type, ally, the host system should only be
tions need. The resources the abstrac- including CPUs without VT exten- a minimal system. You will probably
tion layer itself needs are negligible. sions. It supports snapshots of active want to dedicate a separate partition
The Linux installation on the guest containers as well as the Live migra- to OpenVZ and to mount this below,
only consumes hard disk space. How- tion of containers to a different host /ovz, for example Besides this, you

Container Context

Resource Resource Resource

Container 1 Container 2 Container 3

Host Context
Abstraction Layer

Host System Kernel

Figure 1: In virtualization based on resource containers, the host and guest use Figure 2: openSUSE with Ubuntu: System virtualization with resource containers
the same kernel; therefore, they must be of the same type. This means that a is an interesting option if you need to host (multiple) Linux guest systems as
Linux host can only support Linux guests. efficiently as possible on a Linux host system.

w w w. A d m i n - m AgA z i n e .co m Admin 01 53

V i rt ua l i z at i o n OpenVZ

Figure 3: I installing OpenVZ from the package sources for Ubuntu 8.04 – the Figure 4: The OpenVZ developers provide container templates for various guest
last version of Ubuntu to officially include an OpenVZ kernel. The only package systems; this makes installing a guest system a quick and easy experience.
needed for this was the linux‑openvz meta-package. Templates from the community are also available.

should have at least 5GB of hard disk detailed information on this, refer to check of service vz status should
space, a fair amount of RAM (at least the sysctl section in the quick install now tell you that OpenVZ is running.
4GB), and enough swap space. guide, which covers providing net-
work access to the guest systems, in- Container Templates
Starting OpenVZ volving setting up packet forwarding
for IPv4 [5]. Then, you need to re- OpenVZ users don’t need to install
Installing OpenVZ is simple. Users on boot with the new kernel. If you edit an operating system in the traditional
RPM-based operating systems such as sysctl after rebooting, you can reload sense of the word. The most con-
RHEL or CentOS can simply include by typing sudo sysctl ‑p. Typing venient approach to set up OpenVZ
the Yum repository specified in the containers is with templates (i.e.,
sudo /etc/init.d/vz start
quick install manual on the project tarballs with a minimal version of the
homepage. Ubuntu 8.04 users will wakes up the virtualization machine. distribution you want to use in the
find a linux‑openvz meta-package Next, you should make sure all the container). Administrators can create
in the multiverse repository, which OpenVZ services are running; this is templates themselves, although it’s
installs the required OpenVZ kernel, done easily (on Ubuntu) by issuing: not exactly trivial [6]. Downloading
including the kernel modules and prebuilt templates [7] and copying
sudo sysv‑rc‑conf ‑list vz
header files (Figure 3). At the time of them to the template folder is easier:
writing, no OpenVZ kernel was avail- If the tool is missing, you can type
sudo cp path_to_template U
able for Ubuntu 10.04. If you are in- /var/lib/vz/template/cache
sudo apt‑get install sysconfig
terested in using OpenVZ with a cur-
rent version of Ubuntu, you will find to install it. Debian and Red Hat users Besides templates provided by the
a prebuilt deb package in Debian’s can run the legacy chkconfig tool. A OpenVZ team, the page also offers
unstable branch. To install type:

sudo dpkg ‑i linux‑base_2.6.32‑10_all.debU


The sudo apt‑get ‑f install com-

mand will automatically retrieve any
missing packages. You will also need
to install the vzctl tool, which has a
dependency for vzquota.
Before setting up the containers and
configuring the OpenVZ host envi-
ronment, you need to modify a few
kernel parameters that are necessary
to run OpenVZ in the /etc/sysctl.
conf file on the host system. For more Figure 5: A couple of clear-cut commands are used for creating and starting a VE and for entering the VE.

54 Admin 01 w w w. a d m i n - m aga z i n e .co m

OpenVZ V i rt ua l i z at i o n

a number of community templates subnet and tell them the DNS server The ‑‑ipadd option lets you assign a
(Figure 4). address, which lets OpenVZ create local IP address. If you need to install
venet devices. All of the following a large number of VEs, use VEID as
Configuring the Host commands must be given in the host the host part of the numeric address.
Environment context. To do this, you first need to
sudo vzctl set VEID ‑‑ipadd U
stop the VE and then set all the basic IP-Address ‑‑save
The /etc/vz/vz.conf file lets you parameters. For example, you can set
configure the host environment. This the hostname for the VE as follows: The DNS server can be configured us-
is where you specify the path to the ing the ‑‑nameserver option:
container and template data on the sudo vzctl set VEID U
sudo vzctl set VEID U
host filesystem. If you prefer not to ‑‑hostname Hostname ‑‑save ‑‑nameserver Nameserver-address ‑‑save E
use the defaults of


you can set your own paths. VE_ROOT

is the mountpoint for the root direc-
tory of the container. The private
data for the container are mounted in
VE_PRIVATE. VEID is a unique ID that
identifies an instance of the virtual
environment. All OpenVZ tools use Figure 6: The virtual environment uses venet devices to communicate with the outside world.
this container ID to address the re-
quired container.

Creating Containers
The vzctl, which is only available
in the host context, creates contain-
ers and handles most management Figure 7: The vzlist command outputs a list of active VEs.
tasks, too. In the following example,
I used it to create a new VE based on
a template for openSUSE 11.1 that I

sudo vzctl create VEID U

‑‑ostemplate suse‑11.1‑x86_64

The template name is specified with-

out the path and file extension. The
sudo vzctl start VEID starts the
VE, and sudo vzctl stop VEID stops
it again (Figure 5). The commands
sudo vzctl enter VEID and exit let
you enter and exit the VE.
Entering the VE gives you a working
root shell without prompting you for
a password. Unfortunately, you can’t
deny root access in the host context.

Network Configuration
The next step is to configure network
access for the container. OpenVZ
supports various network modes for
this. The easiest option is to assign Figure 8: User Bean Counters, a set of configuration parameters, allow the administrator to limit resources
the VEs an IP on the local network/​ for each virtual environment.

w w w. a d m i n - m aga z i n e .co m Admin 01 55

V i rt ua l i z at i o n OpenVZ

After restarting the VE, you should be

able to ping it from within the host
context. After entering the VE, you
should also be able to ping the host
or another client (Figure 6). For more
details on the network configuration,
see the “Network Modes” box.

OpenVZ Administration
Figure 9: Virtual Ethernet devices make the VE a full-fledged member of the network with all its advantages
The vzctl tool handles a number and disadvantages.
of additional configuration tasks.
Besides starting, stopping, entering, Without additional configuration, the ues lets you specify a minimum and
and exiting VEs, you can use the use of VEs is a security risk because maximum limit:
‑set option to set a number of op- only one host kernel exists, and each
sudo vzctl set 100 ‑‑diskspace 8G:10G U
erational parameters. Running vzlist container has a superuser. Besides ‑‑quotatime 300
in the host context displays a list of this, you need to be able to restrict
the currently active VEs, including the resources available to each VE, Incidentally, sudo vzlist ‑o lists all
some additional information such as such as the disk and memory and the set UBC parameters. Note that
the network parameter configuration the CPU cycles in the host context. some UBC parameters can clash, so
(Figure 7). OpenVZ has a set of configuration pa- you will need to familiarize your-
In the VE, you can display the pro- rameters for defining resource limits self with the details by reading the
cess list in the usual way by typing known as User Bean Counters (UBCs) exhaustive documentation. To com-
ps. And, if the package sources are [8]. The parameters are classified by pletely remove a container from the
configured correctly, patches and importance in Primary, Secondary, system, just type the
software updates should work in the and Auxiliary. Most of these param-
sudo vzctl destroy
normal way using apt, yum, or yast eters can also be set with vzctl set
depending on your guest system. (Figure 8). command.
For the next step, it is a good idea to For example, you can enter
enter the VE by typing vzctl enter
sudo vzctl set 100 ‑‑cpus 2 Conclusions
VEID. Then, you can set the root pass-
word, create more users, and assign to set the maximum number of CPUs Resource containers with OpenVZ
the privileges you wish to give them; for the VE. The maximum permitted offer a simple approach to running
otherwise, you can only use the VEs disk space is set by the ‑‑diskspace virtual Linux machines on a Linux
in trusted environments. parameter. A colon between two val- host. According to the developers, the

Network Modes
In many cases, a venet device is all you need that is visible in the host context. Within the sudo ifconfig veth100.0

in the line of network interfaces in a VE. Each container, the administrator can then use Linux A bridge device is the only thing missing for
venet device sets up a point-to-point connec- tools to configure the network interface with a host network access; to set this up host-side,
tion to the host context and can be addressed static address or even use DHCP. give the sudo brtcl addbr vmbr0 com-
using an IP address from the host context. The kernel module is loaded when the OpenVZ mand, then sudo brctl addif vmbr0
Venet devices have a couple of drawbacks, kernel boots. You can check that you have it by
verth100.0 to bind it to the veth device, as-
however: They don’t have a MAC address and issuing the sudo lsmod | grep vzethdev
suming bridge‑utils is installed. Host-side you
thus don’t support ARP or broadcasting, which command. To configure a veth device in the
now have the interfaces l0, eth0, venet0,
makes it impossible to use DHCP to assign IP container, run
and veth100.0. If the bridge device is set up
addresses. Also, network services like Samba
sudo vzctl set 101 ‑‑netif_add eth0 ‑‑save correctly, brctl show gives you a listing simi-
rely on functional broadcasting.
A virtual Ethernet (veth) device solves this where eth0 is the interface name in the con- lar to Figure 10. The additional veth device set
problem (Figure 9). These devices are sup- tainer context. The device name in the host up here, 100.1, is for test purposes only and is
ported by a kernel module that uses vzctl to context defaults to vethVEID. not important to further steps.
present a virtual network card to the VE. The If needed, you can assign MAC addresses and Virtual network devices are slightly slower
vzethdev sets up two Ethernet devices: one in device names explicitly. The device can be than venet devices. Also, security might be
the host context and one in the VE. The devices listed in the host context in the normal way an issue with a veth device – this places a
can be named individually, and you can manu- with ifconfig. The number following the dot full-fledged Ethernet device in the container
ally or automatically assign a MAC address to (0) refers to the device number; here, this is context, which the container owner could
them. The host-side device can also reside on the first veth device in the container with the theoretically use to sniff all the traffic outside
a bridge to give the VE a network environment VEID of 100: the container.

56 Admin 01 w w w. a d m i n - m aga z i n e .co m

OpenVZ V i rt ua l i z at i o n

virtualization overhead with OpenVZ OpenVZ kernel requires just a couple ing resource limits in the form of
is only two to three percent more CPU of simple steps, and the template the GUI-based Parallels Management
and disk load: These numbers com- system gives you everything you need Console [9] or Parallels Infrastructure
pare with the approximately five per- to set up guest Linux distributions Manager [10]. The excellent OpenVZ
cent quoted by the Xen developers. quickly. wiki covers many topics, such as the
The excellent values for OpenVZ OpenVZ has a head start of several installation of Plesk in a VE or set-
are the result of the use of only one years development compared with ting up an X11 system [11]. OpenVZ is
kernel. The host and guest kernels modern hypervisor solutions such as the only system that currently offers
don’t need to run identical services, KVM and is thus regarded as mature. Linux guest systems a level of perfor-
and caching effects for the host and Unfortunately, the OpenVZ kernel mance that can compete with that of
guest kernels do not interfere with lags behind vanilla kernel develop- a physical system without sacrificing
each other. The containers themselves ment. performance to the implementation
provide a complete Linux environ- However, if you are thinking of de- itself. This makes OpenVZ a good
ment without installing an operating ploying OpenVZ commercially, you choice for virtualized Linux servers of
system. The environment only uses might consider its commercial coun- any kind. n
the resources that the applications terpart Virtuozzo. Besides support,
running in it actually need. there are a number of aspects to take
The only disadvantage of operating into consideration when using re- Info
system virtualization compared with source containers. For example, host- [1] Linux VServer: [http://​­linux‑vserver.​­org/​
paravirtualization or hardware virtu- ing providers need to offer customers ­Welcome_to_Linux‑VServer.​­org]
alization is that, apart from the net- seamless administration via a web [2] OpenVZ:
work interfaces, it is not possible to interface, with SSH and FTP, or by [http://​­wiki.​­openvz.​­org/​­Main_Page]
assign physical resources exclusively both methods; of course, the security [3] Virtuozzo: [http://​­www.​­parallels.​­com/​­de/​
to a single guest. concerns mentioned previously can- ­products/​­pvc45]
Otherwise, you can do just about not be overlooked. [4] User-Mode Linux: [http://​
anything in the containers, includ- Parallels offers seamless integration ­user‑mode‑linux.​­sourceforge.​­net]
ing installing packages and providing of OpenVZ with Plesk and convenient [5] OpenVZ quick install guide: [http://​­wiki.​
services. Additionally, setting up the administrations tools for, say, impos- ­openvz.​­org/​­Quick_installation]
[6] Creating your own OpenVZ tem-
plates: [http://​­wiki.​­openvz.​­org/​
[7] Prebuilt OpenVZ templates:
[8] OpenVZ User Bean Counters: [http://​­wiki.​
[9] Parallels Management Console:
[10] Parallels Infrastructure Manager: [http://​
[11] X11 forwarding:
[12] Live migration: [http://​­openvz.​­org/​

The Author
Thomas Drilling has been a freelance journalist
and editor for scientific and IT magazines for
more than 10 years. With his editorial office
team, he regularly writes on the subject of open
source, Linux, servers, IT administration, and
Mac OS X. In addition to this, Thomas Drilling is
also a book author and publisher, a consultant
Figure 10: This example includes one venet and one veth device in the host context. The latter is physically to small and medium-sized companies, and a
connected to the host network via a bridge device. The host-side veth bridge looks like a normal Ethernet regular speaker on Linux, open source and IT
device (eth0) from the container context. security.

w w w. a d m i n - m aga z i n e .co m Admin 01 57

or more!

The New IT
New tools, new threats, new technologies...
Looking for a guide to the changing world
of system administration?
ADMIN Network & Security
Explore the new world
of system administration

It isn’t all Windows anymore – and it isn’t all Linux. A router is more than a router.
A storage device is more than a disk. And the potential intruder who is looking for a way
around your security system might have some tricks that even you don’t know. Keep your
network tuned and ready for the challenges with the one magazine that is all for admins.

Each issue delivers technical solutions to the real-world problems you face every
day. Learn the latest techniques for better:

• network security

• system management
R E A L - W
• troubleshooting
• performance tuning
• virtualization

• cloud computing

on Windows, Linux, Solaris, and popular varieties of Unix.

Special introductory offer!

Order by December 31st to save 10%
off the regular subscription price!
Subscription prices for 1 year of ADMIN Magazine (4 issues + 4 DVDs)
are $39.95 for the USA and £24.90/29.90 for UK/Europe
(other regions please see our website).
V i rt uA l i z At i o n Virtual machine manager

Microsoft System Center Virtual Machine Manager 2008 R2

© Tono Balaguer,

Virtual Windows
in theory, virtualizing all your old servers is a good idea, but managing them won’t necessarily become any
easier. Virtual machine manager gives windows administrators an easy option. By Björn Bürstinghaus

Service virtualization is no longer only available as virtual entities and managing an unlimited number of
just an interesting topic for large as the number of virtual machines virtual machines.
corporations and data centers. In and virtualization hosts continues to
fact, virtualization of multiple server rise, administrators need to consider System Requirements
systems on a single physical machine centralizing their management.
has become an affordable option for Microsoft System Center Virtual Ma- To install SCVMM, you need a 64-
small and medium-sized businesses, chine Manager 2008 R2 (SCVMM) is bit version of Windows Server 2008
too. With virtualization and the a management solution for Hyper-V (R2), which you can run as a virtual
consolidation benefits that it offers, (R2) hosts, Virtual Server 2005 R2 machine in smaller environments
system management also changes. hosts, and VMware ESX hosts that (with a maximum of five hosts). The
The machines you are managing are use the VMware VCenter. SCVMM [1] system on which you install SCVMM
offers excellent must be a member of an Active Direc-
scalability, easy tory domain, but you can use it to
management of manage host systems in non-member
hosts and virtual networks. In this case, you’ll need to
machines and install the agents manually because
many benefits automatic installation only works
for the adminis- inside the domain. SCVMM relies on
trator. A Work- Microsoft SQL Server to store data.
group Edition Depending on the size of your envi-
is available for ronment, you can use the free SQL
deployment in Server 2005 Express Edition supplied
small and me- with the bundle, which has a data-
dium-sized busi- base size limit of 4GB, or you can use
nesses: If you an instance of SQL Server 2005 or
use a maximum SQL Server 2008.
of five hosts, this It makes sense to use a separate
Figure 1: The Virtual Machine Manager startup screen shows a selection of the version is a cost- server for the database in larger envi-
components to install. effective way of ronments. You can install the SCVMM

60 Admin 01 w w w. A d m i n - m AgA z i n e .co m

Virtual Machine Manager V i rt ua l i z at i o n

database component on a separate

system; it doesn’t need to reside on
the management server. To install the
management component in SCVMM,
you also need the Windows Auto-
mated Installation Kit (WAIK) 1.1,
which is automatically installed when
you install the management server;
Windows PowerShell 1.0 or 2.0; Win-
dows Remote Management (WinRM)
1.1 or 2.0; and the .NET framework
3.0 (SP1).
The web-based SCVMM Self-Service
Portal additionally requires Internet
Information Server (IIS) version 7.0 or
7.5. The SCVMM Administrator Con-
sole can also be installed on client
systems such as Windows Vista and
Windows 7. If you also want to send
command-line jobs to SCVMM via the
client, you additionally need to install
Windows PowerShell client-side.
Figure 2: The management console gives you an overview of the virtual machines in the central panel and the

Installation system load for the selected VM below.

After starting the installation, you’ll The agents for managing hosts via Microsoft offers a free configuration
be given a choice of components to SCVMM can be installed through the analyzer for SCVMM; after the install,
install for SCVMM (Figure 1). The management console or manually. the analyzer checks whether all the
management server, management If you use the management console components are installed optimally.
console, and self-service portal are for the install, an automatic check is Also, you can use the configuration
all installed separately. When you performed to see whether the host analyzer to check the configuration
install the management server, you has a hypervisor. If not, the Hyper-V on remote systems that you will be
can place the individual modules, role will be installed automatically on deploying as hosts and systems in-
such as the database and the library Windows Server 2008 (R2), and Vir- tended for P2V conversion. If any
server, on different systems. This ar- tual Server 2005 R2 will be installed issues are detected, you’ll be given
rangement improves performance if on Windows Server 2003 (R2). a detailed description and possible
you have a large number of hosts and
virtual machines.
Before the installation starts, an au-
tomatic check is performed to make
sure that the hardware and software
requirements are fulfilled. If this is
the case for all the components, pro-
visioning SCVMM will take less than
20 minutes.
Before installing the self-service por-
tal, you must enable the web server
role on Windows Server 2008 (IIS 7.0)
or Windows Server 2008 R2 (IIS 7.5),
as well as the ASP.NET, IIS-6 meta-
compatibility and IIS 6 WMI compat-
ibility role services. If the role or one
of the additional services is not in-
stalled, you will see an error message
to this effect. Portal access privileges
are configured in the management
console preferences. Figure 3: The Job queue shows modified and outstanding jobs.

w w w. a d m i n - m aga z i n e .co m Admin 01 61

V i rt ua l i z at i o n Virtual Machine Manager

groups access chines on VMware ESX hosts, SSL

to virtual ma- authentication must be enabled on
chines or provide the host side or the VMware ActiveX
templates for control must be installed on the client
creating new from which you will be managing the
systems by way host.
of a web-based
interface. This Command Line-Based
system means Controls
you can allow
developers to PowerShell [2] is an extension of the
restart a test sys- well-known Windows command line;
tem themselves it offers a plethora of administrative
Figure 4: Privileges in the self-service portal: The Web Admins user group is or allow them commands for script-based Windows
allowed to manage its own virtual machines. to create a new management.
virtual machine SCVMM includes more than 150
solutions. To use the configuration based on templates from the library, PowerShell commands, or Cmdlets,
analyzer for SCVMM, you also need without requiring that they access the which you can use for command
the free Microsoft Baseline Security management console. Additionally, line-based management and admin-
Analyzer. you can assign privileges for the por- istration without having to launch
tal individually for various groups or the management console (Figure 6).
The Management Console users, thus allowing certain users to Thus, you can use scheduled tasks on
manage or access a virtual machine Windows to run tasks at a predefined
The SCVMM management console but not restart it or switch it off (Fig- time on the management server – for
allows administrators to handle a full ures 4 and 5). example, to store the status of a vir-
set of host and virtual machine ad- After logging in to the self-service tual machine.
ministrative tasks. The management portal, the user sees all the virtual
console is clear-cut and configurable machine assignments. To open a Intelligent Placement
in many places. connection and manage a virtual ma-
The console is divided into three ar- chine, an ActiveX control is installed One of SCVMM’s most practical
eas (Figure 2). The left-hand panel on the client side; the control requires features is intelligent placement of
contains the navigation aids for the Internet Explorer. virtual machines. Because the man-
various SCVMM subsections (Hosts, Using shared templates, users can agement server monitors the load on
Virtual Machines, Library, Jobs, and create new virtual machines based on the hosts, it automatically displays
preferences). After selecting a subsec- the template in the portal. The host a host statistic when you add a new
tion, its objects are listed in the cen- is then assigned by the built-in intel- machine so that you can easily see
tral panel of the console. The right- ligent placement function in SCVMM. which host is best suited to the task.
hand panel shows you the actions Also note that to control virtual ma- You can customize the automatic host
available for the selected subsection.
The Hosts and Virtual Machines sub-
sections give you a perfect overview
of the status of all your systems. If
you manage a large number of sys-
tems, you can use the filters in the
navigation area on the left to restrict
the number of systems shown.
The Jobs subsection is used to check
all active SCVMM jobs (Figure 3);
Again, you can use the navigation
aids to filter on various criteria. This
helps you quickly identify and resolve
errors and issues.

The Self-Service Portal

With the SCVMM self-service portal,
you can grant individual users or Figure 5: The self-service portal lets non-privileged users manage virtual machines.

62 Admin 01 w w w. a d m i n - m aga z i n e .co m

Virtual Machine Manager V i rt ua l i z at i o n

evaluation function for intelligent tential issues before using the volume 2007 R2 [3] for monitoring host and
placement. shadow copy service to create an virtual machine availability. In this
image. On-the-fly conversion works case, SCVMM not only uses its own
Libraries and Templates with client systems as of Windows XP agent to monitor the systems but also
and for server systems as of Windows provides performance analysis and
The library component in SCVMM Server 2003. For older systems, you reporting for a host or virtual ma-
is a shared directory of virtual hard have only an offline conversion op- chine. The performance and resource
disks, ISO images, hardware, and tion. After conversion, you can shut optimization (PRO) function built into
guest operating system profiles. down the physical system and boot SCVMM can use Operations Manager
Templates automatically provision the system as a virtual machine. 2007 R2 to collect performance data
Windows client and server systems down to the application layer and
quickly. A template comprises a vir- Higher Availability with thus suggest optimization strategies,
tual hard disk and predefined hard- Clustering which are displayed as PRO tips in
ware and operating system profiles. the management console.
The hardware profile lets you specify Host clustering is a useful way to
the minimum requirements for the guarantee system availability. Instead Conclusions
CPU type or the amount of RAM the of expensive SAN memory, the data is
virtual machine needs. When a new provided by cheaper iSCSI solutions. Microsoft System Center Virtual
virtual machine with the specified To create a Hyper-V cluster, you need Machine Manager 2008 R2 greatly fa-
CPU type is created, SCVMM auto- two host systems, both of which ac- cilitates the management and admin-
matically searches for a host with re- cess the same SAN or iSCSI storage. istration of homogeneous or hetero-
sources to match the hardware profile Live migration introduced in Hyper-V geneous virtual infrastructures under
requirements. The operating system R2 means you can move a virtual ma- Windows. Automated provisioning
profile helps automate operating sys- chine between clusters without taking of new client and server systems can
tem provisioning. Besides selecting the virtual machine offline. The previ- be done in minutes with SCVMM.
the operating system, you can also ous version only supported virtual Thanks to the integration of System
configure the administrator password, machine migration if you used the Center Operations Manager 2007 R2,
a license key, the computer name, same processor type in both clusters. SCVMM also directly supports perfor-
and the domain membership. Although this restriction has not been mance and availability monitoring for
completely lifted, it only applies to hosts and virtual machines.
P2V Conversion the CPU vendor, thus improving sup- Because system management al-
port for a variety of hardware in the ways takes a great deal of your time,
SCVMM also converts physical sys- cluster and offering more flexibility. whether you have five or 50 host
tems to virtual machines on the fly systems, it makes sense to plan a
with physical-to-virtual (P2V) migra- Resource Monitoring centralized solution for all aspects of
tion. For this, simply install a small virtualization, which is exactly what
client on the machine; the client SCVMM can be combined with the SCVMM offers. n
checks the system and displays po- System Center Operations Manager

[1] Microsoft System Center Virtual Machine
Manager 2008 R2: [http://​­support.​
[2] PowerShell: [http://​­www.​­microsoft.​
[3] Microsoft System Center Operations Man‑
ager 2007 R2:

The Author
Björn Bürstinghaus is a system administrator
with simyo GmbH in Düsseldorf, Germany. In his
leisure time, he runs Björn’s Windows Blog, a
blog on Microsoft Windows topics located at
Figure 6: Using the PowerShell to move virtual machines between hosts. [http://​­blog.​­buerstinghaus.​­net].

w w w. a d m i n - m aga z i n e .co m Admin 01 63

Ma n ag e m e n t Teamviewer
© Thor Jorgen Udvang,

Convenient graphical remote control nection impossible, the transfer is

handled directly by the Teamviewer

server. The HTTP label in the window
header, rather than the UDP label,
identifies this kind of connection. If
you are worried about using a third-

party server, Teamviewer will sell you
your own authentication server on
Teamviewer will even let you re-
motely control computers that only
Teamviewer is an impressive demonstration of how easy remote control have a modem connection. The soft-
ware vendor improved compression
across routers and firewalls can be. The popular software is now available in version 5 to reduce the amount of
for Linux. By Daniel Kottmair data crossing the wire. Video, Flash
banners, and other applications that
Some 60 million users already have computer on the opposite end of the permanently change screen content
the Teamviewer [1] commercial re- connection can use to access the local are problematic, but a fast DSL con-
mote control solution running on machine. This scheme prevents any- nection will let even those types of
Windows and Mac OS X. Because of body who has ever logged in to that applications run at an acceptable
the many requests from customers, machine from doing so again without speed.
Teamviewer’s manufacturer now pro- the owner’s authorization. You can Private users can run the program
vides a variant for Linux in version 5. keep the newly generated password free of charge, and the vendor offers
Teamviewer facilitates remote access or define one yourself. commercial licenses for commercial
to other computers across a network. use. Teamviewer is available for Win-
The only requirement is that the ma- Connections dows, Mac OS X, and Linux; any plat-
chine at the other end is also running form can remotely control any other
Teamviewer. Teamviewer provides all Remote access without port forward- platform. An iPhone client, available
this functionality in a standalone pro- ing works across routers and firewalls after registering for free online, lets
gram; special client or server versions thanks to one of the globally distrib- you remotely control a computer as
are not available. uted Teamviewer servers on the web, well.
Teamviewer automatically generates a which initiates a 256-bit encrypted The web and iPhone clients are the
globally unique ID on each machine. UDP connection between the two only versions that can only control,
When it is launched, Teamviewer parties. If a proxy server or a firewall rather than work in both directions.
generates a new password that the with content filtering makes this con- The other variants let you control

64 Admin 01 w w w. a d m i n - m aga z i n e .co m

Teamviewer Ma n ag e m e n t

and be controlled, and you can even

change directions mid-flow.

Linux Specifics
Teamviewer offers downloads of deb
and RPM versions 5.0.8206 pack-
ages for Linux, along with an X64
deb package and a simple tarball that
you don’t even need to install. Team-
viewer for Linux is still beta, and the
vendor asks for feedback and bug
The program is based on a modified
version of Wine, although the ven-
dor has made some Linux-specific
changes (e.g., to accelerate reading of
X server graphics). Although it uses
Wine, it is not just a copy of the Win- Figure 1: The Teamviewer window at program launch.
dows version. A native Linux version
is not available right now, but the ing the wallpaper (to avoid unneces- a connection between a v4 Mac client
vendor is considering creating one de- sary data traffic). and a v5 Linux client worked without
pending on acceptance and popularity problems.
of the Wine-based version. In the Lab The program offers three operating
The program offers a plethora of modes when it launches (Figure 1):
built-in remote control solutions, such The Linux version works fairly well Remote support, Presentation, and
as the ability to change direction, despite its beta status. One thing that File transfer. Remote support lets
reboot, simulate a Ctrl+Alt+Del key always worked during testing – no you remotely control a system, and
press, and transfer files conveniently matter what network the comput- presentation mode lets you demon-
between two machines. Multiple ers used or what firewalls they were strate an action to one or multiple
logins on a single machine are also hiding behind – was the connection users on their own machines. File
supported (e.g., for training purposes setup. Teamviewer has thus mas- transfer leaves out the administrative
in which you need to demonstrate tered the most important discipline and graphic functions and simply
something to a group of users). in remote control with flying colors. sends files to, or retrieves them from,
The VoIP and video chat function Also, no version problems emerged; a remote machine. This mode is ac-
introduced in version 5 is also use-
ful, and the application relies on free
codecs: Speex for audio and Theora
for video. Video on Linux only works
in the receiving direction right now.
V4L-connected Linux webcams are
not currently supported by Team-
The Linux version has a couple of
other restrictions: The whiteboard
function, which lets users draw on
a whiteboard at the same time, and
VPN support both fail to provide the
goods. The program does not transmit
virtual consoles, so you need an X
server. The reboot, Ctrl+Alt+Delete
key sequence, and Disable Input/​
Display on remote computer functions
all require the remote machine to run
Windows – Mac users have a similar
problem. And, the same thing applies
to changing the resolution and remov- Figure 2: File transfers between computers are convenient and easy to keep track of.

w w w. a d m i n - m aga z i n e .co m Admin 01 65

Ma n ag e m e n t Teamviewer

cessible at any time during normal On the bottom right is the connection vided by Compiz. Teamviewer only
remote support. monitor, which you can also hide and transfers the window content and not
From the Teamviewer startup win- which tells you who is accessing the the windows, so window zooms and
dow, use Extras | Options to change computer across the wire (Figure 3). soft fades just create unnecessary traf-
various settings, such as your own If you so desired – again this could be fic. A Flash blocker for the browser is
computer name, or to assign a fixed useful for training purposes – the pro- also a good idea for the same reason.
password for remote logins. Also, you gram will use screencasting to moni- Animated ads cause an unnecessarily
can specify which privileges you want tor activities on the remote computer. high level of data – and the bigger the
to grant a remote user accessing a The screencast files you can save only ad space, the worse the problem.
system across the wire. Teamviewer contain the data stream transmitted
will also accept a whitelist or blacklist across the wire by Teamviewer and Beta Blockers
of computers that are allowed or not are thus quite compact. Administra-
allowed to access your computer. tors can create a list of computers The older the distribution I tried, the
for single-click access to remote ma- more difficulties I had in testing. The
File Transfers chines. program performs better and is more
If needed, you can change the trans- stable on more recent versions; for ex-
File transfers occur in a separate win- mission focus. The High speed option ample, virtually no problems occurred
dow (Figure 2) that features a two- reduces the color depth to speed up in connections between Ubuntu 9.04
column view, with your own com- the transmission. If you prefer perfect and 9.10. I definitely advise against
puter on the left and the remote com- image quality at the cost of smoother changing the resolution on remote
puter on the right. To transfer files, operations, you can opt for this in Linux clients, because it typically
just select them and click the button the View menu. The Automatic set- caused Teamviewer to crash on the
above the column. This seems a little ting changes the mode to reflect the client. Also, you shouldn’t change the
convoluted; drag-and-drop, or at connection. Unfortunately, Team- native system settings during remote
least double-clicking, to transfer files viewer changes the standard gray of access, because that can interrupt the
would make more sense. The vendor some desktops (including Gnome on data stream from the remote machine.
aims to change this soon. Also, the Ubuntu) to a horrible pink. Because Another failing: The mouse cursor
Windows-style Wine symlinks and I didn’t notice any speed boosts in doesn’t change its appearance from
drive letters are a little irritating for reduced color mode, you might pre- an arrow to a hand, for example, if
Linux users. fer to keep a high-quality mode, for it moves outside of a window or title
In normal remote support mode, a Linux-to-Linux connections, at least. bar. The vendor has promised to fix
hideable Teamviewer function bar is A couple of options make the remote this before the final release.
displayed on the remote desktop, and support experience smoother. To
you can use it to access a full set of begin with, you will want to disable Conclusions
important remote control functions. desktop effects, such as those pro-
Teamviewer is an easy-to-use and
practical piece of software. Even if
you aren’t an administrator or con-
sole jockey, it gives you a really sim-
ple approach to managing machines
remotely – or for accessing your com-
puter when you’re on the road. Team-
viewer works quite well on Linux,
even though the beta had a few bugs.
The vendor promises to have every-
thing resolved by the time of the final
release. One of Teamviewer’s main
strengths is its cross-platform compat-
ibility between Linux, Windows, Mac
OS X, and even web browsers and the
iPhone. Also, the connection always
works. n

[1] Teamviewer:
Figure 3: View of the remote desktop in Teamviewer.¡ [http://​­www.​­teamviewer.​­com/​­index.​­aspx]

66 Admin 01 w w w. a d m i n - m aga z i n e .co m

NOV. 7–12 24th Large Installation San Jose
2010 System Administration Conference california

Uncovering the Secrets

System Administration

s e 2 0 10 re
San J o Baltimo 9

a n D ie go 200
S Dallas

Program Includes:
Unraveling the Mysteries oF Twitter Infrastructure,
Legal issues in the Cloud, and Huge NFS at Dreamworks



Tony Cass,
in cooperat
ion with LO
* 6-day Virt
Track by in TECHNICAL
John Arrasj
id and Rich structors PROGRAM
ard McDoug including
* Advanced Ti all
me Manageme Invited Tal

* Dovecot an nt: Team Ef

ficiency by ks
d Postfix Tom Limonc Refereed Pa
* 5-day Linu by Patrick elli pers
x Security Be n Koetter an
and Admini d Ralf Hildeb
stration Tr
ack Vendor Exhi
**REGISTER bition
www.usenix 18 AND SAV
.org/lisa1 E**
m A n Ag e m e n t chef

Configuration management with Chef

Chef de Config
© Alistair Cotton,

ever dream of rolling out a complete computer farm with a single mouse over and making a mess on the server
room floor.
click? if you stick to Linux computers and you speak a little Ruby, chef can
go a long way toward making that dream come true. By tim Schürmann Valuable Ingredients
A full-fledged Chef installation com-
Chef is basically a server that stores you deploy a home-grown and home- prises the systems you want to config-
customized configuration guides for tested solution. ure (nodes) and the server that man-
software. Clients connected to the The installation is another obstacle ages and stores the recipes. Chef cli-
server access the recipes and auto- – and a fairly complex one, too, be- ents do all the hard work, picking up
matically configure their systems on cause the Chef server depends on sev- the recipes from the server via a REST
the basis of the rulesets the recipes eral other components, each of which interface and running the scripts.
contain. in turn requires even more software Each client runs on one node but can
To do so, the clients not only modify packages. The Chef server itself is apply recipes to multiple nodes. [Fig-
their configuration files, but – if written in Ruby but relies on the Rab- ure 1] shows you how this works.
needed – launch their package man- bitMQ server and on a Java-based For simplicity’s sake, the following
agers. If the recipes change, or new full-text search engine, at the same examples just use the Chef server and
ones are added at a later date, the time storing its data in a CouchDB a single client. The latter only config-
clients affected automatically update database. ures the computer on which it is run-
to reflect the changes. In an ideal en- Finally, your choice of operating sys- ning. The first thing you need to have
vironment, this just leaves it up to the tem is also important. Chef prefers in place is Ruby version 1.8.5 through
administrator to manage the recipes Linux underpinnings, but it will also 1.9.2 (with SSL bindings). Add to
on the server. run on other Unix-flavored operat- this, RubyGems, which will want to
ing systems such as Mac OS X, Open build various extensions and libraries
Bulk Shopping Solaris, FreeBSD, and OpenBSD, ac- later on, thus necessitating the exis-
cording to the wiki [1]. The fastest tence of make, gcc, g++, and the Ruby
Before you can enjoy the benefits, approach today is offered by Debian developer packages. Additionally, you
the developers behind Chef expect 5, Ubuntu 8.10 or later, or CentOS need the wget tool for various down-
you to put in a modicum of work. For 5.x. Setting up the server on any loads. The following command in-
example, recipes are made up of one other system can be an adventure. stalls the whole enchilada on Debian
or multiple standard Ruby scripts. If This article mainly relates to Debian and Ubuntu Linux:
you need anything beyond the fairly and Ubuntu for this reason. If this is
generic recipes available on the web, the first time you have ever cooked sudo apt‑get install ruby U

you need to have a good command of one of Chef’s recipes, it is also a good ruby1.8‑dev libopenssl‑ruby1.8 U

the Ruby scripting language. In other idea to run your kitchen on a virtual rdoc ri irb build‑essential U

words, your mileage will vary before machine. This prevents things boiling wget ssl‑cert

68 Admin 01 w w w. A d m i n - m AgA z i n e .co m

Chef Ma n ag e m e n t

you need to concentrate on the instal-

lation, particularly server-side.
Collects recipes
Server Client Nodes
Provides recipes and executes scripts
Who’s the Chef?
Figure 1: Overview of the Chef landscape with the server, clients, and nodes. Chef can automate the process of in-
stalling and configuring software, so
The packages for openSUSE are called the case with Debian and Ubuntu), a it only seems logical to let Chef install
ruby, ruby-devel, wget, openssl- symbolic link will improve things: itself. The developers refer to this
certs, make, gcc, and g++. The certifi- process as bootstrapping. Having said
sudo ln ‑sfv /usr/bin/gem1.8 /usr/bin/gem
cates from ssl-cert will be required this, recipes that install the server
later. Now you can issue the following in this way only exist for Debian 5,
According to the how-to [1], Chef Gems command to install the Chef Ubuntu 8.10 or later, and CentOS 5.x.
prefers RubyGems version 1.3.6 or package: On any other distribution, you need
newer, but not 1.3.7. This version to perform all of the steps manually
sudo gem install chef
contains a bug that kills the following as described in the [Manual Server
installation mid-way. Because most When you run a Gem update, keep Installation] boxout.
distributions have an older version of an eye on the JSON Gem. The ver- Life is a little easier with one of the
RubyGems, your best bet is to head sion that now comes with RubyGems, operating systems officially supported
for the source code archive: 1.4.3, causes an error in Chef. If gem by Chef. To begin, make sure the
update installs the offending JSON computers involved have Fully Quali-
cd /tmp
package on your disk, these com- fied Domain Names (FQDNs), such
download.php/69365/rubygems‑1.3.6.tgz mands revert to the original version: as If you
tar zxf rubygems‑1.3.6.tgz don’t, you will be bombarded with
sudo gem uninstall ‑aIx json
cd rubygems‑1.3.6
sudo gem install ‑v1.4.2 json
error messages like Attribute domain
sudo ruby setup.rb
is not defined! (ArgumentError)
If the last command installs the Gems The steps thus far provide the under- later on. Additionally, the repositories
executable as /​usr/​bin/​gem1.8 (as is pinnings for Chef operations. Now, need to provide the runit program in

Manual Server Installation

If you need to set up the Chef server manually, start by installing the openSUSE goes for zlib-devel and libxml-devel. Now, finally, you
RabbitMQ messaging server [2]. openSUSE users should use the open- can install the Chef server
SUSE Build Service to install rabbit-mq [3]. Doing so means that YaST
sudo gem install chef‑server chef‑server‑api chef‑server chef‑solr
automatically adds repositories that you need later on.
Once you have RabbitMQ in place, it’s time to start the Chef configura- and add the really practical web front end:
sudo gem install chef‑server‑webui
sudo rabbitmqctl add_vhost /chef
After completing this work, create the /​etc/​chef/​server.rb con-
sudo rabbitmqctl add_user chef testing
figuration file. [Listing 1] gives you a template. As a minimum, you need
sudo rabbitmqctl set_permissions ‑p /chef chef ".*" ".*" ".*"
to replace the domain name that follows chef_server_url with the
The next task on the list concerns the CouchDB database from the output from hostname ‑f and add a password of your choice after
CouchDB package. If needed, you can start the service manually on web_ui_admin_default_password. All the other defaults you can
openSUSE by typing rccouchdb start. The Chef server also requires keep, particularly the paths, which the server automatically creates
Sun Java SDK version 1.6.0. Some distributions keep this package in an later, should the need arise.
external or special repository. On Debian, you need to enable the non- In the next step, the script shown in [Listing 2] creates a pair of SSL
free package source; on Ubuntu 10.04, you can add the partner reposi- certificates, which you will need. The following command line creates
tory like this: the search index:
sudo add‑apt‑repository "deb U sudo chef‑solr‑indexer
lucid partner"
Another command launches the Chef SOLR Server
sudo apt‑get update
sudo chef‑solr
Now install the JDK. On Debian and Ubuntu, the JDK is hidden away
in the sun-java6-jdk package, whereas openSUSE calls it java- and the Chef server itself,
1_6_0-sun-devel. Users on openSUSE will probably want to delete the
sudo chef‑server ‑N ‑e production
OpenJDK packages java-1_6_0-openjdk and java-1_6_0-openjdk-
devel to be on the safe side.
including the graphical web interface:
Then, you just need to install the developer packages for zlib and
libxml. Debian and Ubuntu call them zlib1g-dev and libxml2-dev; sudo chef‑server‑webui ‑p 4040 ‑e production

w w w. a d m i n - m aga z i n e .co m Admin 01 69

Ma n ag e m e n t Chef

Table 1: Directories in a Repository If the command terminates with a

cryptic error message, try running it
Directory Content
again. During testing, the installation
certificates/​ SSL certificates (typically created by rake ssl_cert) ran without any errors. First, the com-
config/​ General configuration files for the repository mand installs a Chef client, then Rab-
cookbooks/​ Complete cookbooks bitMQ, CouchDB, the developer pack-
roles/​ Role definitions ages for zlib and xml, and the Chef
site-cookbooks/​ Modified cookbooks; any cookbooks stored here will overwrite or server, including the indexer and a
modify the ones stored in cookbooks web GUI you can use later to manage
the Chef server (WebUI). It then goes
a package named runit (don’t install aid to creating simple scripts – for on to create matching configuration
this yourself!). installing the full-fledged server and files and the required directories and
The Chef server also requires Sun clients. To do this, create a ~/​solo.rb adds init script entries for the server
Java SDK version 1.6.0, which the file with the following three lines on to round off the process.
distributions love to hide in a special each of the systems involved: At the end of this procedure, the Chef
repository. Debian users need to en- server should be listening on port
file_cache_path "/tmp/chef‑solo"
able the non-free package source for cookbook_path U
4000; the web GUI is accessible on
this, whereas Ubuntu users can add "/tmp/chef‑solo/cookbooks" port 4040. Java and RabbitMQ use the
the partner repository with the fol- recipe_url U Apache SOLR-based full-text search
lowing two lines: " engine built into the Chef server.
Among other things, it provides infor-
sudo add‑apt‑repository U
"deb U
This tells Chef Solo where the instal- mation about the existing infrastruc-
lucid partner" lation recipes are located. ture, which in turn can be referenced
sudo apt‑get update for recipes. For details of the search
Chef on Call function, see the wiki page [4].
Theoretically, the Chef server will
run with the OpenJDK, although the Now it’s time to move on to the Workers
developers do not give you any guar- server candidate. On this machine,
antees. create a JSON configuration file Once you have the server up and run-
named ~/​chef.json to provide infor- ning, it’s time to turn to the client.
Lonely Kitchen Helper mation about the node. See [Listing Start by creating a ~/​chef.json JSON
3] for the file content. configuration file. [Listing 4] gives
After fulfilling all the requirements, To match your local environment, you you the content. The server_fqdn
you can create configuration files need to modify the server name for entry here must contain the server
for Chef Solo on the server and the server_fqdn. To set up the full-fledged name, not the client’s.
client. This Chef variant runs the Chef server, give the command: Now you can launch Chef Solo:
recipes directly on the client without
involving the server. Without the sudo chef‑solo ‑c ~/solo.rb ‑j U sudo chef‑solo ‑c ~/solo.rb U

server, Chef Solo is only useful as an ~/chef.json ‑j ~/chef.json U

Listing 1: Template for server.rb

log_level :info 19 v
alidation_key "/etc/chef/validation.pem"
log_location STDOUT 20 c
lient_key "/etc/chef/client.pem"
ssl_verify_mode :verify_none 21 w
eb_ui_client_name "chef‑webui"
chef_server_url "" 22 w
eb_ui_key "/etc/chef/webui.pem"
05  23 
signing_ca_path "/var/chef/ca"
24 w
eb_ui_admin_user_name "admin"
couchdb_database 'chef'
25 w
eb_ui_admin_default_password "somerandompasswordhere"
cookbook_path [ "/var/chef/cookbooks", "/var/chef/site‑cookbooks" ]
27 s
upportdir = "/srv/chef/support"
28 s
olr_jetty_path File.join(supportdir, "solr", "jetty")
file_cache_path "/var/chef/cache"
29 s
olr_data_path File.join(supportdir, "solr", "data")
node_path "/var/chef/nodes"
30 s
olr_home_path File.join(supportdir, "solr", "home")
openid_store_path "/var/chef/openid/store"
openid_cstore_path "/var/chef/openid/cstore" 31 s
olr_heap_size "256M"

search_index_path "/var/chef/search_index" 32 

role_path "/var/chef/roles" 33 u
mask 0022

17  34 
validation_client_name "validator" 35 M
ixlib::Log::Formatter.show_time = false

70 Admin 01 w w w. a d m i n - m aga z i n e .co m

Chef Ma n ag e m e n t

Listing 2: SSL Certificates for the Chef Server
The tool creates a couple of directo- 02 
openssl genrsa 2048 > /etc/chef/validation.key
ries, corrects the configuration files, 03 
openssl req ‑subj "${server_ssl_req}" ‑new ‑x509 ‑nodes ‑sha1 ‑days 3650 ‑key /etc/chef/validation.key
and adds chef-client to the init > /etc/chef/validation.crt

scripts. The latter ensures that the cli- 04 

cat /etc/chef/validation.key /etc/chef/validation.crt > /etc/chef/validation.pem

ent will talk to the server on booting 05 

openssl genrsa 2048 > /etc/chef/webui.key

and execute any recipe changes that 06 

openssl req ‑subj "${server_ssl_req}" ‑new ‑x509 ‑nodes ‑sha1 ‑days 3650 ‑key /etc/chef/webui.key > /
have occurred in the meantime.
cat /etc/chef/webui.key /etc/chef/webui.crt > /etc/chef/webui.pem
After this, the client has to register
with the server. To allow this to hap-
pen, copy the /​etc/​chef/​validation. To prepare a recipe for action, run the The target, upload_cookbook, is de-
pem file from the server to the /​etc/​ command fined in the Rakefile provided by the
chef/​directory client-side and then repository.
knife configure ‑i
restart the client manually:

sudo chef‑client
and confirm the default responses by GUI Management
pressing Enter – except, enter your
The client automatically creates a key, own username when asked Your cli- The server now knows the emacs
which you need to add to the /​etc/​ ent user name?, and type . (dot) in cookbook, but the clients don’t. To
chef/​client.pem file and which will response to the Path to a chef reposi- change this, launch a browser and ac-
sign every transaction with the server tory (or leave blank)? query. Knife cess the web front end with http://​
from this point on. Then you want then registers a new client on the Chef
to delete the validation.pem file for Chef server, creates the above-men- does not offer SSL encryption here. If
security reasons. tioned certificate in /​
.chef/​ you prefer a more secure approach,
my-knife.pem, and finally creates the you could use Apache as a proxy.
Librarian /​.chef/​
knife.rb configuration file. In the form that then appears, log in
by typing the admin username [Figure
Now that you have the server and the Convenience Food 2]. The matching password is stored
client running, the next step is to cre- in the web_ui_admin_default_password
ate a repository server-side for your Multiple recipes with the same objec- line of the /​etc/​chef/​server.rb file.
recipes: This is simply a hierarchy of tive can be grouped in a cookbook.
multiple, standardized (sub-)direc- For example, the mysql cookbook con- Listing 3: ~/​c hef.json for the Server
tories. Of course, you could create tains all the recipes required to install 01 

them all manually, but the template and set up the free database. For an 02  "bootstrap": {
03  "chef": {
provided by Opscode does a quicker initial test, it is a good idea to look for
04  "url_type": "http",
job; you just need to download and a simple cookbook [5].
05  "init_style": "runit",
unpack: In the section that follows, I will use
06  "path": "/srv/chef",
the cookbook for emacs from the ap- 07  "serve_path": "/srv/chef",
plications group as an example. In 08  "server_fqdn": "",

tar ‑zxf opscode‑chef‑repo‑U this example, I’ll use the package 09  "webui_enabled": true
123454567878.tar.gz manager to install the popular Emacs 10  }

text editor. 11  },
12  "run_list": [ "recipe[bootstrap::server]" ]
Because this cryptic number is dif- After downloading the Cookbook ar-
ficult to remember in the daily grind, chive, unpack it in the cookbooks sub-
you might want to rename the direc- directory, then introduce the server to
tory (incidentally, the number comes the new recipes: Listing 4: ~/​c hef.json for the Client
from the versioning system and repre- 01 
rake upload_cookbooks 02  "bootstrap": {
sents the Commit ID):
03  "chef": {
The rake command automatically 04  "url_type": "http",
mv opscode‑chef‑repo‑123454567878 U
calls knife with the correct param- 05  "init_style": "runit",

cd chef‑repo eters, and knife then uploads all the 06  "path": "/srv/chef",
07  "serve_path": "/srv/chef",
cookbooks from the corresponding di-
08  "server_fqdn": ""
[Table 1] explains the directory hierar- rectory. To upload a single cookbook 09  }
chy in chef-repo. to the server, do this: 10  },
The recipes stored here are injected 11  "run_list": [ "recipe[bootstrap::client]" ]
into the server by a tool named knife. rake upload_cookbook[emacs]

w w w. a d m i n - m aga z i n e .co m Admin 01 71

Ma n ag e m e n t Chef

at the same time and get in each

other’s way.

To group multiple cookbooks in a
role, create a new file below Roles in
the repository, say, beispiel.rb, with
the following content:

name "beispiel"
description "Example of a role"
run_list("recipe[emacs]", U
"recipe[zsh]", "recipe[git]")

This groups the emacs, zsh and git

recipes under the beispiel role name.
Then send the role to the server like

rake roles
Figure 2: The web front end login page: the default password specified on the right is incorrect.
In the web front end, you can assign
Changing the slightly cryptic default sudo chef‑client roles to a node just like cookbooks
after logging in the first time is a good This command line immediately using drag and drop.
idea. opens a server connection, picks
Now go to the Nodes menu. When up the recipes assigned to the client Freshly Stirred
you get there, click the client name, (only emacs for the time being) and
change to the Edit tab, and finally executes the recipes [Figure 5]. To al- Ready-made recipes and cookbooks
drag the recipe you want to use from low this to happen on a regular basis, off the Internet will only cover stan-
Available Recipes and drop it into you should run the client at regular dard application cases. For special
the Run List (the recipe will slot into intervals as a daemon: cases, or individual configurations,
the top position in the list). In the you will typically need to create your
chef‑client ‑i 3600 ‑s 600
example, you would now see emacs own cookbook.
at the top [Figure 3]. To store this as- In this example, the client contacts The following, extremely simple
signment, press the Save Node button the server every 3,600 seconds. The example, creates a text file on the cli-
bottom left on the page. -s parameter lets you vary the period ent called /​tmp/​thoughts.txt that is
Client-side now, manually launch the slightly. If you don’t set this, all of based on the quick_start cookbook
chef-client tool: your clients might query the server [6], and it adds a sentence that is

G Figure 4: The Status tab lists all the nodes with their last contact attempts
and recipe assignments (following Run List).

F Figure 3: Using drag and drop to assign recipes to a node. In this example, the
client runs the beispiel recipe first, followed by emacs.

72 Admin 01 w w w. a d m i n - m aga z i n e .co m

Chef Ma n ag e m e n t

draw from the full scope of the lan-

guage and on RubyGems. In the case
of the latter, the recipe should first
check to see whether the Gem exists
on the client and, if not, install it.
The beispiel/​metadata.json file
stores metadata on the new cook-
book. Before you roll out the cook-
Figure 5: The client has picked up its assigned recipe from the server and executed it. This puts a pre- book in a production environment,
configured version of Emacs on its disk. you might want to add some details.
As the file extension suggests, the file
generated dynamically in part. Start This should be fairly self-explanatory uses the JSON format.
by creating a new cookbook called for Ruby aficionados: It creates the
beispiel in chef-repo: /​
tmp/​thoughts.txt file from the Conclusions
thoughts.txt.erb template and then
rake new_cookbook COOKBOOK=beispiel
replaces the wildcard with the con- Chef is a complex piece of software,
The command creates a beispiel tent of the thought variable. Now and once you have it running and
folder below cookbooks, populates it you just need to think about what have finished modifying or creating
with the required subdirectories, then thoughts to use here. your recipes, it does make the admin-
creates an empty recipe named de- istrator’s life much easier – at least
fault.rb. Spice on Linux systems. Unfortunately, the
Before you start filling this with con- learning curve is very hard going for
tent, first create a template for the file In this example, thought will be an at- newcomers. The online documenta-
you want to create, /​tmp/​thoughts. tribute. Attributes store node-specific tion for Chef is fairly chaotic and
txt. This will later contain the sen- settings in a cookbook for recipes to incomplete [7]. If you need to know
tence evaluate and use. A typical attribute more about writing cookbooks, it
would be, say, a command-line pa- is a good idea to download prebuilt
Thought for the day:
rameter for a program launched auto- examples and investigate them. The
and the recipe will append an inge- matically by a recipe. The attributes cookbook for emacs shows you how
nious thought on a daily basis. The are identical for each call to the recipe to use action :upgrade to install a
complete template is thus: and, thus, no more than constants package for example.
provided by the recipe author. Additionally, it is hard to find help or
Thought for the day: <%= @thought %>
In contrast to genuine Ruby con- how-tos, even on the web, if you have
The recipe will replace the wildcard stants, attributes can be modified via a problem. Your best option here is
with text later on. The new template the web interface (in the window to post your questions on the mailing
needs to be in templates/​default/​; used to assign cookbooks to nodes). list [8]. n
you can save it as thoughts.txt.erb. A cookbook groups all of the attri-
Most recipes use templates like this, butes in its attributes subdirectory. Info
or, to quote the developers: “We love For the example here, you need to [1] Installation guide: [http://​­wiki.​­opscode.​
templates.” create a beispiel.rb file with the fol- ­com/​­display/​­chef/​­Installation]
lowing content: [2] RabbitMQ:
Hand Mixer thought "Silence is golden ..."
[3] openSUSE build service for Rab-
Now, compose a matching recipe that Now you just need to register the new bitMQ: [http://​­software.​­opensuse.​
picks up the template and uses it to cookbook with the server ­org/​­search?​­q=rabbitmq‑server&​
generate the /​tmp/​thoughts.txt file. ­baseproject=openSUSE%3A11.​­2]
rake upload_cookbooks
To save work here, you can extend [4] Information on full-text searches in Chef:
the existing, but empty, default.rb and assign it to one or multiple nodes [http://​­wiki.​­opscode.​­com/​­display/​­chef/​
recipe in the recipes subdirectory. in the web front end. After running ­Search]
The recipe for this example looks like: chef-client, the /​tmp/​thoughts.txt [5] Repository with prebuilt cookbooks:
file should appear. [http://​­cookbooks.​­opscode.​­com/]
template "/tmp/thoughts.txt" do This recipe leaves much scope for [6] Cookbook quick start: [http://​­wiki.​­opscode.​
source "thoughts.txt.erb" improvement. For example, you could ­com/​­display/​­chef/​­Cookbook+Quick+Start]
variables :thought => U randomly choose the thought of the [7] Chef wiki:
node[:thought] day, which Ruby programmers should [http://​­wiki.​­opscode.​­com/​­display/​­chef/]
action :create handle easily. Because recipes are [8] Chef mailing list: [http://​­lists.​­opscode.​
end full-fledged Ruby scripts, you can ­com/​­sympa/​­lists/​­opensource/​­chef]

w w w. a d m i n - m aga z i n e .co m Admin 01 73

Ma n ag e m e n t Sysinternals

System monitoring with Sysinternals

Health Check
© Denis Tevekov,

Administrators don’t need a massive arsenal of tools just to monitor a couple of

systems. With Microsoft’s free Sysinternals suite, admins can handle all sorts of
tasks. By Thomas Joos

The Sysinternal tools are free tools domain because the new operating AdInsight lists all requests including
from Microsoft that can help Win- systems block access to administra- those that are blocked. This gives
dows administrators handle many tive shares by authentication of local administrators an easy option for ana-
tasks. This article introduces the user accounts. lyzing authentication problems with
Sysinternal tools that are useful for Some Sysinternal tools, such as Active Directory-aware programs and
system monitoring. All of the tools PSInfo.exe, require access to the identifying clients and servers that set
described here can be downloaded admin share and thus will not work up a connection to the domain con-
free of charge from the Microsoft site at first. To allow access, you must troller. AdInsight logs all requests is-
[1], either as individual downloads or enable local logins to administrative sued to domain controllers and stores
as part of the Sysinternals suite. shares in the Registry of standalone them as an HTML report or text file
One advantage of the Sysinternals computers. To do so, launch the Reg- for troubleshooting purposes. The
utilities is that you don’t need to in- istry Editor by typing regedit, then logfile contains the client request and
stall them, so they can be launched navigate to HKEY_LOCAL_MACHINE\SOFT‑ responses that the client received via
conveniently from a USB stick. When WARE\Microsoft\Windows\CurrentVer‑ LDAP.
launched for the first time, the pro- sion\Policies\System. Create a new AdInsight also logs access to system
grams display a license dialog; you Dword entry with the label LocalAc‑ services (Figure 1). When a program
can suppress this dialog with the countTokenFilterPolicy, set the value such as Exchange accesses the do-
/accepteula option, which can be to 1, then restart the computer. main controller, the window fills with
useful in scripting. Unfortunately, this information; then, you can right-click
option does not work for all of the LDAP Microscope to display details of the individual en-
Sysinternal tools. tries, as well as filter the display via
The programs only run on a Windows Insight for Active Directory, also the menu.
system as of Windows 2000 Server. known as AdInsight, lets you monitor The display also includes the name
For this article, I used Windows the LDAP connections on a domain of the accessing user. Unfortunately,
Server 2008 R2 and Windows 7. controller in real time with a GUI. AdInsight only lets you monitor lo-
Windows Server 2008 R2, Windows The user interface is similar to the cal access; over-the-wire diagnostics
Server 2008, Windows Vista, and Sysinternal tools Regmon and File- via remote access are not supported.
Windows 7 do not support access to mon. The tool investigates calls to However, AdInsight’s search function
the hidden System $ shares such as the wldap32.dll file, which most pro- does let you filter by process, error, or
C$, or admin$ as easily as Windows grams, including Exchange, use for request response. The tool selects the
XP or Windows Server 2003; the com- LDAP-based access to Active Direc- response to let you perform specific
puters do not belong to a Windows tory per LDAP. monitoring.

74 Admin 01 w w w. a d m i n - m aga z i n e .co m

Sysinternals Ma n ag e m e n t

AdInsight also supports automated

deployment and offers a variety of
options for this. One useful automa-
tion feature is the ability to write the
log to a file without displaying the
events in the GUI. AdInsight runs on
Windows 2000 Server or newer and
includes a help file, which can be a
useful aid for tasks that require more
thorough analysis.

Filesystem, Registry,
The Process Monitor provides a
graphical user interface for monitor-
ing and color-tagging the filesystem, Figure 1: AdInsight helps you investigate LDAP access to domain controllers.
registry, and process/​thread activity
in real time. The tool combines two entries that contain the same string. to use and does not have an extended
programs standard to Sysinternals: Additionally, you can enable multiple learning curve.
Filemon and Regmon. With a click filters at the same time and save the Process Explorer takes things a step
of the button, you can enable and configuration. further than Process Monitor, in that
disable the individual monitoring For more efficient diagnostics, you Process Explorer lists all the processes
options and restrict your monitoring might want to save the logfile and in a window and includes more
of registry and filesystem access and then load it for analysis, applying ad- detailed information on the current
process calls to an area in which you ditional filters as needed. The Tools process, such as access to directories
are interested. menu gives you a selection of precon- (Figure 3).
Process Monitor is a valuable aid for figured views. In DLL View mode, the tool shows
monitoring stability and identifying Process Monitor can also monitor the which libraries are used and where
bottlenecks; it is capable of logging boot process on a server because it they originated. The process menu
all read and write access to the sys- launches at a very early stage. You contains a Restart entry that first kills
tem and its media. Additionally, it can redirect all the output to a file. a process and then restarts it. Also,
displays comprehensive data on any If Windows fails to boot, analysis of you can temporarily stop individual
process or thread that is launched the output file gives you a fast way to threads and highlight processes in dif-
or terminated. Also, you can moni- identify the issue. Just like all Sysin- ferent colors. Process Explorer nests
tor any TCP/​IP connections that are ternal tools, Process Monitor is easy in the taskbar, which provides an at-
opened, as well as UDP traffic. Note
that Process Monitor does not save
the content of the TCP packets or
payload data; it is not specifically de-
signed for network monitoring. If that
is what you need, you might prefer a
tool such as Wireshark [2].
The Process Monitor tool is also ex-
tremely useful for troubleshooting lo-
cal connection and privilege problems
(Figure 2). If needed, it can display
additional information on active
processes (e.g., their DLL files or the
parameters set when the process was
The filter function allows you to
reduce the volume of data output
generated by Process Monitor. For
example, it lets you hide any pro-
cesses with a specific string in their
names, without filtering out registry Figure 2: Monitoring processes, the registry, and filesystem access with Process Monitor.

w w w. a d m i n - m aga z i n e .co m Admin 01 75

Ma n ag e m e n t Sysinternals

a-glance overview of the current CPU

load and disk utilization.

Logging into the Domain

The LogonSessions tool lists all the
active sessions on a computer at the
command line (Figure 4). If you run
this command without setting op-
tions, the prompt buffer size might
not be sufficient to display all of the
information. In this case, you can use
the logonsessions | more command
or modify the command-line prompt
properties to increase the buffer size.
Alternatively, you can redirect the
output to a file by specifying > logon.
txt. The ‑p option tells LogonSes-
sions to display the active processes
in the individual sessions for login Figure 3: Monitoring and controlling processes with Process Explorer.
users, allowing you to monitor who
is logged in to a terminal server and to the local computer, you can’t actu- thorized access to your network and
which applications the users are run- ally find out where else the user is see the computers on which a user is
ning. The tool is also really useful in logged on in the domain. If you use logged in.
Active Directory environments. psloggedon \\computername, you can Psinfo is another tool in the collec-
also display domain users logged in tion that lets you retrieve a variety of
Monitoring Local Logins remotely. information for the local system, such
To access a computer on the network, as the operating system, the CPU, the
Just like LogonSessions, PsLoggedOn the user account with which you build number, and whether the com-
is a tool for monitoring logged in launch PsLoggedOn must have ad- puter is a member server or a domain
users. If you type psloggedon at the ministrative privileges on the remote controller. Calling psinfo /? lists
command-line prompt, the tool will machine. Although you can check more options. For example, psinfo ‑s
display all the logged in users on the who has an active session on a com- lists the software packages installed
local system with their login times puter without add-on tools, with net on the machine. Although Microsoft
(Figure 5). Also, you can see who is session, for example, you can only provides programs such as system‑
accessing a share on the server. If you monitor local logins, not network log- info.exe and msinfo32.exe with
launch the tool with the username as ins. The tool accesses the HKEY_USERS Windows, Psinfo can also retrieve
an argument, it will investigate all the registry key for this check (Windows information from remote computers
computers in the network environ- creates a separate key for each logged across the wire.
ment or domain and show you where in user by default).
the user is logged in. Unfortunately, The ‑l option lists locally logged in Event Lister
this is not an entirely error-free pro- users only; ‑x leaves out the login
cess in Windows DOS. Although you times. The PsLoggedOn tool is useful The PsLogList utility is a command-
can identify domain users logged in when you need to investigate unau- line tool that retrieves the event

G Figure 5: Listing logged in users with PsLoggedOn.

F Figure 4: Logged in users and services.

76 Admin 01 w w w. a d m i n - m aga z i n e .co m

Sysinternals Ma n ag e m e n t

Table 1: Selected PsLogList Options each file, you are told whether the
user has read (R), write (W), or both
Name Function
(RW) types of access. If you use |
File Runs the command on all the computers listed in the file. Each com-
more to redirect the output from this
puter needs a separate column in the text file.
command, the display will pause on
‑a dd/mm/yy Lists the entries after the specified date.
each page, and you can continue by
‑b dd/mm/yy Lists entries before the specified date.
pressing any key. In a similar fashion,
‑c Deletes the event logs after displaying them in PsLogList, which is
>file.txt redirects the output to a
useful in the case of batch-controlled retrieval.
file. With this approach, you do not
‑d n Displays the entries for the past n days.
see any command-line output.
‑e id1, id3, ... Filters entries with defined IDs.
The accesschk user ‑cw * command
‑f Filters entries with specific types (e.g., ‑f w filters warnings). You
can use arbitrary strings here.
shows you the Windows services to
which a group or user has write ac-
‑h n Lists entries for the past n hours.
cess. If you want to see a user’s ac-
‑i id1, id3, ... Shows entries with IDs defined in a comma-separated list of IDs.
cess privileges for a specific registry
‑l event_log_file Stores entries for the defined event log.
key, the accesschk ‑kns contoso\tami
‑m n Lists entries for the past n minutes.
hklm\software command is your best
‑n n Only shows the n latest defined entries. bet. The AccessChk tool is excellent
for checking computers for vulnera-
display from various computers and is the only account with privileges on bilities, and it also supports scripting.
then displays and compares events. If all the PCs and servers in the domain. AccessEnum gives you a GUI that
you run the tool without any options, In networks without a domain, you presents a full directory tree of user
it will output the entries in the local can use the administrator account for privileges. In other words, AccessE-
system event log. The program has the login; however, you need to set num is the graphical front end for Ac-
numerous options that give you vari- the password to be the same on all cessChk. The download contains both
ous ways of comparing the event logs the computers you want to monitor. files because AccessEnum relies on
that you retrieve. ShareEnum shows you not only the the AccessChk program for perform-
Table 1 lists some of PsLogList’s com- shares but also the local paths for the ing scans. In the GUI, you can select
mand-line options. By default, the shares on the computer. a directory and scan it for privilege
tool uses the system event log; you The Refresh button tells ShareEnum assignments. The tools also show de-
can select the event log by entering to launch a scan. If you want to scan nied privileges.
the first letter or by entering an ab- an individual computer, enter the
breviation, such as sec for security. same IP address as the start and end Conclusions
address of the IP range. The tool will
Monitoring Shares show you all of the shares on the net- The free command-line and GUI tools
work in a single window and list the in the Sysinternals suite are a feature-
The ShareEnum tool lets you monitor access privileges to boot. If you only rich addition to any Windows ad-
shares and their security settings by want to see local privileges, Sysinter- ministrator’s toolbox. The individual
scanning either an IP range or all the nals gives you a choice of two tools: programs are useful for monitoring
PCs and servers in a domain (or all AccessChk and AccessEnum. processes, users, and network con-
the domains in a network) for shares AccessChk outputs an exhaustive list nections, and tools like AdInsight
(Figure 6). To display all of this in- of a user’s rights at file, service, or are indispensable aids if you need to
formation reliably, you need to log in registry level at the command line, troubleshoot Active Directory logins.n
as the domain administrator, which giving you a quick overview of how
access privileges
are defined for a Info
specific user. [1] Sysinternals tools:
To see the privi- [http://​­www.​­sysinternals.​­com]
leges for the ad- [2] Wireshark: [http://​­www.​­wireshark.​­org]
ministrator in
the C:\Windows\ The Author
System32 direc- Thomas Joos is a freelance IT consultant who
tory, you would has worked in the IT industry for more than 20
type accesschk years. Among his many projects, Joos writes
administra‑ practical guides and articles on Windows and
Figure 6: ShareEnum gives you a neat list of all the shares and assigned tor c:\windows\ other Microsoft topics. You can meet him online
privileges on the network. system32. For at [http://​­thomasjoos.​­spaces.​­live.​­com].

w w w. a d m i n - m aga z i n e .co m Admin 01 77

N u ts a n d B o lts PAM and Hardware

© An
a Vas
, Foto

Flexible user authentication with PAM

Turnkey Solution
PAM is a very powerful framework for handling software- and hardware-based user authentication, giving ad-
ministrators a choice of implementation methods. By Thorsten Scherf

Hardware innovations are daily busi- passwd and /etc/shadow files. When the /etc/shadow file but from a direc-
ness in user account authentication. a user runs the login command to tory service. This task can be simpli-
Pluggable Authentication Modules log in to the system with a name and fied by deploying PAM [1].
(PAM) help transparently integrate password, the program creates a cryp-
these new devices into a system. tographic checksum of the password Modular Authentication
This gives experienced administra- and compares the results with the
tors the option of offering a variety checksum stored for this user in the Originally developed in the mid-
of different authentication methods /etc/shadow file. If the checksums 1990s by Sun Microsystems, PAM is
to their users while providing scope match, the user is authenticated; if available on most Unix-style systems
for controlling the total user session not, the login will fail. today. PAM offloads the whole au-
workflow. This approach doesn’t scale well. In thentication process from the appli-
larger environments, user credentials cation itself to a central framework
Old School are typically stored centrally on an comprising an extensive collection
LDAP server, for example. In this of modules (Figure 1). Each of these
User logins on Linux systems are case, the login program doesn’t re- modules handles a specific task;
traditionally handled by the /etc/ trieve the password checksum from however, the application only gets to

78 Admin 01 w w w. a d m i n - m aga z i n e .co m

PAM and Hardware N u ts a n d B o lts

Figure 1: PAM provides a centralized user management framework for the Figure 2: A classic PAM configuration file contains modules and libraries that the
application. administrator can use to customize PAM.

know whether or not the user logged notebooks often include a fingerprint Before you modify the existing PAM
in successfully. In other words, it is reader that allows the owner to use a configuration, you might want to test
PAM’s job to find a suitable method digital fingerprint when logging into the device itself. To do so, scan a fin-
for authenticating the user. The PAM the system. The PAM ThinkFinger li- gerprint by giving the
framework defines what this method brary [2] provides the necessary sup-
tf‑tool ‑‑acquire
looks like, and the application re- port. According to the documentation,
mains blissfully unaware of it. the module will support the UPEK/​ command (Figure 3). Then you can
PAM can use various authentication SGS Thomson Microelectronics fin- use
methods. Besides popular network- gerprint reader used by most recent
tf‑tool ‑‑verify
based methods like LDAP, NIS, or Lenovo notebooks and many external
Winbind, PAM can use more recent devices. to verify the results. You might see
libraries to access a variety of hard- Most major Linux distributions offer a Fingerprint does *not* match mes-
ware devices, thus supporting logins prebuilt packages for the PAM librar- sage at this point; initial attempts can
based on smartcards or the user’s ies. You can use your distribution’s be fairly inaccurate because you will
digital fingerprint. One-time password package manager to install the soft- need to familiarize yourself with the
systems, such as S/​Key or SecurID, ware from the repositories. To install device.
are also supported by PAM, and some the required packages on your hard If you drag your finger too quickly
methods even require a specific Blue- disk, you would give the or too slowly across the scanner, the
tooth device to log in the user. device could fail to identify the fin-
yum install thinkfinger
The way PAM works is fairly simple. gerprint correctly. In this case, it will
Each PAM-aware application (the ap- command on a Fedora system and output an error message and quit.
plication must be linked against the When you achieve reliable results
apt‑get install thinkfinger‑tools U
libpam library) has a separate config- libpam‑thinkfinger
from fingerprint scans, you can delete
uration file in the /etc/pam.d/ folder. the temporary file with the test scan
The file will typically be named after on Ubuntu Hardy. Gentoo admins can in /tmp and create an individual file
the application itself – login, for issue a compact command: for each user on the system that will
example. Within the file, modules dis- contain the user’s fingerprint. The
emerge sys‑auth/thinkfinger
tribute PAM tasks among themselves. command is
Numerous libraries are available in If you’re using openSUSE, you’ll need
tf‑tool ‑‑add‑user username
each group, and they handle a variety the libthinkfinger and pam_thinkfin-
of tasks within the group (Figure 2). ger packages, the repository versions (Figure 4). Users must scan their fin-
Control flags let you manage PAM’s of which are not up to date. gerprints three times for this to work.
behavior in case of error – for ex- You might prefer a manual install If the fingerprint is identified correctly
ample, if a user fails to provide the with the typical ./configure, make, each time, the tool will store it in a
correct password or if the system is make install steps and files from the separate file below /etc/pam_think‑
unable to verify a fingerprint. current source code archive. Debian finger/.
users on Lenny will need to access Once everything is working, you can
Fingerprints the Experimental repository and then begin the PAM configuration. Figure
type 2 shows a PAM configuration for
More recent PAM libraries allow ad- the login program that lists just one
aptitude install libthinkfinger0 U
ministrators to authenticate users by libpam‑thinkfinger thinkfinger‑tools
authentication module: pam_unix. If
means of smartcards, USB tokens, or you want to authenticate against the
biometric features. State-of-the-art for the install. fingerprint scanner first, you need to

w w w. a d m i n - m aga z i n e .co m Admin 01 79

N u ts a n d B o lts PAM and Hardware

Figure 3: tf-tool gives you an option for testing your fingerprint scanner … Figure 4: … and then creating a fingerprint for each user.

call the pam_thinkfinger PAM module pam.d/system‑auth, although other the machine. If so, the user is logged
before pam_unix. Linux distributions call it /etc/pam/ in; if not, access to the system is de-
To prevent PAM from prompting us- common‑auth. You can enter all the nied. The plugged in device is identi-
ers to enter their password despite libraries against which you want to fied by its unique serial number and
passing the fingerprint test, you need authenticate your users in the file model and vendor names. Addition-
add a sufficient control flag. This (Figure 5). ally, a random number is stored on
tells PAM not to call any more librar- The include control flag then includes the USB device and on the computer;
ies once an authentication test has the file in all your other PAM configu- the number changes after each suc-
succeeded and to return PAM_SUCESS ration files. From now on, this makes cessful login attempt.
to the calling program – login in this all the programs in the PAM libraries When a user logs in, PAM checks
example. If the fingerprint-based login listed by the centralized configuration both the specified USB device proper-
fails, pam_unix is called as a last file available in the individual PAM ties and the random number. If the
resort and will prompt for the user’s files, including the pam_thinkfinger number stored on the USB does not
regular password. module. match the number on the disk, the
Manually entering the PAM libraries login fails. This prevents attackers
for all of your PAM-aware applica- USB Tokens from stealing the random number,
tions in every single PAM configura- placing it on their own USB device,
tion file would be fairly tedious. A The pam_usb library supports an- and then modifying the properties of
centralized PAM configuration file other hardware-based approach, in their own device to access the sys-
gives you an alternative. On Fedora which PAM checks to see whether a tem. Because the random number on
or Red Hat, this file is named /etc/ specific USB device is plugged into the system changes after each login,

Figure 5: On Fedora, system‑config‑authentication provides a ba‑ Figure 6: The USB device is identified by its properties. If the user tries to log in without the
sic PAM configuration tool. device, it will not work.

80 Admin 01 w w w. a d m i n - m aga z i n e .co m

November 13-19, 2010 • Ernest N. Morial Convention Center • New Orleans, Louisiana
Conference Dates: November 13-19, 2010 • Exhibition Dates: November 15-18, 2010

The Future of Discovery

IEEE Computer Society

T h e I n t e r n a t i o n a l C o n f e re n c e fo r H i g h Pe r fo r m a n c e C o m p u t i n g , N e t wo r k i n g , S t o ra g e, a n d A n a l ys i s
N u ts a n d B o lts PAM and Hardware

the stolen number will not match the automatically blocks the screen if the the fly; instead, one-time passwords
number on the system. USB device is unplugged: Then You are defined in advance. The pass-
Gentoo and Debian Linux offer pre- need to add the pam_usb PAM library words are stored on the token and
built packages of this PAM library. In to the corresponding PAM configura- in a database on the authentication
both cases, you can use the package tion file, /etc/pam.d/system‑auth or server.
manager to install, as described for /etc/pam.d/common‑auth. If you use When you press the Yubikey button,
pam_thinkfinger. Users on any other the sufficient control flag, users can the key sends one of these OTPs to
Linux distribution can download the log in to the system by plugging in the active application, which then
current source code archive [3] and the USB device, assuming the random uses an API to access the server and
run make; make install to compile number for the user matches on both verify the password. If this fails (Un-
the required files and install them on devices (Listing 2). known Key) or if the password has
the local system. Then you need to To enhance security, you can replace already been used (Replayed Key), an
connect any USB device – it can be the sufficient control flag with re‑ error message is output and the login
a cellphone with an SD card if you quired. This setting first looks for the fails. If the server identifies the key
like – and store its properties in the USB device, but even if the device is as valid, it sets usage‑count to 1 and
/etc/pamusb.conf file. The command identified correctly, PAM still prompts the user is authenticated. The user
for this is the user for a password in the next cannot login with this key anymore
stage of the login process. Both of times.
pamusb‑conf ‑‑add‑device USB‑device‑name
these tests have to complete success- Because of the simple API, more
(Figure 6). The command fully for the user to log in. and more applications are relying
All of the hardware-based login on authentication against the Yubico
pamusb‑conf ‑‑add‑user user
methods I have looked at thus far server. One example is the plugin for
lets you add more users to the con- are easily set up, but they all have the popular WordPress blog, which
figuration and generates a matching vulnerabilities, and it is easy to fake allows users with a Yubikey to log in
random number. The number for fingerprints. Also, USB sticks can be to the blog. A project from Google’s
each user is stored both on the USB stolen, thereby putting an end to any Summer of Code produced a PAM
device and on the system. Also, the security they offered. If you take your module that supports logging in to an
tool adds each user to the XML-based security seriously, you will probably SSH server [5].
/etc/pamusb.conf configuration file. want to use two-factor authentication. Instead of typing your user password
You can use the file to define ac- This method inevitably involves using at the login prompt, you simply press
tions for each user; these actions will chip cards with readers or USB tokens the button on the Yubikey to send a
run when the USB is plugged in or with one-time passwords and PINs. 44-character, modhex-encoded pass-
unplugged. For example, the entry word string to the SSH server. The
in Listing 1 of the configuration file Yubikey server then verifies the string by que-
rying the Yubico server. The first 12
Listing 1: Configuration File for pam_usb A small company from Sweden, characters uniquely identify the user
<user id="tscherf"> Yubico [4], recently started selling on the Yubikey server; the remaining
02  <device> Yubikeys (Figure 7), which are small 32 characters represent the one-time
03  /dev/sdb1 USB tokens that emulate a regular password.
04  </device> USB keyboard. The key has a button You can define a central file on the
05  <
agent event="lock">gnome‑screensaver‑command
on top which, when pressed, tells the SSH server to specify users permitted
token to send a one-time password to log in by producing a Yubikey. To
06  <
agent event="unlock">gnome‑screensaver‑command
(OTP) to the active application, such do so, first create a /etc/yubikey‑us‑
as a login prompt on an SSH server ers.txt file with a username, a colon
or the login window of a web service. separator, and the matching Yubikey
The OTP is verified in real time by a ID (i.e., the first 12 characters of the
Listing 2: USB Device-Based Authentication Yubico authentication server. Because user’s OTP) for each user. Alterna-
[tscherf@tiffy ~]$ id ‑u the software was released under an tively, users can create a file (~/.
open source license, you could theo- yubico/authorized_yubikeys) with
[tscherf@tiffy ~]$ su ‑
retically set up your own authentica- the same information in their home
* pam_usb v0.4.2
* Authentication request for user "root" (su‑l)
tion server on your LAN. This would directory.
* Device "/dev/sdb1" is connected (good). remove the need for an Internet con- You need to configure PAM to verify
* Performing one time pad verification... nection. the OTP against the Yubico server. To
* Verification match, updating one time pads... The way the token works is quite do so, add a line for the Yubikey to
* Access granted. simple. In contrast to popular RSA to- your /etc/pam.d/sshd file (Listing 3).
[root@tiffy ~]# id ‑u
kens, Yubikey doesn’t need a battery The configuration shown in Listing
because the OTP is not generated on 3 runs this authentication in addition

82 Admin 01 w w w. a d m i n - m aga z i n e .co m

PAM and Hardware N u ts a n d B o lts

to the regular, system‑auth-driven on the card. If the certificate is valid, In this example, I’ll use Dogtag [6]
authentication method. But if you the user is mapped onto the system. from the Fedora project as a PKI solu-
replace the required flag with suffi‑ The mapping process can retrieve a tion. Users with other distributions
cient, there is no need for the user to variety of information from the certifi- might prefer OpenSC [7]. The PAM
log in after the Yubikey OTP has been cate, typically the Common Name or library is the same for both variants,
validated. Unfortunately, the Yubikey the UID stored in the certificate. pam_pkcs11.
is not protected by an additional PIN, To make sure the user really is who Dogtag consists of various compo-
and the system is vulnerable if the they claim to be, the system generates nents. For this setup, you’ll also need
token is stolen. An unauthorized user a random 128-bit number. A function a Certificate Authority (CA) to create
in possession of a token would be on the chip card then encrypts the the X.509 certificates. Online Certifi-
able to spoof a third party’s identity. number using the private key, which cate Status Protocol (OCSP) is used
The developers are working on add- is also stored on the card. The user for online validation of the certificates
ing PIN protection for OTPs, and an needs to enter the right PIN to be able on the chip cards. For offline valida-
unofficial patch is already available. to access the private key. The system tion, you just need the latest version
then uses the freely available public of the Certificate Revocation List
X.509 Certificates and PAM key to decrypt the encrypted number. (CRL) on the client system. Of course,
If the results match the random num- you also need a way of moving the
Classic two-factor authentication ber, the user is correctly authenticated user certificate from the certificate au-
typically relies on chip cards. The because the two keys match. thority to the chip card. You can use
cards typically contain a certificate The hardware required for this setup the Enterprise Security Client (ESC)
protected by a PIN. The PAM pam_ is a chip card with a matching reader to open a connection to another PKI
pkcs11 library allows users to log in – for example, the Gemalto e-Gate or component, the Token Processing Sys-
to the system via an X.509 certificate. SCR USB device by SCM. You can use tem, for this.
The certificate contains a private/​ any Java Card 2.1.1 or Global Plat- Assuming correct authentication, the
public key pair. Both can be stored on form 2.0.1-compatible token: Gemalto user certificate is then copied to the
a suitable chip card, with the private Cyberflex tokens are widely available. chip card in the enrollment process.
key protected by an additional PIN Various software solutions are also The ESC tool then gives the user a
to prevent identity spoofing simply available: The approach described in convenient approach to managing the
by stealing a chip card. To log in, this article relies on the pcsc-lite and card. If the user needs to request a
you need both the chip card and the pcsc-lite-libs packages for accessing new certificate from the CA or needs
matching PIN. If the PIN is unknown, the reader. a new PIN for the private key on the
the login fails. card, it’s no problem with ESC.
The details of the login process are as Public Key Infrastructure If you use OpenSC to manage your
follows: The user inserts the chip card
into the reader and enters the PIN. It makes sense to use X.509 certifi- Listing 3: PAM Configuration for a Yubikey
The system searches for the certificate cates, but only if you have a complete auth required authfile=/etc/yubikey‑users.

with the public key and private key Public Key Infrastructure (PKI) set up. txt

auth include system‑auth

account required

account include system‑auth

password include system‑auth

session required close

session required

session required open env_params

session optional force revoke

session include password‑auth

Listing 4: Configuration File for pam_pkcs11

pkcs11_module coolkey {
02  module =;
03  description = "Cool Key"
04  slot_num = 0;
05  ca_dir = /etc/pam_pkcs11/cacerts;
06  nss_dir = /etc/pki/nssdb;
07  crl_dir = /etc/pam_pkcs11/crls;
08  crl_policy = auto;
Figure 7: USB keyboard emulation means that the Yubikey for one-time passwords doesn’t need special
drivers. The token works with the press of a button.

w w w. a d m i n - m aga z i n e .co m Admin 01 83

N u ts a n d B o lts PAM and Hardware

cess should be completely transpar-

ent. The Thunderbird email program
can use the card to sign and encrypt
email; Firefox can use the certificate
for client-side authentication against
a web server. The reward for all this
configuration is a large choice of de-
ployment scenarios. The ESC Guide
[10] has a more detailed description
of the tool and its configuration.

PAM is a very powerful framework
for handling authentication. As
you can see from the PAM libraries
introduced in this article, the func-
tional scope is not just restricted to
Figure 8: The Security Client gives you an easy option for managing chip cards. authenticating users but also covers
tasks such as authorization, password
chip card, you can transfer a prebuilt it possible to validate the user’s au- management, and session manage-
PKCS#12 file [8] to the card using: thenticity. ment. Administrators who take the
time to familiarize themselves with
pkcs15‑init U
‑‑store‑private‑key tscherf.p12 U Certificates for configuring PAM, which isn’t always
trivial, will be rewarded with a feast
‑‑format pkcs12 U
‑‑auth‑id 01
Thunderbird and Co. of feature-rich and flexible options for
Applications that rely on Network password- and hardware-based au-
The PKCS#12 file contains both the Security Services (NSS) for signing or thentication and authorization. n
public key and private key. If you encrypting email with S/​MIME, such
have a user certificate from a public as Thunderbird, use a file in the nss_
certification authority like CACert [9], dir as the CA database; applications Info
you can use your browser’s certificate based on the OpenSSL libraries use [1] Linux PAM: [http://​­www.​­kernel.​­org/​­pub/​
management facility to export the cer- the database in the ca_dir directory. ­linux/​­libs/​­pam/]
tificate to a file and then transfer it to The certutil tool can import the [2] PAM ThinkFinger:
the chip card as described. CA certificate into the NSS database; [http://​­thinkfinger.​­sourceforge.​­net]
If you don’t have a certificate, you OpenSSL-based certificates can sim- [3] pam_usb:
can create a request and send it to ply be appended to the existing file. [http://​­downloads.​­sourceforge.​­net/​
the appropriate certificate authority. Finally, you can define the mappings ­pamusb/​­pam_usb‑0.​­4.​­2.​­tar.​­gz?​­download]
Once the authority has verified your between user certificates and Linux [4] Yubico website: [http://​­www.​­yubico.​­com/​
request, it will return a certificate to users in the pam_pkcs11 configuration ­products/​­yubikey/]
you. file. Various mapping tools are avail- [5] SSH server for Yubikey: [http://​­code.​
For both approaches, you can use the able for this, specified as follows: ­google.​­com/​­p/​­yubico‑pam/​­downloads/]
/etc/pam_pkcs11/pam_pkcs11.conf file [6] Dogtag PKI: [http://​­pki‑svn.​­fedora.​­redhat.​
use_mappers = cn, uid
to define the driver for access to the ­com/​­wiki/​­PKI_Main_Page]
chip card. The driver can be modified Next, you still need to add the PAM [7] OpenSC: [http://​­www.​­opensc‑project.​­org]
in the configuration file, as shown in pam_pkcs11 library to the correct [8] PKCS specifications:
Listing 4. PAM configuration file – that is, /etc/ [http://​­en.​­wikipedia.​­org/​­wiki/​­PKCS]
Here, you must specify the correct pam.d/login or /etc/pam.d/gdm. You [9] CACert certificate authority:
paths to the local CRL and CA certifi- can edit the file manually or use the [http://​­cacert.​­com]
cate repository. The CRL database is system‑config‑authentication tool [10] ESC Guide: [http://​­directory.​
necessary to check that the certificate referred to previously. ­fedoraproject.​­org/​­wiki/​­ESC_Guide]
on a user’s chip card is still valid and When you insert the chip card into
has not been revoked by the certifi- the reader and launch the ESC tool, The Author
cate authority. You need the certifi- you should be able to see the certifi- Thorsten Scherf is a Senior Consultant for Red
cate for the certificate authority that cate (Figure 8). If you now attempt Hat EMEA. You can meet him as a speaker at
issued the user’s certificate from the to log in via the console or a new conferences. He is also a keen marathon runner
CA certificate repository. This makes GDM session, the authentication pro- whenever time permits.

84 Admin 01 w w w. a d m i n - m aga z i n e .co m

Includes a comprehensive Discovery
Guide to help new users install,
configure, and explore clu d e s f r ee Ubuntu
Also in
Ubuntu! x” DVD!
“Lucid Lyn
Find out more on
n u tS A n d b o ltS modSecurity

Protecting web servers with ModSecurity

Apache Protector
© KrishnaKumar Sivaraman,

even securely configured and patched web servers can be compromised NetBSD, AIX, and Windows, with
the later versions only available for
because of vulnerabilities in a web application. modSecurity is an Apache Apache 2.x. This article discusses
extension that acts as a web application firewall to protect the web server version 2.5.10; the successor 2.5.11 is
against attacks. by Sebastian wolfgarten merely a bugfix.
The software’s functional scope is
Security issues on the web are no this to happen, these firewalls ana- enormous but comprehensively docu-
longer typically a result of poor con- lyze incoming and outgoing client re- mented [3]. It logs HTTP requests
figuration or the lack of up-to-date quests and server responses to distin- and gives administrators unrestricted
server software. Tomcat, Apache, guish between benevolent and malev- access to the individual elements of
and even IIS have become extremely olent requests on the basis of rules. a request, such as the content of a
mature over the past few years – so If necessary, they can even launch POST request. It also identifies at-
much so that they don’t have any no- countermeasures; if configured to do tacks in real time based on positive or
ticeable vulnerabilities, although ex- so, the software will also inspect en- negative security models and detects
ceptions can always turn up to prove crypted HTTPS connections. anomalies based on supplied patterns
the rule. Thus, hackers have turned for known vulnerabilities.
their attention to the web applications Accessories en Masse The powerful rules discover whether
and scripts running on the servers. credit cards are in the data stream or
Increasingly complex user require- Where classical network-based fire- use GeoIP to prevent access from cer-
ments are making web applications walls – I’m exaggerating slightly tain regions. ModSecurity checks not
more complex, too: Ajax, interaction here – either permit any or no HTTP only incoming requests but also the
with external databases, back-end connections, WAFs target individual server’s outgoing responses. The soft-
interfaces, and directory services are HTTP connections based on their ware can implement chroot environ-
just part of the package for a modern content. ModSecurity is a high- ments. As a reverse proxy, it protects
application. And, attack vectors grow performance WAF for Apache and a web applications on other web serv-
to match this development (see the complex module for the Apache web ers, such as Tomcat or IIS.
“Attacks on Web Servers” box). server. Originally developed by Ivan Breach also provides a collection of
Ristic, Breach Security handles its dis- core rules that guarantees the basic
Firewalls for the Web tribution and development [2]. security of the web server. Com-
Two variants of the software are avail- prehensive documentation, many
In contrast to legacy packet filters, able: the open source variant released examples, and a mailing list provide
Web Application Firewalls (WAFs) under the GPLv2, and a commercial support for the user. This makes Mod-
don’t inspect data in the network version with professional support, Security a good choice for protecting
or transport layer, but rather at the pre-configured appliances, and man- web servers and their applications
HTTP protocol level (i.e., in OSI Layer agement consoles. ModSecurity runs against vulnerabilities. But before you
7) [1]. They actually speak HTTP. For on Linux, Solaris, FreeBSD, OpenBSD, can even consider tackling the highly

86 Admin 01 w w w. A d m i n - m AgA z i n e .co m

modSecurity n u tS A n d b o ltS

complex configuration, you first need Attacks on web Servers

to install the third-party module.
Compared with local applications, web applications are more vulnerable because they involve so
many different components – from the browser and the Internet infrastructure to the web server
Packages for any and the back ends beyond. Vulnerabilities can occur anywhere, but the server is always at the
Distribution center of this environment.
If the web application doesn’t sufficiently validate user input and instead passes it to a database
If you prefer not to build the pack- running in the background, attackers could use SQL injection to inject their own commands into
age for Apache 2 yourself, you the command chain. Thus, the attacker would be able to read, modify, or delete data and thereby
can pick up pre-built packages for exert a major influence on the application.
Debian, RHEL, CentOS, Fedora, If an application also allows attackers to store files on the web server and execute them over the
FreeBSD, Gentoo, and Windows. The web, the intruder could set up a web shell. Because the server will execute the attacker’s files,
manual install requires the Apache the attacker can run operating system commands on the web server and finally escalate their
mod_unique_id module, which is not privileges to interactive shell access. Although the architecture of a carefully configured Apache
automatically provided by some dis- will not give the attacker root access, this is often unnecessary to access sensitive data. And the
tributions. You can use the more services that are installed on the server, the more likely it is that the attacker will find one
that is vulnerable.
LoadModule security2_module U
without a corresponding action. The filters and actions (see also the “In-
directive to integrate the module into option also expects you to specify a sider Attacks” box). The software of-
the Apache configuration file, httpd. processing phase for the standard ac- fers a number of prebuilt alternatives
conf, if your distribution doesn’t tion, as listed in Table 1. ModSecurity here, including converting request
do this for you. After restarting, the applies filter rules in five different parameters, running external scripts
web server lists the module in its er- phases of processing and respond- (e.g., to perform an antivirus scan),
ror_log. ing to a client request [4]. In real or forwarding malevolent requests.
life, only phase 2, in which the client The latter is useful if you are trying to
Filter Rules requests that content (i.e., incoming investigate attacks or want to forward
data) is filtered, and phase 4, which them to a honeypot.
ModSecurity has a mass of configura- handles the server response (i.e., The basic configuration (Listing 1)
tion options, but to understand how it outgoing data), are relevant. Addi- also logs the contents of incoming
works, all you need is the basic con- tionally, you can use the option to de- requests and the responses given in
figuration. The SecRuleEngine option fine one or multiple actions that the return. It enters the information in the
activates the module’s filter mecha- software will perform when a match auditlog as mentioned previously. The
nism and allows it to process filter occurs [5]. To log incoming client re- first directive, SecAuditEngine, en-
rules. The settings here are On, Off, quests in the auditlog in phase 2 and ables this. The option for the second
and DetectionOnly, which tells the respond to the access attempt with a directive defines whether the software
module to monitor, but not become HTTP 403 (Forbidden) error message, stores the auditlog entries in a single
actively involved with, the client- you would do this: file (Serial) or writes a file for each
server connection, even if individual transaction (Concurrent). Concur-
SecDefaultAction phase:2,log,U
rules are configured to let it do so. auditlog,deny,status:403
rency is necessary if you intend to
This setting is useful for testing the deploy the ModSecurity Console add-
module and your own rules. A massively negative default func- on product. Breach Security offers
To help you get started, or for de- tion like the default deny is highly the software for managing multiple
bugging, you also want to enable restrictive, but it does offer maximum instances, provided you don’t need to
the SecDebugLog option to define a protection if you additionally define monitor more than three servers. E
troubleshooting log (e.g., SecDebugLog
/var/log/httpd/modsec_debug.log). Table 1: modSecurity Processing Phases
Additionally, you can set the SecDe- Number Phase Designator Activities
bugLogLevel parameter to specify 1 Preview phase REQUEST_HEADERS Earliest possible filtering of incoming
verbosity, on a scale of 0 to 9, which requests before access control, authentica‑
tion, authorization, and MIME detection have
issues comments on its own activities
taken place Apache‑side
and the way it processes user-defined REQUEST_BODY
2 Client request Full access to the content of a client request
rules. Levels 4 or 5 are useful for fine (normal case)
tuning or troubleshooting. In produc- 3 POST request RESPONSE_HEADERS Initial option for filtering server responses
tion, set this parameter to 0. 4 Server response RESPONSE_BODY Full access to the content of the server re‑
The SecDefaultAction parameter de- sponse to an incoming client request
fines ModSecurity’s default behavior 5 Logging LOGGING Access to all relevant information before it
for requests that match a filter rule is written to the Apache logfiles

w w w. A d m i n - m AgA z i n e .co m Admin 01 87

N u ts a n d b o lts ModSecurity

Figure 1: Once ModSecurity has been enabled, it will log suspicious activity at the detail level specified in SecAuditLogParts – in this case, an SQL injection attack.

The third instruction defines the au- SecRule REQUEST_HEADERS: User‑Agent "nikto" log the transaction in the auditlog and
ditlog storage location relative to the send a detailed overview of the way
Apache installation path. Finally, the This example tells the web server to the client request was processed to
SecAuditLogParts instruction defines refuse requests from the Nikto secu- the debuglog. This means that your
the information that ModSecurity logs rity scanner [7]. If you want Mod- Apache error_log file will contain a
in the auditlog (Table 2). In this case, Security to run a specific action for note to the effect that a client request
this is the header and the content of a rule, you can overwrite the default was effectively blocked, as shown in
the request, along with the ModSecu- action: Listing 2. Once ModSecurity is work-
rity reaction. The results are shown in ing correctly, you can start adding
SecRule REQUEST_HEADERS:User‑Agent "nikto" U
Figure 1. "phase:2,pass,msg:'Nikto‑Scan logged'"
rules and modifying them for the web
After this preparation, you can add applications you want to protect.
the most important directive to your This rule tells the module to write
configuration: SecRule. This defines a Nikto scan logged message to the The Art of Detecting
a filter rule and optionally an action logfile when it detects the Nikto user Attacks
that the module will perform if it agent in a client request in phase 2.
discovers a match for the rule. If you The rule then overwrites the drop To provide effective protection against
don’t define an action, the tool will default action, which is defined by a huge assortment of attacks, system
run the standard command defined in SecDefaultAction with the pass ac- administrators need to set up a robust
the SecDefaultAction directive. Rules tion. This allows the client request ruleset for ModSecurity. To formulate
always follow this pattern: to pass. To test ModSecurity, Listing rules that protect you against SQL
1 gives you an overview of the basic injection, cross-site scripting, or local
SecRule Variable Operator [Action]
configuration discussed thus far. and remote file inclusion attacks, for
The number of variables is huge, and example, you need in-depth knowl-
they cover every single element of the Practice Session edge of how attacks on web servers
client request (both for POST and for work.
GET), as well as the most important After restarting Apache, a request Of course, not every administrator
server environment details [6]. Ad- like has this knowledge or the time to
ditionally, you can use regular expres- html?file=/etc/passwd would trigger re-invent the wheel. To address this,
sions. For example, to investigate an the sample rule in line 8. Then the the Open Web Application Security
HTTP request to find out whether the action defined in line 9 would block Project (OWASP) offers a predefined
client requests the /etc/passwd string the request. The client sees an HTTP ruleset for ModSecurity [8] that re-
in a GET method, you would use this 403 Forbidden error. At the same time, lies on anomaly detection to protect
rule: lines 3 through 7 tell ModSecurity to web servers against a number of

SecRule REQUEST_URI "/etc/passwd"

Table 2: SecAuditLogParts Arguments
If the request matches the rule, Mod- Abbreviation Description
Security runs the default deny action. A Header for the entry (mandatory)
To filter by browser type, you would B Request header
do this: C Request content; only available if content exists and ModSecurity is configured
to store it
Listing 1: Basic Configuration for ModSecurity D Reserved
E Temporary response content; only available if ModSecurity is configured for this
01 SecRuleEngine On
F Final response header after possible manipulation by ModSecurity; Apache itself
02 SecAuditEngine On
writes the Date and Server headers
03 SecAuditLogType Serial
G Reserved
04 SecAuditLog logs/audit.log
H Auditlog trailer equivalent to C, except where the request contains form data; in
05 SecAuditLogParts ABCFHZ
this case, the software constructs a suitable request that excludes file content to
06 SecDebugLog logs/debug.log simplify matches
07 SecDebugLogLevel 5 J Reserved
08 SecRule REQUEST_URI "/etc/passwd" K Line by line list of all matching rules in the order of their application
09 SecDefaultAction phase:2,log,auditlog,deny,status:403 Z End of entry (mandatory)

88 Admin 01 w w w. a d m i n - m aga z i n e .co m

ModSecurity N u ts a n d b o lts

Insider Attacks Security will mean about a 5 percent

performance overhead for your web
A vulnerability in a web application can thus expose an Apache web server despite its being
hardened and up to date. This is particularly dangerous if the server is deployed in a hosting
environment where many customers share the resources of a physical web server. Protection
mechanisms, such as virtualization or jails, will separate the individual instances; however, the Preventing Attacks on the
security of the whole system depends on its weakest link. With a sniffer, for example, attackers United Nations
can simply sniff password authentication conversations if they’re not encrypted. And, this gives
attackers an inside vector. In some situations, you can’t fix a
Scripting languages and frameworks like PHP or Ruby on Rails help web developers achieve web application vulnerability imme-
results quickly, but they often conceal dangers that occur when security is not given sufficient diately. Imagine a major online store
attention. More complex environments, such as Java, Tomcat, and JBoss, are not necessarily the discovering a security hole a week
answer because they hide many aspects from the developer.
before Christmas and needing several
days to fix the problem, meaning the
standard attacks, such as invalid cli- logging to see exactly what the mod- shop would be offline for that time.
ent requests, SQL injection, cross-site ule is doing. To do so, you need to The owner has to make a decision:
scripting, and email or command include the core rules in httpd.conf Live with the risk and keep the shop,
injection. This gives you basic, fairly as follows: including the vulnerability, online so
robust protection, which you can then you can benefit from lucrative pre-
Include conf/modsecurity/*.conf
modify to match your own applica- Include conf/modsecurity/ base_rules/*.conf
Christmas shopping, or protect the
tion environment as needed. company and its customers by taking
To install these core rules, download After restarting, your Apache web the website down and fixing the vul-
the package and unpack it in your server will have a solid suit of armor nerability.
Apache’s conf configuration direc- that responds with an HTTP 403 to ModSecurity offers a technical work-
tory. Then, move the rules, including any client requests classified as at- around in the form of virtual patch-
the base_rules subdirectory, to the tacks. ing that allows you to define one or
ModSecurity directory. You can look A word of caution: Web masters multiple rules that prevent the vulner-
at the configuration in the modsecu- should first test the core rules exten- ability from being exploited without
rity_crs_10_global_config.conf and sively in a lab environment before actually removing it.
modsecurity_crs_10_config.conf files letting them loose on a production The ModSecurity documentation re-
and modify it to suit your needs. The system. Otherwise, the danger of fers to a case that dates back to 2007,
rules are well documented. Also, you preventing legitimate user access is when attackers were trying to hack
might want to enable audit and debug possible. Also, remember that Mod- the United Nations website [9]. The
sub-page with talks by Secretary-Gen-
eral Ban Ki-moon [10] had a statID
parameter that exposed an SQL injec-
tion vulnerability (Figure 2).
If you discover a vulnerability of this
kind, you can temporarily define a
rule for ModSecurity that will prevent
hackers from exploiting the vulner-
ability even though it still exists.
Although this isn’t a good long-term
solution, it does prevent a disaster
until the web developers can remove
the vulnerability from the source code
on the page.
The following solution would work
for the UN bug: E

Listing 2: Rule Match

AuditLogType Serial [Wed Nov 04 05:39:19 2009]
[error] [client] ModSecurity: Access
denied with code 403 (phase 2). Pattern match
"/etc/passwd" at REQUEST_URI. [file "/usr/local/
httpd‑2.2.14/conf/httpd.conf"] [line "420"] [hostname
Figure 2: The website of UN Secretary-General Ban Ki-moon was affected by an input parameter variable
""] [uri "/index.html"] [unique_id
validation vulnerability. A ModSecurity virtual patching rule protected the website temporarily until the UN
system administrator fixed the problem.

w w w. a d m i n - m aga z i n e .co m Admin 01 89

N u ts a n d b o lts ModSecurity

<Location /apps/news/infocus/ U and although it is not particularly el- mercial products and services such
sgspeeches/statments_full.asp> egant, it does indicate what potential as training. Armed with ModSecurity,
SecRule &ARGS "!@eq 1"
you have. With a carefully crafted administrators can sit up tall in their
SecRule ARGS_NAMES "!^statid$"
SecRule ARGS:statID "!^\d{1,3}$"
regular expression, you can use the saddles, even if attackers are trying to
</Location> same technique to prevent credit card make their horses bolt. n
numbers from being revealed by, for
Three lines embedded in an Apache example, a compromised application
location container state that valid in the aftermath of a successful SQL Info
user requests for the statements_ injection attack. [1] OWASP Best Practices: Use of Web Appli‑
full.asp file are only allowed to cation Firewalls: [http://​­www.​­owasp.​­org/​
have one argument (first rule) called The Chinese Wall in Reverse ­index.​­php/​­Category:OWASP_Best_Prac‑
statid (second rule) with numbers tices:_Use_of_Web_Application_Firewalls]
of one to three digits (third rule) as Another advanced scenario for Mod- [2] ModSecurity: [http://​­modsecurity.​­org]
their parameters. Any requests that Security involves cooperating with [3] ModSecurity Reference Manual:
do not follow this pattern are cleaned the GeoIP provider, Maxmind. GeoIP [http://​­modsecurity.​­org/​­documentation/​
up by the default action, as defined in locates users geographically on the ­modsecurity‑apache/​­2.​­5.​­10/​
SecDefaultAction. This would effec- basis of their IP address, which ­html‑multipage]
tively prevent an attacker exploiting means you can restrict access to a [4] ModSecurity Processing Phases: [http://​
the SQL injection vulnerability. website to a specific region, such as ­www.​­modsecurity.​­org/​­documentation/​
Pennsylvania – if you have a site in ­modsecurity‑apache/​­2.​­5.​­0/​
No Inside Information Pennsylvanian Dutch that nobody ­html‑multipage/​­processing‑phases.​­html]
else would understand – or block [5] ModSecurity Actions: [http://​­www.​
ModSecurity also filters outgoing a country entirely. To do this, you ­modsecurity.​­org/​­documentation/​
data, especially server responses to would install the mod_geoip2 module ­modsecurity‑apache/​­1.​­9.​­3/​
incoming requests. The PHP program- on Apache 2, along with the GeoIP ­html‑multipage/​­05‑actions.​­html]
ming language throws error messages software and GeoLiteCity.dat geo- [6] ModSecurity Variables: [http://​­www.​
such as this: graphical database [11]. ­modsecurity.​­org/​­documentation/​
Imagine a mechanical engineering ­modsecurity‑apache/​­2.​­1.​­0/​
Fatal error: Connecting to MySQL server
'' failed
company in Germany’s Swabian ­html‑multipage/​­05‑variables.​­html]
region that is afraid of industrial [7] Nikto: [http://​­cirt.​­net/​­nikto2]
Although you can disable PHP error espionage from the Far East; in this [8] OWASP ModSecurity Core Rule Set Pro‑
messages in responses, Google still case, they could use the configura- ject: [http://​­www.​­owasp.​­org/​­index.​­php/​
lists a bunch of websites where PHP tion in Listing 3 to prevent access ­Category:OWASP_ModSecurity_Core_Rule_
error messages reveal many juicy de- from China – if the people in China Set_Project]
tails of applications. This information didn’t spoof their origins. The last [9] ModSecurity blog, “Virtual Patching Dur‑
is very useful to an attacker, because two lines form a filter rule chain. Line ing Incident Response: United Nations
it can help them understand the inter- 6 locates the geographical region for Defacement”: [http://​­blog.​­modsecurity.​
nal workings and structure of a web the requesting IP address, then line ­org/​­2007/​­08/​­27/]
application and attack it in a more 7 dumps the request and a message [10] Talks by the UN General Secretary:
targeted way. To tell ModSecurity to into the logfile if the request comes [http://​­www.​­un.​­org/​­apps/​­news/​­infocus/​
catch PHP error messages and pre- from China. This might not be politi- ­sgspeeches/]
vent them from being sent to users, cally correct, but it is technically ef- [11] “Apache ModSecurity with GeoIP blocking
you can define a rule like this: fective. country specific traffic: ModSecurity +
GeoIP” by Suvabrata Mukherjee: [http://​
SecRule RESPONSE_BODY "Fatal error:"
Full Insurance Coverage ­linuxhelp123.​­wordpress.​­com/​­2008/​­12/​­11/​
RESPONSE_BODY refers to the content ­apache]
of the server response to the client, ModSecurity has an enormous feature
scope, and it can take some time to The Author
Listing 3: GeoIP Access understand it completely. But if you Sebastian Wolfgarten works as an IT Security
01 LoadModule geoip_module modules/
go to the trouble to plum the depths Expert with the European Central Bank as an
02 LoadModule security2_module modules/
of the module, it will pay dividends advisor, manager, and auditor of internal proj‑
03 GeoIPEnable On
04 GeoIPDBFile /usr/tmp/GeoLiteCity.dat
with comprehensive methods that ects designed to improve the security of the
05 SecRuleEngine On give you additional protection against IT infrastructure. Before this, he spent several
06 SecGeoLookupDb /usr/tmp/GeoLiteCity.dat attacks on web applications. years working for Ernst & Young AG in Germany,
07 Sec
Rule REMOTE_ADDR "@geoLookup" Thankfully, the prebuilt rulesets and as an Advisor for Information Security in
"chain,drop,msg:'Connection attempt from .CN!'"
make it easier to get started. And, the Ireland. He has also worked as an IT security
08 SecRule GEO:COUNTRY_CODE "@streq CN" "t:none"
vendor behind the project offers com- expert with T-Mobile Germany.

90 Admin 01 w w w. a d m i n - m aga z i n e .co m

Each issue delivers
technical solutions
to the real-world
problems you face
every day.

Learn the latest

techniques for better:

• network security

• system management

• troubleshooting

• performance tuning

• virtualization

• cloud computing

FREE DVD on Windows, Linux,

Solaris, and popular
Inside: varieties of Unix.
Knoppix 6.3 +

. c o m/su bs
aga z i n e
w. a d m in-m
N L I N E AT ww
N U TS A N D B O LTS Daemon Monitoring

Monitoring daemons with shell tools

Watching the
© Shariff Che'Lah,

Administrators often write custom monitoring programs to make sure for accessing the exit code. Error logs
are obtainable by redirecting the er-
their daemons are providing the intended functionality. But simple shell ror output to a file or, if available, by
tools are just as well suited to this task, and not just for systems that are setting the corresponding program
low on resources. By Harald Zisler option.
The only thing left to do is to find
Unix daemons typically go about Because almost every program out- the matching client program test the
their work discreetly in the back- puts standardized exit codes when it functionality of each service.
ground. The process table, which is terminates, you can use Unix conven-
output with the ps command, only tions. 0 stands for error-free process- Web Servers
shows you that these willing helpers ing, whereas 1 indicates some prob-
have been launched, although in the lems were encountered. This value is To check a web server, you could use
worst case they could just be hang- stored in the $? shell variable, which wget. The shell script command line
ing around as zombies. Whether or a shell script evaluates immediately for this would be:
not a daemon is actually working is after launching the sensor.
wget --spider -q ip-address
not something that the process table Various programs are suitable for
will tell you. In other words, you automated, “unmanned” access to The --spider option tells wget to
need more granular diagnostics. The the service provided by a given dae- check that the page exists but not to
underlying idea is to write a “sensor” mon; all of them will run in the shell load it. Defining the IP address in-
script for each service that performs a without a GUI. These programs often stead of the hostname avoids a false
tangible check of its individual func- provide an option (typically -q) that positive if DNS-based name resolution
tionality. suppresses output, and this is fine fails for some reason.

Listing 1: Database Monitoring

01 #! /bin/sh 18 34
02 19 if [ $? -eq 0 ]; 35 then
03 while true 20 36
04 do 21 then
37 time=$(date +%d.%m.%y\ %H:%M\ )
05 22
38 echo "$time: Database online!
06 zeit=$(date +%d.%m.%y\ %H:%M\ ) 23 echo "$time: Database online!
+++++++++" >> dba.log
07 +++++++++" >> dba.log
39 break
08 psql -U monitor -d monitor 24
-c "select * from watch;" 25 else 40

09 26 41 fi
10 if [ $? -eq 2 ]; 27 echo "$time: Database: serious error! 42 sleep 15
11 ***************" >> dba.log 43 done
12 then 28 echo "$time: Unable to restart!
13 ****************" >> dba.log
45 fi
14 echo "$time: Database is not accessible! 29 while true
****************" >> dba.log 30 do
47 fi
15 /usr/local/etc/rc.d/ start 31 psql -U monitor -d monitor
16 sleep 15 -c "select * from watch;" 48 sleep 15

17 psql -U monitor -d monitor 32 49

-c "select * from watch;" 33 if [ $? -eq 0 ]; 50 done

92 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M

Daemon Monitoring N u ts a n d B o lts

The ping com- for the administrator to step in and

mand checks handle the situation.
network con-
nections. The Printer Restart
exit error codes
differ, depending The second sample script relates to
on your operat- the printing service. The one shown
Figure 1: After starting, the script outputs the log at the console: availability, ing system. The here is taken from a production ex-
error, restart, database running. FreeBSD ping ample, in which the cupsd server has
uses 2, the Linux an unknown problem with a network
Almost all known databases include ping uses 1. printer. The printer was disabled time
a client program for the shell – for The number of test packets is re- and time again, causing no end of
example, mysql for MySQL or psql for stricted by the ‑c packets option; frustration to users and unnecessary
PostgreSQL. Alternatively, you can this improves the script run time and work for the system admins. The shell
use ODBC to access the database in avoids unnecessary network traffic. If script shown in Listing 2 doesn’t
your scripted monitoring, such as the you use the IP address as the target, output messages; instead, it simply
isql tool provided by the Unix ODBC you avoid the risk of false positives restarts the service. Either run these
project. from buggy name resolution. scripts manually (for a temporary fix
For ease of access, you might need to or quick check) or as RC scripts.
ping ‑c1 ip_address
set up a (non-privileged) user, a da-
tabase, and a table for the test query Sensor scripts can obviously be ex- Conclusions
on the database server. If you choose tended to cover many other system
the ODBC option, you also need a parameters, such as disk space usage Administrators don’t need a complex
.odbc.ini file with the right access (df), logged in users (who), and much, monitoring framework that covers
credentials. much more. every aspect of the environment and
The psql shell client for the Postgres If an error or threshold value infringe- has a multi-week learning curve.
database also poses the problem of ment occurs, the script can use this With some scripting know-how, you
non-standard exit codes. 1 stands for information to generate a message can easily create your own shell
an error in the query, although the and notify the system administrator. scripts to monitor server daemon pro-
connect attempt has been successful; The message text should include the cesses and restart them autonomously
2 indicates a connection error. hostname, date, and time. Messages if so desired. The use of shell scripts
A connection test with psql would can be stored in a file to which the to monitor daemons and other system
look like this: administrator has permanent access. functions is by no means restricted to
To allow this to happen, you simply small embedded systems. With scripts
psql ‑U User ‑d Database ‑c U
"select * from test_table;"
have to display the logfile in a termi- tailored to match your requirements,
nal and use tail ‑f, but other forms you can establish your own trouble-
For ODBC access, you would need to of communication are also possible – shooting arsenal. n
pipe the SQL query to the client: texting, for example.
If the shell script has the correct
echo "select * from test_table;" | U
isql ODBC_data_source user
privileges, it can become involved The Author
and restart a daemon, remove block Harald Zisler has worked with Unix-flavored
For the cups printer daemon, lpq files, or even reboot the whole sys- operating systems since the early 1990s.
gives you a simple method of check- tem. Because you should avoid run-
ing whether the daemon is alive. If ning this kind of script as root, you Listing 2: CUPS Monitoring
you need to check access to indi- can instead set up special users and 01 
#! /bin/sh

vidual printers, you additionally need groups to own the script and the 02 
while true
to provide the print queue name and process (which is the case with many
then grep the exit code. To make sure daemons). 05 
the exit code complies with this be- 06 
lpq ‑Plp | grep ‑q "lp is ready"
havior, Grep checks the output that Database Restart 07 

you receive if the printer is active: 08 

if [ $? ‑gt 0 ]
09  then
The sample script in Listing 1 moni-
lpq ‑Pprinter | grep ‑q U 10  cupsenable lp
"printer is ready"
tors an active database instance and 11 
notifies the administrator if the da- 12 
To match the output from lpq, you tabase happens to fail and then is 13 
sleep 15

need to modify the search string for successfully restarted (Figure 1). If 14 
grep. it can’t start the daemon, it waits

w w w. a d m i n - m aga z i n e .co m Admin 01 93

N u ts a n d B o lts VPNs with SSTP

z mi

State-of-the-art virtual private networks

Private Affair
Because Microsoft’s legacy VPN protocol, PPTP, has a couple of vulner- connection through a firewall, a NAT
device, or a proxy. Like all SSL VPNs,
abilities, SSTP, which routes data via an SSL connection, was introduced SSTP uses TCP port 443 (HTTPS) for
as the new VPN protocol with Vista, Windows Server 2008, and Windows 7. data transfer. Compared with other
By Thomas Drilling
commercial or proprietary solutions
(e.g., IPsec, L2TP, or PPTP), the ad-
Virtual private networks (VPNs) access. The Secure Socket Tunneling vantage is that port 443 is open in
have established themselves as a Protocol (SSTP), which was intro- almost any router or server configu-
standard solution for convenient re- duced with Microsoft Windows Server ration, and SSTP packets can thus
mote access to enterprise networks. 2008, provides a solution by setting pass through without any additional
However, they can cause some is- up a VPN tunnel that encapsulates configuration overhead. The “Hand-
sues in combination with standard PPP or L2TP traffic on a Secure Sock- shake” box explains what Windows
tunneling protocols like PPTP if, for ets Layer (SSL) channel (Figure 1). does with all of these protocols.
example, NAT routers are involved For administrators, this means that Strong encryption in SSL 3.0 ensures
or you need to work around the lo- SSTP is a new VPN tunnel type in maximum security and performance.
cal firewall. Typically, it is not in the the Windows Server 2008 routing SSTP VPNs are thus a class of SSL
administrator’s best interest to modify and RAS server role. It encapsulates VPNs, like Cisco’s WebVPN or the
the firewall, NAT, or proxy configura- PPP (point-to-point protocol) packets Vigor Router by Draytek, that basi-
tion to suit requirements for remote in HTTPS, thus supporting the VPN cally work in the same way as IPsec,

94 Admin 01 w w w. a d m i n - m aga z i n e .co m

VPns with SSTP n u TS A n d B o LTS

Microsoft’s SSTP is basically another proprietary implementation of an 5. The client sends the encrypted SSL session key to the server.
SSL VPN. SSTP relies on standards such as SSL and TLS for encryption 6. The server decrypts the client’s SSL session key using its own private
and authentication, but Microsoft has modified the tried-and-trusted key and encrypts the communications with the session key. Up to this
SSL handshake by introducing proprietary extensions that bind the pro- point, the procedure is no different from standard SSL communication.
prietary PPP protocol. If you look at the basic handshake, SSTP at first However, Microsoft then implements additional handshake steps that
keeps to the standard SSL handshake procedure: build on what has happened thus far.
1. The client opens a connection to TCP port 443 on the server. 7. The client sends an HTTP-over-SSL request message to the server and
2. The client sends an “SSL Session Setup Message” to indicate that it negotiates an SSTP tunnel with the server.
wants to set up an SSL connection to the server. 8. After this, the client negotiates a PPP connection with the SSTP
3. The server sends an SSL certificate to the client. server, which includes authenticating the user’s login credentials with
4. The client validates the server certificate, identifies the correct en- a PPP authentication method and configuring the settings for the data
cryption method for the SSL session, and creates a session key, which it traffic.
encrypts using the public key from the server certificate. 9. The client starts to transfer data via the PPP connection.

L2TP, or PPTP, but use SSL to handle least one card (preferably both) needs no alternative to configuring a public
the data transfer. Because SSTP en- a static IP address. The client will be key infrastructure. At a minimum,
capsulates complete IP packets, the MS Windows 7. this means installing at least one cer-
connections act just like a PPTP or The server and clients are both mem- tificate on the SSTP server and a root
IPsec tunnel on the client side. Ac- bers of an Active Directory (AD) certificate authority certificate on all
cording to the Microsoft definition, domain. Additionally, the server and SSTP VPN clients. You might have
SSTP is a protocol mainly intended clients should have all the current to modify the packet filter rules, too,
for dialup connections in the applica- updates in place, such as the current even though SSTP doesn’t actually
tion layer that guarantees the confi- Service Pack 2 for Windows Server need any additional NAT configura-
dentiality, authenticity, and integrity 2008 R2. After installing a Windows tion because port 443 is typically
of the data to be transferred. A public server, you need to install the Ac- open. The “Port Customizer” box ex-
key infrastructure (PKI) is used for tive Directory Domain Services to plains how you can use a port other
authentication purposes. Microsoft in- make the server a domain controller. than 443 for SSTP.
troduced SSTP with Windows Server The easiest way to do this is to run Next, you’ll need to configure an
2008 and Vista SP1. Today, Windows dcpromo at the command line and Active Directory-integrated root certi-
Server 2008 R2 and Windows 7 also then follow the wizard. fication authority on the domain con-
support SSTP [1]. But what does SSTP The server also needs to provide troller. In combination with a group
offer the administrator, and how do DNS, DHCP, and certificate services, policy, this causes clients that are do-
you set up a VPN server with SSTP? which you can achieve by configuring main members to request certificates
the matching server roles in the role automatically when they open a con-
Sample Scenario wizard. VPN functionality is also pro- nection. Certificates are then issued
vided via a server role Network policy
In this example, I am using Windows and access services. Port customizer
Server 2008 R2 to provide an SSTP- SSTP normally uses TCP port 443, which is
based VPN server behind a NAT PKI open in most router and NAT configurations.
device. The server is configured as Security-conscious Windows administrators
the domain controller and needs two Because SSTP uses HTTPS (port 443) might prefer to modify the standard port
network cards for the VPN setup. At to handle all the data traffic, there is used by SSTP. To do so, you need to edit the
following registry key
CurrentControlSet\Services\SstpSvc\ U
SSTP VPN connection (TCP port 443)
1. Open TCP connection
2. Set up SSL connection and validate certificate
in the Registry Editor. Look for the Lis-
tenerPort parameter. Changing the view
3. HTTPS request
to Decimal (right-click) lets you specify a
4. Initiate SSL tunnel different port. Then you need to restart the
Routing and RAS service. If you change the
5. Data communication via PPP
ListenerPort, you need to reconfigure
your NAT device to match and forward all
SSTP gateway server SSTP VPN client incoming traffic addressed to port 443 to
the newly configured port on the SSTP-based
Figure 1: The SSTP handshake is not much different from a standard SSL handshake. In contrast to IPsec, SSTP
VPN server.
sends PPP packets (not IP packets) through the tunnel.

w w w. A d m i n - m AgA z i n e .co m Admin 01 95

N u ts a n d B o lts VPNs with SSTP

Setting Up a Certification Authority ment console. Note that you cannot

install the Forefront 2010 server com-
To set up your own (private) certification authority in Active Directory, you need to launch the
ponent on a domain controller. Ad-
MMC console on your Windows server and add the certificate snap-in. Then, in the Certificate
ditionally, you will want to publish a
Snap‑In dialog box, select Computer account and Local computer for Select computer.
n To begin, right-click on Own certificate in the MMC, followed by All tasks | Request new cer- certificate revocation list in a produc-
tificate. tion scenario.
n You will see a selection of templates: Web Server will do the trick here. Now you need to
provide a name, such as fw.example.local. For this certificate, the name of the certificate Revocation
requestor must match the hostname of the device with which the VPN client will open the
connection. This step is essential for a successful SSL handshake. A Certificate Revocation List (CRL)
n If the request worked as intended, the certificate will install automatically and should be details any certificates revoked before
visible in the MMC console: Certificates snap-in below Certificates (local computer) | Own their expiry date. Revocation lists are
certificates. The CA certificate must also be visible in Certificates (local computer) | Trusted always available at CLR Distribution
root certificate authorities; you can check this if you experience difficulty with the certificate
Points (CDPs). A CDP can be set in
the Extensions tab of the CA Proper-
ties dialog box (Figure 3).
by the domain controller and placed this to happen, the firewall must also Additionally, the certificate revocation
in the client’s local certificate store be authorized to request certificates list must be accessible to all clients at
along with the certification authority (Figure 2). You can set this up under all times via the Internet, which will
certificate. If you are unable or prefer the Web Server Properties Security mean configuring the packet filter on
not to purchase a commercial certifi- tab; Read and Enroll privileges are the local firewall. In Forefront TMG
cate, you can use a private certificate required at minimum. 2010, you can use the website pub-
issued by Active Directory’s built-in In production, it often makes more lishing wizard for this.
CA. The “Setting up a Certification sense to purchase a commercial cer-
Authority” box explains how to do tificate and install a dedicated firewall VPN Server
this. server with Microsoft’s new Forefront
You need to configure the firewall to Threat Management Gateway (TMG) Setting up the VPN server itself is
allow network traffic to pass through 2010 [2]. easily done. Once you have added
to the certificate authority in order for The gateway offers a wizard-based the network policy and access ser-
the certificate request to work. To get configuration in the TMG manage- vices role by clicking Add roles in

Figure 2: If you are unable to request a certificate, you can check the privileges Figure 3: A CLR distribution point controls the availability of the certificate
in the certificate template for web servers to see whether Enroll is allowed. revocation list, which all clients need to be able to access at any time.

96 Admin 01 w w w. a d m i n - m aga z i n e .co m

VPNs with SSTP N u ts a n d B o lts

Figure 4: You need to set up the RAS services to run a VPN server.

the Customize this server section of DHCP messages

the Server Manager, the Routing and to RAS clients.
Remote Access tool is available below To do this, you
Management. The Action | Configure need to enable
and enable routing and RAS button the Relay DHCP Figure 5: Configuring Windows to forward DHCP messages to the RAS clients.
takes you to the Routing and Remote packets option in
Access Server Setup Wizard. the DHCP Relay Properties dialog box ets. In IPsec, ESP builds directly on
At the second Configuration step (Figure 5). IP. Microsoft is quite obviously seek-
(Figure 4), you’ll want to enable ing to set itself apart by encapsulating
Virtual private network (VPN) access Conclusions in PPP.
and NAT, then click Next and select Apart from the fairly complex Win-
the required network interface. At the Microsoft’s SSTP encapsulation struc- dows server configuration, which
following step, Address assignment ture is like a Russian doll. Just as mainly involves setting up the cer-
defines how the VPN server assigns with PPTP, Microsoft uses the PPP tificate services, and possibly packet
IP addresses to remote clients. If you protocol with SSTP, which leads to a filters for transporting the certificate
have the DHCP service running, Au- fairly complex encapsulation structure requests, SSTP offers a secure, well-
tomatic is the quickest and cleanest (see Figure 6), in which an IP header performing tunnel technology for the
option. Then you can define an IP contains a TCP header, which in turn future. n
address pool for the DHCP server to contains an SSTP header, which then
use in the next step, Address range contains a PPP header, which finally
assignment. contains the IP packets themselves. Info
In the final step, you can choose Although the method seems to be [1] Microsoft support for SSTP: [http://​
whether or not to use a Radius server slightly more efficient than Encapsu- ­support.​­microsoft.​­com/​­kb/​­947032/]
to authenticate clients on a large-scale lating Security Payload (ESP), with [2] TMG 2010: [http://​­www.​­microsoft.​­com/​
network; this is disabled by default. an overhead of 8 bytes in the PPP ­downloads/​­details.​­aspx?​­familyid=e05a
The wizard will then instruct you to header, compared with 20 bytes in ecbc‑d0eb‑4e0f‑a5db‑8f236995bccd&​
set up the DHCP relay agent for Win- IPsec over HTTPS, there is actually no ­displaylang=en]
dows to support the forwarding of real need to encapsulate in PPP pack-
The Author
Thomas Drilling has been a freelance journalist
and editor for scientific and IT magazines for
IPv4 or IPv6 TCP SSTP PPP IPv4 or IPv6 packet
more than 10 years. With his editorial office
team, he regularly writes on the subject of open
source, Linux, servers, IT administration, and
Mac OS X. In addition to this, Thomas Drilling is
Encapsulated SSL session
also a book author and publisher, a consultant
Figure 6: The SSTP encapsulation structure is like a Russian doll. Microsoft has gone to considerable trouble to small and medium-sized companies, and a
to make something proprietary from what are basically open protocols. From a technical point of view, there regular speaker on Linux, open source, and IT
seems to be no real reason to use PPP. security.

w w w. a d m i n - m aga z i n e .co m Admin 01 97

S E RV I C E Contact Info / Authors

Editor in Chief
Joe Casad,
Managing Editor
Rita L Sooby,
Contributing Editors
Admin: Network and Security is looking • unheralded open source utilities Oliver Frommel, Uli Bantle, Andreas Bohle,
for good, practical articles on system ad- • Windows networking techniques that Jens-Christoph Brendel, Hans-Georg Eßer,
Markus Feilner, Marcel Hilzinger, Mathias Huber,
ministration topics. We love to hear from aren’t explained (or aren’t explained Anika Kehrer, Kristian Kißling, Jan Kleinert,
IT professionals who have discovered well) in the standard documentation. Daniel Kottmair, Thomas Leichtenstern,
Jörg Luther, Nils Magnus
innovative tools or techniques for solving We need concrete, fully developed solu-
Localization & Translation
real-world problems. tions: installation steps, configuration Ian Travis
Tell us about your favorite: files, examples – we are looking for a Proofreading & Polishing
• interoperability solutions complete discussion, not just a “hot tip” Amber Ankerholz
• practical tools for cloud environments that leaves the details to the reader. Klaus Rehfeld, Judith Erb
• security problems and how you solved If you have an idea for an article, send Cover
them a 1-2 paragraph proposal describing your Illustration based on graphics by James Thew,
• ingenious custom scripts topic to:
United Kingdom and Ireland
Penny Wilby,
Phone: +44 1787 211 100
North America
Amy Phalen,
Phone: +1 785 856 3434
All other countries
Hubert Wiest,
Phone: +49 89 9934 1123
Corporate Management (Vorstand)
Hermann Plank,
Brian Osborn,
Management North America
Brian Osborn,
Associate Publisher
Rikki Kite,
Product Management
Hans-Jörg Ehren,
Customer Service / Subscription
For USA and Canada:
AUTHORS Phone: 1-866-247-2802
(Toll Free from the US and Canada)
Falko Benthin 14 Fax: 1-785-274-4305
For all other countries:
Björn Bürstinghaus 25, 60 Email:
Phone: +49 89 9934 1167
Thomas Drilling 52, 94 Fax: +49 89 9934 1199
Admin Magazine • c/o Linux New Media •
Florian Effenberger 42 Putzbrunner Str 71 • 81739 Munich • Germany
Dan Frost 48 While every care has been taken in the content of the
magazine, the publishers cannot be held responsible for
Thomas Joos 74 the accuracy of the information contained within it or any
consequences arising from the use of it. The use of the
DVD provided with the magazine or any material provided
Daniel Kottmair 64 on it is at your own risk.
Copyright and Trademarks © 2010 Linux New Media Ltd.
Caspar Clemens Mierau 20 No material may be reproduced in any form whatsoever
in whole or in part without the written permission of the
James Mohr 28 publishers. It is assumed that all correspondence sent, for
example, letters, emails, faxes, photographs, articles, drawings,
are supplied for publication or license to third parties on a
Thorsten Scherf 8, 44, 78 non-exclusive worldwide basis by Linux New Media unless
otherwise stated in writing.
Tim Schürmann 68 Printed in Germany
Distributed by COMAG Specialist, Tavistock Road,
Udo Seidel 36 West Drayton, Middlesex, UB7 7QE, United Kingdom
Admin Magazine ISSN 2045-0702
Kurt Seifried 34 Admin Magazine is published by Linux New Media USA, LLC,
719 Massachusetts, Lawrence, KS 66044, USA, and Linux
Sebastian Wolfgarten 86 New Media Ltd, Manchester, England. Company registered in
Harald Zisler 92 Linux is a trademark of Linus Torvalds.

98 ADMIN 01 W W W. A D M I N - M AGA Z I N E .CO M





£59 £89 £129 £189 £299

DX-250 per month
D3-240 per month
D3-340 per month
D3-520 per month
D3-620 per month

PROCESSOR Intel® Xeon Intel® Xeon Intel® Xeon Intel® Xeon Intel® Xeon
CORES Quad 2.4Ghz Quad 2.66Ghz Quad 2.66Ghz Quad 2.26Ghz 2 x Quad 2.26Ghz
HARD DISKS 1 x 160GB SATA 2 x 250GB SATA 2 x 500GB SATA 2 x 1000GB SATA 4 x 1000GB SATA
RAID None Hardware RAID Hardware RAID Hardware RAID Hardware RAID
CONNECTION 100 Mbps 100 Mbps 100 Mbps 100 Mbps 100 Mbps




CALL FREE ON: 0800 622 6655 OR VISIT: