Oracle Linux DBA Benefits

Don Burleson

Oracle database administration in a Linux environment is quite different from

Windows. Linux offers a robust command-line interface with a wealth of functions
that go far beyond the limited capabilities of a GUI interface.

Because Linux is open-source, all of the low-level communications between the

software and the hardware are exposed and the Oracle DBA can see virtually every
aspect of Oracle’s behavior as it interfaces with hubs, routers, RAM and disks.

Linux is also very simple in structure. Many of the Linux commands are compiled C
programs, and we have the opportunity to go “behind the scene” and see the
source code behind many Linux commands.

Linux Utilities for the Oracle DBA

Because of its simple structure and low-level utilities, Linux offers the ability to
monitor deep into the environment and examine every aspect of Oracle operations.
Some of these tools include:

vmstat – Provides real-time reports on CPU consumption, CPU dispatcher run queue,
RAM page in, scan rate, RAM page outs

iostat – Provides detailed disk I/O information including disk enqueues and I/O
latency

nestat – Shows TCP/IP traffic and packet details

watch – Shows top server task details including server load average

ps – Shows process-level details on resource consumption

These utilities are exceptional for monitoring Oracle databases and they can be
placed inside Linux shell scripts to take the output from the Linux commands and
store the information inside Oracle tables.

For example, here is a utility that will take the output from Linux vmstat utility and
insert the output into an Oracle table:

PATH=$ORACLE_HOME/bin:$PATH

export PATH

SERVER_NAME=`uname -a|awk '{print $2}'`

typeset -u SERVER_NAME

export SERVER_NAME

# sample every five minutes (300 seconds) . . . .

SAMPLE_TIME=300

while true

do

vmstat ${SAMPLE_TIME} 2 > /tmp/msg$$

# run vmstat and direct the output into the Oracle table . . .

cat /tmp/msg$$|sed 1,3d | awk '{ printf("%s %s %s %s %s %s\n", $1, $8, $9,

14, $15, $16) }' | while read RUNQUE PAGE_IN PAGE_OUT USER_CPU
SYSTEM_CPU
 IDLE_CPU

do

$ORACLE_HOME/bin/sqlplus -s perfstat/perfstat@testsys1<<EOF

insert into perfstat.stats\$vmstat

values (

sysdate,

$SAMPLE_TIME,

'$SERVER_NAME',

$RUNQUE,

$PAGE_IN,

$PAGE_OUT,

$USER_CPU,

$SYSTEM_CPU,

$IDLE_CPU,

);

EXIT

EOF

done

done

rm /tmp/msg$$

Note: This type of operating system script is largely obsolete in Oracle10g because
the Automated Workload Repository (AWR) captures this OS information
automatically.
 We can use the same script technique with the Linux iostat utility and shell scripts
can be used to monitor disk I/O and place the details inside Oracle tables.

#!/bin/ksh

while true

do

iostat -x 300 1|\

sed 1,2d|\

awk '{ printf("%s %s %s\n", $1, $4, $5) }' |\

while read HDISK VMSTAT_IO_R VMSTAT_IO_W

do

if [ $VMSTAT_IO_R -gt 0 ] and [ $VMSTAT_IO_W -gt 0 ]

then

sqlplus -s perfstat/perfstat <<!

insert into

perfstat.stats\$iostat

values

(SYSDATE, 5, '$HDISK', $VMSTAT_IO_R,$VMSTAT_IO_W);

exit

fi

done

sleep 300

done
 As we see, the easy-to-use Linux monitoring utilities open-up a whole new world to
the Oracle DBA. If you take the time to master the Linux command syntax and
utilities you will gain the ability to diagnose any Oracle problem and always know
the exact cause of any OS-related performance issue. Now let’s take a look at some
of the command-line utilities for Linux and see how they can simplify the tasks of
the Oracle DBA.

Command Utilities for Linux

As we have noted, Linux has powerful command-line utilities that provide

exceptional monitoring for every aspects of Oracle’s interactions with the hardware
environment. But there is more to Linux than the commands. The Oracle
professional will also need to master shell scripting, task scheduling and specialized
functions:

The vi file editor – The vi editor is an extremely powerful text editor that is far more
powerful and flexible than Windows-based editors such as notepad.

Utility commands – Linux provides powerful system tools such as grep, sed and awk
that allow you to create comprehensive OS monitoring facilities.

Job scheduling - The Linux crontab utility allows for scheduling OS tasks at specific
time intervals.

Scripting - Shell environments such as Korn shell, C shell, Bourne shell and Bourne
Again shell (bash) allow you to create sophisticated Linux monitoring scripts.

Linux tools are very powerful and flexible, and like any powerful tool, they are
complex. It can take years for the staff to master Linux internals, using the powerful
vi editor, and learning to program with UNIX shell scripts. Let’s take a quick look at
the types of scripts that a Linux DBA might employ for their Oracle environment.
 There are many OS events that need to be monitored by the Oracle DBA and prior
to Oracle10g the DBA would want to be notified of any server-side event.

Operating System Events:

• File system is filling rapidly

• Server is paging RAM

• High CPU enqueues

• Messages in Linux syslog (I/O errors, bus errors)

Oracle OS events:

• Alert log message

• Core dump

• Background dump

• User dump created

For example, here is an example of an actual UNIX script for Oracle to detect a
almost-full filesystem:

#!/bin/ksh

for i in `df -k|grep /u0|awk '{ print $4 }'`

do

filesize=`expr i`

if [ $filesize -lt 100 ]

then
 mailx -s "Oracle filesystem $i has less than 100k free."\

bill.gates@mircosoft.com\

lawrence_ellison@oracle.com

fi

done

One of the most powerful utilities is grep which is used to quickly find files that
contain specific strings. In fact, some Oracle DBA’s using Windows will download a
free copy of grep for PC’s because of its power and ease of use. For example, the
scripts below will cascade through the Linux directory structures, quickly finding all
files that contain a specific string:

find . –print|xargs grep –i string

This script is far faster and easier to use than the Windows GUI search tool.

Linux Configuration

Configuring Linux for optimal performance is highly-dependent on the hardware.

When you use a server like the UNISYS ES7000, the configuration is easy because
the Linux kernel has already been optimized for the specific high-speed hardware
components.

Because UNISYS Linux for Oracle is preconfigured for Oracle, you avoid the
headaches of optimizing the OS for the Oracle software. In proprietary UNIX, proper
configuration for Oracle is often overlooked and SUN, HP and AIX Oracle databases
are often improperly configured.

For example, Oracle data files should be configured to bypass the UNIX server
buffer cache and read the data blocks directly into the Oracle data buffers.
However, a surprising number of proprietary UNIX shops have their I/O improperly
configured, causing multiple buffer access whereby data blocks travel onto the UNIX
Journaled File System (JFS) buffer cache, and then transferred into the Oracle data
buffers.

This additional I/O overhead is a major problem and one that is automatic in Linux.
Linux systems support direct I/O on a per-filehandle basis (which is much more
flexible), and Oracle enables this O_DIRECT feature automatically. This is not true
for those using Sun, HP and Veritas I/O, and tricky utilities must be implemented to
enable direct I/O:

Solaris – Uses the "forcedirectio" option. Oracle DBAs claim this option makes a
huge difference in I/O speed for Sun servers.

AIX - Uses the "dio" option.

Veritas VxFS - (including HP-UX, Solaris and AIX), uses the "convosync=direct". It is
also possible to enable direct I/O on a per-file basis using Veritas QIO; refer to the
"qiostat" command and corresponding man page for hints.

In sum, using Linux with a hardware vendor such as UNISYS ensures that the Linux
environment is optimized to run as quickly as possible. Now let’s look at the issue of
migrating to Linux from Windows and proprietary UNIX.

Migrating to Linux from Proprietary UNIX

Migrating into Linux can be very easy or very difficult, depending on your existing
Oracle configuration. It is not uncommon to see small “dialect” differences between
implementations of UNIX, especially with respect to display output and command
arguments.

Those shops that find a migration difficult are those that have made extensive use
of operating system utilities for their Oracle functions:
External scheduling of Oracle tasks

– Crontab

– Windows “at” scheduling

Extensive use of OS shell scripts

– Automated e-mail alerts (new dumps, alert log messages, Linux syslog
messages)

– File system alerts

– Server overload alerts

If you avoid the OS trap and use Oracle to schedule all Oracle tasks you can easily
port your database across OS platforms. This is especially true if you migrate to
Oracle10g where Oracle automatically collects OS metrics and has a built-in alert
and scheduling mechanism:

Migrate to Oracle10g

Schedule all Oracle tasks with dbms_scheduler

Use the Oracle dbms_alert package to replace OS scripting

Let’s take a closer look at the syntax differences between proprietary UNIX and
Linux so that you can appreciate the challenge of migration.

Linux command syntax

Users of proprietary UNIX will immediately recognize many similarities between

Sun, HP, AIX and Linux commands. However, there are significant syntax issues,
especially with regard to command arguments and the output from Linux utilities:
Linux Command Syntax examples

Display the number of CPUs cat /proc/cpuinfo|grep processor|wc –l

Show top CPU% ps aux|sort -n +2

Display top-10 CPU consumers ps aux|sort -rn +2|head -10

RAM memory display free

Shutdown server as root /sbin/shutdown -r now

Kill all xxx processes pkill [-9] “xxx”

Show swap paging space /sbin/swapon -s

Show Linux syslog errors tail /var/log/messages

Show swap disk details swapon -s

See held memory segments ipcs -m

Show Linux system parms sysctl -a

Linux command history files history|more

Migrating from Sun to Linux can be especially problematic because of the syntax
differences and the different output from the utilities such as vmstat and netstat.
This can cause considerable re-writing of shell scripts.

vmstat Linux:

>vmstat 2 5

procs memory swap io system cpu

r b w swpd free buff cache si … bi bo in cs us sy id

1 0 0 140 90372 726988 26228 0 … 0 0 14 7 0 0 4

0 0 0 140 90372 726988 26228 0 … 0 2 103 11 0 0 100

vmstat Solaris:

>vmstat 2 5

procs memory  page disk faults cpu

r b w swap free re mf pi po … s6 -- -- in sy cs us sy id

0 0 0 2949744 988800 0 4 0 0 … 0 0 0 148 200 41 0 0 99

0 0 0 2874808 938960 27 247 0 1 … 0 0 0 196 434 64 1 2 98

Display number of CPU’s:

Linux:

>cat /proc/cpuinfo|grep processor|wc -l

16

Solaris:

>psrinfo -v|grep "Status of processor"|wc -l

RAM Size in Linux:

>free

total used free shared buffers cached

Mem: 3728668 504688 3223980 41316 430072 29440

-/+ buffers/cache: 45176 3683492

Swap: 265032 608 264424

RAM Size in Solaris:

>prtconf|grep -i mem

Memory size: 2048 Megabytes

memory (driver not attached)

virtual-memory (driver not attached)

Netstat Differences

On any Sun and Linux server the netstat utility provides information about all
network traffic touching the server. However the output from the

Solaris netstat

>netstat
TCP: IPv4

Local Address Remote Address   Swind Send-Q Rwind Recv-Q State

------------- ------------------- ----- ------ ----- ------  -----------

sting.32773 ting.1521 32768 0 32768 0  ESTABLISHED

sting.1521 ting.32773 32768 0 32768 0  ESTABLISHED

sting.32774 ting.1521 32768 0 32768 0  ESTABLISHED

Linux netstat

In Linux, we see that the output from netstat is quite different from Solaris:

>netstat

Proto Recv-Q Send-Q Local Address       Foreign Address State

tcp 0     0     donsrv1.rov:netbios-ssn intranet.janet.com:1351 ESTABLISHED

tcp 0     0     donsrv1.janet.com:1120 sting.janet.com:ssh TIME_WAIT

tcp 0     40    donsrv1.janet.com:ssh h pop3-146.gloryroa:1096 ESTABLISHED

Ironically, the syntax differences between proprietary UNIX and Linux can hinder
your migration and shops that use SFU on Windows often have a far easier
migration. In sum, those Oracle shops that avoid OS utilities such as scheduling
(crontab) and shell scripts will find database migration very easy.

2. Linux commands for Oracle DBA

Posted by Kamran Agayev A. on February 22, 2009

Today I would like to introduce you some articles related to advanced Linux
commands. These commands will be useful when working with Oracle Database

There’re 5 articles that I would like to introduce you