Performance Monitoring on AIX

by
Peter Lee
IT Specialist
Performance Tuning Process

Indentify Workloads

Define and prioritize goals

Identify the required resoureces

Minimize resource requirements

Control the allocation of resource

Apply additional resources as indicated

Performance Analysis Tools
Memory Network
CPU I/O Subsystem
Subsystem Subsystem
vmstat, iostat vmstat iostat lsattr
ps lsps vmstat netstat
sar svmon lsps nfsstat
gprof/prof/tprof filemon lsattr entstat
time/timex bf, bfrpt lsdev netpmon
netpmon lspv/lsls/lsvg ifconfig
stem fileplace iptrace/ipreport
syscalls filemon tcpdump
lockstat
emstat
performance performance performance performance
toolbox toolbox toolbox toolbox
trace, trcrpt, utld trace, trcrpt trace, trcrpt trace, trcrpt
Performance Tuning Tools

Memory Network
CPUs I/O Subsystem
Subsystem Subsystem

nice/renice vmtune vmtune no

schedtune chps/mkps chdev nfso
bindprocessor fdpr migratepv chdev
chdev chdev chlv ifconfig
setpri rmss reorgvg
Performance Tuning Flowchart
YES Is System
CPU-Bound?

Actions NO

YES Is System NO Is System YES

Memory-Bound? Disk-Bound?

Actions NO Actions

YES Is System NO
Network-Bound?

Actions Additonal

Actions
vmstat Command
vmstat Command Reports:
Summary of paging and system events since initialization (-s)
Number of forks since initilization (-f)
Device interrupts since initilization or per interval (-i)
Periodic reports of processes, paging, real memory, and CPU
Usage : vmstat [ -ifs ] [drives] [interval] [count]

vmstat 1 10
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
0 0 2514 24166 0 0 0 0 0 0 121 1 18 0 0 99 0
0 0 2514 24165 0 0 0 0 0 0 126 178 19 1 0 99 0
0 0 2514 24165 0 0 0 0 0 0 118 60 24 0 0 99 0
0 0 2514 24165 0 0 0 0 0 0 114 54 20 0 0 99 0
0 0 2514 24165 0 0 0 0 0 0 121 117 21 0 0 99 0
0 0 2514 24165 0 0 0 0 0 0 119 54 20 0 1 99 0
0 0 2514 24165 0 0 0 0 0 0 126 54 22 1 0 99 0
0 0 2514 24165 0 0 0 0 0 0 127 54 19 0 1 99 0
0 0 2514 24165 0 0 0 0 0 0 125 54 21 0 0 99 0
0 0 2514 24165 0 0 0 0 0 0 127 54 18 0 0 99 0
vmstat Command

root@bmserver[/tmp]>vmstat -i

priority level type count module(handler)

0 0 hardware 0 i_misc_pwr(b0e5c)
0 24 hardware 0 i_epow(9d820)
0 24 hardware 0 /etc/drivers/hscsidd(18bc578)
0 48 hardware 0 i_scu(b0e50)
1 2 hardware 66 /etc/drivers/rsdd(196de4c)
3 4 hardware 3 /etc/drivers/middd_loadpin(19ce63c)
3 5 hardware 1736 /etc/drivers/hscsidd(18bc56c)
3 7 hardware 21636015 /etc/drivers/tokdd(19e78fc)
3 14 hardware 54189 /etc/drivers/hscsidd(18bc56c)
5 62 hardware 117295403 clock(9d670)
10 63 hardware 7647 i_softoff(9d628)
vmstat Command
root@bmserver[/tmp]>vmstat -s

210376 total address trans. faults

5724 page ins
55944 page outs
0 paging space page ins
0 paging space page outs
0 total reclaims
78419 zero filled pages faults
1551 executable filled pages faults
0 pages examined by clock
0 revolutions of the clock hand
0 pages freed by the clock
8790 backtracks
0 lock misses
0 free frame waits
0 extend XPT waits
3760 pending I/O waits
56326 start I/Os
56326 iodones
21347782 cpu context switches
139338639 device interrupts
0 software interrupts
0 traps
1873766 syscalls
vmstat Command
fre : the number of pages of real memory on the free list at the time of the interval sample

re : the number of page reclaims per second observed in the sample interval. If a page fault occurs
and this page is currently on the freelist and has not yet been reused, this is considered a reclaim
since no new I/O request has to be initiated. This is not to be confused with the term repage which
refers to a page which has already incurred a page-fault (the page could be currently on the freelist,
filesystem, or in paging space).

pi or po : the number of page ins or page outs per second from/to paging space during the sample
interval

fr/sr : the number of pages per second that were freed/examined by the page replacement algorithm
during the interval

cy : the number of cycles per second of the clock algorithm

in/sy/cs : the number of device interrupts/system calls/thread context switches per second observed
in the interval

CPU : CPU utilization observed in the interval, divided into us-user, sy-system, id-idle, wa-wait

disk xfer the number of disk requests per second observed to the PV
topas Command
iostat Command

iostat Command Reports:

CPU utilization
is %user + %system approaching 100%?
Terminal I/O
Disk I/O Balanced?
CD-ROM I/O

Hardware Dependent

Local disk information only

No special authority required

iostat Command

usage: iostat [-t] [-d] [drives] [interval [count]]

-t specifies tty/cpu report only
-d specifies drive report only
-t and -d cannot both be specified

root@bmserver[/tmp]>iostat

tty: tin tout avg-cpu: % user % sys % idle

% iowait
0.0 0.5 0.0 0.1 99.8
0.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtn

hdisk0 0.1 0.2 0.0 40264 220277
hdisk1 0.0 0.0 0.0 0 0
cd0 0.0 0.0 0.0 500 0
sar Command

sar Command

Data Types
All data types collected; only specified data types reported on

Timing Intervals: the extraction interval can differ from the

collection interval

Permissions: need to be in system group

sadc: back end for sar, responsible for the data collection

timex -s: displays sar statistics for the duration of the command
executed by timex
sar Command

There are two forms of the sar command:

the first form is used to sample, display, and/or save
systemstatistics;
the second form is used to process and display
previously-captured data.

e.g
Collection: sar -o filename 60 300 > /dev/null &
Extraction: sar -A -f filename
Extraction: sar -u -f filename -s 10:20 -e 14:50
Monitoring: sar -c 60 300
sar Command

Data Types:
-a file access system routines
-b buffer activity
-c system calls
-k kernel process statistics
-m message and semaphore activity
-q statistics on run queue and wait queue
-r paging statistics
-u CPU utilization
-v kernel tables: i-node, process and file
-w process switching activity
-y reports on tty statistics
-A Show all of the above
-P [processor_id[,...] | ALL] Show individual processor
utilization on SMP systems
sar Command
# sar -u 5 3
19:28:02 %usr %sys %wio %idle
19:28:07 0 6 0 94
19:28:12 1 4 0 95
19:28:07 7 4 0 90
Average 3 4 0 93

# sar -c 1 3
19:28:25 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
19:28:26 134 36 1 0.00 0.00 2691306 1517
19:28:27 46 34 1 0.00 0.00 2716922 1531
19:28:28 46 34 1 0.00 0.00 2716922 1531
Average 75 35 1 0.00 0.00 2708329 1527

# sar -q 5 3
19:31:42 runq-sz %runocc swpq-sz %swpocc
19:31:47 1.0 100 1.0 100
19:31:52 2.0 100 1.0 100
19:31:57 1.0 100 1.0 100
Average 1.3 95 1.0 95
ps Command

ps Command
The ps command writes the current status of active processes
and (if the -m flag is given) associated kernel threads to
standard output.
A new option to ps in AIX Version 4 shows if a process is
bound to a particular CPU. This option is -o THREAD. If you
see the -m option as well (-mo THREAD), then you can see the
thread or threads that belongs to each process.
Usage : ps [-Aaedfklm] [-n namelist] [-F Format] [-o
specifier[=header],...] [-p proclist][-G|-g grouplist] [-t termlist]
[-U|-u userlist]
Usage: ps [aceglnsuvwxU] [t tty] [processnumber]
ps Command

root@bmserver[/tmp]>ps -el
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
200003 A 0 1 0 0 60 20 2805 252 - 0:00 init
240001 A 0 2548 1 0 60 20 8831 68 5a6cf58 - 2:32 syncd
40401 A 0 2814 1 0 60 20 1923 276 95378 - 0:00 errdemon
240001 A 0 2902 1 0 60 20 921 244 - 0:00 srcmstr
240001 A 0 3528 1 0 60 20 d21a 168 54aeccc - 0:13 cron
240001 A 0 3694 2902 0 60 20 59eb 608 - 0:00 sendmail
240001 A 0 3968 2902 0 60 20 c9f9 984 - 0:42 snmpd
240001 A 0 4218 2902 0 60 20 a1f4 172 - 0:00 inetd
240001 A 0 4468 2902 0 60 20 79ef 292 - 0:00 portmap
240001 A 0 4712 2902 0 60 20 9e1 128 - 0:02 syslogd
240001 A 0 4998 2902 0 60 20 2204 468 - 0:00 dpid2
240001 A 0 5866 8424 0 60 20 5acb 348 pts/0 0:00 ksh
240001 A 0 6054 2902 0 60 20 8a11 68 - 0:00 biod
240001 A 0 6326 2902 0 60 20 4208 144 - 0:00 rpc.statd
240001 A 0 6598 2902 0 60 20 f21e 68 * - 0:03 rpc.lockd
240001 A 0 6858 1 0 60 20 1a23 32 9ec0170 - 0:00 install_as
200001 A 0 6932 5866 11 65 20 7acf 188 pts/0 0:00 ps
200001 A 0 8424 4218 2 61 20 82d0 268 - 0:00 telnetd
ps Command
root@bmserver[/tmp]>ps -emo THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
root 1 0 - A 0 60 1 - 200003 - 0 /etc/init
- - - 463 S 0 60 1 - 400 - 0 -
root 2548 1 - A 0 60 1 5a6cf58 240001 - 0 /usr/sbin/syncd 60
- - - 3325 S 0 60 1 5a6cf58 400 - 0 -
root 2814 1 - A 0 60 1 95378 40401 - 0 /usr/lib/errdemon
- - - 3335 S 0 60 1 95378 400 - 0 -
root 2902 1 - A 0 60 1 - 240001 - 0 /usr/sbin/srcmstr
- - - 2399 S 0 60 1 - 8400 - 0 -
root 3528 1 - A 0 60 1 54aeccc 240001 - 0 /usr/sbin/cron
- - - 3795 S 0 60 1 54aeccc 400 - 0 -
root 3694 2902 - A 0 60 1 - 240001 - 0 sendmail: accepting
connections
- - - 4727 S 0 60 1 - 8400 - 0 -
root 3968 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/snmpd
- - - 4233 S 0 60 1 - 8400 - 0 -
root 4218 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/inetd
- - - 4483 S 0 60 1 - 8400 - 0 -
root 4468 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/portmap
- - - 4989 S 0 60 1 - 8400 - 0 -
root 4712 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/syslogd
- - - 5233 S 0 60 1 - 8400 - 0 -
root 4998 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/dpid2
- - - 5519 S 0 60 1 - 8400 - 0 -
root 5866 8424 - A 0 60 1 - 240001 pts/0 0 -ksh
- - - 8185 S 0 60 1 - 400 - 0 -
root 6054 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/biod 6
- - - 6833 S 0 60 1 - 400 - 0 -
root 6326 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/rpc.statd
- - - 7105 S 0 60 1 - 8400 - 0 -
root 6598 2902 - A 0 60 3 * 240001 - 0 /usr/sbin/rpc.lockd
- - - 7377 S 0 60 1 9f64c98 400 - 0 -
- - - 7639 S 0 60 1 54aa988 1400 - 0 -
- - - 7897 S 0 60 1 2735dc 1400 - 0 -
root 7008 5866 - A 12 66 1 - 200001 pts/0 0 ps -emo THREAD
- - - 9071 R 12 66 1 - 0 - 0 -
ps Command
root@bmserver[/tmp]>ps aux
USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND
root 5522 0.0 0.0 8 16 - A Aug 20 0:00 kproc
root 5866 0.0 1.0 348 408 pts/0 A 06:01:57 0:00 -ksh
root 6054 0.0 0.0 68 96 - A Aug 20 0:00 /usr/sbin/biod 6
root 5264 0.0 0.0 8 12 - A Aug 20 0:01 kproc
root 4468 0.0 1.0 292 360 - A Aug 20 0:00 /usr/sbin/portmap
root 4712 0.0 0.0 128 172 - A Aug 20 0:02 /usr/sbin/syslogd
root 4998 0.0 2.0 468 548 - A Aug 20 0:00 /usr/sbin/dpid2
root 7308 0.0 0.0 8 12 - A Aug 30 0:00 kproc
root 6598 0.0 0.0 68 108 - A Aug 20 0:03 /usr/sbin/rpc.lock
root 8424 0.0 1.0 268 324 - A 06:01:57 0:01 telnetd
root 7534 0.0 1.0 188 232 pts/0 A 07:27:18 0:00 ps aux
root 6326 0.0 1.0 144 196 - A Aug 20 0:00 /usr/sbin/rpc.stat
root 6858 0.0 0.0 32 44 - A Aug 20 0:00 /usr/sbin/install_
root 7024 0.0 1.0 208 252 pts/0 A 07:27:18 0:00 more
root 1032 0.0 0.0 40 48 - A Aug 20 2:25 kproc
root 1896 0.0 0.0 8 12 - A Aug 20 0:00 kproc
root 2228 0.0 0.0 8 12 - A Aug 20 0:00 kproc
root 774 0.0 0.0 8 12 - A Aug 20 0:25 kproc
root 0 0.0 0.0 4 8 - A Aug 20 2:51 swapper
root 1 0.0 1.0 252 304 - A Aug 20 0:00 /etc/init
root 516 99.9 0.0 0 4 - A Aug 20 19067:57 kproc
emstat Command

Instruction Emulation can be determined

Different processors have different instruction sets

PowerPC architecture deletes about 35 POWER

instruc-tions

To maintain binary compatibility, deleted

instructions are emulated

Instruction emulation increases the pathlength for

the instruction

Emulation can be avoided by compiling applications

using Common Mode option
emstat Command

New tool to detect instruction emulation - shipped

with bos.adt.samples

Usage : /usr/samples/kernel/emstat [interval] [count]

# /usr/samples/kernel/emstat 1 7
emstat total count emstat interval count
21951937 21951937
21967488 15551
21974877 7389
21994354 19477
22007329 12975
22018171 10842
22018171 0
Scheduling Policies

SCHED_FIFO
Non-preemptive scheduling
Fixed piority

SCHED_RR
Round-Robin scheduling
Fixed piority
Run until interrupted or end of time slice

SCHED_OTHER
Default scheduling policy
Same as SCHED_RR but applies to threads with
non-fixed piorities
Piority degrades with CPU Usage
Process and Thread Piority
Process Piorities
High = 0

setpri( )
Real-Time Process Piorities
40 User Process Piorities

Parent fork() Child Process

renice
nice
Process Inherited /Threads
Process
Piority

Low = 127
Initial PRI = NICE + 40
pe -el (to view values)
ps -emo THREAD (to view threads)
nice/renice Commands

Initial priority of non-fixed priority thread is:

40 + NICE
where NICE has a default value of 20. NICE can range from
0-40.

Nice value can be set at thread creation time by using the nice
command. Syntax of nice:
usage: nice [-n increment] command
usage: nice [-increment] command

Nice value can be changed after thread has already been created
using the renice command. Syntax of renice:
usage: renice [-n increment] [ -g| -p| -u] ID...
Nice values are reduced by using negative increment values
and increased by using positive increment values. Reducing
nice value makespriority more favored. Increasing nice value
makes priority less favored.
schedtune Command

Modifies/Disables Memory Load Control

Modifies CPU scheduling algorithms

Modifies CPU time slices

Restores defaults

Must repeat at every boot

schedtune Command
# /usr/samples/kernel/schedtune

THRASH SUSP FORK SCHED

-h -p -m -w -e -f -d -r -t -s
SYS PROC MULTI WAIT GRACE TICKS SCHED_D SCHED_R TIMESLICE MAXSPIN
6 4 2 1 2 10 16 16 1 1

Change the Scheduling policy from default to Round-Robin

using the following command:
/usr/samples/kernel/schedtune -r 0 -d 0

Each timeslice is in units of clock ticks (10ms). Default timeslice

is 1 clock tick and is tunable using
/usr/samples/kernel/schedtune -t <# of ticks>
bindprocessor Command

bindprocessor Process [ ProcessorNum ] | -q | -u Process

Purpose

Binds or unbinds the kernel threads of a process to a processor.

svmon Command

svmon Command is an AIX Specific analysis tool for

virtual memory

Purpose:
Captures a snapshot of the current state of
memory.
The display of information can be analyzed using
four different reports:
global [ -G]
process [-P]
segment [-S]
detailed segment [-D]

In AIX Version 4, svmon is packaged as part of

perfagent.tools fileset.
svmon Command
usage: svmon
[ -G ]
[ -P[n|s|a] [ pid1 ... pidN ] ]
[ -P[n|s|a][u|p|g|r] [ count ] ]
[ -S sid1 ... sidN ]
[ -S{[n|s|a][u|p|g|r]} [ count ] ]
[ -D sid1 ... sidN ]
[ -i interval [ numintervals ] ]
[ -r ]

# svmon -G -i 5 3
m e m o r y i n u s e p i n p g s p a c e
size inuse free pin work pers clnt work pers clnt size inuse
49152 48134 1018 2613 10674 37460 0 2613 0 0 98304 8104
49152 48139 1013 2613 10678 37461 0 2613 0 0 98304 8105
49152 48139 1013 2613 10678 37461 0 2613 0 0 98304 8105
svmon Command
1. To print out global statistics:
$ svmon -G

2. To print out the memory usage statistics for all processes (limit statistics to
non-system segments):
$ svmon -Pn

3. To print out the memory usage statistics for processes 6746 and 10078 (include
statistics on all segments):
$ svmon -Pa 6746 10078

4. To sort system segments by the number of pages in real memory, and print out
the top 10 segments of the resulting list:
$ svmon -Ssu 10

5. To print detailed memory usage statistics on segment 340d:

$ svmon -D 340d

6. To print out the global statistics, repeated 10 times at 30 second intervals:

$ svmon -G -i 30 10
rmss Command

Reduced Memory System Simulator

Simulate RISC System/6000 with various size memories

Run an application over a range of memory sizes

rmss is packaged as part of perfagent.tools

usage: rmss [-s startmemsize] [-f finalmemsize] [-d deltamemsize]

[-n numiterations] [-o outputfile] command
rmss -c memsize
rmss -r
rmss -p
rmss Command
To change memory size
# rmss -c 32
Simulated memory size changed to 32 Mb.

To display the current memory size:

# rmss -p
Simulated memory size is 32 Mb.

To reset the memory size to the real size:

rmss -r
Simulated memory size changed to 192 Mb.

To measure the performance of the command cc -O foo.c

on memory size 32, 24, 16, 8MB, run and measure the
command once at each memory size and write the report to
the file cc.rmss.out
rmss -s 32 -f 8 -d 8 -n 1 -o cc.rmss.out cc -O foo.c
filemon Command
The filemon tool collects and presents trace data on the
various layers of file system utilization including the logical
file system, virtual memory segments, LVM, and physical
disk layers. Data can be collected on all the layers, or
layers can be specified with the -O layer option. The
default is to collect data on the VM, LVM, and physical
layers.

filemon will only collect data for those files opened after
filemon was started unless you specify the -u flag.

Normally, only the top 20 logical files and segments are

reported unless the -v (verbose) flag is used.
filemon Command

Usage: filemon [-i file] [-o file] [-d] [-v] [-u] [-O opt]
-i file: open input file (default is real-time trace)
-o file: open output file (default is stdout)
-d: deferred trace (until ‘ trcon’ )
-T num: set trace kernel buf sz (default 32000 bytes)
-P: pin monitor process in memory
-v: verbose mode (print extra details)
-u: print unnamed file activity via pid
-O opt: other monitor-specific options

valid -O options: lf,vm,lv,pv,all

lf: monitor logical file I/O
vm: monitor virtual memory I/O
lv: monitor logical volume I/O
pv: monitor physical volume I/O
all: short for lf,vm,lv,pv

(default is: vm,lv,pv)

filemon Command

# filemon -O lv,pv -o fmon.out

# dd if=/dev/rhdisk1 of=/dev/null bs=512 count=1024
# cp /unix /tstc/junk
# trcstop
# more fmon.out
Fri Aug 2 15:11:21 1996
System: AIX magic Node: 4 Machine: 000001455C00
5.450 secs in measured interval
Cpu utilization: 72.6%
Most Active Logical Volumes
------------------------------------------------------------------------
util #rblk #wblk KB/s volume description
------------------------------------------------------------------------
0.36 0 1588 145.7 /dev/lv04 /tstc Compressed: Frag_Sz.=1024
0.00 0 16 1.5 /dev/loglv00 jfslog
Most Active Physical Volumes
------------------------------------------------------------------------
util #rblk #wblk KB/s volume description
------------------------------------------------------------------------
0.75 1024 1604 241.1 /dev/hdisk1
netstat Command
netstat Command
Shows state of interfaces
Shows interface statistics
Shows routing statistics
Shows per-protocol statistics
Shows network device driver statistics
Shows network memory usage statistics
netstat Command
root@bmserver[/tmp]>netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lo0 16896 <Link> 473 0 473 0 0
lo0 16896 127 loopback 473 0 473 0 0
tr0 1492 <Link>10.0.5a.c9.62.a4 10832411 0 22191 0 0
tr0 1492 9.184.64 bmserver 10832411 0 22191 0 0

root@bmserver[/tmp]>netstat -I tr0 1
input (tr0) output input (Total) output
packets errs packets errs colls packets errs packets errs colls
10832898 0 22267 0 0 10833371 0 22740 0 0
3 0 1 0 0 3 0 1 0 0
7 0 1 0 0 7 0 1 0 0
4 0 1 0 0 4 0 1 0 0
5 0 1 0 0 5 0 1 0 0
10 0 1 0 0 10 0 1 0 0
7 0 1 0 0 7 0 1 0 0
6 0 1 0 0 6 0 1 0 0
11 0 1 0 0 11 0 1 0 0
10 0 1 0 0 10 0 1 0 0
9 0 1 0 0 9 0 1 0 0
4 0 1 0 0 4 0 1 0 0
9 0 1 0 0 9 0 1 0 0
7 0 1 0 0 7 0 1 0 0
7 0 1 0 0 7 0 1 0 0
netstat Command
root@bmserver[/tmp]>netstat -m

34 mbufs in use:
16 mbuf cluster pages in use
72 Kbytes allocated to mbufs
0 requests for mbufs denied
0 calls to protocol drain routines

Kernel malloc statistics:

******* CPU 0 *******

By size inuse calls failed free hiwat freed
32 108 115 0 20 640 0
64 122 765 0 70 320 0
128 111 1666 0 17 160 0
256 134 21771388 0 42 384 0
512 37 2470 0 11 40 0
1024 1 481 0 3 20 0
2048 0 773 0 6 10 0
4096 21 1510233 0 6 120 0
8192 0 66 0 2 10 0
16384 1 6 0 19 24 7

By type inuse calls failed memuse memmax mapb

mbuf 34 21747482 0 8704 17920 0
mcluster 16 1512910 0 65536 95744 0
socket 111 698 0 18048 19008 0
pcb 78 427 0 12032 12864 0
routetbl 8 13 0 1312 2080 0
ifaddr 5 5 0 576 576 0
mblk 30 1198 0 6272 6528 0
mblkdata 3 1070 0 12288 22528 0
strhead 13 57 0 3776 4800 0
strqueue 21 132 0 10752 13312 0
strmodsw 21 21 0 1344 1344 0
strosr 0 23261 0 0 768 0
strsyncq 29 388 0 2944 3648 0
streams 148 275 0 16000 17280 0
kernel table 14 15 0 45920 46432 0
temp 5 12 0 5312 9920 0

Streams mblk statistic failures:

0 high priority mblk failures
0 medium priority mblk failures
0 low priority mblk failures
netstat Command
root@bmserver[/tmp]>netstat -D

Source Ipkts Opkts Idrops Odrops

-------------------------------------------------------------------------------
ent_dev0 0 0 0 0
tok_dev0 21755956 24738 58384 0
---------------------------------------------------------------
Devices Total 21755956 24738 58384 0
-------------------------------------------------------------------------------
ent_dd0 0 0 0 0
tok_dd0 21697572 24738 0 0
---------------------------------------------------------------
Drivers Total 21697572 24738 0 0
-------------------------------------------------------------------------------
ent_dmx0 0 N/A 0 N/A
tok_dmx0 10836721 N/A 10860851 N/A
---------------------------------------------------------------
Demuxer Total 10836721 N/A 10860851 N/A
-------------------------------------------------------------------------------
IP 5171634 2654588 18061 18061
TCP 26292 21120 0 0
UDP 2628010 473 2 2370
---------------------------------------------------------------
Protocols Total 7825936 2676181 18063 20431
-------------------------------------------------------------------------------
lo_if0 473 473 0 0
tr_if0 10834304 22321 0 7
---------------------------------------------------------------
Net IF Total 10834777 22794 0 7
-------------------------------------------------------------------------------
---------------------------------------------------------------
NFS/RPC Total N/A 12 0 0
-------------------------------------------------------------------------------
(Note: N/A -> Not Applicable)
netstat Command
root@bmserver[/tmp]>netstat -s
ip:
1079372 total packets received icmp:
0 bad header checksums 4559 calls to icmp_error
0 with size smaller than minimum 0 errors not generated because old message was icmp
0 with data size < data length Output histogram:
0 with header length < data size echo reply: 6
0 with data length < header length destination unreachable: 25
0 with bad options 0 messages with bad code fields
0 with incorrect version number 0 messages < minimum length
0 fragments received 0 bad checksums
0 fragments dropped (dup or out of space) 0 messages with bad length
0 fragments dropped after timeout Input histogram:
0 packets reassembled ok echo reply: 37
266328 packets for this host destination unreachable: 13
13 packets for unknown/unsupported protocol echo: 6
0 packets forwarded 6 message responses generated
14007 packets not forwardable igmp:
0 redirects sent 1 message received
14202 packets sent from this host 0 messages received with too few bytes
0 packets sent with fabricated ip header 0 messages received with bad checksum
0 output packets dropped due to no bufs, etc. 0 membership queries received
0 output packets discarded due to no route 0 membership queries received with invalid field(s)
0 output datagrams fragmented 1 membership report received
0 fragments created 1 membership report received with invalid field(s)
0 datagrams that can't be fragmented 0 membership reports received for groups to which we belong
799024 IP Multicast packets dropped due to no receiver 4 membership reports sent
37 successful path MTU discovery cycles
8 path MTU rediscovery cycles attempted
0 path MTU discovery no-response estimates
1 path MTU discovery response timeout
0 path MTU discovery decreases detected
39 path MTU discovery packets sent
0 path MTU discovery memory allocation failures
0 ipintrq overflows
0 with illegal source
0 packets processed by threads
0 packets dropped by threads
0 packets dropped due to the full socket receive buffer
0 dead gateway detection packets sent
0 dead gateway detection packet allocation failures
0 dead gateway detection gateway allocation failures
netstat Command
tcp:
13593 packets sent 0 path MTU discovery terminations due to retransmits
11369 data packets (4102411 bytes) 86 retransmit timeouts
89 data packets (108779 bytes) retransmitted 0 connections dropped by rexmit timeout
1842 ack-only packets (1062 delayed) 2 fast retransmits
0 URG only packets 0 when congestion window less than 4 segments
0 window probe packets 0 newreno retransmits
1 window update packet 0 times avoided false fast retransmits
292 control packets 6 persist timeouts
15930 packets received 0 connections dropped due to persist timeout
11390 acks (for 4104523 bytes) 149 keepalive timeouts
217 duplicate acks 149 keepalive probes sent
0 acks for unsent data 0 connections dropped by keepalive
9910 packets (154901 bytes) received in-sequence 0 times SACK blocks array is extended
195 completely duplicate packets (592 bytes) 0 times SACK holes array is extended
0 old duplicate packets 0 packets dropped due to memory allocation failure
5 packets with some dup. data (1572 bytes duped) 0 connections in timewait reused
307 out-of-order packets (4588 bytes) 0 delayed ACKs for SYN
0 packets (0 bytes) of data after window 0 delayed ACKs for FIN
0 window probes 0 send_and_disconnects
262 window update packets 0 spliced connections
0 packets received after close 0 spliced connections closed
0 packets with bad hardware assisted checksum 0 spliced connections reset
2 discarded for bad checksums 0 spliced connections timeout
0 discarded for bad header offset fields 0 spliced connections persist timeout
0 discarded because packet too short 0 spliced connections keepalive timeout
0 discarded by listeners udp:
579 ack packet headers correctly predicted 250354 datagrams received
3412 data packet headers correctly predicted 0 incomplete headers
10 connection requests 0 bad data length fields
311 connection accepts 0 bad checksums
316 connections established (including accepts) 4559 dropped due to no socket
318 connections closed (including 5 drops) 245436 broadcast/multicast datagrams dropped due to no
0 connections with ECN capability socket
0 times responded to ECN 0 socket buffer overflows
5 embryonic connections dropped 359 delivered
11180 segments updated rtt (of 11262 attempts) 381 datagrams output
0 segments with congestion window reduced bit set
0 segments with congestion experienced bit set
0 resends due to path MTU discovery
entstat Command
The entstat command displays the statistics gathered by the
specified Ethernet device driver.

This command is also invoked when the netstat command is run

with the -v flag. The netstat command does not issue any entstat
command flags.

usage: entstat [-drt] <device name>

-d Displays all the statistics, including the device-specific
statistics.
-r Resets all the statistics back to their initial values. This flag
can only be issued by privileged users.
-t Toggles debug trace in some device drivers.
entstat Command
root@bmserver[/tmp]>entstat -d ent0
-------------------------------------------------------------
ETHERNET STATISTICS (ent0) : General Statistics:
Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020) -------------------
Hardware Address: 00:04:ac:17:74:a0 No mbuf Errors: 0
Elapsed Time: 4 days 18 hours 8 minutes 43 seconds Adapter Reset Count: 0
Adapter Data Rate: 200
Transmit Statistics: Receive Driver Flags: Up Broadcast Running
Statistics: Simplex AlternateAddress 64BitSupport
-------------------- PrivateSegment DataRateSet
-------------------
Packets: 0 Packets: 0 IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics:
Bytes: 0 Bytes: 0 ------------------------------------------------
Interrupts: 0 Interrupts: 0 Chip Version: 26
Transmit Errors: 6 Receive Errors: 0 RJ45 Port Link Status : down
Packets Dropped: 1 Packets Dropped: Media Speed Selected: Auto negotiation
0 Media Speed Running: Unknown
Bad Packets: 0 Receive Pool Buffer Size: 384
Max Packets on S/W Transmit Queue: 2 Free Receive Pool Buffers: 128
S/W Transmit Queue Overflow: 0 No Receive Pool Buffer Errors: 0
Current S/W+H/W Transmit Queue Length: 0 Inter Packet Gap: 96
Adapter Restarts due to IOCTL commands: 1
Broadcast Packets: 0 Broadcast Packets with Transmit collisions:
Packets: 0 1 collisions: 0 6 collisions: 0 11
Multicast Packets: 0 Multicast collisions: 0
Packets: 0 2 collisions: 0 7 collisions: 0 12
No Carrier Sense: 6 CRC Errors: 0 collisions: 0
DMA Underrun: 0 DMA Overrun: 0 3 collisions: 0 8 collisions: 0 13
Lost CTS Errors: 0 Alignment Errors: collisions: 0
0 4 collisions: 0 9 collisions: 0 14
Max Collision Errors: 0 No Resource collisions: 0
Errors: 0 5 collisions: 0 10 collisions: 0 15
Late Collision Errors: 0 Receive Collision collisions: 0
Errors: 0 Excessive deferral errors: 0x0
Deferred: 0 Packet Too Short
Errors: 0
SQE Test: 0 Packet Too Long
Errors: 0
Timeout Errors: 0 Packets Discarded
by Adapter: 0
Single Collision Count: 0 Receiver Start
Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 0
NFS Tuning Concepts
NFS Version 3 or Version 2
(Version 3 is recommended)

UDP or TCP protocol

(NOTE: TCP is recommended)

Increase/decrease number of biods/nfsds

nfs socket size and sb_max

adapter transmit queues

Filesystem read/write buffer sizes

RAM for caching file pages (vmtune -P)

I/O tuning (multiple disks, separate log disk)

Tuning with nfso

nfsstat Command
Displays statistics on:
Network File System (NFS)
Remote Procedure Calls (RPC)

Syntax - /usr/sbin/nfsstat:
-c client information
-s server information
-n NFS information only
-r RPC information only
-z reset statistics (root only)
-m mount statistics
default: -csnr
nfsstat Command

root@bmserver[/tmp]>nfsstat -rc

Client rpc:
Connection oriented:
calls badcalls badxids timeouts newcreds badverfs timers
1 0 0 0 0 0 0
nomem cantconn interrupts
0 0 0
Connectionless:
calls badcalls retrans badxids timeouts newcreds badverfs
1448 0 12 0 12 0 0
timers nomem cantsend
22 0 0
nfsstat Command
# nfsstat -m
/nfs/retain/pmrs from /nfs/retain/pmrs:cia
Flags:
vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,ret
rans=5
Lookups: srtt=2 (5ms), dev=2 (10ms), cur=1 (20ms)
All: srtt=2 (5ms), dev=2400 (12000ms), cur=600 (12000ms)

/nfs/retain/bin from /nfs/retain/bin:cia

Flags:
vers=2,proto=udp,auth=unix,hard,intr,dynamic,rsize=8192,wsize=8192,ret
rans=5
Lookups: srtt=14 (35ms), dev=6 (30ms), cur=4 (80ms)
Reads: srtt=18 (45ms), dev=5 (25ms), cur=4 (80ms)
All: srtt=15 (37ms), dev=7 (35ms), cur=5 (100ms)

/nfs/cust from /nfs/cust:zen

Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,sym-link,
rsize=32768,wsize=32
768,retrans=5
All: srtt=0 (0ms), dev=2400 (12000ms), cur=600 (12000ms)
nfsstat Command
# netstat -p udp

udp:
27136572 datagrams received
0 incomplete headers
0 bad data length fields
0 bad checksums
20 dropped due to no socket
8 broadcast/multicast datagrams dropped due to no socket
39476 socket buffer overflows
27097068 delivered
177636 datagrams output

socket buffer overflows = Too few nfsd daemons and/or too

small nfs_socketsize

Increase the number of nfsd daemons (NFS V2)

Increase value of nfs_socketsize and sb_max

nfso Command
nfso Command
# nfso -a
portcheck= 0
udpchecksum= 1
nfs_socketsize= 60000
nfs_tcp_socketsize= 60000
nfs_setattr_error= 0
nfs_gather_threshold= 4096
nfs_repeat_messages= 0
nfs_udp_duplicate_cache_size= 0
nfs_tcp_duplicate_cache_size= 0
nfs_server_base_priority= 0
nfs_dynamic_retrans= 1
nfs_iopace_pages= 0
nfs_max_connections= 0
nfs_max_threads= 8
nfs_use_reserved_ports= 0
nfs_device_specific_bufs= 1
nfs_server_clread= 1
nfs_rfc1323= 0
nfs_max_write_size= 0
nfs_max_read_size= 0
nfs_allow_all_signals= 0