AIX for System Administrators

http://aix4admins.blogspot.mx/2011/09/load-average-uptime-1801pm-u...

PAGEVIEW S

CPU

620,232
LOAD AVERAGE: uptime 18:01pm
ABOUT

up 217 days,

22:40,

0 users,

load average:

8.78, 8.75, 8.82

FS - LVM FS LV Mirror Pool

shows the load average in the last minute, five minutes, and fifteen minutes In general, each process that is running, waiting for CPU, or waiting on I/O would add one to the load average. These figures are calculated and then averaged over time. load average is the "r" column under vmstat. This is the number of kernel threads (the runnable threads) It has to be compared to tha actual number of CPUs (logical CPU) if the CPUs can service those threads. ----------------------------------

PV VG

GENERAL

ps aux <--shows CPU and memory usage of processes root@aix31: / # ps aux USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND root 61470 8.2 0.0 384 384 - A Aug 27 4711:53 wait <-this wait process is assigned to the CPU if the system is idle oracle 594004 1.0 0.0 29544 30180 - A Aug 27 554:03 /u02/app/oracle %CPU: The percentage of time the process has used the CPU. It is the average CPU utilization of the process since it was first created (If a process consumes 100% for 5 seconds, then sleeps for 5 seconds, ps report at the and of 10 seconds 50%.) (This can be misleading as right now the process is not actually using CPU time.) It can be misleading as it shows the accumulated time for a process since it has been started. (For a process which was started long ago can have a high number) %MEM: The percentage of real memory used by this process wait: In AIX the CPU must always be doing work. If the system is idle, the wait process will be executed. ??kproc: is an idle kernel process created by the UNIX OS to keep the processors doing something void while there's no CPU requirements. ---------------------------------ps ps ps ps aux | head -1; ps aux | sort -rn +2 | head -20 top CPU processes -elf shows priority of processes (PRI) (60 is general, keep an eye on lower than 60 (it means higher priority)) -efk|grep wait k: will show kernel processes -fp <pid> check the time field, if it is constant over time, a probable deadlock or hang could have occurred.

AIX History Backup Commands CPU - Processes Crontab - At Date - Time Devices Dump - Core Errpt - Diag - Alog Syslogd Firmware IO - AIO, DIO, CIO Memory - Pag.Space ODM Printing SRC Startup - Shutdown System - Kernel Tunables User - Group User Login

HACMP - POW ERHA

---------------------------------Check the accumulated system time for key processes: root@aix10: /root # ps -ekf | grep -v grep | egrep "syncd|lrud|nfsd|biod|wait" root 8196 0 0 Mar 24 - 1082:56 wait root 16392 0 0 Mar 24 - 369:35 lrud root 49176 0 0 Mar 24 - 4973:24 wait root 53274 0 0 Mar 24 - 498:10 wait root 57372 0 0 Mar 24 - 3135:26 wait root 61470 0 0 Mar 24 - 358:20 wait root 123092 0 0 Mar 24 - 0:52 kbiod root 147640 1 0 Mar 24 - 267:27 /usr/sbin/syncd 60 root 2560068 229562 0 May 06 - 0:00 /usr/sbin/biod 6 root 3133690 229562 0 May 06 - 0:00 /usr/sbin/nfsd 3891 root 3420312 1 0 May 06 - 5:18 nfsd wait: every cpu has an assigned wait process, and it is in wait state when it has nothing to do. (its a kernel process mapped 1 to 1 to a CPU) You can check these processes how much the accumulated time (for example nfsd, if it has high time value it neeeds some reconfig...) match system times of lrud againd syncd: If lrud is greater than syncd it should get your attention. lrud is a fixed priority kernel process with process priority 16. Once lrud is running not very much else is running ---------------------------------time <any command or script> real 0m0.96s user 0m0.12s sys 0m0.05s

Appl. Monitor Basics Build - Configure Clverify Commands - Cases Config too long DARE - Snapshot Disk Heartbeat Storage - VG, NFS

HMC - ISD ASMI HMC Basics - Console HMC CLI HMC/P7 Install ISD Basics RMC

NETW ORK Basics - Devices, Routing Basics - Protocol, Subnet Basics - Vlan Commands Eth. Chan. Ethernet Adapter IVE - HEA

It shows CPU time spent in user mode, and in system mode and the real time (elapsed time) to execute it.
Netcd

---------------------------------sar: sar -P ALL 1 15 shows CPU usage for all logical CPU (-P : CPU by CPU basis statistics)

NFS RSH - RCP Sendmail SSH - SCP SSH - X11 Telnet - FTP

System configuration: lcpu=4 ent=0.20 mode=Uncapped 13:49:15 cpu 13:49:16 0 1 2 3 %usr 26 0 0 0 %sys 57 15 9 47 %wio 0 0 0 0 %idle 17 85 91 53 physc 0.01 0.00 0.00 0.00 %entc 3.8 0.2 0.0 0.1
NIM Basics Install LPP Source Machines MKSYSB

Check %usr+%sys together and compare it to physc. Some examples when physical consumption of a CPU should be also looked when smt is on.:

Nimadm SPOT

1 of 3

04/05/2013 23:14

AIX for System Administrators

http://aix4admins.blogspot.mx/2011/09/load-average-uptime-1801pm-u...

- usr+sys=16%, but physc=0.56, it means i see 16% is utliized of a CPU, but actually half of the physical CPU (0.56) is used. - if us+sys=100 and physc=0.45 we have to look both. If someone says 100% percent is used, then 100% of what? The 100% of the half of the CPU (physc=0.45) is used. - %usr+%sys=83% for lcpu 0 (output from command sar). It looks a high number at the first sight, but if you check physc, you can see only 0.01 physical core has been used, and the entitled capacityis 0.20, so this 83% is actually very little CPU consumption. ----sar -u 2 10 22:06:25 %usr 22:06:27 89 <--shows system activity info (-u:CPU usage date, 2: interval in seconds, 10:number of intervals)(same as topas) %sys %wio %idle physc %entc 9 1 1 0.49 163.4

PERFORMANCE Basics CPU I/O - Disk, Adapter Memory Network svmon - RAM topas - nmon vmstat - CPU/RAM VMM concepts VIO

The sar command can extract and show previously saved CPU utilization metrics that were previously saved in a file (/var/adm/sa/sadd, where dd refers to current day). The system utilization information is saved by two shell scripts (/usr/lib/sa/sa1 and /usr/lib/sa/sa2) running in the background. These shell scripts are started by the cron daemon using crontab file /var/spool/cron/crontabs/adm. Collecting data in this manner is a useful way to characterize system usage over a period of time and determine peak usage hours. To view the files: sar -f /usr/adm/sa/sa03 sar -P ALL -f /usr/adm/sa/sa03 ---------------------------------mpstat: performance statistics from logical processor viewpoint

STORAGE Adapter Basics - SAN Basics - Settings EMC Hitachi HP EVA - SSA MPIO SDD

UPDATE - INSTALL Basics

root@bb_lpar: / # mpstat 5 cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs 0 0 0 0 185 88 0 1 0 100 66 13 62 0 24 0.00 1.2 133 1 0 0 0 9 0 0 0 0 0 0 2 0 98 0.00 0.4 9 U - - - 0 99 0.20 98.3 ALL 0 0 0 194 88 0 1 0 0 66 0 1 0 99 0.00 1.7 142 -------------------------------------------------------------------------------0 0 0 0 188 90 0 1 0 100 69 14 62 0 24 0.00 1.3 135 1 0 0 0 9 0 0 0 0 100 0 0 4 0 96 0.00 0.5 10 U - - - 0 99 0.20 98.2 ALL 0 0 0 197 90 0 1 0 100 69 0 1 0 99 0.00 1.8 145

Commands IFIX

VIO Basics Commands AME - AMS CDROM - DVDROM LPM NIB - LA NPIV SEA

mig: number of thread migrations to another logical processor lpa: it shows which logical cpu (smt thread) is active ("100" means that smt hread is in use, "-" means it is not used) lcs: logical processor context switches It shows there are standard logical processor context switches (lcs), however, no thread was forced to migrate to another logical processor. ----------------------------------

Shared Storage Pool VIOS update/upgrade Virt. Eth. Ad. Virt. Proc.-Ent. Cap. VSCSI VSCSI - Stor. Pool

+EXTRAS

tprof:
+others

it reports processor usage for individual programs and the system as a whole (it shows which sections of the program are most heavily using the processor) 1. tprof -x sleep 60 run tprof) <--run tprof, execute sleep and run it for 60 seconds (we are not profiling "sleep" we are using the value 60 to

alt_disk awk - sed bash citrix java kdb ksh locale

2. it creates a file, in the dir wher we are: cat sleep.prof

Process ======= wait nfsd /usr/sbin/getty rpc.lockd /usr/bin/tprof ======= Total Process ======= wait wait wait wait nfsd /usr/sbin/getty /usr/bin/tprof rpc.lockd ======= Total PID TID === === 131076 131077 917532 1376299 1048608 1507375 983070 1441837 14221360 30146813 7798814 14680259 11010196 34603221 5177366 9044005 === ===

Freq Total ==== ===== 4 99.83 1 0.07 1 0.03 1 0.03 1 0.03 ==== ===== 8 100.00 Total ===== 57.58 20.09 20.09 2.08 0.07 0.03 0.03 0.03 ===== 100.00

Kernel ====== 99.83 0.07 0.03 0.03 0.03 ====== 100.00

User Shared ==== ====== 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ==== ====== 0.00 0.00 Other ===== 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ===== 0.00

Other ===== 0.00 0.00 0.00 0.00 0.00 ===== 0.00

multibos perl profiles rbac Rsh samba screen script sudo

Kernel ====== 57.58 20.09 20.09 2.08 0.07 0.03 0.03 0.03 ====== 100.00

User Shared ==== ====== 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ==== ====== 0.00 0.00

vi

First section shows the processes, below the threads (with thread id (TID)). It separets Kernel and User usage for each process during the timeframe it was running.

Labels: PERFORMANCE

4 comments:
khan October 10, 2012 at 7:57 PM Very helpful information. Thanks... Reply Replies aix :) October 11, 2012 at 5:10 PM

Reply

Marcelo WonederZ March 14, 2013 at 10:26 AM very nice !!! keep the good work

2 of 3

04/05/2013 23:14

AIX for System Administrators

http://aix4admins.blogspot.mx/2011/09/load-average-uptime-1801pm-u...

Reply Replies aix :-) March 14, 2013 at 2:05 PM

Reply

Comment as:

Newer Post Subscribe to: Post Comments (Atom)

Home

Older Post

Template images by Storman. Powered by Blogger.

3 of 3

04/05/2013 23:14