Tactics For Monitoring Tuning AIX 5L 6.1 Apr28-09 V1.0

IBM System p
Tactics for
Monitoring and
Performance-Tuning
AIX 5L/6.1
All original content by Earl Jew
FTSS Storage and System P
earlj@us.ibm.com (310)251-2907
AIX Virtual Users Group
April 30th, 2009
© 2007 IBM Corporation

IBM System p
Tactical AIX 5L/6.1 Monitoring & Tuning: Strategic Methodology

Can the capacities of the hardware manage the workload? Yes/No
Note the size, scale, technology and implementation of a given LPAR
Note the in-place ratio-of-resources, i.e. CPU-to-RAM-to-SAN I/O
Review the accumulated count-of-events in-light of days-uptime
Determine points of exhaustion, limitation, and over-commitment
Determine surplus resources: CPUcycles, RAM, SAN I/O thruput, etc.
Review exhaustions and surpluses in-light of workload expectations

For example: Is this a Think-Think or a Move-the-Data workload?
Map characteristic AIX dynamic-behaviors to their functional sources

Establish benchmark baselines with routine/repeated batch workloads
Establish dynamic baselines with monitored AIX 5L/6.1 behaviors
Devise tactics to relieve exhaustions by exploiting surplus resources
Singularly Divide-and-Assign: Build Discrete Accountable Constructs

Exploit the Competitive Advantages of Per-Design Features
Note how the implementation of technology defines its daily operation
2 © 2007 IBM Corporation IBM Systems

IBM System p
Tactical AIX 5L/6.1 Monitoring and Tuning: Related Thoughts

Assumptions and Expectations
•size/scale – Capacities vs. workload, aka “tuning-by-hardware”

•2sec vs 300sec - smooth-averaging hides intensities
•count-indicators - offer substantiated empirical diagnosis
•consider - scale, tech-stack, character&intensity, exhaustions
•first-rule - AIX 5L: resize AIX parms to the scale in-place
•demystify - understanding->confidence; insight->competence
•vision - offer theory and concepts, models and tactics, examples
•intimacy - vmstat, mpstat, iostat, ps, perhaps topas/nmon
•formulate - feel-good’s and out-of-whack’s: rules and proportions
•reality – repeatedly finding the same problematic scenarios

IBM System p
Tactical AIX Monitoring: Size&Scale (HMCprofile)

3W’s: Who What When
$ date ; uname -a ; id ; oslevel –s; lparstat -i
Sun Mar 2 10:23:33 PST 2008
AIX sapdb1 3 5 00C52DFE4C00
uid=84081(eyj1) gid=1(staff)
5300-05-03
Node Name : sapdb1
Partition Name : SAPDB1P
Partition Number : 3
Type : Shared-SMT
Mode : Uncapped
Entitled Capacity : 12.00
Partition Group-ID : 32771
Shared Pool ID : 0
Online Virtual CPUs : 12
Maximum Virtual CPUs : 12
Minimum Virtual CPUs : 2
Online Memory : 73728 MB
Maximum Memory : 73728 MB
Minimum Memory : 36864 MB
Variable Capacity Weight : 128
Minimum Capacity : 1.20
Maximum Capacity : 12.00
Capacity Increment : 0.01
Maximum Physical CPUs in system : 64
Active Physical CPUs in system : 32
Active CPUs in Pool : 32
Unallocated Capacity : 0.00
Physical CPU Percentage : 100.00%
Unallocated Weight : 0
$

IBM System p
Tactical AIX Monitoring: prtconf Exam Tech-Stack 1

$ prtconf
System Model: IBM,9119-590
Machine Serial Number: 0299XXX
Processor Type: PowerPC_POWER5
Number Of Processors: 12
Processor Clock Speed: 2102 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 3 SAPDB1P
Memory Size: 73728 MB
Good Memory Size: 73728 MB
Platform Firmware level: Not Available
Firmware Version: IBM,SF240_320
Console Login: enable
Auto Restart: true
Full Core: false
Network Information
Host Name: sapdb1
IP Address: 10.253.999.99
Sub Netmask: 255.255.255.128
Gateway: 10.252.999.9
Name Server: 10.244.999.99
Domain Name: comp.xxx.com
Paging Space Information

Total Paging Space: 130944MB
Percent Used: 1%
…
…

IBM System p
Tactical AIX Monitoring: lsdev Exam Tech-Stack 2

$ lsdev | grep vail
L2cache0 Available L2 Cache
aio0 Available Asynchronous I/O (Legacy)
en6 Available Standard Ethernet Network Interface
en7 Available Standard Ethernet Network Interface
...
ent2 Available 0C-08 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent3 Available 0C-09 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent4 Available Virtual I/O Ethernet Adapter (l-lan)
ent5 Available Virtual I/O Ethernet Adapter (l-lan)
ent6 Available EtherChannel / IEEE 802.3ad Link Aggregation
ent7 Available EtherChannel / IEEE 802.3ad Link Aggregation
...
fcs6 Available 0B-08 FC Adapter
fcs7 Available 0B-09 FC Adapter
...
fscsi6 Available 0B-08-02 FC SCSI I/O Controller Protocol Device
fscsi7 Available 0B-09-02 FC SCSI I/O Controller Protocol Device
hdisk0 Available 08-08-00-8,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 0D-08-00-8,0 16 Bit LVD SCSI Disk Drive
...
hdisk168 Available 06-08-02 Hitachi MPIO Disk USP (Fibre)
hdisk169 Available 06-08-02 Hitachi MPIO Disk USP (Fibre)
...
rmt22 Available 0A-08-02 Other FC SCSI Tape Drive
rmt23 Available 0A-08-02 Other FC SCSI Tape Drive
sa0 Available 0E-08 2-Port Asynchronous EIA-232 PCI Adapter
scsi0 Available 08-08 Wide/Ultra-3 SCSI I/O Controller
scsi1 Available 0D-08 Wide/Ultra-3 SCSI I/O Controller
ses0 Available 08-08-00-15,0 SCSI Enclosure Services Device
ses1 Available 0D-08-00-15,0 SCSI Enclosure Services Device
sys0 Available System Object
sysplanar0 Available System Planar
tty0 Available 0E-08-01-00 Asynchronous Terminal
vio0 Available Virtual I/O Bus
vsa0 Available LPAR Virtual Serial Adapter
vty0 Available Asynchronous Terminal

IBM System p
Tactical AIX Monitoring: lscfg Exam Tech-Stack 3

$ lscfg | grep +
+/- = Added or deleted from Resource List.
+ sys0 System Object
+ sysplanar0 System Planar
+ scsi1 U5791.001.9920C4C-P1-T5 Wide/Ultra-3 SCSI I/O Controller
+ hdisk1 U5791.001.9920C4C-P1-T5-L8-L0 16 Bit LVD SCSI Disk Drive (146800 MB)
+ ses1 U5791.001.9920C4C-P1-T5-L15-L0 SCSI Enclosure Services Device
+ fcs6 U5791.001.9920C4C-P1-C01-T1 FC Adapter
+ ent2 U5791.001.9920C4C-P1-C02-T1 2-Port 10/100/1000 Base-TX PCI-X Adapter
(14108902)
+ ent3 U5791.001.9920C4C-P1-C02-T2 2-Port 10/100/1000 Base-TX PCI-X Adapter
(14108902)
+ fcs2 U5791.001.9920C4D-P2-C01-T1 FC Adapter
+ scsi0 U5791.001.9920C4D-P1-T5 Wide/Ultra-3 SCSI I/O Controller
+ hdisk0 U5791.001.9920C4D-P1-T5-L8-L0 16 Bit LVD SCSI Disk Drive (146800 MB)
+ ses0 U5791.001.9920C4D-P1-T5-L15-L0 SCSI Enclosure Services Device
+ ent0 U5791.001.9920C4D-P1-C02-T1 2-Port 10/100/1000 Base-TX PCI-X Adapter
(14108902)
+ ent1 U5791.001.9920C4D-P1-C02-T2 2-Port 10/100/1000 Base-TX PCI-X Adapter
(14108902)
+ sa0 U5791.001.9920C4D-P1-C03-T1 2-Port Asynchronous EIA-232 PCI Adapter
+ tty0 U5791.001.9920C4D-P1-C03-T1-L0 Asynchronous Terminal
+ L2cache0 L2 Cache
+ mem0 Memory
+ proc0 Processor
+ proc2 Processor
+ proc4 Processor
+ proc6 Processor
+ proc8 Processor
+ proc10 Processor
+ proc12 Processor
…
$

IBM System p
Tactical AIX Monitoring: Count-indicators 1

days-up, counts, waits, interrupts, etc.
$ uptime ; vmstat -s
10:25AM up 133 days, 13:17, 2 users, load average: 18.65, 18.79, 19.07
786936730 total address trans. faults
4214641440 page ins
2917307106 page outs
249724 paging space page ins
27079 paging space page outs
0 total reclaims
2532337046 zero filled pages faults
131416 executable filled pages faults
2659119088 pages examined by clock
4 revolutions of the clock hand
2251628700 pages freed by the clock
620698110 backtracks
120726 free frame waits
65 extend XPT waits
3596479218 pending I/O waits
2792620612 start I/Os
4054308069 iodones
251445496830 cpu context switches
47628136542 device interrupts
29744461415 software interrupts
17223382142 decrementer interrupts
582420837 mpc-sent interrupts
582292649 mpc-receive interrupts
736848246 phantom interrupts
0 traps
488385349970 syscalls
$

IBM System p
Tactical AIX Monitoring: Count-indicators 2

days-up, memstructs, VMM parms, “no buf’s”
$ uptime ; vmstat -v
18874368 memory pages
18165025 lruable pages
49229 free pages
4 memory pools
2810046 pinned pages
80.0 maxpin percentage
30.0 minperm percentage
50.0 maxperm percentage
42.8 numperm percentage
7781399 file pages
0.0 compressed percentage
0 compressed pages
42.8 numclient percentage
50.0 maxclient percentage
7779005 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
0 paging space I/Os blocked with no psbuf
87988 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
11195356 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults
$

IBM System p
Tactical AIX Monitoring: no-load leave-it-up everyday

The Best 6in1: vmstat -I 2
$ vmstat –I 2
System configuration: lcpu=24 mem=73728MB ent=12.00
kthr memory page faults cpu

-------- ----------- ------------------------ ------------ -----------------------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
10 6 0 11507787 49543 3483 823 0 0 5306 5355 5662 45907 26258 33 4 53 10 4.71 39.3
10 7 0 11509796 47989 3824 594 0 0 4116 27771 6252 56592 45959 54 7 28 12 7.65 63.8
9 7 0 11510010 47770 3955 622 0 0 3977 9647 5907 56222 46833 48 7 31 15 6.87 57.2
8 6 0 11510560 50021 4164 2431 0 0 5564 40421 6607 51080 49691 41 7 39 13 6.08 50.7
6 6 0 11512741 46710 4886 1443 0 0 4443 4608 6110 42400 30394 36 5 42 17 5.28 44.0
5 7 0 11514081 48807 4675 227 0 0 6461 7028 4838 34521 11343 33 3 55 9 4.68 39.0
8 7 0 11515469 48531 5679 482 0 0 6445 6593 5686 44979 13230 37 3 48 12 4.99 41.5
9 7 0 11514065 49598 3858 1046 0 0 4128 4255 5807 51871 27521 44 6 32 19 6.27 52.3
8 7 0 11517672 49905 4848 632 0 0 7173 7221 5679 44566 47102 48 6 32 14 6.84 57.0
10 13 0 11520210 50148 4669 692 0 0 6313 6491 6341 47122 45622 52 5 28 15 7.22 60.2
7 9 0 11521192 48222 5087 814 0 0 5194 5790 6211 49553 44306 45 6 34 15 6.45 53.7
10 7 0 11521212 50922 3830 627 0 0 5330 5353 6248 48130 32364 47 4 37 12 6.42 53.5
4 7 0 11521503 49362 3475 573 0 0 3075 3102 5717 47907 13356 42 3 41 14 5.69 47.4
9 8 0 11523055 48731 3502 511 0 0 4143 4176 5884 44391 13427 46 2 41 11 6.01 50.1
10 8 0 11524140 50987 3483 761 0 0 5363 5683 5830 45416 15252 60 3 23 14 7.89 65.7
8 7 0 11524407 45661 3871 351 0 0 1488 1621 5378 34403 13034 54 2 29 15 7.14 59.5
6 7 0 11523652 50033 3325 355 0 0 5229 5448 5434 40780 14372 45 3 36 16 6.06 50.5
8 9 0 11525268 48536 4209 272 0 0 4102 4337 4599 36202 10449 44 4 35 18 6.05 50.4
9 9 0 11525476 48242 4322 521 0 0 4307 4634 5375 33863 13975 44 3 35 18 5.97 49.7
8 11 0 11526444 49830 4988 699 0 0 6351 6828 6743 53110 45620 46 6 32 16 6.63 55.3
…
…

IBM System p
Tactical AIX Monitoring: “memory over-commitment?” Feel-Good 1

Wide Timestamp’d 7in1 plus avm, fre, comp%
$ uptime ; vmstat -Iwt 2 20
kthr memory page faults cpu time

----------- --------------------- ------------------------------------ ------------------ ----------------------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
20 0 0 10957853 48491 66 47 0 0 0 0 363 117648 2422 96 2 2 0 12.00 100.0 10:51:26
20 0 0 10957452 48808 16 81 0 0 0 0 418 94030 4048 97 2 2 0 12.00 100.0 10:51:28
18 0 0 10957456 48524 111 136 0 0 0 0 486 6692 2338 98 0 2 0 12.00 100.0 10:51:30
18 0 0 10957463 48455 25 34 0 0 0 0 152 11637 1348 97 1 2 0 12.00 100.0 10:51:32
18 0 0 10957464 48432 6 14 0 0 0 0 77 3019 1141 98 0 2 0 12.00 100.0 10:51:34
19 1 0 10957470 48298 65 7 0 0 0 0 197 4164 1330 97 0 2 0 11.95 99.6 10:51:36
18 0 0 10957472 48296 0 5 0 0 0 0 39 2842 1028 97 0 3 0 11.86 98.8 10:51:38
19 0 0 10957479 48236 23 13 0 0 0 0 234 5335 1448 98 0 2 0 12.00 100.0 10:51:40
19 1 0 10957487 47686 271 5 0 0 0 0 402 13439 1806 97 1 2 0 12.00 100.0 10:51:42
19 0 0 10957489 47684 0 9 0 0 0 0 37 7145 997 97 1 2 0 12.00 100.0 10:51:44
20 0 0 10957481 47610 39 56 0 0 0 0 167 2837 1061 98 0 2 0 12.00 100.0 10:51:46
19 0 0 10957483 47548 31 1 0 0 0 0 85 3075 1065 98 0 2 0 12.00 100.0 10:51:48
18 0 0 10957481 47500 26 13 0 0 7 135 75 2921 1032 98 0 2 0 12.00 100.0 10:51:50
19 0 0 10957889 49033 53 6 0 0 1025 1031 129 88541 1871 96 2 2 0 11.98 99.8 10:51:52
19 0 0 10957888 48954 40 0 0 0 0 0 89 94550 1869 96 2 2 0 12.00 100.0 10:51:54
20 1 0 10957882 48926 17 6 0 0 0 0 74 123666 2068 96 2 2 0 12.00 100.0 10:51:56
19 0 0 10957880 48916 5 8 0 0 0 0 47 120104 1913 94 3 4 0 11.80 98.4 10:51:58
20 0 0 10957666 49062 34 1 0 0 0 0 80 117384 1849 96 2 2 0 12.00 100.0 10:52:00
18 0 0 10957883 48841 1 7 0 0 0 0 59 130003 1924 95 3 2 0 12.00 100.0 10:52:02
20 0 0 10957889 48779 28 6 0 0 0 0 143 126580 2284 96 3 2 0 12.00 100.0 10:52:04
$ bc
10957889 * 4096
44883513344
quit
$

IBM System p
Tactical AIX Monitoring: vmstat with mpstat

$ uptime ; vmstat -Iwt 2 3 ; mpstat -w 2 1

----------- --------------------- ------------------------------------ ------------------ ----------------------- --------
20 0 0 10957738 50515 233 13 0 0 1031 1031 750 12312 2875 96 1 3 0 11.80 98.3 10:54:03
19 1 0 10957743 50166 170 8 0 0 0 0 652 8864 2596 94 1 5 0 11.58 96.5 10:54:05
19 0 0 10957757 49766 192 12 0 0 0 0 790 14208 2756 96 1 3 0 11.88 99.0 10:54:07
System configuration: lcpu=24 ent=12.0
cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs
0 16 24 85 454 577 278 0 52 100.0 1609 88.7 2.8 1.1 7.4 0.27 2.3 334
1 0 0 85 215 1 1 0 0 100.0 0 99.8 0.2 0.0 0.0 0.73 6.1 0
2 0 0 85 217 0 0 0 0 100.0 681 99.4 0.6 0.0 0.0 0.81 6.8 0
3 24 22 85 242 758 365 0 77 100.0 1936 83.8 2.4 2.5 11.3 0.19 1.5 459
4 0 0 85 222 18 9 0 0 100.0 261 99.7 0.3 0.0 0.0 0.74 6.2 0
5 32 10 85 227 363 142 0 117 100.0 974 91.0 1.0 0.4 7.6 0.26 2.1 247
6 0 0 85 226 102 32 0 49 100.0 484 96.7 1.6 0.0 1.7 0.40 3.3 44
7 0 0 85 227 5 1 0 0 100.0 44 99.9 0.1 0.0 0.0 0.60 5.0 0
8 0 0 85 267 3 2 0 3 100.0 295 99.6 0.4 0.0 0.0 0.90 7.5 16
9 18 4 1955 132 214 90 0 52 100.0 1219 61.1 6.0 0.1 32.9 0.10 0.9 223
10 0 0 85 213 1 1 0 0 100.0 200 99.7 0.3 0.0 0.0 0.73 6.1 0
11 3 3 85 226 344 137 0 103 100.0 624 92.8 0.8 0.6 5.9 0.27 2.2 236
12 1 2 85 233 346 159 0 55 100.0 704 93.9 0.7 0.2 5.2 0.32 2.7 278
13 0 0 85 233 38 16 0 14 100.0 128 99.4 0.2 0.0 0.4 0.67 5.6 26
14 5 7 85 230 413 174 0 66 100.0 527 93.1 0.9 0.1 6.0 0.31 2.6 235
15 0 0 85 234 8 4 0 0 100.0 3 99.8 0.2 0.0 0.0 0.69 5.7 0
16 0 0 85 274 0 0 0 0 100.0 325 99.6 0.4 0.0 0.0 0.52 4.4 0
17 0 0 85 215 1 1 0 0 100.0 0 99.9 0.1 0.0 0.0 0.48 4.0 0
18 0 0 85 214 0 0 0 0 100.0 0 99.9 0.1 0.0 0.0 0.66 5.5 0
19 8 10 85 226 130 69 0 5 100.0 437 95.4 0.4 0.2 4.0 0.34 2.8 103
20 2 3 85 238 266 96 0 101 100.0 271 95.5 0.8 0.2 3.5 0.45 3.7 161
21 8 6 85 236 198 87 0 58 100.0 400 96.1 0.8 0.2 2.9 0.46 3.9 148
22 0 0 85 229 0 0 0 0 100.0 0 99.8 0.2 0.0 0.0 0.67 5.6 0
23 7 16 85 230 357 180 0 30 100.0 1447 93.9 1.1 1.5 3.5 0.33 2.7 273
U - - - - - - - - - - - - 0.1 0.8 0.10 0.9 -
ALL 124 107 3910 5660 4143 1844 0 782 100.0 12569 96.7 0.6 0.2 2.5 11.90 99.1 2783
$

IBM System p
Tactical AIX Monitoring: perhaps only some topas

topas<cr>d n c c
Topas Monitor for host: sapdb1 EVENTS/QUEUES FILE/TTY
Sun Mar 2 11:14:49 2008 Interval: 2 Cswitch 5916 Readch 0.0G
Syscall 24150 Writech 1043.9K
CPU User% Kern% Wait% Idle% Physc Reads 5587 Rawin 0
cpu4 99.9 0.1 0.0 0.0 0.52 Writes 1213 Ttyout 686
cpu5 99.9 0.1 0.0 0.0 0.48 Forks 0 Igets 0
cpu9 99.8 0.2 0.0 0.0 0.72 Execs 0 Namei 46
cpu6 99.8 0.2 0.0 0.0 0.69 Runqueue 20.0 Dirblk 0
cpu11 99.8 0.2 0.0 0.0 0.69 Waitqueue 2.0
cpu15 99.8 0.2 0.0 0.0 0.76
cpu17 99.8 0.2 0.0 0.0 0.48 PAGING MEMORY
cpu20 99.6 0.4 0.0 0.0 0.51 Faults 854 Real,MB 73728
cpu13 99.5 0.3 0.1 0.1 0.62 Steals 1986 % Comp 59.2
cpu16 99.5 0.5 0.0 0.0 0.52 PgspIn 0 % Noncomp 41.4
cpu21 99.5 0.4 0.2 0.0 0.49 PgspOut 0 % Client 41.4
cpu22 99.1 0.4 0.1 0.4 0.59 PageIn 1369
cpu1 98.3 0.8 0.4 0.4 0.64 PageOut 167 PAGING SPACE
cpu23 97.0 0.7 0.9 1.5 0.40 Sios 1436 Size,MB 130944
cpu2 93.0 6.0 0.2 0.8 0.53 % Used 0.7
cpu18 93.0 4.9 0.9 1.2 0.50 NFS (calls/sec) % Free 99.2
cpu3 92.8 2.0 2.4 2.8 0.41 ServerV2 0
cpu7 92.4 2.3 2.5 2.8 0.31 ClientV2 0 Press:
cpu10 91.4 2.2 3.1 3.2 0.31 ServerV3 11 "h" for help
cpu0 90.7 3.4 3.2 2.8 0.32 ClientV3 0 "q" to quit
cpu19 90.3 5.2 2.2 2.3 0.40
cpu8 89.0 2.0 4.4 4.6 0.28
cpu12 87.8 7.7 2.0 2.5 0.38
cpu14 83.9 3.7 5.0 7.4 0.24
Name PID CPU% PgSp Owner

oracle 2424844 6.4 18.1 orapr1
oracle 1474750 6.3 8.9 orapr1
oracle 3256398 5.8 9.4 orapr1
topas 33759394 1.3 40.1 eyj1
oracle 4227258 5.7 11.1 orapr1
oracle 33087570 5.7 8.9 orapr1
oracle 2007246 5.7 8.9 orapr1
oracle 33861652 5.6 12.3 orapr1

IBM System p
Tactical AIX Monitoring: some characterizing “signposts”

Check the accumulated CPU-time of key processes
$ uptime ; ps -ekf | grep -v grep | egrep "syncd|lrud|nfsd|biod|wait"
root 8196 0 0 Oct 20 - 2218:47 wait
root 16392 0 9 Oct 20 - 17178:29 lrud
root 53274 0 0 Oct 20 - 8461:10 wait
root 57372 0 0 Oct 20 - 1335:46 wait
root 61470 0 0 Oct 20 - 6073:45 wait
root 65568 0 0 Oct 20 - 1374:15 wait
root 69666 0 0 Oct 20 - 5560:02 wait
root 73764 0 0 Oct 20 - 1331:34 wait
root 77862 0 0 Oct 20 - 4979:33 wait
root 81960 0 0 Oct 20 - 1260:27 wait
root 86058 0 0 Oct 20 - 4369:40 wait
root 90156 0 0 Oct 20 - 965:50 wait
root 94254 0 0 Oct 20 - 3300:50 wait
root 98352 0 0 Oct 20 - 826:15 wait
root 102450 0 0 Oct 20 - 2657:03 wait
root 106548 0 0 Oct 20 - 681:51 wait
root 110646 0 0 Oct 20 - 2107:25 wait
root 114744 0 0 Oct 20 - 581:12 wait
root 118842 0 0 Oct 20 - 1709:27 wait
root 122940 0 0 Oct 20 - 385:38 wait
root 127038 0 0 Oct 20 - 1047:36 wait
root 131136 0 0 Oct 20 - 294:26 wait
root 135234 0 0 Oct 20 - 731:46 wait
root 139332 0 0 Oct 20 - 185:51 wait
root 143430 0 0 Oct 20 - 382:23 wait
root 266430 1 0 Oct 20 - 1008:28 /usr/sbin/syncd 60
root 1429696 0 0 Oct 20 - 0:25 kbiod
root 1540176 1437714 0 Oct 20 - 0:00 /usr/sbin/nfsd 3891
root 1646808 1 0 Oct 20 - 506:30 nfsd
root 1728526 1437714 0 Oct 20 - 0:00 /usr/sbin/biod 6
$ uptime ; ps -ekf | grep -v grep | egrep "syncd|lrud|nfsd|biod|wait" | grep -c wait
24
$

IBM System p
Tactical AIX Monitoring: investigate by digging deeper

Seek-out and discern anomalies and other atypical events/situations
$ ps -ef | grep -v "Oct 20"
UID PID PPID C STIME TTY TIME CMD
orapr1 1474750 1 61 00:45:40 - 82:55 oraclePR1 (DESCRIPTION=(LOCAL=NO)(SDU=4096))
root 1515764 1 0 Feb 27 - 1:22 ./wrapper ./agent.conf wrapper.pidfile=../tmp/teagent.pid
pr1adm 1810644 3301460 0 00:44:18 - 1:17 dw.sapPR1_DVEBMGS00 pf=/usr/sap/PR1/SYS/profile/PR1_DVEBMGS00_sapdb1
pr1adm 1867872 1 0 00:43:55 - 0:00 ora_reco_PR1
…
…
$ ps -ef | grep -c LOCAL=NO
507
$

IBM System p
Tactical AIX Monitoring: thread migrations S0-S5

Note: POWER5/5+ only; not applicable to POWER6
$ uptime ; mpstat -d 2 2
03:06PM up 133 days, 17:58, 2 users, load average: 5.87, 4.92, 4.66
System configuration: lcpu=24 ent=12.0
cpu cs ics bound rq push S3pull S3grd S0rd S1rd S2rd S3rd S4rd S5rd ilcs vlcs
0 1387 947 0 0 0 0 0 86.9 0.2 0.0 12.9 0.0 0.0 0 1317
1 4 2 0 0 0 0 0 100.0 0.0 0.0 0.0 0.0 0.0 0 1359
2 1165 752 0 0 0 0 0 85.2 0.2 0.0 14.6 0.0 0.0 0 883
3 0 0 0 0 0 0 0 100.0 0.0 0.0 0.0 0.0 0.0 0 576
4 868 576 0 0 0 0 0 83.5 0.3 0.0 16.3 0.0 0.0 0 801
5 3 2 0 0 0 0 0 100.0 0.0 0.0 0.0 0.0 0.0 0 882
6 1259 813 0 0 0 0 0 84.9 0.1 0.0 14.9 0.0 0.0 0 1091
7 2 2 0 0 0 0 0 100.0 0.0 0.0 0.0 0.0 0.0 0 1113
8 1207 809 0 0 0 0 0 85.5 0.0 0.0 14.5 0.0 0.0 0 1082
9 0 0 0 0 0 0 0 - - - - - - 0 1099
10 990 654 0 0 0 0 0 87.0 0.0 0.0 13.0 0.0 0.0 0 804
11 0 0 0 0 0 0 0 - - - - - - 0 802
12 1388 907 0 0 0 0 0 89.4 0.0 0.0 10.6 0.0 0.0 0 986
13 0 0 0 0 0 0 0 - - - - - - 0 990
14 1104 730 0 0 0 0 0 86.7 0.0 0.0 13.3 0.0 0.0 0 949
15 0 0 0 0 0 0 0 - - - - - - 0 960
16 659 411 0 0 0 0 0 88.8 0.0 0.0 11.2 0.0 0.0 0 524
17 0 0 0 0 0 0 0 - - - - - - 0 589
18 863 452 0 0 0 0 0 96.1 0.0 0.0 3.9 0.0 0.0 0 215
19 0 0 0 0 0 0 0 - - - - - - 0 316
20 1175 766 0 0 0 0 0 87.1 0.0 0.0 12.9 0.0 0.0 0 943
21 0 0 0 0 0 0 0 - - - - - - 0 951
22 1224 705 0 0 0 0 0 91.8 0.0 0.0 8.2 0.0 0.0 0 635
23 0 0 0 0 0 0 0 - - - - - - 0 639
ALL 13298 8528 0 0 0 0 0 87.7 0.1 0.0 12.2 0.0 0.0 0 20506
------------------------------------------------------------------------------------------------------
cpu cs ics bound rq push S3pull S3grd S0rd S1rd S2rd S3rd S4rd S5rd ilcs vlcs
0 1969 1277 0 0 0 0 0 83.2 0.0 0.0 16.8 0.0 0.0 0 1940
1 0 0 0 0 0 0 0 100.0 0.0 0.0 0.0 0.0 0.0 0 1968
2 1679 1068 0 0 0 0 0 85.7 0.0 0.0 14.3 0.0 0.0 0 1216
3 0 0 0 0 0 0 0 - - - - - - 0 1236
…

IBM System p
Tactical AIX Monitoring: detailed hdisk I/O dimensions

$ iostat -D
System configuration: lcpu=24 drives=87 paths=172 vdisks=0
hdisk0 xfer: %tm_act bps tps bread bwrtn

0.8 18.7K 2.3 7.0K 11.7K
read: rps avgserv minserv maxserv timeouts fails
0.6 3.0 0.1 267.1 0 0
write: wps avgserv minserv maxserv timeouts fails
1.7 5.5 0.3 320.5 0 0
queue: avgtime mintime maxtime avgwqsz avgsqsz sqfull
8.8 0.0 291.3 0.0 0.0 6349911
0.6 12.9K 1.7 1.2K 11.7K
0.0 4.8 0.1 301.8 0 0
1.7 5.4 0.4 281.1 0 0
11.3 0.0 275.6 0.0 0.0 6102418
…
10.2 789.3K 33.1 753.9K 35.4K
30.6 6.5 0.1 1.3S 0 0
2.5 2.5 0.2 912.0 0 0
4.3 0.0 1.1S 0.0 0.0 73320194
10.1 801.6K 33.7 764.2K 37.4K
31.2 6.3 0.1 1.2S 0 0
2.5 2.5 0.2 913.1 0 0
4.3 0.0 1.2S 0.0 0.0 74160810
…

IBM System p
Tactical AIX Monitoring: iostat -aT 2

$ iostat -aT 2 | grep -v "0.0 0.0 0.0 0 0"
System configuration: lcpu=24 drives=87 ent=12.00 paths=172 vdisks=0
tty: tin tout avg-cpu: % user % sys % idle % iowait physc % entc time
0.0 62.1 92.4 4.8 0.3 2.6 11.8 98.6 15:42:10
Adapter: Kbps tps Kb_read Kb_wrtn time

scsi0 2.0 0.5 4 0 15:42:10
Disks: % tm_act Kbps tps Kb_read Kb_wrtn time

hdisk0 0.0 2.0 0.5 4 0 15:42:10

scsi1 0.0 0.0 0 0 15:42:10

fcs0 150429.9 7234.0 298240 2369 15:42:10

hdisk86 14.0 228.2 16.0 264 192 15:42:10
hdisk87 11.0 88.1 11.0 176 0 15:42:10
hdisk88 9.5 64.1 8.0 128 0 15:42:10
hdisk89 20.0 2886.4 62.1 5760 8 15:42:10
hdisk90 20.5 9203.7 150.1 18392 0 15:42:10
hdisk91 34.0 7190.0 351.8 14368 0 15:42:10
hdisk92 59.5 6081.1 543.0 12152 0 15:42:10
hdisk93 32.5 384.3 42.0 768 0 15:42:10
hdisk94 12.0 260.2 21.0 304 216 15:42:10
hdisk95 8.5 64.1 8.0 128 0 15:42:10
hdisk96 39.0 7742.5 444.4 15280 192 15:42:10
hdisk97 33.5 400.3 45.0 800 0 15:42:10
hdisk98 12.0 260.2 19.5 328 192 15:42:10
hdisk99 10.5 68.1 8.5 136 0 15:42:10
…
…

IBM System p
Tactical AIX Monitoring: lsps ; mount

$ lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
paging01 hdisk80 pagingvg 32736MB 1 yes yes lv
hd6 hdisk0 rootvg 512MB 0 no yes lv
$ lsps -s
Total Paging Space Percent Used
130944MB 1%
$ mount
node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/dev/hd4 / jfs Oct 20 22:06 rw,log=/dev/hd8
/dev/hd2 /usr jfs Oct 20 22:06 rw,log=/dev/hd8
/dev/hd9var /var jfs Oct 20 22:06 rw,log=/dev/hd8
/dev/hd3 /tmp jfs Oct 20 22:06 rw,log=/dev/hd8
/dev/hd1 /home jfs Oct 20 22:08 rw,log=/dev/hd8
/proc /proc procfs Oct 20 22:08 rw
/dev/hd10opt /opt jfs Oct 20 22:08 rw,log=/dev/hd8
/dev/lvsapcds /sapcds jfs2 Oct 20 22:08 rw,log=/dev/lv00
/dev/lvcnvbt /cnv jfs2 Oct 20 22:20 rw,log=/dev/lvjfs00
/dev/lvhrtmpbt /hrtmp jfs2 Oct 20 22:20 rw,log=/dev/lvjfs00
/dev/lvoraclebt /oracle jfs2 Oct 20 22:20 rw,log=/dev/lvjfs00
/dev/lvorapr1bt /oracle/PR1 jfs2 Oct 20 22:20 rw,log=/dev/lvjfs00
/dev/lvmirrlogAp /oracle/PR1/mirrlogA jfs2 Oct 20 22:20 rw,log=/dev/lvjfs20
/dev/lvmirrlogBp /oracle/PR1/mirrlogB jfs2 Oct 20 22:20 rw,log=/dev/lvjfs20
/dev/lvoriglogAp /oracle/PR1/origlogA jfs2 Oct 20 22:20 rw,log=/dev/lvjfs20
/dev/lvoriglogBp /oracle/PR1/origlogB jfs2 Oct 20 22:20 rw,log=/dev/lvjfs20
/dev/lvsaparchbt /oracle/PR1/saparch jfs2 Oct 20 22:20 rw,log=/dev/lvjfs00
/dev/lvsapdata1bt /oracle/PR1/sapdata1 jfs2 Oct 20 22:20 rw,log=/dev/lvjfs01
…
…

IBM System p
Tactical AIX Monitoring: Used 1024-blocks-per-inode

$ df -k
Filesystem 1024-blocks Free %Used Iused %Iused Mounted on
/dev/hd4 262144 129016 51% 3777 3% /
/dev/hd2 3932160 544280 87% 42721 5% /usr
/dev/hd9var 1048576 334980 69% 4293 2% /var
/dev/hd3 1048576 731832 31% 519 1% /tmp
/dev/hd1 262144 63632 76% 2622 5% /home
/proc - - - - - /proc
/dev/hd10opt 262144 213832 19% 849 2% /opt
/dev/lvsapcds 2097152 456840 79% 1246 2% /sapcds
/dev/lvcnvbt 20480000 16993664 18% 715 1% /cnv
/dev/lvhrtmpbt 524288 506984 4% 30 1% /hrtmp
/dev/lvoraclebt 524288 436808 17% 2938 3% /oracle
/dev/lvorapr1bt 8978432 3838252 58% 21476 3% /oracle/PR1
/dev/lvmirrlogAp 3080192 2567348 17% 6 1% /oracle/PR1/mirrlogA
/dev/lvmirrlogBp 3080192 2567348 17% 6 1% /oracle/PR1/mirrlogB
/dev/lvoriglogAp 3080192 2567348 17% 6 1% /oracle/PR1/origlogA
/dev/lvoriglogBp 3080192 2567348 17% 6 1% /oracle/PR1/origlogB
/dev/lvsaparchbt 14680064 14296480 3% 7176 1% /oracle/PR1/saparch
/dev/lvsapdata1bt 268173312 73734764 73% 116 1% /oracle/PR1/sapdata1
/dev/lvsapreorgbt 25165824 19272876 24% 1153 1% /oracle/PR1/sapreorg
/dev/lvostage 2097152 1957092 7% 794 1% /oracle/stage
/dev/lvsapmntbt 2097152 1447736 31% 357 1% /sapmnt/PR1
…
…

IBM System p
Tactical AIX Monitoring: 1024-blocks/inode per “df –k”

avg
1024- blocks-
1024- blocks per- avg
Filesystem blocks Free Iused Mounted on Used inode bytes/inode
/dev/lvoriglogBp 3080192 2567348 6 /oracle/PR1/origlogB 512844 85474.0 87,525,376
/dev/lvsaparchbt 14680064 14296480 7176 /oracle/PR1/saparch 383584 53.5 54,737
/dev/lvsapdata1bt 268173312 73734764 116 /oracle/PR1/sapdata1 194438548 1676194.4 1,716,423,044

IBM System p
Tactical AIX Monitoring: shared memory segments

ipcs -bm: Comp% includes allocated shmemsegs, i.e. Oracle SGA
$ ipcs -bm | wc -l
164
$ ipcs -bm | grep -v sapsys
IPC status from /dev/mem as of Sun Mar 2 16:23:03 PST 2008
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 1048576 0xffffffff --rw-rw---- root system 8192
m 1048578 0x7800004f --rw-rw-rw- root system 16777216
m 4 0x670017bc --rw-r--r-- root system 12
m 5 0x680017bc --rw-r--r-- root system 106548
m 6 0x700017bc --rw------- root system 3152
m 7 0xffffffff --rw-rw---- root 219 4096
m 62914569 0x8aa6abe8 --rw-r----- orapr1 dba 12903030784
$ ipcs -bm | grep -c sapsys
152
$ ipcs -bm | grep sapsys
m 23068682 0xffffffff --rw-r----- pr1adm sapsys 1024
m 11 0x0382be84 --rw-rw-rw- pr1adm sapsys 4096
m 46137357 0xffffffff --rw------- pr1adm sapsys 2147483648
m 57671711 0x00002725 --rw-r----- pr1adm sapsys 562192
m 56623136 0x00002722 --rw-r----- pr1adm sapsys 656416
…

IBM System p
Tactical AIX Monitoring: netstat –ss (non-zero values)

$ netstat -ss
ip:
343290037 total packets received
257309 fragments received
5 fragments dropped after timeout
128641 packets reassembled ok
337255449 packets for this host
5871563 packets for unknown/unsupported protocol
396 packets not forwardable
3083156934 packets sent from this host
71209 output datagrams fragmented
149137 fragments created
2 packets dropped due to the full socket receive buffer
2115340 dead gateway detection packets sent
icmp:
5884385 calls to icmp_error
Output histogram:
echo reply: 1792801
destination unreachable: 5884312
707 messages with bad code fields
1 bad checksum
Input histogram:
echo reply: 2118565
destination unreachable: 5871963
echo: 1792801
1792801 message responses generated
igmp:
6 messages received
6 membership reports received
6 membership reports received for groups to which we belong
4 membership reports sent
tcp:
2416410139 packets sent
2830792567 data packets (3879350818 bytes)
14816541 data packets (1123365316 bytes) retransmitted
614103879 ack-only packets (290291730 delayed)
19 URG only packets
56895 window probe packets
…
…

IBM System p
Tactical AIX Monitoring: netstat -v (verbose)

$ netstat -v
…
…
-------------------------------------------------------------
ETHERNET STATISTICS (ent7) :
Device Type: EtherChannel
Hardware Address: 00:14:5e:48:c1:cc
Elapsed Time: 134 days 0 hours 21 minutes 27 seconds
Transmit Statistics: Receive Statistics:

-------------------- -------------------
Packets: 34718204002 Packets: 44862435158
Bytes: 48050223811333 Bytes: 9016251735773
Interrupts: 0 Interrupts: 27620561406
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 2 Packets Dropped: 0
Bad Packets: 0
Max Packets on S/W Transmit Queue: 227
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 2
Elapsed Time: 0 days 0 hours 0 minutes 0 seconds

Broadcast Packets: 14384 Broadcast Packets: 627555
Multicast Packets: 1 Multicast Packets: 3
No Carrier Sense: 0 CRC Errors: 0
DMA Underrun: 0 DMA Overrun: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors: 0
Deferred: 0 Packet Too Short Errors: 0
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by Adapter: 0
Single Collision Count: 0 Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 2
General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
…
…

IBM System p
Tactical AIX Monitoring: nfsstat

$ nfsstat
Server rpc:
Connection oriented
calls badcalls nullrecv badlen xdrcall dupchecks dupreqs
715102799 128 0 0 0 58378077 20
Connectionless
calls badcalls nullrecv badlen xdrcall dupchecks dupreqs
536 0 0 0 0 0 0
Server nfs:
calls badcalls public_v2 public_v3
715102602 27 0 0
Version 2: (256 calls)
null getattr setattr root lookup readlink read
256 100% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
wrcache write create remove rename link symlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
mkdir rmdir readdir statfs
0 0% 0 0% 0 0% 0 0%
Version 3: (715102346 calls)
null getattr setattr lookup access readlink read
208 0% 56755889 7% 1397024 0% 430178070 60% 4733070 0% 0 0% 55037597 7%
write create mkdir symlink mknod remove rmdir
43093505 6% 2409219 0% 7 0% 0 0% 0 0% 2329235 0% 14 0%
rename link readdir readdir+ fsstat fsinfo pathconf
110306 0% 63 0% 17771684 2% 9038704 1% 65544670 9% 206 0% 95 0%
commit
26702780 3%
Client rpc:
Connection oriented
calls badcalls badxids timeouts newcreds badverfs timers
272059 0 0 0 0 0 0
nomem cantconn interrupts
0 0 0
Connectionless
calls badcalls retrans badxids timeouts newcreds badverfs
79 0 0 0 0 0 0
timers nomem cantsend
0 0 0
…
…

IBM System p
Tactical AIX Monitoring: vmo ; ioo (root-use only)

$ vmo –L ; ioo –L
NAME CUR DEF BOOT MIN MAX UNIT TYPE

DEPENDENCIES
--------------------------------------------------------------------------------
cpu_scale_memp 8 8 8 1 64 B
--------------------------------------------------------------------------------
data_stagger_interval 161 161 161 0 4K-1 4KB pages D
lgpg_regions
--------------------------------------------------------------------------------
defps 1 1 1 0 1 boolean D
--------------------------------------------------------------------------------
force_relalias_lite 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
framesets 2 2 2 1 10 B
--------------------------------------------------------------------------------
htabscale n/a -1 -1 -4 0 B
--------------------------------------------------------------------------------
kernel_heap_psize 4K 4K 4K 4K 16M bytes B
--------------------------------------------------------------------------------
kernel_psize 16M 0 0 0 16M bytes B
--------------------------------------------------------------------------------
…
…
NAME CUR DEF BOOT MIN MAX UNIT TYPE
DEPENDENCIES
--------------------------------------------------------------------------------
j2_atimeUpdateSymlink 0 0 0 0 1 boolean D
--------------------------------------------------------------------------------
j2_dynamicBufferPreallocation
16 16 16 0 256 16K slabs D
--------------------------------------------------------------------------------
j2_inodeCacheSize 400 400 400 1 1000 D
--------------------------------------------------------------------------------
j2_maxPageReadAhead 128 128 128 0 64K 4KB pages D
--------------------------------------------------------------------------------
j2_maxRandomWrite 0 0 0 0 64K 4KB pages D
--------------------------------------------------------------------------------
j2_maxUsableMaxTransfer 512 512 512 1 4K pages M
--------------------------------------------------------------------------------
j2_metadataCacheSize 400 400 400 1 1000 D
--------------------------------------------------------------------------------
…

IBM System p
Tactical AIX Monitoring: lvmo -a -v `lsvg` (root-use only)
$ vmstat -v | grep pbuf

176559 pending disk I/Os blocked with no pbuf
$
$ for VG in `lsvg`
do
lvmo -a -v $VG
echo
done
vgname = rootvg
pv_pbuf_count = 512
total_vg_pbufs = 1024
max_vg_pbuf_count = 16384
pervg_blocked_io_count = 0
pv_min_pbuf = 512
global_blocked_io_count = 176559
vgname = oravg
pv_pbuf_count = 512
total_vg_pbufs = 8192
max_vg_pbuf_count = 16384
pervg_blocked_io_count = 176559
pv_min_pbuf = 512
global_blocked_io_count = 176559
…
…

IBM System p
Strategic Performance-Tuning Considerations

•Use AIX better, deeper, symbiotically; dare to extend the intrinsic elegance of Unix itself
•Assign and build discrete architectures that enhance unambiguous monitoring and tracking
•Use available resources to improve the performance/throughput of bottlenecked resources

•Maintain “sysminfree” somewhat beyond the greatest observed sum of vmstat:page:fi+fo
•Distinguish Enterprise workloads from Infrastructure workloads, and “architect” accordingly

•Performance-tuning ventures are seemingly only concerned with Enterprise workloads
•Adjust to the ever-greater scale, range and power inherent with each generation of technology
•Devise tactics that exploit the features of new technology for competitive advantage
•Or rather, don’t tolerate traditions that confound the value and benefits of new technology
•Trends toward greater virtualization and consolidation often create/insert more layers
•Tuning should thus entail the alignment/optimization of an infrastructure’s interfacing layers
•Most AIX 5L performance-tuning issues are founded on accepting the default values of AIX 5L
•Resize AIX 5L values to the specific scale, character and intensities of monitored workloads
•Demystifying the magic of AIX perf-tuning engenders confidence, confidence, confidence

•Enhancing your AIX perf-tuning competence makes daily-monitoring less of a chore/burden
•Successful performance-tuning is not unlike “money-for-nothing” – i.e. cost-effective efficiency

IBM System p
A typical resizing of AIX 5L (not 6.1) default values

•Most AIX 5L performance-tuning issues are founded on using the default values of AIX 5L
•Resize AIX 5L values to the specific scale, character and intensities of monitored workload
Assuming an Enterprise workload on a POWER5 p590 LPAR with an HMCprofile resembling:

eCPU=0.1/0.8/1.2 # consider using an eCPU-to-vCPU ratio of 0.1-to-1
vCPU=1/8/12 # with SMT=On, this grants 16 logical CPUs
gbRAM=1.0/36.0/48.0 # 1.0gbRAM/min ensures an LPAR can be booted for examination
uncapped/priority=128
VMO and IOO tuning parms

AIX vmo:memory_affinity=0 # Controversial -- but this does ensure equal-sized memory_pools
AIX vmo:cpu_scale_memp=4 # with 16 logical CPUs, this will create four mempools (9gbRAM each)
AIX vmo:lru_file_repage=0 # default=1; all Enterprise workload LPARs must have this disabled (=0)
AIX vmo:strict_maxperm=0 # default

AIX vmo:strict_maxclient=0 # thus lrud is only active when freemem falls below vmo:minfree per mempool
AIX vmo:maxperm%=80
AIX vmo:maxclient%=80
AIX vmo:minperm%=20 # default; IBM Best Practices is 3%-5% (we should discuss ramifications)
AIX vmo:minfree=[5*2048]=10240 # sysminfree= four mempools*10240= 40960 * (4k mempages)

AIX vmo:maxfree=[6*2048]=12288 # sysmaxfree= four mempools*12288= 48152 * (4k mempages)
AIX ioo:pv_min_pbuf=2048 # given today’s larger SAN LUNs, this ensures a sufficient allocation of pbufs per volume-group.
JFS2 tuning parms

AIX ioo:j2_nBufferPerPagerDevice=2048
AIX ioo:j2_dynamicBufferPreallocation=256
AIX ioo:j2_maxPageReadAhead=2048
AIX ioo:sync_release_ilock=1
JFS tuning parms

AIX ioo:numfsbufs=2048
AIX ioo:maxpgahead=2048

IBM System p
VMM:vmo nitty-gritty…
•Distinguish Enterprise workloads from Infrastructure workloads, and “architect” accordingly
AIX vmo:memory_affinity=0
Default=1 generally assumes an Infrastructure workload of short-lived processes. An Enterprise workload is comprised of long-lived
enduring user-processes, i.e. Oracle rdbms; as such, their comprising threads face never-ending migrations. Disabling this (=0)
ensures all memory_pools will be close-to-equal in size, and serve all migrated/migrating threads equally. Also, (=0) activates the
ability to use vmo:cpu_scale_memp below.
AIX vmo:cpu_scale_memp=(see below) # Use requires vmo:memory_affinity=0

Dividing the count of logical CPUs by cpu_scale_memp determines the number of created memory_pools. Assuming POWER5 and
the default (SMT=On), desired_vCPUs * 2 = logical_CPUs. Given an HMCprofile of desired:vCPU=8, there are 16 logical_CPUs;
16 divided-by (cpu_scale_memp=4) creates four 9gbRAM memory_pools, based on desired:RAM=36.0gb.
AIX vmo:lru_file_repage=0
Default=1 ensures comp and noncomp memory are treated equally. Setting =0 assumes an Enterprise workload that greatly favors
computational memory over noncomp JFS/JFS2 filesystem buffer-cache. Setting =0 causes lrud to steal only non-computational
mempages – until numclient<=minperm% or numperm<=minperm% -- when both comp and noncomp mempages can be stolen.
AIX vmo:strict_maxperm=0
AIX vmo:strict_maxclient=0
Disabling both (=0) causes lrud to be active only when needed to maintain minfree mempages per mempool.
AIX vmo:maxperm%=40
AIX vmo:maxclient%=40
AIX vmo:minperm%=20
When both vmo:strict_maxperm=0 and AIX vmo:strict_maxclient=0, maxperm% and maxclient% become merely
symbolic watermarks. That is, lrud scanning&stealing is no longer triggered when numperm>=maxperm% or
numclient>=maxclient%. This limits lrud scanning&stealing to maintaining minfree mempages per mempool.
AIX ioo:pv_min_pbuf=2048
Given today’s larger SAN LUNs, this ensures a sufficient allocation of pbufs per volume-group.

IBM System p
JFS/JFS2 filesystem I/O tuning parameters 1

•Most AIX performance-tuning issues are founded on accepting the default values of AIX 5L
•Adjust to the ever-greater scale, range and power inherent with each generation of technology
For optimal performance/throughput, we must assume LPARs housing Enterprise workloads are granted more than sufficient gbRAM
to say, just house the Oracle rdbms’ SGA (i.e. its shmemseg) and little else. Unfortunately, the optimum performance/throughput of
both POWER5 and any given enterprise SAN subsystem is dependent on seemingly surplus RAM. This unaccounted excess
includes the filesystem buffer-cache (noncomp), AIX pinned memory structures (including pbuf/fsbuf/psbuf’s), and any immediately-
available memory, (aka free memory pages, or simply fre from AIX:vmstat:memory:fre).
Example: All JFS/JFS2 filesystem fi/fo’s, as well as, pagingspace pi/po’s are conveyed to/from main memory via pbuf’s-per-vg,
fsbuf’s-per-filesystem, psbuf’s-per-pagingspace. If the intensity of coincident and nearly-coincident I/O’s exhaust entitlements of
pbuf/fsbuf/psbuf’s, then I/O’s slow to rates of real-time de-staging, i.e. I/O’s blocked with no fsbuf. For a worthwhile trade
of lruable memory, greater allocations of “buf’s” -- matched to subsystem capacities -- can greatly temper I/Os blocked.
Typical JFS/JFS2 SAN reads:

Enterprise SAN cacheÆFCfabricÆAIX pbuf/fsbuf/psbuf’sÆavailable vmstat:memory:freÆbecomes vmstat:page:fi
Typical JFS/JFS2 SAN writes:

Available vmstat:memory:freÆbecomes vmstat:page:foÆAIX pbuf/fsbuf/psbuf’sÆFCfabricÆenterprise SAN cache
Addressing vmstat I/Os blocked from slide 9:

0 pending disk I/Os blocked with no pbuf # volume_group
0 paging space I/Os blocked with no psbuf # pagingspace
87988 filesystem I/Os blocked with no fsbuf # JFS
0 client filesystem I/Os blocked with no fsbuf # NFS and VxFS
11195356 external pager filesystem I/Os blocked with no fsbuf # JFS2
This example of JFS/JFS2 tuning values is scaled to 3yr-old enterprise SAN storage technology.
•AIX ioo:numfsbufs=2048 # default 196 vs 2048 JFS fsbuf’s
•AIX ioo:j2_nBufferPerPagerDevice=2048 # default 512 vs 2048 JFS2 fsbuf’s
•AIX ioo:j2_dynamicBufferPreallocation=256 # default 16 vs 256 dynamic fsbuf’s

IBM System p
JFS/JFS2 filesystem I/O tuning parameters 2

•Most AIX performance-tuning issues are founded on accepting the default values of AIX 5L
•Note the ever-greater scale, range and power inherent with each generation of technology
Recommendations for resizing the default:

AIX ioo:maxpgahead=2048 # JFS default 8
AIX ioo:j2_maxPageReadAhead=2048 # JFS2 default 128
In general, each successive sequential JFS/JFS2 filesystem read doubles its page read-ahead request until it reaches its
JFS:maxpgahead or JFS2:j2_maxPageReadAhead value. These pages are the familiar 4096byte memory pages.
•The JFS default (maxpgahead=8) “page read-ahead” series resembles:

24888888888888888888888888888…
This is essentially a series of: 8 * 4k-mempgs/read = 8 * 4096bytes/read = 32kb/read.
•The JFS2 default (j2_maxPageReadAhead=128) “page read-ahead” series resembles:

2 4 8 16 32 64 128 128 128 128 128 128 128 128 128 …
•The JFS2 recommendation (j2_maxPageReadAhead=2048) “page read-ahead” series resembles:

2 4 8 16 32 64 128 256 512 1048 2048 2048 2048 2048 …
Note: The column-units of vmstat:memory (avm fre) and vmstat:page (fi fo pi po fr sr) are in 4096byte memory pages.

-------- ----------- ------------------------ ------------ ----------------------- --------
9 31 0 7704384 47859 25421 1693 1 0 26406 32079 10952 58448 49189 24 10 45 21 4.41 36.8 16:51:01
10 29 0 7703858 52000 24699 1796 0 0 27820 81344 10500 56043 47717 27 11 40 22 4.96 41.3 16:51:03
7 37 0 7703875 50080 17785 1394 1 0 17675 18239 11725 66938 51732 31 12 21 36 5.66 47.2 16:51:05

IBM System p
Formula for tuning vmo:minfree and vmo:maxfree

•Use AIX better, deeper, symbiotically; dare to extend the intrinsic elegance of Unix itself
•Maintain “sysminfree” somewhat beyond the greatest observed sum of vmstat:page:fi+fo
-------- ----------- ------------------------ ------------ ----------------------- --------
6 27 0 7703121 46805 16326 1784 0 0 15360 15493 10862 49438 39443 26 10 41 23 4.73 39.5 16:53:02
5 28 0 7704199 49096 17880 1667 0 0 20860 21274 8790 44411 33839 26 8 47 19 4.41 36.8 16:53:04
8 29 0 7703104 47639 19724 944 0 0 19067 19082 8888 43816 32166 28 7 48 17 4.52 37.7 16:53:06
9 22 0 7703094 47085 18003 834 0 0 18290 18322 9486 63670 36327 27 8 47 18 4.54 37.8 16:53:08
6 33 0 7702739 47932 14074 1762 0 0 15552 15572 9293 65875 32604 30 6 47 17 4.67 38.9 16:53:10
8 22 0 7703907 49302 21413 1479 0 0 23607 23658 9242 56128 36406 26 8 49 17 4.44 37.0 16:53:12
6 30 0 7702650 51610 20085 2260 0 0 22295 22358 8715 47855 36858 22 8 56 15 3.86 32.1 16:53:14
7 28 0 7702691 52352 17968 11672 0 0 26896 29316 8897 38839 61368 25 13 33 29 5.07 42.3 16:53:16
5 35 0 7702715 47761 18939 12076 0 0 25486 106439 9040 51783 46374 26 13 27 34 5.17 43.0 16:53:18
5 40 0 7702440 48472 19811 12096 0 0 28863 81215 9829 48196 46682 28 12 37 24 5.19 43.2 16:53:20
8 26 0 7702448 47810 19487 1688 0 0 20050 83134 10013 60880 43185 24 9 43 24 4.39 36.6 16:53:22
4 26 0 7702416 52917 16696 1555 0 0 20583 58802 8862 50358 33387 23 9 55 13 4.17 34.7 16:53:24
•Calculating sysminfree/sysmaxfree using the AIX default values:

AIX default vmo:minfree=960 # sysminfree= four mempools*960 = 3840 * (4k mempages) = 15.7mb
AIX default vmo:maxfree=1088 # sysmaxfree= four mempools*1088 = 4352 * (4k mempages) = 17.8mb
•Calculating sysminfree/sysmaxfree using the resized example values -- plus arbitrarily formulating the use of
AIX ioo:j2_maxPageReadAhead=2048
as a multiplying factor:
AIX vmo:minfree=[5*2048]=10240 # sysminfree= four mempools*10240 = 40960 * (4k mempages) = 167.7mb
AIX vmo:maxfree=[6*2048]=12288 # sysmaxfree= four mempools*12288 = 48152 * (4k mempages) = 197.2mb
Essentially, this range of sysminfree-to-sysmaxfree means 20-24 concurrent streams of JFS2-sequential reads can be
accommodated at any given moment, i.e. AIX pbuf/fsbuf/psbuf’sÆavailable vmstat:memory:fre.

IBM System p
Summary, Miscellany and Closing Thoughts

•One can’t tune what is not taxed; One can’t tune what is not tracked, thus build in discretely trackable layers
•Ensure sufficient allocations of pbuf/fsbuf/psbuf’s for all manner of JFS/JFS2 SAN I/O
•Aim to maintain at least 5-digits of free_mempages at all times
•Exploit the concurrency of multiple 8gbRAM-18gbRAM memory_pools with at least one vCPU per mempool
•Ensure calculations of vCPU’s also includes one lrud thread per memory_pool
•Manpage and practice using variations of vmstat, mpstat, iostat, ps and perhaps topas/nmon
•Collect an hourly execution of “vmstat -sv” to a 14-day rewriting textfile
•Be mindful that vmstat:memory:avm*4096 is best maintained at no higher than 80% of LPAR gbRAM.
•POWER5/5+: Be mindful of S4/S5 thread migration latencies; monitor “mpstat –dw 2”
•Limit the count of active LPARs to less-than/equal-to the count of CPU-cores in any given SysP frame
•Execute “iostat –D” for a sense of hdisk I/O dimensions, i.e. read/write: minsrv, avgsrv, maxsrv.
•Implement a spectrum JFS2-mount options: -o rbw, -o rbr, -o rbrw, -o cio
•A surplus of unused CPU-cycles may be due to insufficient gbRAM, and not “there’s no work to be done”
•Use vmstat and mpstat to discern&characterize a persistent count of active threads
•Maintain the persistent count of active threads between 1-and-2 times the count of vCPUs, and not higher
•Meet the challenge of discerning the bottlenecking resource of any given active workload
•Be mindful of Folding vCPU’s when deploying VIOS LPARs serving I/O-intensive client-LPARs
•Be mindful to apply appropriate-technology appropriately; do not engineer complexity without need
•Always thoroughly understand and exercise the implementations of new technologies and operations
•Note how an explosive burst of pagingspace-pageout’s can appear like a system hang
•Note how bursts of pagingspace pi/po’s can gravely confound performance, yet leave few clues it happened
•The trends of technology tend toward every manner of consolidation, virtualization, mobility, autonomy
•IT over time grows bigger baskets holding more eggs -- managed by fewer smarter hens

Tactics For Monitoring Tuning AIX 5L 6.1 Apr28-09 V1.0

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Tactics For Monitoring Tuning AIX 5L 6.1 Apr28-09 V1.0

Încărcat de

Drepturi de autor:

Formate disponibile

IBM System p

© 2007 IBM Corporation

Tactical AIX 5L/6.1 Monitoring & Tuning: Strategic Methodology

 Review exhaustions and surpluses in-light of workload expectations

 Map characteristic AIX dynamic-behaviors to their functional sources

 Devise tactics to relieve exhaustions by exploiting surplus resources

 Singularly Divide-and-Assign: Build Discrete Accountable Constructs

2 © 2007 IBM Corporation IBM Systems

Tactical AIX 5L/6.1 Monitoring and Tuning: Related Thoughts

•size/scale – Capacities vs. workload, aka “tuning-by-hardware”

3 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: Size&Scale (HMCprofile)

4 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: prtconf Exam Tech-Stack 1

Paging Space Information

5 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: lsdev Exam Tech-Stack 2

6 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: lscfg Exam Tech-Stack 3

7 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: Count-indicators 1

8 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: Count-indicators 2

9 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: no-load leave-it-up everyday

kthr memory page faults cpu

10 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: “memory over-commitment?” Feel-Good 1

System configuration: lcpu=24 mem=73728MB ent=12.00

kthr memory page faults cpu time

11 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: vmstat with mpstat

System configuration: lcpu=24 mem=73728MB ent=12.00

kthr memory page faults cpu time

System configuration: lcpu=24 ent=12.0

12 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: perhaps only some topas

Name PID CPU% PgSp Owner

13 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: some characterizing “signposts”

14 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: investigate by digging deeper

15 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: thread migrations S0-S5

System configuration: lcpu=24 ent=12.0

16 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: detailed hdisk I/O dimensions

hdisk0 xfer: %tm_act bps tps bread bwrtn

17 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: iostat -aT 2

System configuration: lcpu=24 drives=87 ent=12.00 paths=172 vdisks=0

Adapter: Kbps tps Kb_read Kb_wrtn time

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time

Adapter: Kbps tps Kb_read Kb_wrtn time

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time

Adapter: Kbps tps Kb_read Kb_wrtn time

Disks: % tm_act Kbps tps Kb_read Kb_wrtn time

18 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: lsps ; mount

19 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: Used 1024-blocks-per-inode

20 © 2007 IBM Corporation IBM Systems

Tactical AIX Monitoring: 1024-blocks/inode per “df –k”

Review exhaustions and surpluses in-light of workload expectations

Map characteristic AIX dynamic-behaviors to their functional sources

Devise tactics to relieve exhaustions by exploiting surplus resources

Singularly Divide-and-Assign: Build Discrete Accountable Constructs

AIX vmo:minfree=[52048]=10240 # sysminfree= four mempools10240= 40960 * (4k mempages)