AIX 10Gbe Recomended Performance Set-up

Daniel Martin-Corben | June 26 2013 | Visits (2557)

inShare1

Out of the box 10G Ethernet is not tuned at all, which can result in total system throughput only slightly better than 1G. To max out 10G's capabilities, you need to make some adjustments. I hope that this blog will give you a number of tips to tuning your system for the best 10Gbe performance. Bring down the adapter # ifconfig en1 down detach Set jumbo frames # chdev -l ent1 -a jumbo_frames=yes NOTE - The transmission speed of Virtual Ethernet adapters is in the range of 1-3 Gigabits per second, depending on the transmission (MTU) size. The Virtual Ethernet Adapter supports, as Gigabit (Gb) Ethernet, Standard MTU-Sizes of 1500 Byte and Jumbo frames with 9000 Byte. Additionally for Gb Ethernet, the MTU-Size of 65280 Bytes is also supported in Virtual Ethernet. So, the MTU of 65280 Bytes can be only used inside a Virtual Ethernet ONLY. Set the MTU # chdev -l en1 -a mtu=9000 The final tuning step is to enable Ethernet flow control throughout your network. Flow control prevents retransmissions, and the resulting network congestion, by throttling transmissions at their source when any buffer on the path between two endpoints approaches overflow. For AIX, use the following command, though your also need to speak to your networking team to see if its been enable on the related switches: # chdev -l ent0 -a flow_ctrl=yes Bring up the adapter # chdev -l en0 -a state=up Testing the network bandwidth with ftp between two AIX (or two Unix) LPARs, first the 10Gbe adapter: # ftp <AIX box="box"> provide login credentials 230 User root logged in. ftp> put "|dd if=/dev/zero bs=1m count=100" /dev/null 200 PORT command successful. 150 Opening data connection for /dev/null. 100+0 records in. 100+0 records out. 226 Transfer complete. 104857600 bytes sent in 0.3988 seconds (2.567e+05 Kbytes/s) local: |dd if=/dev/zero bs=1m count=100 remote: /dev/null ftp> by 221 Goodbye. Then on the traditional 1Gb adapter: # ftp <AIX box="box"> provide login credentials 230 User root logged in. ftp> put "|dd if=/dev/zero bs=1m count=100" /dev/null 200 PORT command successful.

150 Opening data connection for /dev/null. 100+0 records in. 100+0 records out. 226 Transfer complete. 104857600 bytes sent in 3.956 seconds (2.588e+04 Kbytes/s) local: |dd if=/dev/zero bs=1m count=100 remote: /dev/null The following table shows suggested sizes for the tunable values to obtain optimal performance, based on the type of adapter and the MTU size:
Device Speed MTU size tcp_sendspace tcp_recvspace sb_max1 rfc1323

Gigabit Gigabit Gigabit 10 Gigabit 10 Gigabit

1500 9000 9000 1500 9000

131072 131072 262144 131072 262144

65536 65535 1310722 65536 131072

131072 262144 524288 131072 262144

0 0 1 0 1

There are five ISNO parameters for each supported interface: rfc1323, tcp_nodelay, tcp_sendspace, tcp_recvspace, and tcp_mssdflt. When set, the values for these parameters override the system-wide parameters of the same names that had been set with the no command. The network option use_isno, set with the no command, must have a value of 1 for the ISNOs to take effect. The default value for use_isno is 1. Verify general system and interface support using the no and lsattr commands. Ensure the use_isno option is enabled using a command similar to the following: # no -a | grep isno use_isno=1 Enable isno # no -p -o use_isno=1 Modification to restricted tunable use_isno, confirmation required yes/no: yes Warning: a restricted tunable has been modified Setting use_isno to 1 Setting use_isno to 1 in nextboot file NOTE This seems to be only true on older OS levels, use_isno seems to be enabled as default in AIX7. Ensure the interface supports the five new ISNOs using the lsattr -El command, as shown in the following: # lsattr -E -l en0 -H attribute value description rfc1323 N/A tcp_nodelay N/A tcp_sendspace N/A tcp_recvspace N/A tcp_mssdflt N/A Set the interface specific values, using either the ifconfig or chdev command. The ifconfig command sets values temporarily, which is recommended for testing. The chdev command alters the ODM, so customized values remain valid after reboot. Set the tcp_recvspace and tcp_sendspace to 64K and enable tcp_nodelay by using one of the following:

# ifconfig en0 tcp_recvspace 65536 tcp_sendspace 65536 tcp_nodelay 1 # chdev -l en0 -a tcp_recvspace=65536 -a tcp_sendspace=65536 -a tcp_nodelay=1
Enable the rfc1323 Option, this enables TCP enhancements as specified by RFC 1323, TCP Extensions for hight perforrmance. To do this AIX system wide -

# no -p -o rfc1324=1 Alternatively, assuming the no command reports an rfc1323=1 global value, the root user can turn rfc1323 off for all connections over en0 with the following commands: # ifconfig en0 rfc1323 1 # chdev -l en0 -a rfc1323=1 Verify the settings using the ifconfig or lsattr command, as shown in the following example: # ifconfig en0 <UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT> en0: flags=e080863 inet 9.19.19.19 netmask 0xffffff00 broadcast 9.19.19.255 tcp_sendspace 65536 tcp_recvspace 65536 tcp_nodelay 1 rfc1323 1 # lsattr -El en0 rfc1323 0 N/A True tcp_nodelay 1 N/A True tcp_sendspace 65536 N/A True tcp_recvspace 65536 N/A True tcp_mssdflt N/A True

Tags: aix 10gbe performance network networking power