Performance and Recommended Use of AB287A, HP PCI-X 133MHz 10Gigabit Ethernet SR Adapter

From Results on an HP Integrity rx7640 server

May 2006

Introduction ....................................................................................................3 Performance Summary.....................................................................................3 Recommended Use Based on Performance and Design.......................................6 How We Measured 10 Gigabit Ethernet Efficiency ............................................7 Features and Benefits of the AB287A................................................................9 For More Information.....................................................................................10 Legal Notices................................................................................................11

Introduction
This article highlights the excellent performance of HPs PCI-X 133MHz10 Gigabit Ethernet SR Adapter, AB287A (Figure 1). The performance measurements were done on an HP Integrity rx7640 server. This article also provides suggestions based on the performance and technology that will help you to best meet your business needs.

Figure 1

AB287A: PCI-X 133MHz 10 Gigabit Ethernet SR Adapter

Performance Summary
The AB287A card provides exceptional performance when used in accordance with the recommendations in this paper. Figure 2 shows the average of sustained transmit, receive and bidirectional throughput with a Maximum Transmission Unit (MTU) of 9000 bytes and 1500 bytes as a percentage of the maximum possible bandwidth of the PCI-X

133MHz bus for different types of traffic combinations (Ethernet transmit, Ethernet receive, Ethernet bi-directional) when the recommended models are used. The AB287A has excellent sustained performance at both 9000bytes MTU and 1500bytes MTU.

Figure 2

Performance Summary

10 Gigabit 9000MTU Performance

Following are the highlights of the excellent performance achieved when operating at 9000MTU:
Receive traffic achieved a sustained throughput of 7.90 gigabits per second, amounting to about 92.8% of the theoretical maximum bandwidth of the PCI-X 133MHz bus. The service demand (which the amount of time it takes the CPU to handle one kilobyte of data) was 1.4s/KByte. The transmit throughput was 7.56 gigabits per second, which is about 88.9% of the theoretical maximum bandwidth of the PCI-X 133MHz bus. The service demand was 1.1s/KByte. The bi-directional throughput was 7.64 gigabits per second, which is approximately 89.8% of the theoretical maximum bandwidth of the PCI-X 133MHz bus. The sevice

demand was 1.3s/KByte.

Figure 3 shows the throughput achieved by the 10Gigabit adapter operating at 9000MTU.

Figure 3:10GigE throughput at 9000MTU

10 Gigabit 1500MTU Performance

Following are highlights of the impressive performance achieved during traffic testing at 1500MTU:
Receive traffic achieved 7.05 gigabits per second throughput with a service demand of 4.3s/KByte. A throughput of 6.9 gigabits per second was achieved with Transmit traffic; the service demand was 2.7s/KByte. Bi-directional traffic achieved 7.3 gigabits per second throughput with a service demand of 3.1s/KByte.

Figure 4:10GigE throughput at 1500MTU

Recommended Use Based on Performance and Design

HP recommends the following usage model to achieve the best performance:
Run the AB287A cards in the highest performing PCI-X slots. Slots 3, 4, 5 and 6 on

each of the I/O partitions are the recommended high performance PCI-X slots in the HP Integrity rx7640 used in our performance testing. Please refer to the appropriate Server Installation Guide for information on how to identify high performance slots. Configure Jumbo Frames (9000MTU) if possible. Jumbo Frames can provide excellent throughput with reduced CPU overhead. However, Jumbo Frames may not be usable in all topologies. Please refer to the HP document Boosting server-to-server Gigabit Throughput with Jumbo Frames for more information on advantages and recommended usage models. The 10Gigabit software is optimized for multiprocessor support. Best performance will be achieved when running multiple data streams (TCP connections, for example) on a system with four or more processors configured. The software will automatically assign appropriate processor resources to each data stream. This feature needs the following patches and enhancement products to be installed: TOUR 3.0 (Transport Optional Update release), PHNE_32277 (Streams Cumulative Patch), PHNE_33429 (LAN cumulative patch) and AR0512 release of IP Filter.

The recommendations mentioned here were followed to achieve the results shown in this article. Card throughput will be affected with non-recommended configurations. Please contact an HP representative for additional help in understanding how to best deploy the AB287.

How We Measured 10 Gigabit Ethernet Efficiency

This article highlights the AB287A throughput. Throughput is the data transfer rate, or data rate--the amount of time it takes data to move from one place to another. In this article, its shown for one-way transfers as well as two-way. Throughput measures how well programs run with a certain workload and how quickly user requests can be handled. This article also provides the Service Demand for each throughput test. Service demand is the amount of time (in microseconds) it takes one CPU to handle one kilobyte of data. It s a normalized measurement because it eliminates disparities due to differences in the quantities, types, or frequencies of CPUs. Service Demand is an important capacity planning & performance metric that is sometimes considered when comparing different server models. In this paper, the results for service demand are mentioned in text but not shown in graphs. The performance results shown in this article were measured with the netperf benchmarking software. Tests were run with a single AB287A card in a 2-cell rx7640 with four processors (with two processors in each cell). The adapter was installed in slot 3. The December 2005 release of the driver for HP-UX 11i v2, supports multi-queue using destination port based steering. The driver programs a card resident table to look-up the destination port of the received packets and directs them to appropriate queues. It updates the table dynamically as operating conditions change. This feature is supported only for TCP packets and provides good performance improvement when there is a mix of applications that do not use the same destination port. Details of the systems used and the software versions are shown in Table 1. The message and socket sizes used with netperf for the 10 Gigabit Ethernet transmit, receive, and bidirectional tests are as follows. At 1500 MTU, the transmit test used a message size 64240 bytes and socket size of 512K. Receive and bi-directional tests used message size of 64240 bytes, socket size of 256K. At 9000 MTU, all the tests used a message size of 71680 bytes and socket size of 512K. Performance will vary when this product is used on different systems or software. NOTES:

The observed performance results are consistent across all of the same type of I/O slots of the system; On HP Integrity rx7640, the high performance slots are slots 3, 4, 5 and 6 on each of the I/O Partitions. Performance on other slots will be lower. The core I/O cards in the rx7640 handled minimal site LAN traffic during performance tests. Processor affinity was used to assign netperf processes to specific CPUs.

Table 1 summarizes the products used to measure the performance.

Table 1: Products Used in the Performance Measurement Tests

Servers Tested

rx7640 Server. 2-cell configuration. Four 1.6 GHz CPU Itanium2 processors with 6 MB cache Two CPUs per cell System Memory: 20GB. Eight DIMM slots in each of the cells were populated. Operating System - HP-UX 11i version 2 of Dec 2005 (B.11.23.0512).

Cards Tested

AB287A PCI-X 133MHz 10Gigabit Ethernet SR Adapter

PCI-X (64-bit, 133 MHz, 3.3v) 10GigE LAN Driver (ixgbe)

version B.11.23.06

Clients generating the test load for Gigabit Ethernet

Eight rx2600 servers, each with:

Two, 1.5 GHz Itanium2 CPUs HP-UX 11i version 2

(B.11.23.0409)

A7012A PCI-X 1000Base-T card LAN Driver (iether)

version B.11.23.05

Benchmark software for Gigabit Ethernet tests

Neterf 2.3pl1 is the benchmarking software suite that generated LAN traffic for the 10 Gigabit Ethernet performance tests. For more information about netperf or to get a free copy, go to http://www.netperf.org

Features and Benefits of the AB287A

Features and benefits of the AB287A include:
PCI-X operation in 133 MHz, 64-bit mode. Conformance to IEEE 10GBASE-SR using multi-mode fiber. Operating distances from 7 to 984 (2 to 300 meters). Support for several features that improve CPU utilization: Jumbo Frames with a maximum transmission unit (MTU) of 9000 bytes. On board TCP Segmentation Offload (TSO) of IPv4. On board Checksum Offload (CKO) for TCP, UDP and IPv4. Support for multiple inbound and outbound queues. This allows traffic to be distributed to multiple processors and prevents saturation of a single processor. VLAN priority and destination port based packet steering are supported. All onboard memory and storage are ECC protected; with single bit errors automatically corrected and double bit errors detected. Support of HP Serviceguard and Auto-Port Aggregation (APA). APA is currently supported only for high availability. Virtual LANs (VLANs to provide increased network flexibility). Online addition/replacement (OLAR) on supported systems. Support of 64-bit management information base (MIB) statistics. Directs the data link service (DLS) provider to return 64 bit statistics.

Configuration through graphic user interface SAM or command line. Support for subnetwork-access protocol (SNAP) network-layer protocol encapsulation.

For More Information

For more information about the products described in this paper such as a current list of tested HP products or supported systems, please go to:
http://www.hp.com/products1/unixserverconnectivity.

This paper is the latest in a series of white papers detailing the performance of HPs link and server products. For a complete list of white papers on HPs networking and I/O products including Gigabit Ethernet and Fibre Channel solutions, go to: http://docs.hp.com. For further assistance including a detailed analysis of your specific requirements and needs, please contact your local HP Sales Representative.

10

Legal Notices

2006 Hewlett-Packard Company, L.P. The information contained herein is subject to change without notice.
The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. HP-UX , Serviceguard are registered trademarks of the Hewlett-Packard Corporation. PCI-X is a registered trademark of the PCI SIG. All other trademarks and registered trademarks are the property of the respective corporations.

11