Spirent TestCenter Application Note #1

10-Gigabit Ethernet Testing

April 2006

P/N 71-000689 REV A

Spirent Communications, Inc.
26750 Agoura Road
Calabasas, CA
91302 USA

Support Contacts
U.S. Europe
E-mail: support@spirentcom.com E-mail: support.europe@spirentcom.com
Web: http://support.spirentcom.com Web: http://support.spirentcom.com
Toll Free: +1 800-SPIRENT (+1 800-774-7368) Phone: +33 (0) 1 61 37 22 70
Phone: +1 818-676-2616 Fax: +33 (0) 1 61 37 22 51
Fax: +1 818-880-9154
China
E-mail: supportchina@spirentcom.com
Web: http://support.spirentcom.com.cn
Toll Free: +1 800-810-9529 (mainland China only)
Phone: +86 10 8233 0033 (rest of the world)
Fax: +86 10 8233 0022

Copyright
© 2006 Spirent Communications, Inc. All Rights Reserved.
All of the company names and/or brand names and/or product names referred to in this document, in particular, the
name “Spirent” and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries,
pending registration in accordance with relevant national laws. All other registered trademarks or trademarks are the
property of their respective owners. The information contained in this document is subject to change without notice
and does not represent a commitment on the part of Spirent Communications. The information in this document is
believed to be accurate and reliable, however, Spirent Communications assumes no responsibility or liability for any
errors or inaccuracies that may appear in the document.

Limited Warranty
Spirent Communications, Inc. (“Spirent”) warrants that its Products will conform to the description on the face of
order, that it will convey good title thereto, and that the Product will be delivered free from any lawful security interest
or other lien or encumbrance.
Spirent further warrants to Customer that hardware which it supplies and the tangible media on which it supplies
software will be free from significant defects in materials and workmanship for a period of twelve (12) months, except
as otherwise noted, from the date of delivery (the “Hardware Warranty Period”), under normal use and conditions.
To the extent the Product is or contains software (“Software”), Spirent also warrants that, if properly used by Customer
in accordance with the Software License Agreement, the Software which it supplies will operate in material
conformity with the specifications supplied by Spirent for such Software for a period of ninety (90) days from the date
of delivery (the “Software Warranty Period”). The “Product Warranty Period” shall mean the Hardware Warranty
Period or the Software Warranty Period, as applicable. Spirent does not warrant that the functions contained in the
Software will meet a specific requirement or that the operation will be uninterrupted or error free. Spirent shall have no
warranty obligations whatsoever with respect to any Software which has been modified in any manner by Customer or
any third party.
Defective Products and Software under warranty shall be, at Spirent's discretion, repaired or replaced or a credit issued
to Customer's account for an amount equal to the price paid for such Product provided that: (a) such Product is
returned to Spirent after first obtaining a return authorization number and shipping instructions, freight prepaid, to
Spirent's location in the United States; (b) Customer provides a written explanation of the defect or Software failure
claimed by Customer; and (c) the claimed defect actually exists and was not caused by neglect, accident, misuse,
improper installation, improper repair, fire, flood, lightning, power surges, earthquake, or alteration. Spirent will ship
repaired Products to Customer, freight prepaid, within ten (10) working days after the receipt of defective Products.
Except as otherwise stated, any claim on account of defective materials or for any other cause whatsoever will
conclusively be deemed waived by Customer unless written notice thereof is given to Spirent within the Warranty
Period. Spirent reserves the right to change the warranty and service policy set forth above at any time, after
reasonable notice and without liability to Customer.
TO THE EXTENT PERMITTED BY APPLICABLE LAW, ALL IMPLIED WARRANTIES, INCLUDING BUT
NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT AND FITNESS
FOR A PARTICULAR PURPOSE, ARE HEREBY EXCLUDED, AND THE LIABILITY OF SPIRENT, IF ANY,
FOR DAMAGE RELATING TO ANY ALLEGEDLY DEFECTIVE PRODUCT SHALL BE LIMITED TO THE
ACTUAL PRICE PAID BY THE CUSTOMER FOR SUCH PRODUCT. THE PROVISIONS SET FORTH ABOVE
STATE SPIRENT'S ENTIRE RESPONSIBILITY AND CUSTOMER'S SOLE AND EXCLUSIVE REMEDY WITH
RESPECT TO ANY BREACH OF ANY WARRANTY.
10-Gigabit Ethernet Testing
The development of high throughput and low latency 10-Gigabit network switches and routers
is challenging. Network equipment manufacturers of switches and routers with 10-Gigabit
Ethernet network interfaces are challenged to provide switch fabrics that have high availability
10 Gigabits-per-second line rate forwarding that meets the requirements of converged, multi-
service networks that demand low latency performance. This application note introduces
Spirent Communications’ new test tool platform, Spirent TestCenter™, how it addresses the
critical aspects of 10-Gigabit Ethernet performance testing, and the network protocols that
affect 10-Gigabit Ethernet layer 2 and layer 3 switch performance. Spirent TestCenter is
uniquely suited to assist network equipment manufacturers to meet the challenges of
successfully delivering 10GbE network devices.

In this document...
• Overview of 10-Gigabit Ethernet Network Devices . . . . 4

• 10-Gigabit Ethernet Network Interfaces for Enterprise Networks . . . . 5

• 10-Gigabit Ethernet Network – Two Rates in One . . . . 6

• New Applications and Protocols for 10-Gigabit Ethernet . . . . 8

• Scalable 10GbE Networks Create Test Challenges . . . . 12

• 10-Gigabit Ethernet Test Considerations – 10GbE is not Gigabit Ethernet . . . . 12

• Traditional Switch Testing and 10GbE Considerations . . . . 13

• Test Setup for 10GbE using the RFCs . . . . 14

• Line Rate Testing for 10-Gigabit Ethernet Switches and Routers . . . . 14

• Key Considerations for 10GbE Tests . . . . 17

• Conclusion . . . . 18

• Spirent TestCenter Products for 10GbE Switch and Router Testing . . . . 19

Spirent Communications Application Note | 3

10-Gigabit Ethernet Testing
Overview of 10-Gigabit Ethernet Network Devices

Overview of 10-Gigabit Ethernet Network Devices

The deployment of 10-Gigabit Ethernet (10GbE) ports in network switching and routing
equipment is increasing rapidly. Most major network equipment manufacturers (NEMs)
offer layer 2 switches, layer 3 aware switches, and layer 3 routers. These network devices
have various combinations of copper 10/100/1000Mbps and fiber Gigabit Ethernet (GE)
combined with 10-Gigabit Ethernet (10GbE) ports. 10GbE is becoming a prevalent
interface on network servers, firewalls, and appliances. Typically, the switch
configurations that are equipped with 10GbE network interfaces are:
• Fixed, managed switches – these switches have a fixed number of copper or fiber GE
ports with two or four 10GbEports. Thye cannot be expanded with additional ports.
They can be stacked (interlinked) to form a larger network.
• Modular switches/ routers – these are switches with a chassis that have individual
blades for GE and 10GbE. The blades are removable, and the user can expand or
change the port type and rate mix as required. These switches have the ability to
connect directly to GE and 10GbE servers, 10GbE storage area networks, network
attached storage, network appliances such as firewalls, and intrusion detection/
protection devices running GE links.
• Data center switches – These are fixed multiple port, data center switches, some with
up to 24 ports of 10GbE that are used in machine room applications in main
distribution facilities in large networks that aggregate multiple 10GbE links into a
single switch for distribution to different devices for multiple applications. These
switches are used to run high performance applications in parallel.
• Network appliances – Today, there are 10GbE intrusion detection devices, and upper-
layer application switches that provide multiple-gigabit throughput over 10GbE.
There is a large variety of switch and router products that offer between 12 and 48 ports of
Gigabit-rate capable ports combined with two or four ports of 10GbE. These switches are
used in workgroup and stackable switch applications in small to medium sized Local Area
Networks (LAN) that may have up to 2,000 thousand users (host clients) on the network.
In larger LANs, multiple stacks of interlinked switches are connected with a 10GbE
backbone or trunk line. In some networks, 10GbE backbones are used to interconnect two
or more campus networks within 40 kilometers of each other, and even up to 80
kilometers with special 10GbE optical transport interfaces. These are the kinds of LAN
switches that most of us are familiar with and have seen as new product introductions
since early in 2003.

4 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
10-Gigabit Ethernet Network Interfaces for Enterprise Networks

10-Gigabit Ethernet Network Interfaces for Enterprise Networks

The target applications for Enterprise network equipment that deploys 10GbE optical and
copper network interfaces are determined by the IEEE803.2ae and IEEE802.3ak
standards. Typical applications of 10GbE interfaces are shown in Table 1. Included in
Table 1 are the fundamental interface specifications. Many of these interface
specifications are found in product literature for network equipment and are relevant to
where the link is deployed.
The 10-Gigabit Ethernet story does not end in the LAN with the LAN layer 2 protocol.
The IEEE802.3ae standard has defined a Wide Area Network protocol.

Table 1. Network Interface Specifications for 10gbe in Enterprise Network Applications Driven by the
IEEE802.32ae Standard

10GbE Enterprise Layer 2 Nominal Standard Type of Maximum

Network Network 10GbE Line Rate Laser Optical Fiber, or Reach
Interfaces Applications Protocol & Clock Wavelengths Copper (Meters)
Tolerance Cable

10GBASE-SR Data Center, Serial 10.3125 850nm, 62.5m MMF 2 to 33m

wiring closet, LAN
Gb/s,
Short
machine room, ±100ppm wavelength
stackable 50m MMF 2 to 300m
serial
switches

10GBASE-LR Campus or Serial 10.3125 310nm, 10m SMF 2 to 10km

Metro Gb/s,
LAN Long
±100ppm
wavelength
serial

10GBASE-ER Metro Serial 10.3125 1550nm, 10m SMF 2 to 40km

Gb/s,
LAN Extra long
±100ppm
wavelength
serial

10GBASE- Campus or LAN 3.125 Gb/s 4 different 62.5 m 2 to 300m

LX4 Data Center per lane, wavelengths, MMF 2 to 240m
±100ppm long wave laser 50 m MMF
2 to 300m
50 m MMF 2 to 10,000m
10 m SMF

Spirent Communications Application Note | 5

10-Gigabit Ethernet Testing
10-Gigabit Ethernet Network – Two Rates in One

Table 1. Network Interface Specifications for 10gbe in Enterprise Network Applications Driven by the
IEEE802.32ae Standard (continued)

10GbE Enterprise Layer 2 Nominal Standard Type of Maximum

Network Network 10GbE Line Rate Laser Optical Fiber, or Reach
Interfaces Applications Protocol & Clock Wavelengths Copper (Meters)
Tolerance Cable

10GBASE- Data Center, LAN 3.125 Gb/s ae Twinaxial, 15m

CX4 wiring closet, per lane, shielded,
machine room, ±100ppm crossover
stackable (IEC 61076-
switches 3-113 plug
standard)

10-Gigabit Ethernet Network – Two Rates in One

10-Gigabit Ethernet supports two layer 2 protocols, a LAN and a WAN. The most widely
implemented protocol is commonly called 10GbE LAN. It runs at a 10.3125Gbps line
rate. In terms of the IEEE802.3ae standard, it is called 10GBASE-R. The LAN protocol is
used in LANs, wiring closets, machine rooms, and campus and enterprise network
applications. The LAN protocol may be implemented from a CE router interface to a
service provider edge router interface (see Table 1 on page 5).
A second layer 2 protocol for 10GbE is defined and is commonly known as 10GbE WAN.
In 2005, 10GbE WAN made its appearance in switches and routers. In IEEE802.3
terminology, this protocol is called 10GBASE-W. It is supported by a frame encapsulation
mechanism, defined by the IEEE802.3ae in Clause 50, called the WAN Interface Sublayer
(WIS). The WIS mechanism is the 10GBASE-W definition that has been adapted from the
ANSI T1.416-1999 (SONET STS-192c/SDH VC-4-64c) physical layer specifications.
10GBASE-W specifies three network interfaces: 10GBASE-SW, 10GBASE-LW, and
10GBASE-EW (see Table 2 on page 7).

6 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
10-Gigabit Ethernet Network – Two Rates in One

Table 2. Network Interface Specifications for 10GbE in Metropolitan Network Applications Driven by the
IEEE802.32ae Standard

10GbE Metropolitan Layer 2 Nominal Standard Laser Type Of Maximum

Network Network 10GbE Line Rate & Optical Fiber, or Reach
Interfaces Applications Protocol Clock Wavelengths Copper (Meters)
Tolerance Cable

10GBASE-SW Data Center, Serial 9.95328Gbps, 850nm, 62.5m MMF 2 to 33m

wiring closet, ±100ppm
WAN Short
machine room
wavelength
trunks to 50m MMF 2 to 300m
serial
routers

10GBASE-LW Campus or Serial 9.95328Gbps, 1310nm, 10m SMF 2 to 10km

Metro ±100ppm
WAN Long
wavelength
serial

10GBASE-EW Metro Serial 9.95328Gbps, 1550nm, 10m SMF 2 to 40km

WAN ±100ppm Extra long
wavelength
serial

There are some significant differences between 10GBASE-R and 10GBASE-W. The
10GBASE-W protocol operates at a 9.95328Gbps line rate that conforms to the
requirements of SONET STS-192c and SDH VC-4-64c frame rates. It matches the
9.58464 Gb/s payload rate of SONET. The IEEE802.3ae standard states, “The purpose of
the WIS is to allow 10GBASE-W equipment to generate Ethernet data streams that may
be mapped directly to STS-192c or VC-4-64c streams at the PHY level, without requiring
MAC or higher-layer processing.” This means that its design is meant for a direct PHY-to-
PHY interface, a point-to-point connection. A 10GBSAE-W port should be connected to
another 10GBASE-W port.
The 10GBASE-W implements the minimum framing, scrambling, and error detection for
SONET. It implements the sufficient Path, Section and Line overhead fields that are
required to maintain a SONET link. 10GBASE-W emulates a SONET link using SONET
encapsulation of the Ethernet frame. On the network, it looks like a SONET frame, but it
does not have the jitter, synchronous operation, electrical interface, or various
performance and line controls that a standard fully SONET compliant link would possess.
10GBASE-W is not directly interoperable with OC-192c SONET/SDH; one cannot attach
a 10GBASE-W interface to an OC-192c SONET/SDH interface. The IEEE8023.ae
standard requires that one 10GBASE-W interface be connected to another 10GBASE-W
interface.

Spirent Communications Application Note | 7

10-Gigabit Ethernet Testing
New Applications and Protocols for 10-Gigabit Ethernet

An example of this is the connection of two routers that have 10GbE Ethernet and OC-
192c SONET interfaces. When the two routers are connected by two 10GBASE-SW links,
the router’s switching fabric can easily move the SONET encapsulated Ethernet frames
seamlessly to the OC-192c interface that is connected to a core optical network. Rates,
framing, and error detection are aligned between the two different PHY. Within a service
provider network 10GbE is used in LAN or WAN modes depending on the requirement.
The WAN protocol can be used to link to a core network at 10Gb/s rates. The WAN
protocol encapsulates the Ethernet frame in a fixed SONET header to allow SONET
devices to forward it over OC-192c SONET/SDH links.
The operator of the router, from a maintenance viewpoint, would see two SONET links,
not an Ethernet link and a SONET link. This is intended to make links easier to maintain
in a SONET environment.
In metropolitan network applications this translates into the ability to attach Ethernet
switches and routers to core network devices that are attached to the SONET cloud. A
service provider point-of-presence can connect a 10GBASE-W link to a carrier’s central
office network equipment that is using SONET/SDH or optical transport equipment.
Now – and here is the catch – the whole scheme of the IEEE802.3as standard is to enable
Ethernet to participate in true end-to-end applications that makes use of compatible
infrastructure in the LAN and allow Ethernet to be readily adapted to MAN environments
with facility to interconnect with a network device that is attached to the optical core
network. This is made possible because the underlying rate at the MAC layer is 10Gb/s,
the MAC layer rate is constant and is common to both 10GbE LAN and WAN protocols.
The WIS layer adapts the MAC’s 10Gb/s rate to be compatible with SONET/SDH. By
doing this, a 10GBASE-W link inherits the lower cost and ease of maintenance inherent in
10GASE-R links. Due to the constant rate MAC, a switch or router fabric can handle
switching 10GBASE-R to 10GBASE-W across the fabric in the same box.
This flavor of 10GbE, 10GBASE-W, adds new complexities to performance testing of
switches and routers, depending upon where the equipment is deployed in the network.

New Applications and Protocols for 10-Gigabit Ethernet

There are several areas of networking and internetworking that drive the growth of
10GbE. Fundamentally, a number of new, IP-based network protocols, and time-sensitive
network applications, are increasing the need for more network bandwidth. Data center
applications add to the need for higher data throughput and low latency switching because
switches are increasingly requested to interconnect with GE and 10GbE servers and
network appliances that have multi-gigabit data rate throughput. Today, every type of
network device for machine room and data center application has 10GbE interfaces
available for deployment. The switch must handle these rates with layer 2 and/or layer 3
traffic without adding performance degradation at the higher network layers. IT
infrastructure managers expect that layer 2 switching is never to be a part of their problem.
The implementation of several bandwidth management and provisioning protocols that
affect GE and 10GbE performance are shown in Table 3 on page 9. The measurement of

8 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
New Applications and Protocols for 10-Gigabit Ethernet

throughput, forwarding rates, and latency with these various protocols, turned on or off,
with various combinations, is critical to the design of the network. There are more GE and
10GbE links present in the network than ever before, so it is important to know that
network devices can handle planned traffic loads under segmented and prioritized
conditions. The switch must maintain the quality of the user’s experience on the network.

Table 3. Layer 2 and Layer 3 Protocols that Affect Forwarding Performance in Switches

Protocols OSI Layer Effect on the Switch or Router

Affected

IEEE 802.1Q, Virtual LAN (VLAN)
Layer 2 Increase the length of packet headers by 4 bytes for each
(MAC layer) packet. At high 10Gb/s speeds switches must forward
larger packets with very low latency.
VLAN control field data must be inspected for each
packet and this increases packet-processing time.
Multiple VLANs per port force switches to be tested for
throughput per VLAN, VLAN leakage (mis-forwarding),
and over subscription that can flood ports
In Ethernet end-to-end applications over the WAN it
requires Layer 2 to Layer 3 mapping to maintain CoS.
Switches spend more time tracking and forwarding.

IEEE 802.1D Quality of Service
management, MAC bridges, support
for multicast networks in
multimedia applications (grouped
MAC addresses)
Layer 2 More segregation of bandwidth and network domains
(MAC layer) over individual LANs, that cause more packet inspection,
increase in forwarding table sizes, and buffer memory
management. This increases potential for higher latency.
Switch must maintain throughput in the presence of mis-
ordered and lost frames per VLAN.
Switch must inspect VLAN priorities and maintain
throughput (forwarding rate) over all separate VLANs

Spirent Communications Application Note | 9

10-Gigabit Ethernet Testing
New Applications and Protocols for 10-Gigabit Ethernet
Table 3. Layer 2 and Layer 3 Protocols that Affect Forwarding Performance in Switches (continued)
Protocols
IEEE802.3-Clause 43 Link
Aggregation and IEEE 802.1ad
Bridged Local Area Networks,
Virtual Bridged Local Area
Networks,
Internet Protocol version 6 (IPv6)
• 40-byte header length
• Extension headers
• Dual Stack operations
• IPv6 tunneling
RFC 2474 Definition of the
Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers
10 | Spirent Communications Application Note
OSI Layer Effect on the Switch or Router
Affected
Layer 2 Traffic on a single port or across multiple ports must
(MAC layer) support different customer LANs on the service provider
network. Links can be aggregates with up to 8 GE links
aggregating to 1 10GbE link, and 10GbE links to 40Gbps
links.
Forwarding performance and maintenance of QoS over
segregated LANs combined with the dedication of
bandwidth and network domains per customer, because
more packet inspection, increased packet lengths due to
keys and tags and increased complexity of forwarding
tables, and buffer memory management.
This has potential for higher latency, incorrect forwarding,
and VLAN leakage must be measured. Both Port and per
stream QoS and forwarding rates should be measured on
aggregated links.
Other measurements include port and aggregated stream
failover.
Layer 3 Longer standard packet header that supports header
extensions. This requires additional packet inspection of a
(IP Layer)
larger packet header with more fields.
IPv6 and IPv4 protocol must be supported in a dual-stack
mode on a per port basis. This requires additional packet
inspection, which increases the possibility for higher
latency and stresses memory buffers.
Layer 3 Uses the DS field of the IP header to mark packets with
values (called code points) that prioritize one type of
(IP Layer)
traffic over another based on class and cost. It is used in
QoS, CoS and layer 2 to Layer 3 mapping applications
(MPLS).
This, although not a guaranteed service, requires switches
to perform additional header inspection and can cost
processing time and increase latency.
 10-Gigabit Ethernet Testing
New Applications and Protocols for 10-Gigabit Ethernet

Table 3. Layer 2 and Layer 3 Protocols that Affect Forwarding Performance in Switches (continued)

Protocols OSI Layer Effect on the Switch or Router

Affected

Multi-Protocol Label Switching Layer 2 and Service provider networks have two types of routers that
(MPLS) Layer 3 interact. Label Edge Routers (LER) at the provider edge
• MPLS VPN and Label Switching Routers (LSR) within the provider
• VPLS core. MPLS provides a mechanism for IP flows from
many LAN customers to enter a large service provider IP/
MPLS network and to separately maintain QoS for each
customer over the backbone links between the routers in
the provider network.
LERs must inspect and keep track of many different
network domains, and perform deep and complex, packet
inspection. This increases processing time and forwarding
latency.
LSRs label switch the packets and forward them. The
LSRs must demonstrate low Layer 2 latency.
Time sensitive applications like IP telephony and video
are affected by MPLS services.

IP Multicast Layer 2 and A one-to-many protocol used for switching and routing of
Layer 3 audio, video, and conferencing of web casts. Switches
must keep track of grouped MAC addresses and VLANs,
where there can be many users per group and thousands of
groups per port. The switch must be able to allow users to
join and leave these groups without affecting QoS and
throughput. Poor latency performance may disrupt audio
and video transmissions.
Switch fabrics must be able to handle IPv4, IPv6, VLANs,
Unicast, and Multicast across the entire fabric.

Routing Protocols Layer 3 Switches must be able to maintain Layer 2 throughput,

forwarding rate and low latency in the presence of routing
(IP Layer)
protocol overhead (control plane traffic).
A single GE and 10GbE router may be aware of thousands
of networks, with several hundred users per network that
equates to forwarding over a few million flows.

Spirent Communications Application Note | 11

10-Gigabit Ethernet Testing
Scalable 10GbE Networks Create Test Challenges

Scalable 10GbE Networks Create Test Challenges

Today, there is an ever-growing increase in the number of GE and 10GbE ports in a single
blade. The challenge for NEMs is that they must test and measure the performance of
these high capacity, high port count devices with all of the protocols listed in Table 3 on
page 9. The simple layer 2 and layer 3 test over the data plane that measure throughput,
packet loss, and latency may not be sufficient in this scalable 10GbE environment. The
selection of the right test tool is critical. The test tool must be able to scale, while tracking
multiple protocol details within every port, over large numbers of ports. As outlined in
Table 3 on page 9, being able to test switches over their ports, with multiple VLANs per
port, and with different QoS priorities over different bandwidth provisioning levels per
port, requires the ability to discretely track the test traffic. When a failure occurs, it is
insufficient to only know the port the errored traffic came from. It is much better and
faster to find the source of errors, if the test traffic is tracked on a per stream basis, as
opposed to only a per port basis. If the test Engineer knows that error traffic was sourced
from the exact port, with the exact individual stream, with the exact VLAN, and the exact
QoS setting for that stream, then the chances of isolating the source of the error increases
greatly. This saves the test engineer a considerable time in debugging problems.
The test tools built by Spirent Communications are designed with 10GbE scalability
testing in mind. Spirent’s new test tool platform, Spirent TestCenter™, provides the ability
to track up to 24 different parameters on a per stream basis in real time with real-time
feedback while running a test. Every stream is tracked, on every port, in a scalable
manner. For bandwidth provisioning applications, every stream’s transmit rate, and every
stream’s receive rate on every stream, on every port can be determined with Spirent
TestCenter in a single test. Up to 32,767 clients can be tracked with stream traffic with real
time measurement on a single port in the Spirent TestCenter test tool; it is the most
scalable test tool in the test and measurement industry.

10-Gigabit Ethernet Test Considerations – 10GbE is not

Gigabit Ethernet
The approach to testing 10GbE is similar to the testing of GE. However, there are several
important differences between 10GbE and GE. The traditional switch test strategies for
GE are not comprehensive enough for 10GbE. When testing 10GbE, the differences
between 10GbE and GE must be taken into account. Further, since 10GbE is readily
configured as a LAN or WAN interface, this aspect must be considered in the test plan
strategy.
Another important consideration is interframe gap. In GE, the interframe gap (IFG) is a
constant value; it is a single byte length value. Once the IFG is set to 12 bytes for a given
packet size, that translates the packet’s transmit speed into a line rate transmission at 1Gb/
s. This is not the case for 10GbE. The IFG at 10GbE is derived over a predefined range of
IFG lengths; it presents an average IFG, and therefore an average line rate. The result is
that the IFG ranges from as small as 9 bytes to as large as 15 bytes while maintaining an
average 12-byte IFG. This creates problems for network devices with their packet
handling mechanisms. This key difference from GE must be considered as part of the
10GbE test plan.

12 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
Traditional Switch Testing and 10GbE Considerations

The test tools built by Spirent Communications are designed with 10GbE testing in mind.
Spirent TestCenter has capabilities and features that are critical to the testing of 10GbE.
Spirent TestCenter does account for the differences in GE and 10GbE testing. The tool
accommodates the nuances in testing the 10GbE interface. This section discussed what is
required in a test tool to accurately test 10GbE and its common interaction with GE.
The sum of these considerations is that: (1) 10GbE requires additional scalability testing,
(2) 10GbE testing requires higher resolution test tools with improved accuracy, and (3) the
test tool must be flexible enough to allow the test engineer to discretely test and measure
the different implementations of 10GbE (i.e 10GBASE-R and 10GBASE-W interfaces).

Traditional Switch Testing and 10GbE Considerations

The tests defined in RFC 2889 and RFC 2544 are applicable, in principal, for all Ethernet
speeds and network interface types from 10Mbps to 10Gbps. RFC 2544 and RFC 2889
will be referenced as “the RFCs” for efficiency, or as “RFC test” when general points are
applied.
The basis for the following discussion is centered over the traditional Ethernet switch tests
Requests For Comment (RFC) written by the IETF:
• Layer 2 data plane tests: RFC 2889 Benchmarking Methodology for LAN Switching
Devices with its associated RFC 2285 Benchmarking Terminology for LAN Switching
Devices.
• Layer 3 data and control plane tests: RFC 2544 Benchmarking Methodology for
Network Interconnect Devices with its associated RFC 1242 Benchmarking
Terminology for Network Interconnection Devices.
These four RFCs were written between 1991 and 2000. While all of the tests defined in
the RFCs are critically important to quantify the performance of network switches, major
new functionality has been added to both layer 2 and layer 3 network devices since their
standardization. Table 3 on page 9 points to the most significant of these new feature sets,
industry protocols, and bridging standards.
The IEEE802.3ae standard was ratified in 2002, written well after the RFC tests. The RFC
tests have no concept of 10GbE. They do a nice job of covering 10Mbps through 1Gbps.
The major point here is to run the mandatory RFC switch tests on 10GbE and on GE with
the considerations described in the following section. As discussed on page 12, 10GbE
does not behave on the network like GE. So, while the RFC switch tests are applicable, in
principal, to 10GbE, there are serious practical considerations that must be added to the
test plan for 10GbE.

Spirent Communications Application Note | 13

10-Gigabit Ethernet Testing
Test Setup for 10GbE using the RFCs

Test Setup for 10GbE using the RFCs

A few fundamentals on 10GbE are noteworthy for 10GbE RFC test setup. These
considerations point out how 10GbE differs from the test RFCs, and the requirements for
10GbE.
• The speed of 10-Gigabit Ethernet is 10,000,000,000 bits/sec (10Gb/s, 10Gbps) at the
medium access layer (MAC). A bit time is 1/10,000,000,000 seconds.
• The interframe gap is 9.6 nanoseconds for 10-Gigabit Ethernet. This is still 96-bit
times.
• 10GbE is a full-duplex protocol. There is no half-duplex mode in 10GbE.
• 10GbE supports two protocols: LAN and WAN. Only one protocol can be used in
comparative tests between two DUTs. This is due to different data rates per protocol.
10GbE WAN has a different frame encapsulation scheme and line rate than 10GbE
LAN.
• 10GbE WAN and 10GbE LAN must be tested separately on the switch. One cannot
test one of the protocols and assume the other protocol will work. Frame sizes that
pass RFC performance test specifications with 10GbE LAN may fail on 10GbE WAN
and vice versa.
• The RFCs specify a 30-second test duration per frame size iteration. This is often not
long enough to stress packet buffers on switches and routers. Longer test durations of
up to 300 seconds are quite common. Test for the larger more scalable 10GbE devices
may be run for multiple hours, or even days, with expectations of no packet loss.
While these tests are not per any RFC, they are being conducted. Special
considerations are required for these tests.
• At 10GbE rates, there are carrier class switches that forward 10Gb/s traffic to 40Gb/s
interfaces. To do this, they have an extremely large packet buffer memory. Buffer
depth presents an issue if the test does not run long enough to fill up the buffer. A
special technique that monitors frame loss in real time may be required. The test tool
should offer this capability to test deep packet buffer memory. One technique to
reduce test time is to report packet loss in real time while the test is running, as
opposed to reporting man hours later at the end of a batch mode test.

Line Rate Testing for 10-Gigabit Ethernet Switches and Routers

In line rate tests with 10-Gigabit Ethernet, there is a critical consideration that affects the
setup of the traditional RFC test and any other special performance test. The decision on
how to set up the performance test must take into account the IFG mechanism for 10GbE
that was previously mentioned.
A brief explanation is provided for clarity. Recall the earlier discussion in which it was
pointed out that 10GbE does not have a constant IFG byte length. The IFG varies between
12 and 15 bytes to produce an average of 12 bytes at line rate. The variation of the IFG is
beyond the scope of this document.

14 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
Line Rate Testing for 10-Gigabit Ethernet Switches and Routers

The net result is that the actual IFG, which affects the final transmission rate on the line,
will be slower than the theoretical 10GbE line rate. Certain packet sizes are affected by the
average IFG generating mechanism in 10GbE. A 10GbE interface will transmit between
12 and 15 bytes of interfame gap at a given frame size. What this means is that in some
frame sizes, the theoretical 10GbE line rate will not be achieved. So, how does one
perform a 10GbE line rate test on a switch port with so many different frame sizes that
almost ensure that line rate will be never be achieved?
To test at the 10GbE line rate, the test tool must have Deficit Idle Count (DIC)
compensation. The test tool must allow the test to be run with DIC enabled or disabled.
Fundamentally, DIC adds or subtracts up to 3 bytes to or from the nominal average 12-
byte interframe gap in order to maintain the 10GbE frame rate. The range of IFGs
transmitted by the 10GbE interface can be from 9 to 15 bytes, but the average is 12 bytes.
In order to achieve the maximum 10GbE throughput, the test tools and devices under test
must have DIC enabled. The pattern in Table 4 on page 15, will faithfully repeat. There
will be less difference in frames per second rates between DIC enabled and DIC disabled
as the frames increase in size. If the switch is not tested with DIC enabled, and the device
is placed into a DIC-enabled network, the probability increases that the DIC-enabled
network interface will overrun the receive frame buffer of the device with DIC
compensation disabled. This problem occurs with smaller frame lengths.
DIC provides a higher gradient for frames per second rates, so in the event of packet loss,
a more accurate line rate can be measured and reported with DIC enabled.

Table 4. Transmit Rate Differences at the 10Gb/s Line Rate with Deficit Idle Count

Frame Size Frames per Frames per Difference

(bytes) second rate second rate in Frame Rates
(DIC enabled) (DIC disabled)

64 14,880,952 14,880,952 0

65 14,705,882 14,204,545 501,337

66 14,534,883 14,204,545 330,338

67 14,367,816 14,204,545 163,271

68 14,204,545 14,204,545 0

69 14,044,943 13,586,956 457,987

70 13,888,888 13,586,956 301,932

71 13,736,263 13,586,956 149,307

72 13,586,956 13,586,956 0

Spirent Communications Application Note | 15

10-Gigabit Ethernet Testing
Line Rate Testing for 10-Gigabit Ethernet Switches and Routers

The first recommendation that comes from this is to initially test the switch with DIC
enabled. This will stress the switch far more than with DIC disabled by subjecting it to
many varying frame rates. The second recommendation is to run through a considerable
range and number of small frame sizes, for both odd and even frame sizes. Throughput,
packet loss, and forwarding rates must be evaluated until one is satisfied that the switch is
forwarding at the 10GbE line rate over many frame sizes. The third recommendation is to
make sure the range of packet sizes that are test is wide. The switch should be tested in
small frame size steps (increments) from 64 bytes to 16Kb frame lengths.
In Table 5 on page 16, the standard RFC frame sizes and frames per second rates are
shown with DIC enabled and disabled with a 12-byte IFG. Notice that with frame sizes
that are modulo 4, that no impact from DIC is seen until the 1518 frame size is
encountered (reference blue highlighted table cells). This may appear like a subtle
anomaly of the RFC test, or 10GbE, but it is not. When all possible frames sizes are
considered and the differences in frame rates between DIC enabled and DIC disabled
(Table 4) is considered there will be many frame sizes that will pass line rates tests with
DIC disabled but may fail line rate tests with DIC enabled. Spirent recommends that
throughput, frame loss and forwarding rate tests be conducted with DIC enabled.

Table 5. 10-Gigabit Ethernet Line Rates With Standard RFC 2544 and 2889 Frame Sizes

Frame Size Frames / second rates Frames / second rates Interframe Gap
(bytes) (DIC enabled) (DIC disabled) (100% line rate)

64 14,880,952 14,880,952 12 bytes

128 8,445,945 8,445,945 12 bytes

256 4,528,985 4,528,985 12 bytes

512 2,349,624 2,349,624 12 bytes

7681 1,586,294 1,586,294 12 bytes

1024 1,197,318 1,197,318 12 bytes

1280 961,538 961,538 12 bytes

1518 812,743 811,688 12 bytes

1 The 768 frame size is not specified as MUST in the RFCs.

For example, if a routine run of the standard RFC frame sizes performed over a 10GbE
interface without DIC, the DUT may pass 100%. Trouble may arise if that same port
interconnects with another port with DIC enabled running off-standard frame sizes. The
DUT may have been tested at less than line rate for non modulo 4 frame sizes. It is easy to
be caught off one’s guard, even in a well known test like RFC 2544 or 2889.
As shown in Table 3 on page 9, the newer network protocols create frame sizes that are
not part of RFC 2889 and 2544, as they were originally written. Table 5 on page 16,
selects a few fundamental frame sizes that exceed the RFC test requirements. These frame

16 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
Key Considerations for 10GbE Tests

sizes should be tested across the entire switch fabric with their header fields populated
with realistic forwarding values seen in virtual and bridged networks. Today’s switches
must be tested beyond the traditional RFC, using protocols with headers that segment
traffic in support of QoS as well as in converged network applications.
Table 6 does not list all of the possible influences on critical Ethernet frame sizes that
stress switch performance with the many protocols listed in Table 3 on page 9. The point
illustrated is this: simply running a vanilla-flavored RFC 2889 and RFC 2544 switch test
and claiming that one’s device is ready for the world of 10GbE is not advisable.

Table 6. 10-Gigabit Ethernet Frame Rates with Non-standard RFC Frame Sizes

Frame Size(s) Frames per Interframe Gap Application driver

(bytes) second
(100% line rate)

1280 961,538 12 bytes IPv6 minimum MTU

1500 822,368 12 bytes Ethernet MTU

1522 812,743 12 bytes VLAN tag

1526 808,538 12 bytes VLAN tag and MPLS Label

9022 138,243 12 bytes Jumbo frame

9216 135,339 12 bytes Popular jumbo frame

Key Considerations for 10GbE Tests

The items listed here reinforce previous points and provide cautions for testing the
performance of 10GbE switches and routers.
• For line rate tests Deficit Idle Count (DIC) should be enabled unless the DUT/SUT
cannot support it, or an agreement is in place to test without it.
• The switch or router should be performance tested with DIC enabled and with DIC
disabled. This applies to the 10GbE LAN and WAN protocols
• Most test equipment uses statistical transmit and receive event counters. These often
display results in real time (e.g. real time event counters). At the high data rates of
10GbE, real-time statistical event counters may not be capable of displaying what is
actually being transmitted or received on the line. They will report rates other than the
theoretical line rate for that frame size. The test equipment should have special
transmit and receive counters. These special counters are required to accurately reflect
the transmit and receive rates that are determined in the actual test and reflected in the
test report. Note that, at the slower speed of 1Gbps, sampling mechanisms that
produce real-time statistical event counters can keep pace with the data rate. The
problem is usually not seen at 1Gbps.

Spirent Communications Application Note | 17

10-Gigabit Ethernet Testing
Conclusion

• Latency measurements are dramatically different between 10GbE LAN and 10GbE
WAN. The 10GbE WAN will have a larger average latency and range compared to
10GbE LAN. Average, minimum and maximum latency for both 10GbE WAN and
10GbE LAN should be measured.
• For latency measurements, the test equipment should have 10-nanosecond
measurement resolution and an accuracy rating of 40 nanoseconds.
• 10GbE supports several types of optical and copper transceivers. Latency test results
may vary according to the technology of the transceivers that are used in the test. A
low quality transceiver can add latency to the performance results.

Conclusion
What 10GbE deployment means for NEMS, service providers, carriers and their
customers, is that new test methodology is required to test switches with 10GbE interfaces
in a comprehensive manner. The RFC tests should be run, as well as tests that include
Virtual LANs (VLANs), IPv6 addressing and tunneling, IP multicasting, and Quality of
Service. Another influence that stresses switch fabrics is subnets per VLAN for IPv4 and
IPv6. The summary point is that switches and routers are being asked to simultaneously
perform more tasks on a single port, with more ports, at higher speeds, at lower cost.
The primary challenges for switch and router vendors are:
• Deeper frame and packet header inspections due to more active controls in frame
headers
• High speed, tagging, mapping, and tracking of many more header field data to
network segments
• Perform the functions in bullets 1 and 2 for thousands of clients on a single port,
across multiple ports, over an entire switch fabric
• To maintain large memory buffers for forwarding tables. This is intermixed between
physical and virtual networks.
• Switches and routers with 10GbE network interfaces must be able to forward layer 2
traffic at line rate at all packet sizes, not just the standard packet sizes, and with jumbo
frames.
Table 7 on page 19 and Table 8 on page 19 provide product information for your
reference.
For more details on Switch Test Methodologies, please refer to Spirent’s Test
Methodology Journals for RFC 2544 and RFC 2889.

18 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
Spirent TestCenter Products for 10GbE Switch and Router Testing

Spirent TestCenter Products for 10GbE Switch and Router Testing

The Spirent TestCenter products that support 10GbE testing are shown in Table 7 and
Table 8 on page 19.

Table 7. 10-Gigabit Ethernet Test Hardware Modules and Related Transceivers

Model Description Application 10GbE Test

Technologies with LAN
and WAN Protocol
Support

XFP-1001A Standard scalability and performance, Traffic generator and XFP transceivers: 850,
1-port, 10GbE test module statistical measurement 1310nm, and 1550nm
analyzer

XFP-2001A High scalability and performance, Traffic generator and XFP transceivers: 850,
1-port, 10GbE test module statistical measurement 1310nm, and 1550nm
analyzer

MSA-1001A Standard scalability and performance, High port density traffic XENPAK LAN
2-port Multi-MSA test module generator and statistical transceivers: 850nm,
measurement analyzer 1310nm, and 1550nm
XENPAK LAN/WAN
transceiver: 1310nm
CX-4 Copper, LAN
transceiver

XFP-1001A XFP-2001A MSA-2001A

Spirent Communications Application Note | 19

10-Gigabit Ethernet Testing
Spirent TestCenter Products for 10GbE Switch and Router Testing

Table 8. Spirent TestCenter Software Packages for 10-Gigabit Ethernet Testing

Model Description Application 10GbE Test Technologies

with LAN and WAN Protocol
Support

BPK-1001A Packet generator and analyzer base Traffic generator and IPv4, IPv6, layer 2 and layer 3
package A statistical traffic generation, QoS,
measurement analyzer Diffserv, error injection,
custom packet generator,
capture, statistics and real-time
measurements/charts

TPK-1000 RFC-2544 with VLAN network device Layer 3 RFC switch Deficit Idle Count
benchmark test package and router testing and
beyond with IPv6,
VLAN support

TPK-1001 RFC-2889 with VLAN switching Layer 2 RFC switch Deficit Idle Count
benchmark test package and router testing and
beyond with IPv6,
VLAN support

BPK-1002A STP/RSTP/PVST+ base package A Key metropolitan and STP/RSTP/PVST+ and per-
enterprise protocols VLAN Spanning Tree Protocol
state machines

BPK-1014A Multiple Spanning Tree base package A Key metropolitan and Multiple Spanning Tree
enterprise protocols Protocol

BPK-1003A IGMP/MLD host IP multicast base Multicast routing Multicast registration IGMPv1/
package A v2/v3 and MLDv1/v2

Deficit Idle Count

BPK-1004A Unicast routing base package A Unicast routing IPv4 and IPv6 interior and
exterior gateway routing
protocols: RIPv1/v2, RIPng,
OSPFv2/v3, IS-IS, IS-ISv6,
BGP-4 and BGP+

Deficit Idle Count

BPK-1005A Multicast routing base package A Multicast routing Multicast registration IGMPv1/
v2/v3 and MLDv1/v2
Deficit Idle Count

20 | Spirent Communications Application Note

 10-Gigabit Ethernet Testing
Spirent TestCenter Products for 10GbE Switch and Router Testing

Table 8. Spirent TestCenter Software Packages for 10-Gigabit Ethernet Testing (continued)

Model Description Application 10GbE Test Technologies

with LAN and WAN Protocol
Support

BPK-1006A MPLS/LDP/RSVP-TE base package A MPLS support RFC 2547bis Layer 3 VPNs,
Martini-draft Layer 2 VPNs
(PWE emulation), Virtual
Private LAN Service - LDP
(VPLS – LDP), Virtual Private
LAN Service - BGP (VPLS –
LDP), Layer 3 IPv6 VPNs
Deficit Idle Count

Spirent Communications Application Note | 21