HOME

About
SAM's Guide to Linux Administration

Just another Scott Alan Miller Sites site

Search

RSS Twitter

Working with NIC Ring Buffers

June 20th, 2011
Posted in Networking
Write comment
Making configuration changes to the settings of a network interface card is not one that we typically do on an average day but it can be an important
component of network performance tuning. The most common change that we might need to make is to increase the receive (RX) ring buffer size.
Most often we will find that the ring buffer is set rather small and might need to be increased on systems that are receiving a high volume of network
traffic.
We can examine our current and available NIC settings using the ethtool command combined with the -g flag.

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:
2040
RX Mini:
0
RX Jumbo:
8160
TX:
255
Current hardware settings:
RX:
255

RX Mini:
RX Jumbo:
TX:

0
0
255

Here we have two sections in our output. The first section is Pre-set maximums which tells us the maximum values that could be set for each
available parameter. The second section shows us to what each parameter is currently set. We are most interested in the top most parameter labeled
simply RX which is our receive ring buffer.
Buffers are generally tuned small for latency reasons. The smaller the buffer the lower the latency. But low latency comes at a price and that price is
maximum throughput. For greater throughput we need a larger buffer. Factory defaults are good, generally, for most systems but dont be afraid to
tune this for your own scenario.
To modify the current parameter settings we use the -G flag as opposed to the -g flag used to read them. A common setting is to set the RX parameter
to 1020. Here is the command to do that:
# ethtool -G eth0 rx 1020
The change will take affect immediately and requires no restart to your system or even your network stack. Read your current settings again with -g
to verify the change.
Remember that these changes made with ethtool are being made to the network card itself and not to the operating system. We are not changing the
kernel network stack parameters but the NIC parameters in the firmware.

Working with NIC Ring Buffers

Comments ( 15 )
Trackbacks ( 1 )
Write comment
1.
linuxadmin
August 4th, 2011
REPLY

QUOTE
Will changing the buffer size cause an outage on a Prod server?

Angus
August 2nd, 2012
REPLY
QUOTE
Actually I notice the interface goes down/up whilst the change is applied. This took around 1 second.
We use the bonding module in active/backup mode. The bond failed over when applying the change on the active card, and back again
when applying to the passive (which was active at the time due to the previous fail-over)
Relevant log messages with around the commands I ran:
# ethtool -G eth0 rx 1024
kernel: bonding: bond0: link status definitely down for interface eth0, disabling it
kernel: bonding: bond0: making interface eth2 the new active one.
kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
kernel: bonding: bond0: link status definitely up for interface eth0.
# ethtool -G eth2 rx 1024
kernel: bonding: bond0: link status definitely down for interface eth2, disabling it
kernel: bonding: bond0: making interface eth0 the new active one.
kernel: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
kernel: bonding: bond0: link status definitely up for interface eth2.
The bonding module also notices link failures (which werent there before)
# grep -E Slave Int|Failure /proc/net/bonding/bond0
Slave Interface: eth0
Link Failure Count: 1
Slave Interface: eth2
Link Failure Count: 1

2.
Scott Alan Miller
August 4th, 2011
REPLY
QUOTE
No, the change can happen at any time. Of course, if you change it to a value that causes problems that will impact production, so be careful. If
the buffer is too small you will drop packets. If it is too large you increase latency. The latency is generally very small and normally only affects
low latency applications such as financial trading and only rare systems within that realm.
3.
Al
December 21st, 2011
REPLY
QUOTE
Hello Sam,
First, Thank YOU for this great post!
Can you please explain why The smaller the buffer the lower the latency?
For some reason, there seems to be little info out there on this specific topic. Any pointers to a good source specifically explaining how the ring
buffers are used would be appreciated to.
Thanks again,
Al.

Scott Alan Miller

December 22nd, 2011
REPLY
QUOTE

Wikipedia has a great overview of ring buffers here: http://en.wikipedia.org/wiki/Circular_buffer

The latency of a ring buffer comes from the nature of buffer processing. If you think of data streaming into the buffer and the buffer
processor spinning the buffer one notch at a time to get the next element from it then you can see that the larger the buffer, the more
notches need to be processed before you get to the latest data in the stream. Larger data sets always take longer to process, it is the nature
of large data sets.
You find this even with excess system memory. The rule of thumb is more memory is always better. But this is not true. People are just
used to being memory constrained and dont think about the risks of excess memory. But if you have far too much memory you will, for
example, introduce unnecessary garbage collection overhead to manage that memory for which you have no particular use.
Buffers, especially, you want to keep as small as possible without being too small. Like most things, the selection of buffer size is a
tradeoff between throughput and latency. Big buffers mean big throughput while small buffers mean log latency.
A good example of too much memory can be seen in the old Linux kscand issue: http://www.sheepguardingllama.com/2008/04/linuxkscand/
4.
Al
December 22nd, 2011
REPLY
QUOTE
Thank you very much for the very quick response. Im still missing something.
If I have the same rate of data coming into a given buffer and being consumed, does the buffer size really matter? There will be some empty
space in it but unless the consumer of the buffer is adjusting his consumption speed based on the buffer size, shouldnt a larger buffer just mean
that worst case part of it is effectively not used but doesnt introduce any additional latency? My understanding is that the data is not flowing
from one end of the buffer to the other end before it exists and gets consumed. When the consumer is keeping up with the producer, the pointers
of the former chase the pointers of the latter in the ring and is picking data from the location/cell in the buffer where it was dropped.
Its very clear that consuming a larger buffer would take more time and introduce latency but only if the consumer is waiting longer before it
consumes the buffer. If the consumer is working in parallel to the producer and is not changing his behavior knowingly based on the buffer size,
then his rate of consumption would not depend on the buffer size and there should be no significant change in the latency of processing the data
. by any chance is the latency in question here because a larger buffer is occupying more memory space and the more physical memory pages
are involved, the more memory paging/mapping and cache gets into play?

I had found your post as I was surprised to see the latency jump significantly when I increased the rx ring buffer size. I intentionally ran a low
enough packet rate that doesnt result in any packet drops with some smaller ring size and measured the latency. Then I increased the size of the
rx ring (and kept everything else as is) and expected that this additional memory will not be used and that the latency will not change. It
changed significantly. I initially suspected the driver may have adjusted its speed of consumption dynamically (perhaps through some interrupt
moderation) knowing that it has a larger buffer that can hold more data and can be visited for consumption at a lower rate (for efficiency but at
latency cost). Your response thou seem to suggest a different reason for the increased latency.
Thanks for the link you sent. I was rather looking for more info on the arch of NIC, their FIFO memory, data transfer over PCI and so forth,
including where the mentioned ring buffers fit in the overall picture of data transfer from the wire to the main memory and vice versa.
Thanks,
Al.
5.
Scott Alan Miller
December 22nd, 2011
REPLY
QUOTE
Every NIC runs its own code so they will differ NIC to NIC. There is no single standard, each vendor implements things as they see fit.
You should not see a large jump in latency from increasing your NICs ring buffer, just a small one (how large did you make it?) But if you
make it too large you may be timing out and getting retransmissions.
Just like the NIC ring buffer, Linux TCP/IP stack has a ring buffer as well (the one governed by sysctl) and there is a great explanation here of
why the buffers need to be kept small there:
http://www.ece.virginia.edu/cheetah/documents/papers/TCPlinux.pdf
6.
Scott Alan Miller
December 22nd, 2011
REPLY
QUOTE
In the FreeSwitch Wiki (FreeSwitch is the main open source VoIP competitor for Asterisk) they talk about issues with buffering vs. discarding

for real time traffic:

http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations

Klash
March 5th, 2014
REPLY
QUOTE
Im guessing thats because SIP traffic is UDP and connectionless. In a TCP conversation (which is most enterprise traffic), discarding is
not wise.
For TCP traffic, buffering is going to be better than losing/discarding packets and triggering re-transmits.
7.
Al
December 22nd, 2011
REPLY
QUOTE
Thanks a lot Scott!!
The Linux paper is exactly the type of information I was looking for. Thanks again.
I have ran into the FreeSwitch link, it describes a symptom/result of the latency but doesnt explain the root cause.
In my case, I did try extreme values (since the purpose was understanding the behavior) like the 4096 max value allowed in the driver I was
using.
My tests are a bit convoluted to explain in a short post. Perhaps Ill report back with a more representative environment later. Most notably, Im
testing Linux is a VMware VM and Im looking at the latency of passing packet THROUGH the system (like a switch/router. The packets get
routed by the kernel since they are not destined to the box). So the latency results in this case are in micro seconds (rather than milliseconds).
Here is a glimpse of the results from one experiment:
using: ethtool G eth4 rx
Ring size Latency [uS]
32 140

64 171
96 198
128 204
256 (default) 232
512 242
1024 239
2048 301
4096 484
The absolute value of 484us (example) is small but its double the latency for the default-setting case. For pass-thru traffic, this is relatively
significant impact.
Al.
8.
Jon Miller
July 5th, 2012
REPLY
QUOTE
What kind of NIC / driver are you using? Im currently using Jumbo packets on a private inter-connect between a few machines and was
looking to bump up my RX Jumbo value but was disappointed to see that my pre-set maximum was 0.
My setup: (via lspci excerpt)
Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01)
Subsystem: Hewlett-Packard Company NC553i 10Gb 2-port FlexFabric Converged Network Adapter
Then driver info: (via ethtool -i)
driver: be2net
version: 2.102.518r
firmware-version: 3.102.517.701
9.
AK

July 3rd, 2014

REPLY
QUOTE
Need solution on the issue.
10.
AK
July 3rd, 2014
REPLY
QUOTE
My SetUp
filename: /lib/modules/2.6.18-53.el5/extra/hp-e1000/e1000.ko
version: 8.0.35-NAPI
license: GPL
description: Intel(R) PRO/1000 Network Driver
author: Intel Corporation,
srcversion: 8E4E478C6472DA14AA960BC

[root@123_server ~]# ethtool -g eth2

Ring parameters for eth2:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256
eth2 Link encap:Ethernet HWaddr 00:1F:29:5C:32:8C

inet addr:7.7.27.1 Bcast:7.7.31.255 Mask:255.255.240.0

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9211233 errors:0 dropped:0 overruns:0 frame:0
TX packets:27339732 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1277876211 (1.1 GiB) TX bytes:40481379165 (37.7 GiB)
Interrupt:177 Memory:fdfe0000-fe000000

Is it safe to change the RX value to 4096?

11.
Scott Alan Miller
July 16th, 2014
REPLY
QUOTE
Yes, 4096 is a fine value.
12.
Daqing Yun
September 4th, 2014
REPLY
QUOTE
Hi guys,
This is a really great post!
I am wondering that is there a way to check out the RX/TX value in a C/C++ program? Thanks!
TrackBack URL

http://www.scottalanmiller.com/linux/2011/06/20/working-with-nic-ring-buffers/trackback/

1. November 26th, 2011

Trackback from : Elektrische Zahnbuerste

NAME( required )
E-MAIL( required ) - will not be published URL

Submit Comment

Generating an SSH Key

Change Kernel Panic Behavior
Return top

INFORMATION
Change this sentence and title from admin Theme option page.

RECENT ENTRY
Installing Chef 12 on CentOS 6 or RHEL 6
Better Scripting with BASH
Finding a File by MD5 Sum
Making an RC Script
Scripting FDISK

CATEGORY
Firewall
HowTo
Kernel

Networking
Security
Storage
Uncategorized

ARCHIVES
September 2014
August 2014
June 2014
May 2014
January 2014
August 2013
April 2013
March 2013
January 2013
November 2012
September 2012
August 2012
April 2012
December 2011
August 2011
June 2011
April 2011
January 2011
October 2010

LINKS
Documentation
Plugins
Suggest Ideas
Support Forum
Themes
WordPress Blog
WordPress Planet

Copyright 2010-2014 SAM's Guide to Linux Administration

Theme designed by mono-lab
Powered by WordPress