LINUX Network Bonding Explained

This document does not cover all bonding configurations and options. Only the default setting as
seen in PACS SE and/or Linux PACS to this date.

Bonding serves several very useful purposes. It allows you to create a redundant, fault tolerant, or
"failover" solution, or to aggregate multiple ports into a single group, effectively combining the
bandwidth into a single connection. Bonding can also allow you to create multi-gigabit pipes to
transport traffic through the highest traffic areas of your network. In our current PACS SE or Linux
PACS configuration, the default setup is a redundant "failover" solution.

Bonding Configuration
The bonding configuration type is set in the bonding kernel module entries in 'modules.conf' (RH
Enterprise AS 2x 3x) or modprobe.conf (RH Enterprise AS 4x).

[root@GELINUX root]# cat /etc/modules.conf

alias parport_lowlevel parport_pc
alias scsi_hostadapter cciss
alias eth0 tg3
alias eth1 tg3
alias usb-controller usb-uhci
alias bond0 bonding
options bond0 miimon=100 mode=1

In the above example note the line that states 'options bond0 miimon=100 mode=1'. This indicates
that bonding mode #1 is the chosen bonding configuration mode. This is an "active-backup"
configuration. In this configuration, only one slave adapter in the bond is active at a time. A different
slave adapter becomes active if, and only if the active slave fails. The bond's MAC address is
externally visible on only one port (network adapter) to avoid confusing the switch. This mode
provides fault tolerance.

Sequence of Events
The basic sequence of bonding initialization is as follows.

Set bonding device first

Set network information to master device
ifenslave command attaches slave devices to master device.
Acquire network information to apply to master device
Master sets the information that was acquired to slave device.

Bonding Interface Setup

A good example of a correct bonding interface setup is included below. Note that there are no
IPADDR (IP Address) lines defined in the actual 'eth0' and/or 'eth1' configs.

[root@GELINUX log]# cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
ONBOOT=yes
BOOTPROTO=static
IPADDR=159.212.93.137
NETMASK=255.255.255.0

[root@GELINUX log]# cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
ONBOOT=yes
MASTER=bond0
SLAVE=yes

[root@GELINUX log]# cat /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Bonding Status
How to check the status of the bond:

[root@GELINUX root]# cat /proc/net/bond0/info (RH Enterprise AS 2x 3x) or

/proc/net/bonding/bond0 (RH Enterprise AS 4x)
Bonding Mode: fault-tolerance (active-backup)
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Multicast Mode: all slaves

Slave Interface: eth1

MII Status: down
Link Failure Count: 0

Slave Interface: eth0

MII Status: up
Link Failure Count: 0

Show Speed and Duplex

Showing Speed, and Duplex on a bonded system:

mii-tool and ethtool do not read the speed and duplex of ifcfg-bond0 because it's not a true
interface. mii-tool is actually fairly obsolete and has not been updated very much in the last few
years. ethtool is smart enough to know that ifcfg-bond0 is really just a software bond, not a "real"
interface, and cannot really have a "speed" per-se. You will need to check the actual interface eth0
and eth1 with ethtool to get the actual speed and duplex. I suggest ethtool only as stated before mii-
tool may not give accurate results.

[root@GELINUX root]# mii-tool

eth0: negotiated 1000baseTx-FD, link ok
eth1: no link
[root@GELINUX root]# ethtool eth0
Settings for eth0:
 Supported ports: [ MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Current message level: 0x000000ff (255)
Link detected: yes

Configuring Duplex and Speed

Locking bonded configuration to 1000/FULL:
(note: If there is a poor connection to the switch for any reason, the results of the following may be a disconnection from the
network. Please make sure all hardware including cabling is capable of Gigabit connect before performing these steps.
Additionally, if a disconnect occurs you likely have additional issues not related to configuration.)

1) Make sure you are on-site, have someone on-site that can reverse the change, or have
guaranteed and tested ILO access.
2) Test locking the eth0 device to 1000/full using ethtool by entering the following from the
command line as root.

ethtool -s eth0 speed 1000 duplex full autoneg off

If the connection stays active/connected, use ethtool to check the speed and duplex. If the speed
and duplex are 1000/Full and you wish to make the change permanent, continue to step 4 after
verifying that setting speed and duplex for eth1 in the next step was also successful. If the
connection drops, simply do the following from the command line locally as root to reset the network
to it's previous state.

service network restart

3) Test locking the eth1 device to 1000/full using ethtool by entering the following from the
command line as root.

ethtool -s eth1 speed 1000 duplex full autoneg off

If the connection stays active/connected, use ethtool to check the speed and duplex. If the speed
and duplex are 1000/Full and you wish to make the change permanent, continue to step 4 after
verifying that setting speed and duplex for eth0 in the previous step was also successful. If the
connection drops, simply do the following from the command line locally as root to reset the network
to it's previous state.
service network restart

4) To make this change permanent and active on start/reboot, add the following line to /etc/rc.local
on a new line at the bottom of the file.

# Added to lock network down at 1000/FULL (YOUR_NAME,THE_DATE)

ethtool -s eth0 speed 1000 duplex full autoneg off
ethtool -s eth1 speed 1000 duplex full autoneg off

5) Check both interfaces seperately with ethtool. ex: 'ethtool eth0' and 'ethtool eth1'.
6) You may also wish to check the bonding status. 'cat /proc/net/bond0/info'.

Advanced Notes:

* To restore a slave's MAC address, you need to detach them from the bond (`ifenslave -d bond0
eth0'). The bonding driver will then restore the MAC addresses that the slaves had before they were
enslaved.
* The bond MAC address will be the taken from its first slave device.

Resources:
The Linux Channel Bonding Page
http://sourceforge.net/projects/bonding/