How to setup SEA failover with Load Sharing configuration

Education

Abstract
Use this Techdoc to help you to configure primary and backup Shared Ethernet Adapters for load sharing in the Virtual I/O Server.

Content
Authors: Bianca Rademacher ( Bianca.Rademacher@de.ibm.com ) Michael Guha Thakurta ( michael.guha@de.ibm.com ) Reviewer: Rajendra D Patel ( rajpat@us.ibm.com ) Last Updated: July 31, 2012 For Changes / Updates - ref to Changes Section Below: Table of contents: Introduction Requirements Functionality Simple test scenario from scratch - Setup - How do I know that load sharing is working and how do I know which VLANs are bridged by each SEA? - Some outputs in disaster cases Enhanced test scenario (adding additional trunk adapter dynamically to the SEA) - Setup - Some outputs in disaster cases Switching from SEA Load Sharing to SEA failover Switching from SEA failover to SEA Load Sharing FAQs References

Introduction: The first implementation to fulfil redundancy requirements for Power System with network virtualization was the introduction of NIB (Network Interface Backup, s. Figure 1)

Figure 1: Network Interface Backup (NIB)

The client LPAR is connected to two separate VLANs. VLAN tagging cannot be supported in this case is because you cant have one (virtual) switch with two connections. To get around this problem, a different configuration needs to be used, namely, Shared Ethernet Adapter (SEA) Failover (s. Figure 2)

Figure 2: SEA Failover

The Shared Ethernet Adapter failover configuration provides redundancy only by configuring a backup Shared Ethernet Adapter on a different Virtual I/O Server. This backup Shared Ethernet Adapter is in the standby mode and can be used only if the primary Shared Ethernet Adapter fails. Hence, the bandwidth of the backup Shared Ethernet Adapter is not used. When using very expensive 10 Gb/s ethernet adapter the utilization of both, primary and backup adapter, would be highly appreciated. This can be obtained with use of SEA together with virtual switches (s. Figure 3).

Figure 3: Multiple Virtual Ethernet Switches

A significant benefit to this design is that both Virtual I/O Servers can be active at the same time. Half of the clients could be configured to use Virtual I/O Server 1 and the other half to use Virtual I/O Server 2 as their primary paths. Each client would failover to its respective secondary path in the case that its primary path was lost. So the customer's investment in hardware is more effectively utilized. Different to SEA with vswitch mode is the SEA with load sharing configuration, where the SEA algorithm itself decides which VLAN is bridged by the primary or the backup SEA (s. Figure 4).

Figure 4: SEA with Load Sharing

Functionality In the Shared Ethernet Adapter failover with load sharing configuration, the primary and the backup Shared Ethernet Adapters negotiate the set of virtual local area network (VLAN) IDs that they are responsible for bridging. After successful negotiation, each Shared Ethernet Adapter bridges the assigned trunk adapters and the associated VLANs. Thus, both the primary and the backup Shared Ethernet Adapter bridge the workload for their respective VLANs. If a failure occurs, the active Shared Ethernet Adapter bridges all trunk adapters and the associated VLANs. This action helps to avoid disruption in network services (s. Figure 4).

Requirements --> Both of primary and backup Virtual I/O Servers are at Version 2.2.1.0, or later. --> Two or more trunk adapters are configured for the primary and backup SEA pair. --> Load Sharing mode must be enabled on both primary and backup SEA pair. --> The virtual local area network (VLAN) definitions of the trunk adapters are identical between the primary and backup SEA pair. --> You need to set the same priority to all trunk adapters under one SEA. The primary and backup priority definitions are set at the SEA level, not at trunk adapters level.

Simple test scenario from scratch If you already have traditional SEA failover configured refer to section How to switch from ha_mode to sharing mode. For maintainability we only have tagged traffic in our szenario.

After doing below steps your setup will look like this:

Figure 5: SEA Load Sharing Scenario

p72vio1: State: PRIMARY_SH Bridge Mode: Partial VID shared: 1 10 11 High Availability Mode: Sharing Priority: 1

p72vio2: State: BACKUP_SH Bridge Mode: Partial VID shared: 2 12 13 High Availability Mode: Sharing Priority: 2

If the ha_mode is set to "sharing" on both the primary and the backup SEA but the state of the SEA is "PRIMARY" or "BACKUP" as opposed to "PRIMARY_SH" or "BACKUP_SH" something went wrong. An error must have occurred which forced SEA to go back to non-sharing state even though

Load sharing is enabled. Try to enable Load sharing again by running a chdev command on the backup SEA device. Check the SEA state again. If the SEA has recovered from its error, then it must be operating in Load sharing mode again. If the SEA is still not operating in Load sharing mode, then further detail investigation is required.

Setup 1.) Create virtual ethernet adapters on p72vio1 HMC --> select p72vio1 --> select Configuration --> select Manage Profiles --> select the correct profile --> Actions --> Edit --> Select Virtual Adapters --> Actions --> Create Virtual Adapter --> Ethernet Adapter Create first trunk adapter on p72vio1:

Create second trunk adapter on p72vio1:

Create Control Channel on p72vio1:

2.) Create virtual ethernet adapters on p72vio2 HMC --> select p72vio2 --> select Configuration --> select Manage Profiles --> select the correct profile --> Actions --> Edit --> Select Virtual Adapters --> Actions --> Create Virtual Adapter --> Ethernet Adapter

Create first Trunk Adapter on p72vio2:

Create second Trunk Adapter on p72vio2:

Create Control Channel on p72vio2:

3.) Power on both VIO servers p72vio1: # lsdev -Cc adapter |grep Ethernet ent0 Available Logical Host Ethernet Port (lp-hea) ent1 Available Virtual I/O Ethernet Adapter (l-lan) ent2 Available Virtual I/O Ethernet Adapter (l-lan) ent3 Available Virtual I/O Ethernet Adapter (l-lan) lhea0 Available Logical Host Ethernet Adapter (l-hea) # entstat -d ent1 |grep Trunk Trunk Adapter: True # entstat -d ent2 |grep Trunk Trunk Adapter: True # entstat -d ent1 |grep VLAN Port VLAN ID: 1 VLAN Tag IDs: 10 11 # entstat -d ent2 |grep VLAN Port VLAN ID: 2 VLAN Tag IDs: 12 13 p72vio2: # lsdev -Cc adapter |grep Ethernet ent0 Available Logical Host Ethernet Port (lp-hea) ent1 Available Virtual I/O Ethernet Adapter (l-lan) ent2 Available Virtual I/O Ethernet Adapter (l-lan) ent3 Available Virtual I/O Ethernet Adapter (l-lan)

lhea0 Available Logical Host Ethernet Adapter (l-hea) # entstat -d ent1 |grep Trunk Trunk Adapter: True # entstat -d ent2 |grep Trunk Trunk Adapter: True # entstat -d ent1 |grep VLAN Port VLAN ID: 1 VLAN Tag IDs: 10 11 # entstat -d ent2 |grep VLAN Port VLAN ID: 2 VLAN Tag IDs: 12 13

On HMC: hscroot@p7hmc:~> lshwres -r virtualio --rsubtype eth -m p72 --level lpar |grep p72vio1 lpar_name=p72vio1,lpar_id=1,slot_num=2,state=1,is_required=0, is_trunk=1,trunk_priority=1,ieee_virtual_eth=1, port_vlan_id=1,vswitch=ETHERNET0," addl_vlan_ids=10,11",mac_addr=761FA3934802,allowed_os_mac_addrs=all,qos_priority=none lpar_name=p72vio1,lpar_id=1,slot_num=3,state=1,is_required=0, is_trunk=1,trunk_priority=1,ieee_virtual_eth=1, port_vlan_id=2,vswitch=ETHERNET0," addl_vlan_ids=12,13",mac_addr=761FA3934803,allowed_os_mac_addrs=all,qos_priority=none lpar_name=p72vio1,lpar_id=1,slot_num=4,state=1,is_required=0, is_trunk=0,ieee_virtual_eth=0, port_vlan_id=99,vswitch=ETHERNET0,addl_vlan_ids=,mac_addr=761FA3934804,allowed_os _mac_addrs=all,qos_priority=none hscroot@p7hmc:~> lshwres -r virtualio --rsubtype eth -m p72 --level lpar |grep p72vio2 lpar_name=p72vio2,lpar_id=2,slot_num=2,state=1,is_required=0, is_trunk=1,trunk_priority=2,ieee_virtual_eth=1, port_vlan_id=1,vswitch=ETHERNET0," addl_vlan_ids=10,11",mac_addr=761FA4F6A602,allowed_os_mac_addrs=all,qos_priority=non e lpar_name=p72vio2,lpar_id=2,slot_num=3,state=1,is_required=0, is_trunk=1,trunk_priority=2,ieee_virtual_eth=1, port_vlan_id=2,vswitch=ETHERNET0," addl_vlan_ids=12,13",mac_addr=761FA4F6A603,allowed_os_mac_addrs=all,qos_priority=non e lpar_name=p72vio2,lpar_id=2,slot_num=4,state=1,is_required=0, is_trunk=0,ieee_virtual_eth=0, port_vlan_id=99,vswitch=ETHERNET0,addl_vlan_ids=,mac_addr=761FA4F6A604,allowed_os _mac_addrs=all,qos_priority=none

4.) Creating SEA with Load Sharing mode

If all the Load sharing criteria are satisfied, then Load sharing can be enabled by setting the ha_mode attribute of SEA device to "sharing". This value must be set on primary SEA first before it is set for the backup SEA, because the backup SEA initiates the request for load sharing. If this sequence is not followed (means not activated on the primary first), then a chdev must occur on the backup SEA for Load sharing to work. - When the backup SEA is configured in Load sharing mode it initiates the request for Load sharing by sending a special packet over the control channel to the primary SEA. - The packet contains the list of VLANs the backup SEA proposes to take over for bridging. - When the primary SEA receives the request for sharing it verifies the request and grants the request if it meets the sharing criteria. - After the primary SEA grants the request, it switches over to sharing mode and sends an ACK packet to backup SEA via the control channel. - When the backup SEA receives the ACK packet it switches to sharing mode and starts bridging for VLANs it had proposed to bridge. - From then on, heartbeats are exchanged between the primary and the backup SEA in load sharing mode. - Different to the traditional SEA failover mode, both the primary and the backup SEAs send and receive heartbeats. Note that these heartbeats are in addition to heartbeats sent by the primary SEA in traditional SEA failover mode - When either side (primary or backup) fails to receive load sharing heartbeats for a predetermined period of time, it is assumed that the other SEA has encountered a problem and it falls back to non-sharing mode and starts bridging traffic for all VLANs. - To restart Load sharing, a chdev must occur on the backup SEA for the backup SEA to reinitiate Load sharing request. On p72vio1: $ mkvdev -sea ent0 -vadapter ent1,ent2 -default ent1 -defaultid 1 -attr ha_mode=sharing ctl_chan=ent3 ent4 Available en4 et4 On p72vio2: $ mkvdev -sea ent0 -vadapter ent1,ent2 -default ent1 -defaultid 1 -attr ha_mode=sharing ctl_chan=ent3 ent4 Available en4 et4

How do I know that load sharing is working and how do I know which VLANs are bridged by each SEA? --> Check the value of ha_mode attribute of the SEA device. It must be set to "sharing" on both the primary SEA and the backup SEA --> Run entstat -all (as padmin) or entstat -d (as root) for SEA device. If the SEA is operating in Load sharing state, then its state must be either PRIMARY_SH or BACKUP_SH --> Run the entstat on the SEA device and look for values for VID Bridged. These are the VLANs bridged by the SEA while SEA is operating in Load sharing mode. On p72vio1: # lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode sharing High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True ... virt_adapters ent1,ent2 List of virtual adapters associated with the SEA (comma separated) # entstat d ent4 State: PRIMARY_SH Bridge Mode: Partial VID shared: 1 10 11 High Availability Mode: Sharing Priority: 1 On p72vio2: # lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode sharing High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True

... virt_adapters ent1,ent2 List of virtual adapters associated with the SEA (comma separated) # entstat d ent4 State: BACKUP_SH Bridge Mode: Partial VID shared: 2 12 13 High Availability Mode: Sharing Priority: 2 Some outputs in disaster cases Shutdown primary vio server (p72vio1) p72vio1 (primary): p72vio2 (backup): # errpt E136EAFA 0112173112 I H ent4 BECOME PRIMARY # entstat d ent4 State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 2 Power on primary vio server (p72vio1) p72vio1 (primary): p72vio2 (backup): # errpt # errpt E136EAFA 0112105212 I H ent4 BECOME 40D97644 0112174812 I H ent4 BECOME PRIMARY BACKUP # entstat d ent4 # entstat d ent4 State: PRIMARY_SH State: BACKUP_SH Bridge Mode: Partial Bridge Mode: Partial VID shared: 1 10 11 VID shared: 2 12 13 High Availability Mode: Sharing High Availability Mode: Sharing Priority: 1 Priority: 2 Shutdown backup vio server (p72vio2) p72vio1 (primary): p72vio2 (backup):

# errpt # entstat d ent4 State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 1 Power on backup vio server (p72vio2) p72vio1 (primary): p72vio2 (backup): # errpt On backup vio server: # errpt 40D97644 0112180112 I H ent4 BECOME BACKUP # entstat d ent4 # entstat d ent4 State: PRIMARY_SH Bridge Mode: Partial State: BACKUP_SH VID shared: 1 10 11 Bridge Mode: Partial High Availability Mode: Sharing VID shared: 2 12 13 Priority: 1 High Availability Mode: Sharing Priority: 2

Enhanced test scenario (adding additional trunk adapter dynamically to the SEA)

Figure 6: Enhanced SEA Load Sharing Scenario

p72vio1: State: PRIMARY_SH Bridge Mode: Partial VID shared: 3 4 33 44 333 High Availability Mode: Sharing Priority: 1 Setup

p72vio2: State: BACKUP_SH Bridge Mode: Partial VID shared: 1 2 10 11 12 13 High Availability Mode: Sharing Priority: 2

1.) create additional trunk adapter on the vio server with dlpar HMC --> select p72vio1 --> select Dynamic Logical Partitioning Select Virtual Adapter --> Actions --> Create Virtual Adapter --> Ethernet Adapter Create additional trunk adapter on p72vio1:

Create additional trunk adapter on p72vio1:

1.) create additional trunk adapter on the vio server with dlpar HMC --> select p72vio2 --> select Dynamic Logical Partitioning Select Virtual Adapter --> Actions --> Create Virtual Adapter --> Ethernet Adapter Create additional trunk adapter on p72vio2:

Create additional trunk adapter on p72vio2:

3.) add adapter to the profile (starting with HMC 7.7.3 the existing profile can be overwritten, previously you needed to specify new profile name ) On both vio server (p72vio1 and p72vio2):

HMC --> select Configuration --> seclect Save Current Configuration Overwrite Existing Profile 4.) run cfgmgr (or cfgdev as padmin) on both vio server (p72vio1 and p72vio2): # cfgmgr # lsdev -Cc adapter |grep Ethernet ent0 Available Logical Host Ethernet Port (lp-hea) ent1 Available Virtual I/O Ethernet Adapter (l-lan) ent2 Available Virtual I/O Ethernet Adapter (l-lan) ent3 Available Virtual I/O Ethernet Adapter (l-lan) ent4 Available Shared Ethernet Adapter ent5 Available Virtual I/O Ethernet Adapter (l-lan) ent6 Available Virtual I/O Ethernet Adapter (l-lan) lhea0 Available Logical Host Ethernet Adapter (l-hea) new virual adapter ent5 and ent6

5.) Add the new trunk adapter to the SEA On p72vio1: $ chdev -dev ent4 -attr virt_adapters=ent1,ent2,ent5,ent6 ent4 changed On p72vio2: $ chdev -dev ent4 -attr virt_adapters=ent1,ent2,ent5,ent6 ent4 changed How do I know that load sharing is working and how do I know which VLANs are bridged by each SEA? On p72vio1: # lsattr El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode sharing High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True ... virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)

# entstat d ent4 ... State: PRIMARY_SH Bridge Mode: Partial VID shared: 3 4 33 44 333 High Availability Mode: Sharing Priority: 1 ... On p72vio2: # lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode sharing High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True ... virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated) # entstat d ent4 ... State: BACKUP_SH Bridge Mode: Partial VID shared: 1 2 10 11 12 13 High Availability Mode: Sharing Priority: 2 ...

Some outputs in disaster cases Shutdown primary vio server (p72vio1) p72vio1 (primary): p72vio2 (backup): # errpt E136EAFA 0113115612 I H ent4 BECOME PRIMARY

# entstat d ent4 State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 2 Power on primary vio server (p72vio1) p72vio1 (primary): p72vio2 (backup): # errpt: # errpt: E136EAFA 0113050512 I H ent4 BECOME 40D97644 0113120112 I H ent4 BECOME PRIMARY BACKUP # entstat d ent4 # entstat d ent4 State: PRIMARY_SH State: BACKUP_SH Bridge Mode: Partial Bridge Mode: Partial VID shared: 3 4 33 44 333 VID shared: 1 2 10 11 12 13 High Availability Mode: Sharing High Availability Mode: Sharing Priority: 1 Priority: 2

Switching from SEA Load Sharing to SEA failover Switching to auto or to load sharing, you have to change ha_mode on the primary SEA first before changing ha_mode on the backup SEA.

On p72vio1 (Primary):

$ chdev -dev ent4 -attr ha_mode=auto ent4 changed # lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode auto High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True

... virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated) # entstat d ent4 State: PRIMARY Bridge Mode: All High Availability Mode: Auto Priority: 1

On p72vio2 (Backup):

$ chdev -dev ent4 -attr ha_mode=auto ent4 changed # lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode auto High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True ... virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated) # entstat d ent4 Limbo Packets: 0 State: BACKUP Bridge Mode: None High Availability Mode: Auto Priority: 2

Switching from SEA failover to SEA Load Sharing

On p72vio1 (Primary):

$ chdev -dev ent4 -attr ha_mode=sharing ent4 changed

# lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode sharing High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True ... virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated) # entstat d ent4 State: PRIMARY Bridge Mode: All High Availability Mode: Sharing Priority: 1

On p72vio2 (Backup): $ chdev -dev ent4 -attr ha_mode=sharing ent4 changed # lsattr -El ent4 ... ctl_chan ent3 Control Channel adapter for SEA failover True ... ha_mode sharing High Availability Mode True ... pvid 1 PVID to use for the SEA device True pvid_adapter ent1 Default virtual adapter to use for non-VLAN-tagged packets ... real_adapter ent0 Physical adapter associated with the SEA True ... virt_adapters ent1,ent2,ent5,ent6 List of virtual adapters associated with the SEA (comma separated)

# entstat d ent4 State: BACKUP_SH Bridge Mode: Partial VID shared: 1 2 10 11 12 13 High Availability Mode: Sharing Priority: 2

On p72vio1 (Primary): State: PRIMARY_SH Bridge Mode: Partial VID shared: 3 4 33 44 333 High Availability Mode: Sharing Priority: 1

FAQs 1) What is the algorithm that the VIOS pair uses to determine which VIOS should "own" the VID's? VIOS divides the number of trunk adapters by 2. One VIOS bridges all VLANs of one half, and the other VIOS does bridging for VLANs of the other half. 2) As time goes on will the list of VID's change for each VIOS? (related to my algorithm question). Not if the SEA configuration remains the same. 3) In load sharing mode is there any potential for broadcast storm if either of the VIOS are rebooted? (one at a time). No. 4) Before rebooting each VIOS (one at a time) do I need to switch it back to "auto" mode? No. 5) If I want to rmdev the SEA can I do this when it's in "sharing" mode on either VIOS? Based on what I've read I'd have to do this on the Primary SEA VIOS (with the lowest priority) first. In "auto" mode I can run rmdev against the SEA on either VIOS. Yes.

Changes Section:

July 31, 2012 a) Correction to figures 4,5 and 6 b) Added, "For maintainability we only have tagged traffic in our szenario" in the section Simple test scenario from scratch. c) Figure 2 on the first VIOS the PVID was empty in the trunk adapter and is now added. d) Figure 3 (multiple virtual ethernet switches) had incorrect PVID in the SEA on the second VIOS . (PVID 2 instead of PVID 1) July 12, 2013 a) Removed dead link