HP BladeSystem Networking Reference Architecture

HP Virtual Connect FlexFabric Module and VMware vSphere 5

Technical white paper

Table of contents
Executive Summary .............................................................................................................................. 2 Virtual Connect FlexFabric Module Hardware Overview .......................................................................... 3 Designing an HP FlexFabric Architecture for VMware vSphere .................................................................. 4 Designing a vSphere Network Architecture with the Virtual Connect FlexFabric module ............................... 7 vNetwork Distributed Switch Design ................................................................................................... 7 Network I/O Control ........................................................................................................................ 9 Hypervisor Load Balancing Algorithms ............................................................................................... 9 Modifying Server Profiles with Online Servers .................................................................................... 10 HP NC551 and NC553 Minimum Supported Firmware and Drivers .................................................... 10 Appendix A: Virtual Connect Bill of Materials ....................................................................................... 11 Appendix B: Terminology cross-reference ............................................................................................. 11 Appendix C: Glossary of Terms ........................................................................................................... 12 For more information .......................................................................................................................... 14

Executive Summary
HP has revolutionized the way IT thinks about networking and server management. With the release of the HP ProLiant BladeSystem Generation 6 servers, along with Virtual Connect Flex-10 Ethernet modules, HP provided a great platform for VMware vSphere. Virtual Connect Flex-10 is the world's first technology to divide and fine-tune 10Gb Ethernet network bandwidth at the server edge. When combined with Virtual Connect, the BladeSystem architecture streamlines the typical change processes for provisioning in the datacenter. HP has since evolved Virtual Connect Flex-10 to the next level: Virtual Connect FlexFabric modules. By combining the power of ProLiant BladeSystem Generation 7 servers and Virtual Connect, the Virtual Connect FlexFabric module allows customers to consolidate network connections and storage fabrics into a single module. This further reduces infrastructure cost and complexity by eliminating HBA adapters and Fibre Channel modules at the edge. These servers include virtualization friendly features such as larger memory capacity, dense population, room for additional mezzanine cards and 4 - 64 (with Intel Hyper-Threading technology enabled and AMD Opteron 6200-series) processing cores. The following ProLiant BL Servers ship standard with the NC551i FlexFabric Adapter: BL465 G7 BL685 G7

The following ProLiant BL Servers ship with the NC553i FlexFabric Adapter: BL460 G7 BL490 G7 BL620/680 G7

Additionally, the NC551m and NC553m provide support for the FlexFabric Adapter in ProLiant BladeSystem G6 servers. NOTE: Please check the latest NC551m1 and NC553m2 QuickSpecs for the official server support matrix. The ProLiant Generation 8 servers do not ship with a built-in adapter. Instead, HP offers a FlexLOM option to where customers can chose which FlexFabric Adapter vendor they want. Please review the server models QuickSpecs for support FlexLOM adapters. The FlexFabric Adapter introduces a new Physical Function called the FlexHBA. The FlexHBA, along with the FlexNIC, adds the unique ability to fine-tune each connection to adapt to your virtual server channels and workloads on-the-fly. The effect of using the Virtual Connect FlexFabric module is a reduction in the number of interconnect modules required to uplink outside of the enclosure, while still maintaining full redundancy across the service console, VMkernel, virtual machine (VM) networks and

1 2

http://h18004.www1.hp.com/products/blades/components/emulex/nc551m/index.html http://h18004.www1.hp.com/products/blades/components/emulex/nc553m/index.html

storage fabrics. This translates to a lower cost infrastructure with fewer management points, switch modules, adapter cards and cables. This whitepaper will focus on designing a highly available network and vSphere Cluster architecture by providing redundant uplinks across physical modules, but also across enclosures. This design will also maximize the management features of Virtual Connect, while providing customers with the flexibility to provide any networking to any host within the Virtual Connect domain. Simply put, this design will not over-provision servers, while keeping the number of uplinks used to a minimum. This helps reduce infrastructure cost and complexity by trunking the necessary VLANs (IP Subnets) to the Virtual Connect domain, and minimizing potentially expensive 10Gb uplink ports. Finally, this document will provide key design best practices for vSphere 5 network architecture with HP FlexFabric, including: vDS design for Hypervisor networking vSwitch and dvPortGroup load balance algorithms

Virtual Connect FlexFabric Module Hardware Overview

The Virtual Connect FlexFabric module is the first Data Center Bridging (DCB) and Fibre Channel over Ethernet (FCoE) solution introduced into the HP BladeSystem portfolio. It provides 24 line rate ports, Full-Duplex 240Gbps bridging, single DCB-hop fabric. As shown in Image 1, there are 8 faceplate ports. Ports X1-X4 are SFP+ transceiver slots only; which can accept a 10Gb or 8Gb SFP+ transceiver. Ports X5-X8 are SFP and SFP+ capable, and do not support 8Gb SFP+ transceivers. NOTE: The CX-4 port provided by the Virtual Connect Flex-10 and legacy modules has been depreciated.
Image 1: HP Virtual Connect FlexFabric Module

Ports X7 and X8 are shared with internal Stacking Link ports. If the external port is populated with a transceiver, the internal Stacking Link is disabled. At least one Stacking Link is required between modules. Please refer to the Virtual Connect User Guide or Multi-Enclosure Stacking Reference Guide3 for more information.

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02102153/c02102153.pdf

Important: Even though the Virtual Connect FlexFabric module supports Stacking, stacking only applies to Ethernet traffic. FC uplinks cannot be consolidated, as it is not possible to stack the FC ports, nor provide a multi-hop DCB bridging fabric today.

Designing an HP FlexFabric Architecture for VMware vSphere

In this design, two HP ProLiant c-Class 7000 Enclosures with Virtual Connect FlexFabric modules are stacked to form a single Virtual Connect management domain4. By stacking Virtual Connect FlexFabric modules, customer can realize the following benefits: Management control plane consolidated More efficient use of WWID, MAC and Serial Number Pools Provide greater uplink port flexibility and bandwidth Profile management across stacked enclosures

Shared Uplink Sets provide administrators the ability to distribute VLANs into discrete and defined Ethernet Networks (vNet.) These vNets can then be mapped logically to a Server Profile Network Connection allowing only the required VLANs to be associated with the specific server NIC port. This also allows customers the flexibility to have various network connections for different physical Operating System instances (i.e. VMware ESX host and physical Windows host.) As of Virtual Connect Firmware 3.305 release, the following Shared Uplink Set rules apply per domain: 1000 Unique VLANs (Ethernet Networks) per Virtual Connect Domain 162 Unique Server Mapped VLANs per Server Profile Network Connection

Important: When using FlexNICs, the 162 Unique Server Mapped VLAN maximum is an aggregate of all FlexNICs that belong to the same 10Gb Physical Interface. A FlexHBA also counts as a single VNET against the 162 total. By providing two stacked Enclosures, this will allow for not only Virtual Connect FlexFabric module failure, but also Enclosure failure. The uplink ports assigned to each Shared Uplink Set (SUS) were vertically offset to allow for horizontal redundancy purposes, as shown in Figure 1-2. IP-based storage (NFS and/or iSCSI) can be dedicated and segregated by a separate vNet and assigned uplink port. This design approach provides administrators to dedicate a network (physically switched, directly connected or logical within a Shared Uplink Set) to provide access to IP-based storage arrays.

Only available with Virtual Connect Manager Firmware 2.10 or greater. Please review the Virtual Connect Manager Release Notes for more information regarding domain stacking requirements: http://h18004.www1.hp.com/products/blades/components/c-class-tech-installing.html Only when the Expanded VLAN Capacity feature is enabled in the Domain (Ethernet -> Advanced Settings)

Directly connecting an IP-based Storage array has certain limitations: Each storage array front-end port will require a unique vNet Each defined vNet will require separate server network connections You are limited to the number of IP-based arrays based on the number of unassigned uplink ports

Note: Please review the Virtual Connect with iSCSI Cookbook Third Edition or newer for more information on how to directly attach an IP-based storage device to Virtual Connect. Virtual Connect has the capability to create an internal, private network without uplink ports, by using the low latency mid-plane connections to facilitate communication. This vNet can be used for cluster heartbeat networks, or in this case VMotion and/or Fault Tolerance traffic. Traffic will not pass to the upstream switch infrastructure, which will eliminate the bandwidth otherwise consumed.
Figure 1-1: Physical VMware vSphere Cluster Design

Figure 1-2 show the physical cabling. The X5 and X6 Ethernet ports of the FlexFabric module are connecting to a redundant pair of Top of Rack (ToR) switches, using LACP (802.3AD) for link redundancy. The ToR switches can be placed End of Row to save on infrastructure cost. Ports X7 are used for vertical External Stacking Links, while X8 are used for Internal Stacking Links.

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02533991/c02533991.pdf

As noted in the previous section, Virtual Connect FlexFabric Stacking Links will only carry Ethernet traffic, and do not provide any Fibre Channel stacking options. Thus, ports X1 and X2 from each module are populated with 8Gb SFP+ transceivers, providing 16Gb net FC bandwidth for storage access. Ports X3 and X4 are available to provide additional bandwidth if FC storage traffic is necessary. If additional Ethernet bandwidth is necessary, ports Enc0:Bay2:X5, Enc0:Bay2:X6, Enc1:Bay1:X5, and Enc1:Bay1:X6 can be used for additional Ethernet Networks or Shared Uplink Sets.
Figure 1-2: Physical cabling design

Figure 1-3: Logical design

Designing a vSphere Network Architecture with the Virtual Connect FlexFabric module
The vNetwork Distributed Switch7 (vDS) allows an administrator to create a centralized distributed vSwitch, which share a common networking configuration across hosts. Port Groups are still utilized in this model, but have a different association to host uplink ports. Host uplink ports are added to Uplink Groups (dvUplinkGroup), where a logical association between the dvUplinkGroup and a PortGroup (dvPortGroup) is formed. vDS can service any of the vmkernel functions; Service Console, VMotion, IP Storage, and Virtual Machine traffic. In this section, we will outline the overall vDS design.

vNetwork Distributed Switch Design

Each pair of redundant pNICs will be assigned to dvUplinkGroups, which then are assigned to a specific vDS. This will simplify host network configuration, while providing all of the benefits of a vDS. Table 2-1 and Figure 2-1 shows which hypervisor networking function will be assigned to vDS configuration.
Table 2-1 VMware vDS Configuration Example

VMkernel Function

vDS Name

dvPortGroup Name
dvPortGroup1_Mgmt dvPortGroup2_vmkernel dvPortGroup3_vmnet100

ESXi Host Management dvs1_mgmt VMotion VM Networking1 dvs2_vmkernel dvs3_vmnet

Requires vSphere 5.0 Enterprise Plus licensing; Please refer to the vSphere 5 Maximums document on supported configuration maximums http://www.vmware.com/pdf/vsphere5/r50/vsphere-50-configuration-maximums.pdf

Table 2-1 VMware vDS Configuration Example

VMkernel Function
VM NetworkingN

vDS Name
dvs3_vmnet

dvPortGroup Name
dvPortGroupN_vmnetNNN

Figure 2-1: Hypervisor Networking Design

Important: Please know that for HP CloudSystem Matrix installations, the dvPortGroup for the Service Console must be named Management Network. Mixing vSphere Standard Switches and vSphere Distributed Switches is possible. The current scalability limits to the vDS may require some designs to incorporate both. HP would recommend vmkernel functions (vMotion, Management, IP Storage, FT) be assigned to Standard vSwitches, and use vDS for Virtual Machine networking. VMware Fault Tolerance (FT) could introduce more complexity in to the overall design. VMware states that a single 1Gb NIC should be dedicated for FT logging, which would have the potential to starve any shared pNIC with that of another vmkernel function (i.e. VMotion traffic.) FT has not been taken into consideration within this document. Even though FT could be shared with another vmkernel

function, and if FT is a design requirement, then the overall impact of its inclusion should be examined. With the design example given, there are three options one could choose to incorporate FT Logging:
Table 2-2 VMware Fault Tolerance Options

FT Design Choice
Share with VMotion network

Justification
The design choice to keep VMotion traffic internally to the Domain allows the use of low latency links for inter-Domain communication. NetIOC could be used to control and provide VMotion or FT traffic higher shares, or priority, over the other traffic.

Rating
***

Non-redundant VMotion Dedicate one pNIC for VMotion traffic, and the other for FT and FT networks logging traffic. Neither network will provide pNIC redundancy. Add additional FlexFabric Adapters and Modules This option increases the overall CapEx to the solution, but will provide more bandwidth options.

** *

Network I/O Control

Network I/O Control8 (NetIOC), is a feature of the vDS that provides QoS-like capabilities. NetIOC can identify the following types of traffic: Virtual Machine FT Logging iSCSI (Software Initiator only; HW iSCSI Offload not supported) NFS Service Console Management vMotion

NetIOC can be used to control identified traffic, when multiple types of traffic are sharing the same pNIC. In our design example above, FT Logging could share the same vDS as the vmkernel, and NetIOC would be used to control the two types of traffic. NetIOC can also be used to apply an 802.1p tag for DataCenter QOS, which is not supported by Virtual Connect at this time.

Hypervisor Load Balancing Algorithms

VMware provides a number of different NIC teaming algorithms, which are outlined in Table 2-3. As the table shows, any of the available algorithms can be used, except IP Hash. IP Hash requires switch assisted load balancing (802.3ad), which Virtual Connect does not support 802.3ad with server downlink ports. HP and VMware recommend using Originating Virtual Port ID with Standard vSwitch, and Physical NIC Load when using vDS and NetIOC, as shown in Table 2-3.

http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf (vSphere 4.1) or Managing Network Resources chapter in http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-501-networking-guide.pdf

Table 2-3 VMware Load Balancing Algorithms

Name
Originating Virtual Port ID Source MAC Address IP Hash

Algorithm
Choose an uplink based on the virtual port where the traffic entered the virtual switch. MAC Address seen on vnic port Hash of Source and Destination IPs. Requires switch assisted load balancing, 802.3ad. Virtual Connect does not support 802.3ad on server downlink ports, as 802.3ad is a Point-to-Point bonding protocol. Introduced in vSphere 4.1 and only available with a vDS, Load-Based Teaming policy monitors the flow when the mean send or receive utilization on a dvUplink exceeds 75% capacity over 30-sec intervals. Highest order uplink from the list of Active pNICs.

Works with VC
Yes Yes No

Physical NIC Load

Yes

Explicit Failover

Yes

Modifying Server Profiles with Online Servers

HP fully supports modifying Server Profiles while hosts are still online. For older generation servers, there are specific minimum NIC driver and firmware revisions required. Please review the Virtual Connect Release Notes, and the Service Pack for Proliant Release Notes.

HP NC551 and NC553 Minimum Supported Firmware and Drivers

The table lists the minimum support firmware and driver revisions for the Emulex OneConnect-based adapters from HP:
Table 2-4 Supported minimum versions

Component
Firmware Driver (Ethernet) Driver (FC/FCoE)

Version
4.0.360.15 4.0.355.1 8.2.2.105.36

When multiple NC55x devices are used on a single host, you may experience an issue where not all of the FlexNICs will appear within the ESXi kernel when using the 4.0.3551. To resolve this, you must adjust the be2net driver heap stack allocation. Please refer to the HP Customer Advisory c032201409 on the process and any future updates.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c03220140

10

Appendix A: Virtual Connect Bill of Materials

Table A-1 Virtual Connect FlexFabric module Mapped VLAN BoM

Partnumber
571956-B21 AJ716A 487649-B21 455883-B21 Or 487655-B21

Description
HP Virtual Connect FlexFabric Module HP StorageWorks 8Gb B-series SW SFP+ .5m 10Gb SFP+ DAC Stacking Cable 10Gb SR SFP+ transceiver

Qty
4 8 2 4

3m SFP+ 10Gb Copper DAC

Appendix B: Terminology cross-reference

Table C-1 Terminology cross-reference

Customer term
Port Bonding or Virtual Port VLAN Tagging

Industry term

IEEE term

Cisco term
Etherchannel or channeling (PaGP) Trunking

Nortel term
MultiLink Trunking (MLT) 802.1Q

HP Virtual Connect term

802.3ad LACP

Port Aggregation 802.3ad or Port-trunking LACP LACP VLAN Trunking 802.1Q

Shared Uplink Set

11

Appendix C: Glossary of Terms

Table C-1 Glossary Term vNet/Virtual Connect Ethernet Network Definition A standard Ethernet Network consists of a single broadcast domain. However, when VLAN Tunnelling is enabled within the Ethernet Network, VC will treat it as an 802.1Q Trunk port, and all frames will be forwarded to the destined host untouched. An uplink port or a group of uplink ports, where the upstream switch port(s) is configured as an 802.1Q trunk. Each associated Virtual Connect Network within the SUS is mapped to a specific VLAN on the external connection, where VLAN tags are removed or added as Ethernet frames enter or leave the Virtual Connect domain. Let VC automatically determine best FlexNIC speed Manually set FlexNIC speed (up to Maximum value defined) Device Control Channel: method for VC to change either a FlexNIC or FlexHBA Adapter port settings on the fly (without power no/off) A Cisco proprietary technology that combines multiple NIC or switch ports for greater bandwidth, load balancing, and redundancy. The technology allows for bi-directional aggregated network traffic flow. One of four virtual NIC partitions available per FlexFabric Adapter port. Each capable of being tuned from 100Mb to 10Gb The second Physical Function providing an HBA for either Fibre Channel or iSCSI functions An industry standard protocol that enables multiple virtual networks to run on a single link/port in a secure fashion through the use of VLAN tagging. An industry standard protocol that allows multiple links/ports to run in parallel, providing a virtual single link/port. The protocol provides greater bandwidth, load balancing, and redundancy. Link Aggregation Control Protocol (see IEEE802.3ad) LAN-on-Motherboard. Embedded network adapter on the system board Maximum FlexNIC speed value assigned to vNet by the network administrator. Can NOT be manually overridden on the server profile. Global Preferred and Maximum FlexNIC speed values that override defined vNet values when multiple vNets are assigned to the same FlexNIC Mezzanine Slot 1; LAM on Motherboard/systemboard NIC

Shared Uplink Set (SUS)

Auto Port Speed** Custom Port Speed** DCC**

EtherChannel*

FlexNIC**

FlexHBA*** IEEE 802.1Q

IEEE 802.3ad

LACP LOM Maximum Link Connection Speed** Multiple Networks Link Speed Settings** MZ1 or MEZZ1; LOM

12

Network Teaming Software

A software that runs on a host, allowing multiple network interface ports to be combined to act as a single virtual port. The software provides greater bandwidth, load balancing, and redundancy. Physical NIC port. A FlexNIC is seen by VMware as a pNIC Combining ports to provide one or more of the following benefits: greater bandwidth, load balancing, and redundancy. A Cisco proprietary protocol aids in the automatic creation of Fast EtherChannel links. PAgP packets are sent between Fast EtherChannel-capable ports to negotiate the forming of a channel. A term typically used in the Unix/Linux world that is synonymous to NIC teaming in the Windows world. Preferred FlexNIC speed value assigned by a vNet by the network administrator. 802.1Q VLAN tagging Combining ports to provide one or more of the following benefits: greater bandwidth, load balancing, and redundancy. See also Port Aggregation. A virtual network within a physical network. Tagging/marking an Ethernet frame with an identity number representing a virtual network. A Cisco proprietary protocol used for configuring and administering VLANs on Cisco network devices. Virtual NIC port. A software-based NIC used by VMs
**The feature was added for Virtual Connect Flex-10

pNIC Port Aggregation Port Aggregation Protocol (PAgP)*

Port Bonding Preferred Link Connection Speed** Trunking (Cisco) Trunking (Industry)

VLAN VLAN Tagging VLAN Trunking Protocol (VTP)* vNIC

*The feature is not supported by Virtual Connect. ***The feature was added for Virtual Connect FlexFabric modules

13

For more information

To read more about the Virtual Connect FlexFabric module, go to www.hp.com/go/virtualconnect To learn more about HP BladeSystem, go to www.hp.com/go/bladesystem For additional HP BladeSystem technical documents, go to www.hp.com/go/bladesystem/documentation

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Trademark acknowledgments, if needed. c03278211, March 2012