Introduction to vSphere Networking

Day 2

VMware vSphere:
Install, Configure, Manage
Content

• Virtual Networking
• Introduction to vSphere Standard Switches
• Troubleshooting of vSS
• Scenarios
• Introduction to vSphere Distributed Switches
• Troubleshooting of vDS
• NSX
Introduction to vSphere
Standard Switches
Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
• Describe the virtual switch connection types
• Configure and view standard switch configurations, such as virtual machine
port group, VMkernel port, VLAN, and so on
About Virtual Networks
A virtual network is a network of virtual machines running on a physical machine
that are connected logically to one another so that they can send data to and
receive data from one another. Virtual machines can be connected to the virtual
networks that you create when you add a network.

Application Application Application Workload Workload Workload

x86 Environment L2, L3, L4-7 Network Services

Software
Virtual Virtual Virtual Virtual Virtual Virtual
Machine Machine Machine Network Network Network

Server Hypervisor Decoupled Network Hypervisor

Requirement: x86 Requirement: IP Transport

Hardware

General Purpose Server Hardware General Purpose IP Hardware

(Dell, HP, IBM, OpenCompute, Quanta) (Arista, Cisco, HP, Juniper, Accton)
Types of Virtual Switch Connections
A virtual switch has specific connection types:
• Virtual machine port groups
• VMkernel port:
– For IP storage, VMware vSphere® vMotion® migration, VMware vSphere® Fault
Tolerance, VMware Virtual SAN™, and VMware vSphere® Replication™
– For the ESXi management network

Virtual Machine Port Groups VMkernel Ports

Production TestDev DMZ vSphere Management

vMotion
Virtual Switch

Uplink Ports
Virtual Switch Connection Examples
More than one network can coexist on the same virtual switch. Or
networks can exist on separate virtual switches.

Management vSphere vMotion Production TestDev iSCSI

Virtual Switch

Management vSphere vMotion Production TestDev iSCSI

Virtual Switch Virtual Switch Virtual Switch Virtual Switch Virtual Switch
Types of Virtual Switches
A virtual network supports these types of virtual switches:
• Standard switches:
– Virtual switch configuration for a single host
• Distributed switches:
– Virtual switches that provide a consistent network configuration for virtual machines
as they migrate across multiple hosts
Standard Switch Components
A standard switch provides connections for virtual machines to
communicate with one another, whether they are on the same host or on
different hosts.

VM VM VM IP Management
1 2 3 storage Network

VNIC VNIC VNIC VNIC

VMkernel

Test VLAN 101

Production VLAN 102
IP Storage VLAN 103
Management VLAN 104
Viewing the Standard Switch Configuration
You can view a host’s standard switch configuration by clicking
Networking on the Manage tab.

Display port group

properties.

Delete the
port group.

Display Cisco Discovery

Protocol information.
About VLANs
Virtual Local Area Networks
(VLANs) allow you to have many VM VM
virtual networks running over a
single physical network
Uses a standard format to “tag” VMkernel VLAN VLAN
Ethernet frames IEEE 802.1Q.This 105 106

information in the header tells the Virtual Switch

network device which VLAN the
frame belongs in. Physical
NIC
While VLANs do logically separate
Physical Switch
traffic if someone has access to a Trunk Port
network segment they can see all
traffic on all VLANs in that
segment
VLAN RECOMMENDATIONS
VLAN Recommendations
The number of VLANs you can use is often
dictated by your networking equipment
VLAN 0 is reserved
VLAN 1 is usually the default but recommended not to use
Most equipment can go over 4,000 VLANs, you may not be able to use
them all at once
Make sure you know which (if any) VLAN on the physical uplink
switch is set as the native VLAN
Native VLANs are sent out untagged which can cause problems

Suggested to only trunk the VLANs you actually need

Does require more administration to add a pass an additional VLAN

As mentioned, it is suggested not to use VLAN 1 unless necessary

VLAN TAGGING
There are three places where a frame can be tagged with a VLAN.

•All VLAN tagging of packets is performed on the physical switch.

External Switch •ESXi/ESX host network adapters are connected to access ports on the physical
switch.
Tagging •The portgroups connected to the virtual switch must have their VLAN ID set to 0.

•All VLAN tagging of packets is performed by the virtual switch before leaving the
ESXi/ESX host.
Virtual Switch •The ESXi/ESX host network adapters must be connected to trunk ports on the
physical switch.
Tagging •The portgroups connected to the virtual switch must have an appropriate VLAN ID
specified.

•All VLAN tagging is performed by the virtual machine.

Virtual Guest •You must install an 802.1Q VLAN trunking driver inside the virtual machine.
•VLAN tags are preserved between the virtual machine networking stack and
Tagging external switch when frames are passed to/from virtual switches.
•Physical switch ports are set to trunk port.
VLAN Header
Network Adapter Properties
A physical adapter can become a bottleneck for network traffic if the
adapter speed does not match application requirements.
Review of Learner Objectives
You should be able to meet the following objectives:
• Describe the virtual switch connection types
• Configure and view standard switch configurations, such as virtual machine
port group, VMkernel port, VLAN, and so on
Configuring Standard Switch
Policies
Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
• Explain how to set the security policies for a standard switch port group
• Explain how to set the traffic shaping policies for a standard switch port group
• Explain how to set the NIC teaming and failover policies for a standard switch
port group
Network Switch and Port Policies
Policies that are set at the standard switch level apply to all port groups
on the standard switch by default.
Available network policies:
• Security
• Traffic shaping
• NIC teaming and failover

Policies are defined at the following levels:

• Standard switch level:
– Default policies for all the ports on the standard switch.
• Port group level:
– Effective policies: Policies defined at this level override the default policies that are
set at the standard switch level.
Configuring Security Policies
Administrators can define security policies at both the standard switch
level and the port group level:
• Promiscuous mode: Allows a virtual switch or port group to forward all traffic
regardless of the destination.
• MAC address changes: Accept or reject inbound traffic when the MAC
address has been altered by the guest.
• Forged transmits: Accept or reject outbound traffic when the MAC address
has been altered by the guest.
Traffic-Shaping Policy
Network traffic shaping is a mechanism for limiting a virtual machine’s
consumption of available network bandwidth.
Average rate, peak rate, and burst size are configurable.
Outbound Bandwidth

Peak Bandwidth

Average

Time
Burst Size = Bandwidth x Time
Configuring Traffic Shaping
A traffic-shaping policy is defined by average bandwidth, peak
bandwidth, and burst size. You can establish a traffic-shaping policy for
each port group and each distributed port or distributed port group:
• Traffic shaping is disabled by default.
• Parameters apply to each virtual NIC in the standard switch.
• On a standard switch, traffic shaping controls only outbound traffic.
NIC Teaming and Failover Policies
Administrators can edit the NIC teaming and failover policy by
configuring specific options.
Load-Balancing Method: Originating Virtual Port ID
The diagram shows routing based on the originating port ID, called virtual
port ID load balancing.

Virtual
Switch
Physical
Switch

Virtual NICs Physical NICs

Load-Balancing Method: Source MAC Hash
The diagram shows routing based on source MAC hash.

Internet

Virtual
Switch Physical
Switch

Virtual Physical
NICs NICs
Load-Balancing Method: Source and Destination IP
Hash

The diagram shows routing based on IP hash.

Internet

Virtual Physical
Switch Switch

Virtual NICs Physical NICs

Detecting and Handling Network Failure
The VMkernel can use link status or beaconing or both to detect a
network failure.
Network failure is detected by the VMkernel, which monitors the link state
and performs beacon probing.
VMkernel notifies physical switches of changes in the physical location of
a MAC address.
Failover is implemented by the VMkernel based on configurable
parameters:
• Failback: How the physical adapter is returned to active duty after recovering
from failure.
• Load-balancing option: Use explicit failover order. Always use the vmnic uplink
at the top of the active adapter list.
SR-IOV (Single Root IO Virtualization)
SR-IOV is a specification that allows a single Peripheral
Component Interconnect Express (PCIe) physical device under a single
root port to appear as multiple separate physical devices to the
hypervisor or the guest operating system.

SR-IOV uses physical functions (PFs) and virtual functions (VFs)

to manage global functions for the SR-IOV devices. PFs are full PCIe
functions that are capable of configuring and managing the SR-IOV
functionality. It is possible to configure or control PCIe devices using PFs,
and the PF has full ability to move data in and out of the device. VFs are
lightweight PCIe functions that support data flowing but have a restricted
set of configuration resources.

The number of virtual functions provided to the hypervisor or the

guest operating system depends on the device. SR-IOV enabled PCIe
devices require appropriate BIOS and hardware support, as well as SR-
IOV support in the guest operating system driver or hypervisor instance.
Switch Discovery Protocol
Switch discovery protocols help vSphere administrators to
determine which port of the physical switch is connected to a vSphere
standard switch or vSphere distributed switch.

vSphere 5.0 and later supports Cisco Discovery Protocol (CDP)

and Link Layer Discovery Protocol (LLDP). CDP is available for vSphere
standard switches and vSphere distributed switches connected to Cisco
physical switches. LLDP is available for vSphere distributed switches
version 5.0.0 and later.

When CDP or LLDP is enabled for a particular vSphere

distributed switch or vSphere standard switch, you can view properties of
the peer physical switch such as device ID, software version, and timeout
from the vSphere Web Client.
CDP

Cisco Discovery Protocol (CDP) enables vSphere administrators

to determine which port of a physical Cisco switch connects to a vSphere
Standard Switch or vSphere Distributed Switch. When CDP is enabled
for a vSphere Distributed Switch, you can view the properties of the
Cisco switch such as device ID, software version, and timeout.
LLDP
The Link Layer Discovery Protocol (LLDP) is a vendor-neutral link layer
protocol in the Internet Protocol Suite used by network devices for
advertising their identity, capabilities, and neighbors on an IEEE 802
local area network, principally wired Ethernet.
Jumbo Frames
Jumbo frames let ESXi hosts send larger frames out onto the physical network. The
network must support jumbo frames end-to-end that includes physical network adapters,
physical switches, and storage devices.

Before enabling jumbo frames, check with your

hardware vendor to ensure that your physical
network adapter supports jumbo frames.

You can enable jumbo frames on a vSphere

distributed switch or vSphere standard switch by
changing the maximum transmission unit (MTU) to
a value greater than 1500 bytes. 9000 bytes is the
maximum frame size that you can configure.
Review of Standard Switch

If a virtual machine loses network connectivity, the cause of the problem

might be anywhere from the virtual machine’s NIC to the ESXi host’s
physical network.

Virtual Virtual Virtual

ESXi NIC NIC NIC
Host Management
Network

vmnic0 vmnic1 Physical NICs

Review of Learner Objectives
You should be able to meet the following objectives:
• Explain how to set the security policies for a standard switch port group
• Explain how to set the traffic shaping policies for a standard switch port group
• Explain how to set the NIC teaming and failover policies for a standard switch
port group
Network Troubleshooting
ESXCLI Command
To troubleshoot networking configurations from the ESXi command line,
ESXCLI is the tool to use or PUTTY can be used.
There are a number of options available when running ‘esxcli’ in terms of
network settings:
~ esxcli network
Netcat Command
Netcat can be used to test connectivity to and from your ESXi host.

~ nc -h
VMPING Command
You can test connectivity to remote ESXi host using the ping and
vmkping utilities. Using vmkping to test connectivity via vMotion
interfaces is a common practice. For example:
~ vmkping 192.168.1.20
OpenSSL Command
You can use the open ssl client present on an ESXi host to test
connectivity to an ssl port – for example to vCenter or to another host. To
do so:
~ openssl s_client -connect 192.168.1.100:443
TCPDUMP Command
This command is used to identify the packet flow in an NIC. To display
packets on interface vmk0 you can run:
~ tcpdump-uw -i vmk0 | more
ESXCFG Command
The esxcfg-nics command provides information about the physical NICs
in use by the VMkernel.
This prints the VMkernel name for the NIC, its PCI ID, driver, link state,
speed, duplex, and a short PCI description of the card. It also allows
users to set speed and duplex settings for a specific NIC.
~ esxcfg-nics <options> [nic]
PKTCAP-UW Tool
The pktcap-uw tool is an enhanced packet capture and analysis tool that
can be used in place of the legacy tcpdump-uw tool. The pktcap-uw tool
is included by default in ESXi 5.5.
vMA
The vSphere Management Assistant (vMA) allows administrators
and developers to run scripts and agents to manage ESXi hosts and
vCenter Server systems. vMA is a virtual machine that includes
prepackaged software, a logging component, and an authentication
component that supports non-interactive login.
As an alternative to esxcli, you can also use the vicfg-dns command from
the vMA or vSphere CLI. Running the command without any parameters
will display a host’s DNS configuration:

vi-admin@vma:~> vifptarget --set

192.168.88.134
vi-admin@vma:~[192.168.88.134]>
vicfg-dns
DNS Configuration

Host Name esxi1

Domain Name vmlab.local
DHCP false
DNS Servers 10.0.0.1
Network Problem 1

The ESXi host has intermittent or no network connectivity to other

systems.

As an initial check from VMware vSphere® ESXi™ Shell, ping a system

that is known to be up and accessible by the ESXi host.

DCUI
Command
Prompt
Identifying Possible Causes
If you know that your hardware is functioning correctly, take the top-down
approach to troubleshooting, starting with the ESXi host configuration.

Possible Causes

The ESXi host network configuration is incorrect.

The VLAN ID of the port group is incorrect.
ESXi The speed and duplex of the network links are not
Host consistent.
The network link is down.
NIC teaming is not configured properly.

The network adapter or server hardware is not supported.

Hardware
The physical hardware is faulty or misconfigured.
(Network, Server)
Network performance is slow.
Possible Cause: ESXi Network Misconfiguration (1)
Verify that your ESXi host network is configured properly:
• Check vSphere standard switches, vmnics, port groups, and VMkernel ports:
– In VMware vSphere® Management Assistant, use vicfg-vswitch –l
– In vSphere ESXi Shell, use esxcfg-vswitch –l and esxcfg-vmknic –l

• Check VLAN IDs of port groups:

– esxcli network vswitch standard portgroup list
Possible Cause: ESXi Network Misconfiguration (2)
Verify that your ESXi host network configuration is configured properly:
• Speed and duplex:
– vicfg-nics –l
• Network uplink and NIC status (up or down):
– esxcfg-nics –l
– vicfg-nics –l
– esxcli network nic list
Resolving ESXi Network Misconfiguration
Adjust settings in your ESXi network configuration that are not configured
properly:
• Standard switches, vmnics, port groups:
– Add standard switch: vicfg-vswitch –a vswitch#
– Add port group: vicfg-vswitch –A pg_name vswitch#
– Add uplink: vicfg-vswitch –L vmnic# vswitch#
• VLAN IDs of port groups:
– esxcli network vswitch standard portgroup set
–p pg_name -v vlan_ID
• Speed and duplex:
– vicfg-nics –d duplex -s speed vmnic#

• Network link status (up or down):

– Connect network adapters to the intended physical switch ports.
Possible Cause: NIC Teaming Misconfiguration
Verify that NIC teaming is configured properly.
Possible Cause: Unsupported or Faulty Hardware
Verify that you are not encountering the following ESXi network hardware
issues:
• The network adapter or server hardware is not supported:
– vicfg-nics –l
– Verify that the network hardware is listed in VMware Compatibility Guide.
• The physical hardware is faulty or misconfigured:
– lspci –p
Possible Cause: Slow Network Performance
Use esxtop (or resxtop) to view key network metrics that can help
identify network performance problems.

Sample resxtop Output

The esxtop command is available in both vSphere ESXi Shell and

VMware vSphere® Command-Line Interface.
Review of Virtual Machine Connectivity
If your virtual machine loses network connectivity, the cause of the
problem might be in the physical layer, the virtual layer, or the guest
operating system itself.

APP
Virtual
FIREWALL Machine
OS

Virtual NIC
Port Groups

Virtual
Switch

Uplink Ports
Physical NICs
Network Problem 2

The virtual machine has no network connectivity.

As an initial check, ping the virtual machine from another system.

If the ping command fails, ping other virtual machines on the same
network to determine the scope of the problem.
Identifying Possible Causes
Take a top-down approach to troubleshooting, from the guest operating
system to the virtual machine and the ESXi host.

Possible Causes

Application or IP settings are misconfigured.

Guest OS The firewall in the guest OS is blocking traffic.

Virtual The port group name does not exist.

Machine The virtual network adapter is not connected.

Underlying issues with ESXi network connectivity exist.

ESXi Storage or resource contention on the ESXi host exists.
Host
Possible Cause: IP Settings and Firewall Problems
IP settings and problems with firewalls might cause the problem.
Check IP settings to ensure that the TCP/IP settings in the guest
operating system are correct.
The firewall in the guest operating system might be blocking traffic.
Ensure that the firewall does not block required ports.
Possible Cause: Port Group Misconfiguration
The port group name that the virtual machine uses is incorrect:
• View the standard switch port group names on the ESXi host:
– vicfg-vswitch –l
• Verify that the virtual machine is using the correct port group.

The virtual network adapter is not connected to the port group:

• Verify that the network adapter is connected to the correct port group.
Possible Cause: ESXi Network Connectivity Problems
Storage or resource contention on the ESXi host can cause network
connectivity issues:
• Ensure that the virtual machine has no underlying issues with storage and that
it is not in resource contention.
Problems might exist with the ESXi host network, the port group ID, the
speed or duplex settings, the physical network link, or the NIC teaming
configuration.
To eliminate a NIC failure or physical configuration issue, connect the
virtual machine to a virtual switch that uses NIC teaming.
Possible Cause: No Available Ports on Virtual Switch
If your vSphere version is earlier to version 5.5, you might encounter a
problem. The virtual switch might not have an available port for the virtual
machine to connect to:
• This situation can occur during a VMware vSphere® vMotion® migration.

Verify the number of configured ports:

The vicfg-vswitch command also shows the number of used ports.

Resolving the Issue of Unavailable Ports on a Virtual Switch
If the virtual switch does not have available ports for virtual machines to
connect to, resolve the issue in one of the following ways:
• Increase the number of ports for the virtual switch and reboot the host to make
the changes effective.
• Create a new virtual switch and spread the virtual machines and port groups
across the two switches.
This problem is not relevant to vSphere version 5.5 and above.
Network Problem 3

An ESXi host frequently disconnects from VMware vCenter Server™.

Another symptom is that the ESXi host is successfully added to the

vCenter Server inventory but disconnects 30 to 90 seconds after the task
completes.
The problem is that dropped, blocked, or lost heartbeat packets are
occurring between vCenter Server and the ESXi host.
Heartbeat Communication Between vCenter Server and ESXi
The ESXi host sends a heartbeat to vCenter Server to signal that the
host is accessible by the management network.

vCenter Server
Management
Firewall ESXi Network
Windows (vmk0)

Heartbeat Sent over UDP Port 902

Identifying Possible Causes
Take a top-down approach to troubleshooting, from the vCenter Server
system to the ESXi host and the hardware.

Possible Causes

Windows Firewall is enabled on the vCenter Server system,

vCenter Server
and UDP port 902 is blocked.

ESXi vCenter Server is not using port 902 for receiving heartbeats,
Host or the ESXi firewall is blocking that port.

Hardware
(CPU, Memory, The network between ESXi and vCenter Server is congested.
Network, Storage)
Possible Cause: Port Blocked by Windows Firewall
If Windows Firewall is enabled and UDP port 902 is blocked, view the
ports blocked by Windows Firewall.
To resolve this problem, adjust Windows Firewall settings:
• If ports are not configured, disable Windows Firewall.
• If the firewall is configured to affect ports, ensure that Windows Firewall is not
blocking UDP port 902.
Possible Cause: vCenter Server Not Using Port 902
By default, the vpxa agent on the ESXi host sends heartbeats to vCenter
Server (vpxd) through UDP port 902.
A problem might exist if the host is configured to send heartbeats over a
port other than 902.
Use the less /etc/vmware/vpxa/vpxa.cfg command on the host
to determine the port that is used to send heartbeats:

…
Resolving the Use of a Port Other Than 902 (1)
If you prefer to use a nondefault port for heartbeats, ensure that the ESXi
firewall is not blocking that port.
Contents of heartbeat.xml
Resolving the Use of a Port Other Than 902 (2)
Check the vCenter Server configuration to verify the port number used
for heartbeats.
Possible Cause: Network Congestion
If the network between ESXi and vCenter Server is congested, dropped
heartbeats might occur.
To verify whether your management network is congested, use a network
packet analyzer:
• You can use the resxtop utility or graphical views to analyze traffic.
• The pktcap-uw command is an enhanced packet capture and analysis tool.
• The tcpdump-uw command is a legacy network traffic capture tool:
– Available in vSphere ESXi Shell, based on the standard tcpdump utility
– For example, to display packets on the VMkernel interface vmk0, run the command:
• tcpdump-uw –i vmk0

• Wireshark is a publicly available network analyzer:

– Captures live network traffic
– Displays packets with detailed protocol information
Resolving Network Congestion
Resolving network congestion has both short-term and long-term
solutions.
Short-term solution to this problem:
• Increase the timeout limit in vCenter Server to keep the ESXi host continuously
connected.
Long-term solution to this problem:
• Resolve the underlying network congestion problems.
• If using distributed switches, use VMware vSphere® Network I/O Control to
reprioritize traffic and increase the number of shares for management traffic.
Network Problem 4

The ESXi host cannot be managed by vCenter Server.

This problem can occur if the ESXi host’s management network was
misconfigured or manipulated from the command line.
For example, you can bring a physical network card up or down with the
esxcli command:
• esxcli network nic up –n vmnic0
• esxcli network nic down –n vmnic0
• esxcli network nic list
Preventing Loss of Management Network Connectivity
vSphere network rollback prevents accidental misconfiguration of
management networking and loss of connectivity:
• For example, if you try to change the IP address of your management
VMkernel interface, VMware vSphere® Web Client returns the error message
in the screenshot.
Host Networking Rollback
Rollback enables you to roll back to a previous valid configuration.
The host networking rollback is triggered when a network configuration
change is made that disconnects the host
Several events can trigger a host networking rollback:
• Updating DNS and routing settings
• Updating the speed or duplex of a physical NIC
• Changing the IP settings of a management VMkernel network adapter
• Updating teaming and failover policies to a port group that contains the
management VMkernel network adapter
If a network disconnects for any of these reasons, the task fails and the
host reverts to the last valid configuration.
Recovering a Lost Management Network: Standard Switch
If your management network is on a standard switch and you lose
management network connectivity, the solution uses the Configure
Management Network option in the DCUI.
Network Restore Options in the DCUI
To restore the network through the DCUI:
1. Select Network Restore Options.
2. Perform a full network restore.
3. Repair the Management network on a misconfigured standard or
distributed switch.

The Restore Network Settings option deletes all the current network
settings except for the Management network.
Review of Learner Objectives
You should be able to meet the following objectives:
• Provide a network troubleshooting overview
• Analyze and troubleshoot standard switch problems
• Analyze and troubleshoot virtual machine connectivity problems
• Analyze and troubleshoot management network problems
Key Points
• Virtual network connectivity problems might occur with standard switches,
distributed switches, virtual machines, or management networks.
• A virtual machine connectivity problem might exist in the physical layer, the
virtual layer, or the guest operating system.
• The ping command is useful when troubleshooting ESXi host and virtual
machine connectivity issues.
• When an ESXi host frequently disconnects from vCenter Server, heartbeat
packets are being lost between vCenter Server and the ESXi host.
• vSphere network rollback prevents accidental misconfiguration of management
networking and loss of connectivity.
• A good practice is to back up your distributed switch configuration with the
vSphere Web Client whenever you make a change to the configuration.
• You can use the restore or the import function to reset the distributed switch
configuration.
Questions?