Storage Troubleshooting: Module Number 6-1

Storage Troubleshooting
Module 6
VMware vSphere: Troubleshooting Revision A

Copyright 2009 VMware, Inc. All rights reserved.
Module Number 6-1
You Are Here
VMware vSphere: Troubleshooting

Course Introduction
Storage Troubleshooting
ESXi Command-Line Troubleshooting
VMotion Troubleshooting
ESX/ESXi and vCenter Server Log Files
VMware Infrastructure Troubleshooting
Network Troubleshooting
Management Troubleshooting

Module Number 6-2
Importance
Nearly every service depends on access to storage. It is
critical that you understand the Pluggable Storage
Architecture in VMware ESX 4.x in order to be able to
quickly restore lost access to storage.

Module Number 6-3
Module Lessons
Lesson 1:
Pluggable Storage Architecture Review
Lesson 2:
Viewing Storage Components
Lesson 3:
iSCSI and NAS Storage

Module Number 6-4
Lesson 1:
Pluggable Storage
Architecture Review

Module Number 6-5
Lesson Objectives
Describe the Pluggable Storage Architecture
Describe the operation of the VMware-supplied PSA plug-ins
Describe the available load-distribution policies
Describe the purpose of the VMware vStorage APIs
Describe asymmetric logical unit access and target port
group support

Module Number 6-6
Pluggable Storage Architecture Overview

The Pluggable Storage Architecture (PSA) uses software called plug-ins to
manage and access storage.
The top-level plug-in is the multipathing plug-in (MPP).
The VMware default MPP is the native multipathing plug-in (NMP), which
includes Storage Array Type Plug-ins (SATPs) and Path Selection Plug-ins
(PSPs).
These plug-ins manage
ESX/ESXi storage on
the VMware hardware
compatibility list.
The default MPP
is the NMP, which
includes the
SATPs and PSPs.

PSA
VMware NMP
VMware SATP
VMware PSP
VMware SATP
VMware PSP
VMware SATP
VMware PSP
Module Number 6-7
Overview of MPP Tasks

The PSA:
Discovers available storage (physical paths)
Uses predefined claim rules to assign each device to an MPP
An MPP claims a physical path and exports a logical device.

Details of path failover for a specific path are delegated to the
Storage Array Type Plug-in (SATP).
Details for determining which physical path is used to a
storage device (load balancing) are handled by the Path
Selection Plug-in (PSP).

Module Number 6-8
Overview of VMware NMP Tasks

The native multipathing plug-in (NMP):
Is the default VMware-supplied MPP
Claims physical storage paths based on claim rules listed in
the /etc/vmware/esx.conf file
Associates an SATP with each physical storage path
Associates physical paths with logical devices.
Processes I/O requests to logical devices
Supports management tasks like abort or reset of logical devices

Module Number 6-9
NMP Illustration
SATPs handle path failover, monitor
path health, and report changes to the
NMP.
PSPs choose the best available path.
SATP
HBA 1

PSP
HBA 2
Module Number 6-10
NMP
PSA
VMkernel
storage stack
Overview of SATP Tasks

The Storage Array Type Plug-in (SATP) handles path failover
operations.
Primary functions of an SATP:
To implement the switching of physical paths to the array when a
path has failed
To determine when a hardware component of a physical path
has failed
To monitor the hardware state of the physical paths to the
storage array

Module Number 6-11
Overview of PSP Tasks

The Path Selection Plug-in (PSP) is responsible for choosing
a physical path to issue an I/O request to a logical device.
A PSP determines how load distribution or load balancing occur.
VMware provides three PSPs: Fixed, MRU and Round Robin.

The NMP specifies a default PSP for every logical device
based on the SATP associated with the physical paths for
that device.
The NMP allows the default PSP for a device to be overridden.
There is the capability to install third-party PSPs in order to

take advantage of more complex I/O load-balancing
algorithms if they are available.

Module Number 6-12
Most Recently Used PSP

This load-distribution PSP selects the first working path
discovered at system boot time.
If this path becomes unavailable, the PSP selects an
alternate path and continues to use the new path while it is
available.
The MRU PSP is the default and required PSP for activepassive arrays and arrays that use asymmetric logical unit
access.

Module Number 6-13
Fixed PSP
The Fixed PSP uses a designated preferred path, if it has
been configured.
If the ESX host cannot use the preferred path, the PSP
selects a random alternative available path.
The PSP automatically fails back to the preferred path as
soon as that path becomes available.
This is the default PSP for active-active arrays.

Module Number 6-14
Round Robin PSP

The Round Robin PSP rotates through all available paths,
spreading I/O requests across all available paths.
The concepts of failback and a preferred path do not apply to
the Round Robin PSP.
The Round Robin PSP is most effective on active-active
arrays with multiple available paths.
This PSP is not the default for any array type.

Module Number 6-15
Storage I/O Example

Information about all functional
paths is forwarded by the SATP to
the PSP. The PSP chooses which
path the use.
PSP
1
SATP
5
HBA 1
2
HBA 2
3

Module Number 6-16
NMP
PSA
VMkernel
storage stack
Third-Party Plug-Ins
PSA
VMware NMP
third-party
MPP
third-party
MPP
For unique performance

and fault-tolerance
behavior

VMware SATP
VMware PSP
VMware SATP
VMware PSP
VMware SATP
VMware PSP
third-party SATP
third-party PSP
To accommodate
specific storage
arrays
Module Number 6-17
For more complex

I/O load-balancing
algorithms
Asymmetric Logical Unit Access

Asymmetric logical unit access (ALUA):
Used with arrays that have multiple physical controllers with
multiple target ports
Asymmetric access to a LUN occurs when the access
characteristics of one port differ from those of another port.

Module Number 6-18
Target Port Group Support

Target port group support:
Provides a method for determining the access characteristics of
a path to a LUN through a target port
Supports soliciting information about different target port
capabilities
Supports routing I/O to the particular port or ports that can
achieve the best performance
Often used with arrays that handle load balancing and failover
within the array controllers and that present one physical port at
a time as the target port
Allows new storage devices to be discovered automatically

Module Number 6-19
Lab 20
In this lab, you will identify components of the PSA.
Identify PSA components.

Module Number 6-20
Lesson Summary
Described the PSA
Described the operation of the VMware-supplied PSA plugins
Described the available load-distribution policies
Described the purpose of the vStorage APIs
Described asymmetric logical unit access and target port
group support

Module Number 6-21
Lesson 2:
Viewing Storage Components

Module Number 6-22
Lesson Objectives
Describe storage device naming
View storage plug-ins in the VMware vSphere Client
Use the esxcli command to view and manage storage
Use the esxcfg-mpath command to view and manage
storage
Use the vmkload_mod command to view VMkernel modules
Configure storage path masking
Troubleshoot storage problems

Module Number 6-23
Storage Naming
Identifiers are universally unique IDs associated with devices
that are persistent across reboots and across hosts. Depending
on the type of storage, different algorithms are used.
Friendly names are generated

by the host and can be changed
by the administrator.
Module Number 6-24
Storage Identifier Examples

mpx ID generated by the
ESX host because the
device did not return an
naa* or t10 ID
naa ID returned by the

Fibre Channel storage
array
t10 ID returned by the

iSCSI storage array
*naa = Network Address Authority
Module Number 6-25
Runtime Names
Field
Definition
The name
of the first
path to the
device,
assigned by
the host,
and not
persistent
across
reboots.
vmhba<n>
The physical storage adapter on the host
C<n>
The channel number. Software iSCSI initiators use the channel number to show
multiple paths to the same target.
T<n>
The target number. Target numbering is decided by the host and might change if
there is a change in the mappings of targets visible to the host. Targets that are
shared by different ESX hosts might not have the same target number.
L<n>
The LUN number that shows the position of the LUN within the target. The LUN
number is provided by the storage system.

Module Number 6-26
Viewing Storage Plug-Ins in the vSphere Client

The MPP that
claimed and
controls access
to the storage
device.
The PSP and the

SATP for the
device.

Properties > Manage Paths
Module Number 6-27
esxcli Command
The esxcli command is primarily used for managing storage.
It is available in the service console and in VMware vSphere
Command-Line Interface (vCLI).
Command syntax:
esxcli <conn_options> <namespace> <app> <cmd> [cmd_options]
<conn_options>
vCLI connection parameters, not needed in the service

console
<namespace>
nmp VMware NMP commands

swiscsi Commands in the software iSCSI name space
corestorage VMware core storage commands
<app>
Area within the name space to which the command applies
<cmd>
Command to be called
[cmd_options]
Command options

Module Number 6-28
esxcfg-mpath Command
The service console esxcfg-mpath command lists information
about Fibre Channel or iSCSI LUNs and changes a storage
paths state.
The vicfg-mpath command is the equivalent in vCLI.
To manage ESX 3.5 hosts using the ESX 4.x vCLI, use the vicfgmpath35 command.
Command syntax:
esxcfg-mpath <conn_options> <cmd_options>
<conn_options> are vCLI connection parameters that not
needed in the service console.
To display a list of command specific options, use esxcfgmpath h.

Module Number 6-29
vmkload_mod Command
The vmkload_mod command is used to load, unload, and
list the loaded VMkernel modules.
It can also list the available parameters for each module.
It is available in the ESX service console.

For available options and syntax, see the vmkload_mod man
page.

Module Number 6-30
Listing Loaded VMkernel Modules

# vmkload_mod -l | grep satp
vmw_satp_local
0x418017811000
vmw_satp_default_aa 0x418017812000
vmw_satp_alua
0x41801783c000
vmw_satp_cx
0x418017840000
vmw_satp_default_ap 0x418017846000
vmw_satp_eva
0x418017848000
vmw_satp_lsi
0x41801784a000
vmw_satp_symm
0x41801784e000
vmw_satp_inv
0x41801784f000
vmw_satp_eql
0x418017852000
vmw_satp_msa
0x418017853000
vmw_satp_svc
0x418017854000
vmw_satp_alua_cx
0x418017855000
0x1000
0x1000
0x4000
0x6000
0x2000
0x2000
0x4000
0x1000
0x3000
0x1000
0x1000
0x1000
0x3000
0x417fd8676270
0x417fd8680e80
0x417fd8684460
0x417fd868d9b0
0x417fd868e9c0
0x417fd868f9d0
0x417fd86909e0
0x417fd86919f0
0x417fd8692a00
0x417fd8693a10
0x417fd8694a20
0x417fd8695a30
0x417fd8696a40
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
0x1000
10
11
17
18
19
20
21
22
23
24
25
26
27
# vmkload_mod -l | grep psp

vmw_psp_fixed
0x418017813000
vmw_psp_rr
0x418017858000
vmw_psp_mru
0x41801785b000
0x2000
0x3000
0x2000
0x417fd8681e90
0x417fd8697a80
0x417fd8698aa0
0x1000
0x1000
0x1000
12 Yes
28 Yes
29 Yes

Module Number 6-31
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Storage Path Masking

You can prevent an ESX/ESXi host from accessing:
A storage array
A LUN
An individual path to a LUN
Use the esxcli command to mask access to storage:

esxcli corestorage claimrule add -r
<claimrule_ID> -t <type> <required_option> -P
MASK_PATH

Module Number 6-32
Lab 21
In this lab, you will examine and configure the storage
of your ESX host.
1. List all multipathing plug-ins installed on your ESX host.
2. Examine devices that have been claimed by the native
multipathing plug-in on your ESX host.
3. Record information about one of the Storage Array Type Plugins loaded into the NMP.
4. Examine the SATP configuration on your ESX host.
5. Determine the default PSP and the active path for a specific
storage NAA device.
6. Change the configuration of the PSP for a storage device.

Module Number 6-33
Lab 22
In this lab, you will examine and configure the storage
of your ESXi host.
1. List all multipathing plug-ins installed on your ESXi host.
2. List which devices have been claimed by the native
multipathing plug-in on your ESXi host.
3. Examine Storage Array Type Plug-ins on your ESXi host.
4. Examine a specific storage device on your ESXi host.

Module Number 6-34
Lesson Summary
Described storage device naming
Viewed storage plug-ins in the vSphere Client
Used the esxcli command to view and manage storage
Used the esxcfg-mpath command to view and manage
storage
Used the vmkload_mod command to view VMkernel
modules
Resolved storage problems

Module Number 6-35
Lesson 3:
iSCSI and NAS Storage

Module Number 6-36
Lesson Objectives
Configure iSCSI Challenge Handshake Authentication
Protocol settings
Configure software iSCSI initiator multipathing
Describe iSCSI message digests
Troubleshoot iSCSI and NAS storage problems

Module Number 6-37
Advanced iSCSI Parameter Review

You might sometimes have to configure several iSCSI initiator
parameters in order to match the settings on a storage array.
ESX/ESXi 4.x has several iSCSI parameters that can be modified.
The parameters reviewed here include:
Bidirectional Challenge Handshake Authentication Protocol (CHAP)
Per-target CHAP
Software iSCSI initiator multipathing
Message digests
Do not make any changes to these iSCSI parameters unless you

have thorough information about the values to provide for the
parameters.

Module Number 6-38
CHAP Security Settings

Security setting
Description
Supported
Do not use CHAP
The host does not use CHAP

authentication. Select this option to disable
authentication if it is currently enabled.
HW/SW
Do not use CHAP

unless required by
target
The host prefers a non-CHAP connection,

but can use a CHAP connection if required
by the target.
SW
Use CHAP unless

prohibited by target
The host prefers CHAP, but can use nonCHAP connections if the target does not
support CHAP.
HW/SW
Use CHAP
The host requires successful CHAP

authentication. The connection fails if
CHAP negotiation fails.
SW

Module Number 6-39
Bidirectional CHAP
iSCSI Storage Adapter > Properties
Bidirectional CHAP
cannot be enabled until
Use CHAP is selected.
The only available

items are Do not use
CHAP and Use CHAP.
For bidirectional CHAP,

you should choose
different CHAP secrets.

Module Number 6-40
Per-Target CHAP
Per-target
CHAP is
configured
by selecting
a target on
the
Dynamic
Discovery
tab or the
Static
Discovery
tab.

Module Number 6-41
Software iSCSI Multipathing

If you have two or more physical NICs, you can create multiple paths for a
software iSCSI initiator by using the port-binding technique.
With port binding, you create a separate VMkernel port for each physical NIC with
a one-to-one mapping.
Port binding can be configured with a single virtual switch or a pair of virtual
switches.
vmhba32
vmk1
vmk3
vmk2
PortGroup A
vmnic1
vmhba32
vmnic2

PortGroup C
PortGroup B
vmnic3
Module Number 6-42
vmk4
PortGroup D
vmnic4
Configuring Software iSCSI Multipathing

Configuring software iSCSI multipathing is a two-step
process:
1.
Configure the VMkernel ports and connect them to the physical

NICs.
2.
Bind the VMkernel ports to the software iSCSI adapter by using

the esxcli command:
# esxcli swiscsi nic add -n <port_name> -d <vmhba>
Example:
# esxcli swiscsi nic add -n vmk1 -d vmhba32
# esxcli swiscsi nic add -n vmk2 -d vmhba32

Module Number 6-43
iSCSI Message Digests

ESX/ESXi provides two digest parameters that are configurable
on software iSCSI.
iSCSI Storage Adapter > Properties > Advanced
Advanced parameter
Description
Header Digest
Increases data integrity. When Header Digest is enabled,

the system performs a checksum over each header part of
each iSCSI Protocol Data Unit (PDU) and verifies with the
CRC32C algorithm.
Data Digest
Increases data integrity. When Data Digest is enabled,

the system performs a checksum over each PDUs data
part and verifies using the CRC32C algorithm.
Note: Systems that use Intel Nehalem processors off-load
the iSCSI digest calculations for software iSCSI, thus
reducing the impact on performance.

Module Number 6-44
Lab 23
In this lab, you will configure your ESX host to use
iSCSI storage with CHAP and digest settings enabled.
1. Configure your ESX host to use iSCSI storage with CHAP
authentication.
2. Configure your ESX host to use iSCSI storage with digests.
3. Examine iSCSI-related log files and diagnostic data.

Module Number 6-45
Lab 24
In this lab, you will configure your iSCSI storage to use
port binding and multipathing.
1. Configure your ESX/ESXi host to use iSCSI port binding.
2. Configure your ESX/ESXi host to use storage multipathing.

Module Number 6-46
Lab 25
In this lab, you will configure path masking.
Configure your ESX/ESXi host to use path masking.

Module Number 6-47
Lab 26
In this lab, you will diagnose and correct a variety of
FC-SCSI problems.
Troubleshoot FC-SCSI storage problems.

Module Number 6-48
Lab 27
In this lab, you will diagnose and correct a variety of
NAS and iSCSI problems.
Troubleshoot NAS and iSCSI problems.

Module Number 6-49
Lesson Summary
Configured iSCSI CHAP settings
Configured software iSCSI initiator multipathing
Described iSCSI message digests
Configured storage path masking
Resolved iSCSI and NAS storage problems

Module Number 6-50
Key Points
The Pluggable Storage Architecture uses software plug-ins to
manage and access storage.
The default VMware-supplied multipathing plug-in is the
native multipathing plug-in.
A Storage Array Type Plug-in is available to manage all
VMware arrays on the hardware compatibility list.
The three Path Selection Plug-ins support the Most Recently
Used, Fixed, and Round Robin load-distribution policies.
esxcli and esxcfg-mpath (vicfg-mpath) are primary
storage management commands.
ESX/ESXi 4.x supports several CHAP configurations.
ESX/ESXi 4.x supports software iSCSI initiator multipathing.

Module Number 6-51

Storage Troubleshooting: Module Number 6-1

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Storage Troubleshooting: Module Number 6-1

Încărcat de

Drepturi de autor:

Formate disponibile

Storage Troubleshooting

VMware vSphere: Troubleshooting Revision A

Module Number 6-1

You Are Here

VMware vSphere: Troubleshooting

ESXi Command-Line Troubleshooting

ESX/ESXi and vCenter Server Log Files

VMware Infrastructure Troubleshooting

VMware vSphere: Troubleshooting Revision A

Module Number 6-2

VMware vSphere: Troubleshooting Revision A

Module Number 6-3

Pluggable Storage Architecture Review

Viewing Storage Components

iSCSI and NAS Storage

VMware vSphere: Troubleshooting Revision A

Module Number 6-4

VMware vSphere: Troubleshooting Revision A

Module Number 6-5

VMware vSphere: Troubleshooting Revision A

Module Number 6-6

Pluggable Storage Architecture Overview

VMware vSphere: Troubleshooting Revision A

Module Number 6-7

Overview of MPP Tasks

An MPP claims a physical path and exports a logical device.

VMware vSphere: Troubleshooting Revision A

Module Number 6-8

Overview of VMware NMP Tasks

VMware vSphere: Troubleshooting Revision A

Module Number 6-9

VMware vSphere: Troubleshooting Revision A

Module Number 6-10

Overview of SATP Tasks

VMware vSphere: Troubleshooting Revision A

Module Number 6-11

Overview of PSP Tasks

VMware provides three PSPs: Fixed, MRU and Round Robin.

There is the capability to install third-party PSPs in order to

VMware vSphere: Troubleshooting Revision A

Module Number 6-12

Most Recently Used PSP

VMware vSphere: Troubleshooting Revision A

Module Number 6-13

VMware vSphere: Troubleshooting Revision A

Module Number 6-14

Round Robin PSP

VMware vSphere: Troubleshooting Revision A

Module Number 6-15

Storage I/O Example

VMware vSphere: Troubleshooting Revision A

Module Number 6-16

For unique performance

VMware vSphere: Troubleshooting Revision A

Module Number 6-17

For more complex

Asymmetric Logical Unit Access

VMware vSphere: Troubleshooting Revision A

Module Number 6-18

Target Port Group Support

VMware vSphere: Troubleshooting Revision A

Module Number 6-19

VMware vSphere: Troubleshooting Revision A