Vsphere4 Storage

TA1394 – vSphere 4 Storage
Troubleshooting and Log Analysis
Mostafa Khalil, VCP, VCDX, VMware Product

Support Engineering, VMware, Inc.
Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents

no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or features

discussed or presented have not been determined.
“These features are representative of feature areas under development. Feature commitments are
subject to change, and must not be included in contracts, purchase orders, or sales agreements of
any kind. Technical feasibility and market demand will affect final delivery.”
Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.
Agenda
About the Speaker

Collecting logs
vSphere 4.0 event handling (vProb)
Logs Locations
Interpreting log entries
Sample Logs
Q&A
About the Speaker
Mostafa Khalil, PSE

Staff Engineer, VMware Global Support Services (GSS)
The most senior member of the GSS team. Supported all VMware
Products from Workstation to the current version of vSphere 4.0
Worked for VMware for over 10 years.
Wide Range of expertise on all VMware products
Specialized in Storage
Collecting Logs – User Interface
Log in to VC using the VI

Client as an Administrator
Select: File  Export 
Export System Logs (or
Administration  Export
System Logs)
Select servers from which to
collect the logs including
VC Server
Select “Include information ..”
Checkbox
Specify location for storing
the files.
vSphere 4.0 Event Handling
vSphere provides a framework that defines the following use cases
Each of the use cases link to corresponding KBs that describe where the error occurred
(e.g. affected vmnic#, portgroup, vSwitch, storage path etc.) and provides
troubleshooting tips to fix the issue
Updates to these events will be provided as part of vSphere 4.0 updates
Events also logged to /var/log/vmkernel (or /var/log/messages on ESXi) with VOB prefix
Networking
vprob.net.connectivity.lost
vprob.net.redundancy.lost
vprob.net.redundancy.degraded
vprob.net.e1000.tso6.notsupported
Storage
vprob.storage.connectivity.lost
vprob.storage.redundancy.lost
vprob.storage.redundancy.degraded
Events Use Cases
VMFS specific:
vprob.vmfs.nfs.server.disconnect
vprob.vmfs.nfs.server.restored
vprob.vmfs.heartbeat.timedout
vprob.vmfs.heartbeat.recovered
vprob.vmfs.heartbeat.unrecoverable
vprob.vmfs.lock.corruptiondisk
vprob.vmfs.resource.corruptiondisk
vprob.vmfs.volume.locked
Migration Specific:
vprob.net.migrate.vmknic
Networking vProb
vprob.net.connectivity.lost
http://kb.vmware.com/kb/1009515
Connectivity to a physical network has been lost, all the affected portgroups
are part of the message (e.g. >Lost network connectivity on virtual switch
"system". Physical NIC vmnic1 is down. Affected port groups: "COS", "VM
Network".<)
Networking vProb
vprob.net.redundancy.lost
Only one physical NIC is currently connected, one more failure will result in a
loss of connectivity (e.g. >Lost uplink redundancy on virtual switch "system".
Physical NIC vmnic0 is down. Affected port groups: "COS", "VM Network".<)
Networking vProb
vprob.net.redundancy.degraded
One of the physical NICs in your NIC team has gone down, you still
have n-1 NICs available (e.g. Uplink redundancy degraded on virtual
switch "vSwitch0". Physical NIC vmnic1 is down. 2 uplinks still up.
Affected portgroups: "VM Network“.)
Networking vProb
vprob.net.e1000.tso6.notsupported
Guest e1000 driver is misbehaving and sending TSO IPv6
packets, which will be dropped. The vProb specifies the affected
VM, and the KB article discusses ways to fix this.
"Guest-initiated IPv6 TCP Segmentation Offload (TSO) packets
ignored. Manually disable TSO inside the guest operating system in
virtual machine "XYZ", or use a different virtual adapter."
Storage vProb
vprob.storage.connectivity.lost
The connectivity to a specific device has been lost (e.g. "Lost connectivity to
storage device naa.60a9800043346534645a433967325334. Path
vmhba35:C1:T0:L7 is down")
Storage vProb
vprob.storage.redundancy.lost
Only one path is remaining to a device and you no longer have any
redundancy (e.g. "Lost path redundancy to storage device
naa.60a9800043346534645a433967325334. Path vmhba35:C1:T0:L7 is
down.")
Storage vProb
vprob.storage.redundancy.degraded
http://http://kb.vmware.com/kb/1009555
One of your paths to a device has been lost but you still have n-1
paths remaining (e.g. "Path redundancy to storage device
naa.60a9800043346534645a433967325334 degraded. Path
vmhba35:C1:T0:L7 is down. 3 remaining active paths.")
VMFS vProb
vprob.vmfs.nfs.server.disconnect
vprob.vmfs.nfs.server.restored
VMFS vProb
vprob.vmfs.heartbeat.timedout
VMFS Volume Connectivity Degraded 496befed-1c79c817-6beb-
001ec9b60619 san-lun-100
VMFS vProb
vprob.vmfs.heartbeat.recovered
VMFS Volume Connectivity Restored 496befed-1c79c817-6beb-
001ec9b60619 (san-lun-100)
VMFS vProb
vprob.vmfs.heartbeat.unrecoverable
VMFS Volume Connectivity lost 496befed-1c79c817-6beb-
001ec9b60619 san-lun-100
VMFS vProb
vrpob.vmfs.lock.corruptiondisk
Volume 4976b16c-bd394790-6fd8-00215aaf0626 (san-lun-100) may
be damaged on disk. Corrupt lock detected at offset 0
vprob.vmfs.resource.corruptiondisk
Volume 4976b16c-bd394790-6fd8-00215aaf0626 (san-lun-100) may
be damaged on disk. Resource cluster metadata corruption detected
VMFS vProb
vprob.vmfs.volume.locked
Volume on device naa.60060160b3c018009bd1e02f725fdd11:1
locked, possibly because remote host 10.17.211.73 encountered an
error during a volume operation and couldn’t recover.
Migration Specific
vprob.net.migrate.vmknic
The ESX advanced config option /Migrate/Vmknic is set to an invalid
Vmknic: vmk0. /Migrate/Vmknic specifies a Vmknic that VMotion
binds to for improved performance. Please update the config option
with a valid Vmknic or, if you don't want VMotion to bind to a specific
Vmknic, remove the invalid Vmknic and leave the option blank.
vProb KB Mapping
VProb ID KB #
vprob.net.connectivity.lost 1009515
vprob.net.redundancy.lost 1009542
vprob.net.redundancy.degraded 1009546
vprob.net.e1000.tso6.notsupported 1009548
vprob.net.migrate.bindtovmk 1009550
vprob.storage.connectivity.lost 1009553
vprob.storage.redundancy.lost 1009554
vprob.storage.redundancy.degraded 1009555
vprob.vmfs.heartbeat.recovered 1009556
vprob.vmfs.heartbeat.timedout 1009557
vprob.vmfs.heartbeat.unrecoverable 1009559
vprob.vmfs.lock.corruptiondisk 1009565
vprob.vmfs.nfs.server.disconnect 1009566
vprob.vmfs.nfs.server.restored 1009568
vprob.vmfs.resource.corruptiondisk 1009569
vprob.vmfs.volume.locked 1009570
VMware ESX 3.5/4.0 common Logs
(collected via vm-support)
Most logs are located under /var/log directory.
Locations listed here are relative to that directory
vmkernel vmware/esxcfg-boot.log
messages vmware/esxcfg-firewall.log
dmesg vmware/vmware-cim.log
boot.log vmware/esxupdate.log
initrdlogs/* oldconf/esx.conf.*
vmksummary rpmpkgs
vmware/hostd.log vmkernel-version
vmware/vpx/vpxa.log
VMware ESXi 3.5/4.0 Common Log Files
config.log vmware/hostd.log
messages vmware/aam/*
slpd.log vmware/vpx/vpxa.log
wsmand.log
configRP.log
VMware ESX 4.0 Specific Log Files
esxcfg-* and esxcli commands outputs
Stored in /tmp or a directory referenced in
/tmp/working_dir.txt of vm-support dump
Usually in /usr/lib/vmware/hostd/docroot/downloads
esxcfg-dumppart esxcfg-swiscsi
esxcfg-info esxcfg-vmknic
esxcfg-info-xml esxcfg-vswitch
esxcfg-mpath-devices esxcli-corestorage-claimrules
esxcfg-mpath-paths esxcli-nmp-devices
esxcfg-nas esxcli-nmp-paths
esxcfg-nics esxcli-nmp-satp-rules
esxcfg-resgrp esxupdate-patch-history
esxcfg-route esxupdate-vib-view
esxcfg-route6
esx_install.log
•Located in /var/log directory
•sym-linked to /var/log/weasel.log
•All events from upgrade or fresh install
ESX 4.0 COS Location
COS VMDK
•COS VMDK is located at

/vmfs/volumes/<COS-Home>/esxconsole-<uuid>
Example:
/vmfs/volumes/storage1/esxconsole-4a0a1e4f-f9fa-ef4e-70fa-001ec9b60619/esxconsole.vmdk
•Defined in /etc/vmware/esx.conf  /boot/cosvmdk
boot-log directory
• Located in /var/log directory
• Replaced /var/log/initrdlogs (ESX 3.x)
• sym-linked to /vmfs/volume/<COS-home>/esxconsole-
<UUID>/logs
• contains
• sysboot-dmesg-boot.log
• sysboot-dmesg-late.log
• sysboot.log
• sysboot-vmkernel-boot.log
• sysboot-vmkernel-late.log
Events generated from vmkernel boot image
Sample sysboot.log
sysboot: Loading VMkernel Module 'nmp'

Module nmp loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_local'
Module vmw_satp_local loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_default_aa'
Module vmw_satp_default_aa loaded successfully
sysboot: Loading VMkernel Module 'vmw_psp_fixed'
Module vmw_psp_fixed loaded successfully
sysboot: Executing 'esxcli nmp boot restore'
sysboot: Loading VMkernel Module 'libata'
Module libata loaded successfully
sysboot: Loading VMkernel Module 'usb-storage'
Module usb-storage loaded successfully
sysboot: vmfs ...
Sample sysboot.log
sysboot: Executing 'mkdir /vmfs'

sysboot: Executing 'mount -t vmfs /vmfs /vmfs'
sysboot: advanced-config ...
sysboot: Executing 'esxcfg-init -A'
sysboot: Executing 'esxcfg-init -V'
Restoring vmkernel uuid
sysboot: psa-mask-path ...
sysboot: Loading VMkernel Module 'mask_path_plugin'
Module mask_path_plugin loaded successfully
sysboot: psa-nmp ...
sysboot: Loading VMkernel Module 'vmw_satp_alua'
Module vmw_satp_alua loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_cx'
Module vmw_satp_cx loaded successfully
Sample sysboot.log
sysboot: Loading VMkernel Module 'vmw_satp_default_ap'

Module vmw_satp_default_ap loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_eva'
Module vmw_satp_eva loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_lsi'
Module vmw_satp_lsi loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_symm'
Module vmw_satp_symm loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_inv'
Module vmw_satp_inv loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_eql'
Module vmw_satp_eql loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_msa'
Module vmw_satp_msa loaded successfully
Sample sysboot.log
sysboot: Loading VMkernel Module 'vmw_satp_svc'

Module vmw_satp_svc loaded successfully
sysboot: Loading VMkernel Module 'vmw_satp_alua_cx'
Module vmw_satp_alua_cx loaded successfully
sysboot: Loading VMkernel Module 'vmw_psp_rr'
Module vmw_psp_rr loaded successfully
sysboot: Loading VMkernel Module 'vmw_psp_mru'
Module vmw_psp_mru loaded successfully
sysboot: psa-nmp-pre-claim-config ...
sysboot: Loading NMP SATP rules
sysboot: Executing 'esxcli nmp boot restore'
sysboot: psa-load-rules ...
sysboot: Executing 'esxcli corestorage claimrule load'
sysboot: psa-nmp-post-claim-config ...
vmkernel Log
Located in /var/log directory

Contains all events generated by vmkernel
vmkwarning log is a subset of the vmkernel log and
contains only the warning events
Rotated with a numeric extension. The current log
without extension and the next newest one with “.1”
extension
All events since last vmkernel load are also in
memory in /proc/vmware/log
VMware ESXi 4.0 Specific Log Files
/var/log
• messages
• Equivalent to /var/log/vmkernel on ESX 4.0
• sysboot.log
• Same as ESX 4.0
• The rest of boot logs do not apply to ESXi 4.0
/var/log/vmware
• Same as VMware ESX 4.0
Messages Log Files (VMware ESXi 4.0)
Located in /var/log directory

Contains all events generated by vmkernel
Rotated and compressed with a numeric extension.
The current log without extension and the next
newest one with “.0.gz” extension
SCSI Errors Verbose logging
VMware ESX and VMware ESXi 4.0 are enabled by default for
reporting the SCSI Error Strings “Scsi.LogCmdErros” (set to “1”)
To increase the verbosity of MultiPath events logging, ESX and
ESXi 4.0 advanced vmkernel option “Scsi.LogMPCmdErrors” is
enabled by default (set to “1”)
It is recommended that you redirect the logs to a vMA appliance
or a syslog host to collect historic logs and (for ESXi 4.0) to
preserve them between reboots
Redirecting VMware ESX/ESXi Logs to vMA
vMA (vSphere Management Assistant) is a Virtual

Appliance for managing multiple VMware ESX hosts
from a single Service Console
To enable log forwarding to vMA use:
sudo vifp addserver <server-name or ipAddr>
vilogger enable --server <VMware ESX name>
Redirecting VMware ESX/ESXi 4.0 Server Logs to vMA
Location of forwarded logs on vMA

/var/log/vmware/<ESX-Hostname>
Notice that the log files are all suffixed with “.log”
extension
Redirecting VMware ESXi 4.0 Logs
to a syslog server
Log on to VMware ESXi directly using vSphere Client
Configuration  Advanced Options  Syslog  Remote
Enter the syslog server‟s host name
vmkernel Log - Components
Jun 19 09:12:54 giza vmkernel: 14:22:31:50.009 cpu3:1033)scsi-qla0: Scheduling SCAN for new luns....
System Message Device

Date/Time source
CPU:World ID
Host uptime Message
name
Rescan Event - LUN Discovery – ESX 4.0
ScsiScan: 839: Path 'vmhba2:C0:T0:L21': Vendor: 'EMC ' Model: 'SYMMETRIX '
Rev: '5772'
ScsiScan: 842: Path 'vmhba2:C0:T0:L21': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)
ScsiPath: 3685: Plugin 'NMP' claimed path 'vmhba2:C0:T0:L21'
LUN 21 on target 0 on vmhba2 channel 0 discovered.

Storage Vendor’s ID, Array Model and Microcode Rev.
Reported ANSI version is 4 = SCSI-3
TPGS (Target Port Group Support) is off (0)
If the Target supports ALUA, this would be on (1, 2 or 3)
NMP Plugin claimed the path. If 3rd party MPIO is installed, it
would claim paths on its configured arrays instead of NMP
Rescan Event – LUN Discovery - ESX 4.0
VMWARE SCSI Id: Id for vmhba2:C0:T0:L21

0x01 0x90 0x10 0x05 0x60 0x53 0x30 0x30 0x32 0x32 0x46 0x53 0x59
0x4d 0x4d 0x45 0x54
ScsiDevice: 1756: Successfully registered device
"naa.60060480000190100560533030323246" from plugin "NMP" of type 0
Inquiry Commend returns Device ID

A logical device representing the discovered LUN is registered.
Notice the LUN ID is part of the logical device name
Native Multipathing Plugin (NMP) is use here
Device type 0 means “Block Device”
Rescan Event – LUN Discovery - ESX 4.0
Device Type Codes
Code Description
00h Direct access block device
01h Sequential-access device (e.g. Tape Drive)
02h Printer device
03h Processor device
04h Write-once device
05h CD/DVD device
08h Tape Library
LUN Discovery - ESX 4.0 - More Examples
Path 'vmhba2:C0:T1:L0': Vendor: 'DGC ' Model: 'RAID 5 ' Rev: '0428„
Path 'vmhba2:C0:T1:L0': Type: 0x0, ANSI rev: 4, TPGS: 0 (none)
Path 'vmhba2:C0:T2:L0': Vendor: 'IBM' Model: 'ULT3580-TD4' Rev: '89B2„
Path 'vmhba2:C0:T2:L1': Vendor: 'IBM' Model: '3573-TL ' Rev: '7.30„
LUN on an EMC Clariion with ALUA not enabled

Tape Drive “IBM: ULT3580-TD4”
Notice type 0x1
Fibre Attached Tape Library “IBM:3573-TL”
Notice type 0x8
vmkernel log entries - Understanding SCSI Error Strings
Format:
Device/Host Sense_buffer[2] 12 13
Abbrev: D/H S ASC ASCQ
Extended:
“Device Status”/”Host Status” “Sense Key” “Additional Sense
Code” “Additional Sense Code Qualifier”
Example:
2/0 0x6 0x29 0x0
vmkernel log entries - Understanding SCSI Error Strings
New format (Extended):

Host Status: <hex> Device Status: <hex> Plugin Status: <hex>
Example:
H:0x1 D:0x0 P:0x0
Host No_Connect, Device OK, Plugin OK
Possible sense data: 0x0 0x0 0x0
No Sense Data
Understanding SCSI Error Strings – Device Status
Device Status: (Displayed in decimal values)
Code Meaning
0 No errors
2 Check Condition
8 Device Busy
24 Reservation Conflict
Understanding SCSI Error Strings – Host Status
Host Status (displayed in decimal values)
Code Meaning
0 Host_OK
1 Host No_Connect
2 Host_Bus_Busy
3 Host_Timeout
4 Host_Bad_Target
5 Host_Abort
Understanding SCSI Error Strings –
Host Status (cont.)
Host Status (displayed in decimal or Hex values) listed here
in decimal
Code Meaning
6 Host_Parity
7 Host_Error
8 Host_Reset
9 Host_Bad_INTR
10 Host_PassThrough
11 Host_Soft_Error
Understanding SCSI Error Strings – Sense Key
Sense Key (displayed in hex)
Code Meaning
0x0 No Sense Information
0x1 Last command completed but used error correction
0x2 Unit Not Ready
0x3 Medium Error
0x4 Hardware error
0x5 ILLEGAL_REQUEST (Passive SP)
0x6 LUN Reset
Understanding SCSI Error Strings –
Sense Key – (cont.)
Sense Key (displayed in hex)
Code Meaning
0x7 Data_Protect – Access to data is blocked
0x8 Blank_Check – Reached an unexpected region
0xa Copy_Aborted
0xb Aborted_Command – Target aborted command
0xc Comparison for SEARCH DATA unsuccessful
0xd Volume_Overflow – Medium is full
0xe Source and Data on Medium do not agree
Understanding SCSI Error Strings – ASC/ASCQ
ASC and ASCQ are always in pairs (in hex) (ASQ usually 0)
Code Meaning
0x4 Unit Not Ready
0x3 Unit Not Ready – Manual Intervention Required
0x2 Unit Not Ready - Initializing Command Required
0x29 Device Power on or SCSI Reset
0x8b ASC_QUIESCENCE_HAS_BEEN_ACHIEVED (IBM FAStT)
0x94 ASC_Invalid_Req_due_To_Current_LU_Ownership (IBM FAStT)
0x01 ASCQ_Invalid_Req_due_To_Current_LU_Ownership (IBM FAStT)
0x02 ASCQ_QUIESCENCE_HAS_BEEN_ACHIEVED (IBM FAStT)
Understanding SCSI Error Strings - Examples (4.0)
Plugin Code Translation
Code Translation Meaning

0x0 No Error OK
0x1 VMK_SCSI_PLUGIN_TRANSIENT transient plugin failure, retry.
(e.g. out of memory)
0x2 VMK_SCSI_PLUGIN_SNAPSHOT Device is a deactivated
Snapshot
0x3 VMK_SCSI_PLUGIN_RESERVATION_LOST Host lost a reservation of a
previously reserved LUN
nmp_CompleteCommandForPath:
Command 0x2a (0x410004237c00) to NMP device
"naa.60060160432017005c97aea1b32fdc11" failed on physical
path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0 Possible sense
data: 0x0 0x0 0x0.
H:0x1 D:0x0 P:0x0 (Host No_Connect/Device OK/Plugin OK)

Possible sense data: 0x0 0x0 0x0. (No sense data)
Physical Path: vmhba0:C0:T1:L1
NMP: nmp_CompleteCommandForPath: Command 0x25 (0x410004134a80)
to NMP device "naa.60060160b4111600826120bae2e3dd11" failed on
physical path "vmhba0:C0:T1:L0" H:0x0 D:0x2 P:0x0 Valid sense
data: 0x6 0x29 0x0.
NMP: nmp_CompleteCommandForPath:
Command 0x25 to device "naa.60060160b4111600826120bae2e3dd11"
failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
H:0x0 D:0x2 P:0x0 (Host OK/Device Check Condition/Plugin OK)

Valid sense data: 0x6 0x29 0x0. (LUN Reset)
ScsiDeviceIO: 747: Command 0x16 to device
"naa.6006016004351f00bc913175af58de11" failed H:0x0 D:0x8 P:0x0
Possible sense data: 0x0 0x0 0x0.
H:0x0 D:0x8 P:0x0 (Host OK/Device Busy/Plugin OK)

Possible sense data: 0x0 0x0 0x0. (No sense date)
Device name: naa.6006016004351f00bc913175af58de11
NMP: nmp_CompleteCommandForPath: Command 0x25 (0x4100051f7bc0) to
NMP device " naa.60060160e6341f00988aef0a4f4fde11" failed on
physical path "vmhba0:C0:T1:L2" H:0x0 D:0x2 P :0x0 Valid sense
data: 0x6 0x3f 0xe.
H:0x0 D:0x2 P:0x0 (Host OK/Device Check Condition/Plugin OK)

Valid sense data: 0x6 0x3f 0xe. (LUN Removed) (Clariion)
Device name: naa.60060160e6341f00988aef0a4f4fde11
NMP: nmp_DeviceAttemptFailover: Retry world failover device
"naa.600601600ade2200aa3f43e90e7dde11" - failed to issue command
due to Not found (APD), try again...
I/O Failover for device naa.600601600ade2200aa3f43e90e7dde11

failed
Reason: Device not found
Cause: All Paths are Dead (APD)
Identifying path state
From ESX console, vMA or vCLI, run:

# esxcli nmp path list
NOTE: vm-support collects esxcli-nmp-paths.<pid>.txt which is the same as
above
For a specific device:
# esxcli nmp path list -d <device-name>
# esxcli nmp path list -d naa.600601608cc01c002d45db8b0fb8dd11
fc.2001001b32351707:2101001b32351707-fc.50060160c1e04536:5006016841e04536-
naa.600601608cc01c002d45db8b0fb8dd11
Runtime Name: vmhba2:C0:T1:L3
Device: naa.600601608cc01c002d45db8b0fb8dd11
Device Display Name: DGC Fibre Channel Disk (naa.600601608cc01c002d45db8b0fb8dd11)
Group State: standby
Storage Array Type Path Config:
Path Selection Policy Path Config: {non-current path}
Identifying Path State - cont.
fc.2001001b32351707:2101001b32351707-fc.50060160c1e04536:5006016141e04536-
naa.600601608cc01c002d45db8b0fb8dd11
Group State: active
Path Selection Policy Path Config: {current path}
fc.2000001b32151707:2100001b32151707-fc.50060160c1e04536:5006016041e04536-
naa.600601608cc01c002d45db8b0fb8dd11
Group State: active
Path Selection Policy Path Config: {non-current path}
Any Questions?
Extra Slides
Appendix -ESX 4.0 Failover Logs With
Additional Logging
14:13:47:59.878 cpu7:4109)NMP: nmp_HasMoreWorkingPaths: STANDBY path(s) only to device
"naa.60060160432017005c97aea1b32fdc11".
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_PspSelectPathForIO: Plugin VMW_PSP_MRU selectPath() returned
path "vmhba0:C0:T1:L1" for device "naa.60060160432017005c97aea1b32fdc11" which is in state standby
instead of ON. Status is Bad parameter
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_SelectPathAndIssueCommand: PSP select path "vmhba0:C0:T1:L1"
in a bad state on device "naa.60060160432017005c97aea1b32fdc11".
14:13:47:59.887 cpu7:4374)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004237c00) to NMP device
"naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_DeviceRetryCommand: Device
"naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O
blocked...
14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_DeviceStartLoop: NMP Device
"naa.60060160432017005c97aea1b32fdc11" is blocked. Not starting I/O from device.
14:13:48:00.069 cpu7:4109)NMP: nmp_DeviceUpdatePathStates: Activated path "vmhba0:C0:T1:L1" for NMP
device "naa.60060160432017005c97aea1b32fdc11".
14:13:48:00.888 cpu1:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device
"naa.60060160432017005c97aea1b32fdc11" - issuing command 0x410004237c00
14:13:48:00.888 cpu2:4373)WARNING: NMP: nmp_CompleteRetryForPath: Retry command 0x2a (0x410004237c00) to
NMP device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x0
D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
14:13:48:00.888 cpu2:4373)WARNING: NMP: nmp_CompleteRetryForPath: Retry world restored device
"naa.60060160432017005c97aea1b32fdc11" - no more commands to retry
14:13:48:00.888 cpu2:4373)ScsiDeviceIO: 746: Command 0x2a to device
"naa.60060160432017005c97aea1b32fdc11" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.
Last slide magnified
NMP: nmp_HasMoreWorkingPaths: STANDBY path(s) only

to device "naa.60060160432017005c97aea1b32fdc11“
NMP: nmp_PspSelectPathForIO: Plugin VMW_PSP_MRU
selectPath() returned path "vmhba0:C0:T1:L1" for
device "naa.60060160432017005c97aea1b32fdc11"
which is in state standby instead of ON. Status
is Bad parameter
cpu7:4374)NMP: nmp_CompleteCommandForPath: Command
0x2a (0x410004237c00) to NMP device
"naa.60060160432017005c97aea1b32fdc11" failed on
physical path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0
NMP: nmp_DeviceRetryCommand: Device
"naa.60060160432017005c97aea1b32fdc11": awaiting
fast path state update for failover with I/O
blocked...
Appendix - ESX 4.0 Failover Logs – FC
cable unplugged
14:13:32:16.716 cpu3:4099)<6>qla2xxx 003:00.1: LOOP DOWN detected mbx1=2h mbx2=5h mbx3=0h.
14:13:32:24.425 cpu6:4195)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004286980) to NMP
device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba1:C0:T0:L1" H:0x5 D:0x0
P:0x0 Possible sense data: 0x2 0x3a 0x0.
14:13:32:24.425 cpu6:4195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device
"naa.60060160432017005c97aea1b32fdc11" state in doubt; requesting fast path state update...
14:13:32:24.425 cpu6:4195)ScsiDeviceIO: 746: Command 0x2a to device
"naa.60060160432017005c97aea1b32fdc11" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.
14:13:32:26.718 cpu4:4198)<3> rport-4:0-0: blocked FC remote port time out: saving binding
14:13:32:26.718 cpu4:4198)<3> rport-4:0-1: blocked FC remote port time out: saving binding
14:13:32:26.718 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004286980) to NMP
P:0x0 Possible sense data: 0x0 0x0 0x0.
14:13:32:26.718 cpu5:4101)WARNING: NMP: nmp_DeviceRetryCommand: Device
"naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O
blocked...
14:13:32:26.718 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100042423c0) to NMP
P:0x0 Possible sense data: 0x0 0x0 0x0.
14:13:32:26.718 cpu3:4281)WARNING: VMW_SATP_CX: satp_cx_otherSPIsHung: Path "vmhba1:C0:T1:L1" MODE
SENSE PEER SP command failed 0/1 0x0 0x0 0x0.
14:13:32:26.719 cpu1:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device
"naa.60060160432017005c97aea1b32fdc11" - issuing command 0x410004286980
14:13:32:26.752 cpu2:4237)NMP: nmp_CompleteRetryForPath: Retry world recovered device
"naa.60060160432017005c97aea1b32fdc11"
Messages Log (VMware ESX 4.0)
Jul 24 19:13:33 giza sshd[18915]: Connection from 10.16.112.24 port 1396
Jul 24 19:13:36 giza sshd[18915]: Accepted password for root from 10.16.112.24 port
1396 ssh2
Jul 24 19:13:36 giza sshd(pam_unix)[18915]: session opened for user root by (uid=0)
Jul 29 01:01:03 giza iscsid[32725]: cannot make connection to 10.16.95.161:3260: No
route to host
 Console events
 Logon events
hostd.log
Located in /var/log/vmware
Sym-linked to the current rotated hostd log file
Hostd events
VI Client communications when directly connected to
VMware ESX
Events done on behalf of
VPXA
System Services
Firewall System
HA services
VMware Converter
vm-support collects hostd-support*.tgz files
hostd-support dump
vm-support collects hostd-support*.tgz files located
in /var/log/vmware directory
Generated by /usr/sbin/vmware-hostd-support script
Rotated up-to 4 copies (blank, -1, -2 and -3 suffix)
Extracted to hostd-support-<pid> directory which includes
Config
Content of /etc/vmware/hostd directory
Logs
/var/log/vmware/hostd-*.log file with current hostd’s PID
/var/log/vmware/hostd-trace.log file if exists
Runtime
Content of /var/lib/vmware/hostd/journal directory
Content of /var/lib/vmware/hostd/stats directory
hostd.log
Log for VMware ESX, pid=3049, version=4.0.0, build=build-140815, option=Release,
section=10
Compression of /var/log/vmware//hostd-3.log to /var/log/vmware//hostd-3.log.gz took 167 ms
[2009-04-27 08:39:13.157 F624BB90 info 'ha-eventmgr'] Event 3558 : Lost access to volume
492c94b9-7b4f6ee8-bb0b-001a645dba02 (PSE-i144-100gb) due to connectivity issues.
Recovery attempt is in progress and outcome will be reported shortly.
[2009-04-27 08:36:14.927 F56EBB90 info 'ha-eventmgr'] Event 3549 : Successfully restored
access to volume 492c94b9-7b4f6ee8-bb0b-001a645dba02 (PSE-i144-100gb) following
connectivity issues.
[2009-04-27 08:36:15.307 F6188B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
[2009-04-27 08:36:15.307 F6188B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
[2009-04-27 08:36:34.081 F61C9B90 verbose 'PropertyProvider'] RecordOp 1:
latestPage[3540], session[524622d3-a4f2-0704-e090-fedc59dbc2f5]52553360-a9c2-63b0-7138-
0486ae7d4b0f
ESX Version reported

Log compression listed
VMFS volume access events logged and reported
vSphere Distributed Switch discovery events
oldconf Files
Backup copies of /etc/vmware/esx.conf file

Located in /var/log/oldconf directory
Created prior to updating the existing file
Only when changes done via VMware vCenter, vSphere Client
or esxcfg-* scripts
Date and time of backup used as the extension of the file name
esx.conf.2009-03-11_16:39:06 esx.conf.2009-04-14_10:01:57

Vsphere4 Storage

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Vsphere4 Storage

Încărcat de

Drepturi de autor:

Formate disponibile

TA1394 – vSphere 4 Storage

Troubleshooting and Log Analysis

Mostafa Khalil, VCP, VCDX, VMware Product

This session/overview of the new technology represents

Features are subject to change, and must not be included in

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

About the Speaker

Mostafa Khalil, PSE

Log in to VC using the VI

•COS VMDK is located at

•Defined in /etc/vmware/esx.conf  /boot/cosvmdk

sysboot: Loading VMkernel Module 'nmp'

sysboot: Executing 'mkdir /vmfs'

sysboot: Loading VMkernel Module 'vmw_satp_default_ap'

sysboot: Loading VMkernel Module 'vmw_satp_svc'

Located in /var/log directory

Located in /var/log directory

vMA (vSphere Management Assistant) is a Virtual

Location of forwarded logs on vMA

System Message Device

LUN 21 on target 0 on vmhba2 channel 0 discovered.

VMWARE SCSI Id: Id for vmhba2:C0:T0:L21

Inquiry Commend returns Device ID

LUN on an EMC Clariion with ALUA not enabled

New format (Extended):

Code Translation Meaning

H:0x1 D:0x0 P:0x0 (Host No_Connect/Device OK/Plugin OK)

H:0x0 D:0x2 P:0x0 (Host OK/Device Check Condition/Plugin OK)

H:0x0 D:0x8 P:0x0 (Host OK/Device Busy/Plugin OK)

H:0x0 D:0x2 P:0x0 (Host OK/Device Check Condition/Plugin OK)

I/O Failover for device naa.600601600ade2200aa3f43e90e7dde11

From ESX console, vMA or vCLI, run:

NMP: nmp_HasMoreWorkingPaths: STANDBY path(s) only

ESX Version reported

Backup copies of /etc/vmware/esx.conf file

S-ar putea să vă placă și