vsphere4 storage

TA1394 – vSphere 4 Storage

Troubleshooting and Log Analysis

Mostafa Khalil, VCP, VCDX, VMware Product

Support Engineering, VMware, Inc.

Copyright © 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents.

Disclaimer

This session may contain product features that are currently under development.

This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features discussed or presented have not been determined.

“These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”


Agenda

About the Speaker

Collecting logs

vSphere 4.0 event handling (vProb)

Logs Locations

Interpreting log entries

Sample Logs

Q&A


About the Speaker

Mostafa Khalil, PSE

Staff Engineer, VMware Global Support Services (GSS)

The most senior member of the GSS team. Supported all VMware

Products from Workstation to the current version of vSphere 4.0

Worked for VMware for over 10 years.

Wide Range of expertise on all VMware products

Specialized in Storage


Collecting Logs – User Interface

Log in to VC using the VI

Client as an Administrator

Select: File Export

Export System Logs (or

Administration Export

System Logs)

Select servers from which to

collect the logs including

VC Server

Select “Include information ..”

Checkbox

Specify location for storing

the files.


vSphere 4.0 Event Handling

vSphere provides a framework that defines the following use cases

Each of the use cases link to corresponding KBs that describe where the error occurred (e.g. affected vmnic#, portgroup, vSwitch, storage path etc.) and provides troubleshooting tips to fix the issue

Updates to these events will be provided as part of vSphere 4.0 updates

Events also logged to /var/log/vmkernel (or /var/log/messages on ESXi) with VOB prefix

Networking

vprob.net.connectivity.lost

vprob.net.redundancy.lost

vprob.net.redundancy.degraded

vprob.net.e1000.tso6.notsupported

Storage

vprob.storage.connectivity.lost

vprob.storage.redundancy.lost

vprob.storage.redundancy.degraded


Events Use Cases

VMFS specific:

vprob.vmfs.nfs.server.disconnect

vprob.vmfs.nfs.server.restored

vprob.vmfs.heartbeat.timedout

vprob.vmfs.heartbeat.recovered

vprob.vmfs.heartbeat.unrecoverable

vprob.vmfs.lock.corruptiondisk

vprob.vmfs.resource.corruptiondisk

vprob.vmfs.volume.locked

Migration Specific:

vprob.net.migrate.vmknic


Networking vProb

vprob.net.connectivity.lost http://kb.vmware.com/kb/1009515

Connectivity to a physical network has been lost, all the affected portgroups

are part of the message (e.g. >Lost network connectivity on virtual switch

"system". Physical NIC vmnic1 is down. Affected port groups: "COS", "VM

Network".<)

http://kb.vmware.com/kb/1009515


Networking vProb

vprob.net.redundancy.lost http://kb.vmware.com/kb/1009542

Only one physical NIC is currently connected, one more failure will result in a

loss of connectivity (e.g. >Lost uplink redundancy on virtual switch "system".

Physical NIC vmnic0 is down. Affected port groups: "COS", "VM Network".<)



Networking vProb

vprob.net.redundancy.degraded


One of the physical NICs in your NIC team has gone down, you still

have n-1 NICs available (e.g. Uplink redundancy degraded on virtual

switch "vSwitch0". Physical NIC vmnic1 is down. 2 uplinks still up.

Affected portgroups: "VM Network“.)



Networking vProb

vprob.net.e1000.tso6.notsupported


Guest e1000 driver is misbehaving and sending TSO IPv6

packets, which will be dropped. The vProb specifies the affected

VM, and the KB article discusses ways to fix this.

"Guest-initiated IPv6 TCP Segmentation Offload (TSO) packets

ignored. Manually disable TSO inside the guest operating system in

virtual machine "XYZ", or use a different virtual adapter."



Storage vProb

vprob.storage.connectivity.lost http://kb.vmware.com/kb/1009553

The connectivity to a specific device has been lost (e.g. "Lost connectivity to

storage device naa.60a9800043346534645a433967325334. Path

vmhba35:C1:T0:L7 is down")



Storage vProb

vprob.storage.redundancy.lost http://kb.vmware.com/kb/1009554

Only one path is remaining to a device and you no longer have any

redundancy (e.g. "Lost path redundancy to storage device

naa.60a9800043346534645a433967325334. Path vmhba35:C1:T0:L7 is

down.")



Storage vProb

vprob.storage.redundancy.degraded

http://http://kb.vmware.com/kb/1009555

One of your paths to a device has been lost but you still have n-1

paths remaining (e.g. "Path redundancy to storage device

naa.60a9800043346534645a433967325334 degraded. Path

vmhba35:C1:T0:L7 is down. 3 remaining active paths.")

http://http/kb.vmware.com/kb/1009555


VMFS vProb

vprob.vmfs.nfs.server.disconnect


vprob.vmfs.nfs.server.restored





VMFS vProb

vprob.vmfs.heartbeat.timedout


VMFS Volume Connectivity Degraded 496befed-1c79c817-6beb-

001ec9b60619 san-lun-100



VMFS vProb

vprob.vmfs.heartbeat.recovered


VMFS Volume Connectivity Restored 496befed-1c79c817-6beb-

001ec9b60619 (san-lun-100)



VMFS vProb

vprob.vmfs.heartbeat.unrecoverable


VMFS Volume Connectivity lost 496befed-1c79c817-6beb-

001ec9b60619 san-lun-100



VMFS vProb

vrpob.vmfs.lock.corruptiondisk


Volume 4976b16c-bd394790-6fd8-00215aaf0626 (san-lun-100) may

be damaged on disk. Corrupt lock detected at offset 0

vprob.vmfs.resource.corruptiondisk


Volume 4976b16c-bd394790-6fd8-00215aaf0626 (san-lun-100) may

be damaged on disk. Resource cluster metadata corruption detected




VMFS vProb

vprob.vmfs.volume.locked


Volume on device naa.60060160b3c018009bd1e02f725fdd11:1

locked, possibly because remote host 10.17.211.73 encountered an

error during a volume operation and couldn’t recover.



Migration Specific

vprob.net.migrate.vmknic


The ESX advanced config option /Migrate/Vmknic is set to an invalid

Vmknic: vmk0. /Migrate/Vmknic specifies a Vmknic that VMotion

binds to for improved performance. Please update the config option

with a valid Vmknic or, if you don't want VMotion to bind to a specific

Vmknic, remove the invalid Vmknic and leave the option blank.



vProb KB Mapping

VProb ID KB #

vprob.net.connectivity.lost 1009515

vprob.net.redundancy.lost 1009542

vprob.net.redundancy.degraded 1009546

vprob.net.e1000.tso6.notsupported 1009548

vprob.net.migrate.bindtovmk 1009550

vprob.storage.connectivity.lost 1009553

vprob.storage.redundancy.lost 1009554

vprob.storage.redundancy.degraded 1009555

vprob.vmfs.heartbeat.recovered 1009556

vprob.vmfs.heartbeat.timedout 1009557

vprob.vmfs.heartbeat.unrecoverable 1009559

vprob.vmfs.lock.corruptiondisk 1009565

vprob.vmfs.nfs.server.disconnect 1009566

vprob.vmfs.nfs.server.restored 1009568

vprob.vmfs.resource.corruptiondisk 1009569

vprob.vmfs.volume.locked 1009570


VMware ESX 3.5/4.0 common Logs(collected via vm-support)

Most logs are located under /var/log directory.

Locations listed here are relative to that directory

vmkernel

messages

dmesg

boot.log

initrdlogs/*

vmksummary

vmware/hostd.log

vmware/vpx/vpxa.log

vmware/esxcfg-boot.log

vmware/esxcfg-firewall.log

vmware/vmware-cim.log

vmware/esxupdate.log

oldconf/esx.conf.*

rpmpkgs

vmkernel-version


VMware ESXi 3.5/4.0 Common Log Files

config.log

messages

slpd.log

wsmand.log

configRP.log

vmware/hostd.log

vmware/aam/*

vmware/vpx/vpxa.log


VMware ESX 4.0 Specific Log Files

esxcfg-dumppart

esxcfg-info

esxcfg-info-xml

esxcfg-mpath-devices

esxcfg-mpath-paths

esxcfg-nas

esxcfg-nics

esxcfg-resgrp

esxcfg-route

esxcfg-route6

esxcfg-swiscsi

esxcfg-vmknic

esxcfg-vswitch

esxcli-corestorage-claimrules

esxcli-nmp-devices

esxcli-nmp-paths

esxcli-nmp-satp-rules

esxupdate-patch-history

esxupdate-vib-view

esxcfg-* and esxcli commands outputsStored in /tmp or a directory referenced in

/tmp/working_dir.txt of vm-support dump

Usually in /usr/lib/vmware/hostd/docroot/downloads



esx_install.log

•Located in /var/log directory

•sym-linked to /var/log/weasel.log

•All events from upgrade or fresh install


ESX 4.0 COS Location

COS VMDK

•COS VMDK is located at/vmfs/volumes/<COS-Home>/esxconsole-<uuid>

Example:/vmfs/volumes/storage1/esxconsole-4a0a1e4f-f9fa-ef4e-70fa-001ec9b60619/esxconsole.vmdk

•Defined in /etc/vmware/esx.conf /boot/cosvmdk



boot-log directory

• Located in /var/log directory

• Replaced /var/log/initrdlogs (ESX 3.x)

• sym-linked to /vmfs/volume/<COS-home>/esxconsole-<UUID>/logs

• contains

• sysboot-dmesg-boot.log

• sysboot-dmesg-late.log

• sysboot.log

• sysboot-vmkernel-boot.log

• sysboot-vmkernel-late.log

Events generated from vmkernel boot image



Sample sysboot.log

sysboot: Loading VMkernel Module 'nmp'

Module nmp loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_local'

Module vmw_satp_local loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_default_aa'

Module vmw_satp_default_aa loaded successfully

sysboot: Loading VMkernel Module 'vmw_psp_fixed'

Module vmw_psp_fixed loaded successfully

sysboot: Executing 'esxcli nmp boot restore'

sysboot: Loading VMkernel Module 'libata'

Module libata loaded successfully

sysboot: Loading VMkernel Module 'usb-storage'

Module usb-storage loaded successfully

sysboot: vmfs ...



Sample sysboot.log

sysboot: Executing 'mkdir /vmfs'

sysboot: Executing 'mount -t vmfs /vmfs /vmfs'

sysboot: advanced-config ...

sysboot: Executing 'esxcfg-init -A'

sysboot: Executing 'esxcfg-init -V'

Restoring vmkernel uuid

sysboot: psa-mask-path ...

sysboot: Loading VMkernel Module 'mask_path_plugin'

Module mask_path_plugin loaded successfully

sysboot: psa-nmp ...

sysboot: Loading VMkernel Module 'vmw_satp_alua'

Module vmw_satp_alua loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_cx'

Module vmw_satp_cx loaded successfully



Sample sysboot.log

sysboot: Loading VMkernel Module 'vmw_satp_default_ap'

Module vmw_satp_default_ap loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_eva'

Module vmw_satp_eva loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_lsi'

Module vmw_satp_lsi loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_symm'

Module vmw_satp_symm loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_inv'

Module vmw_satp_inv loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_eql'

Module vmw_satp_eql loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_msa'

Module vmw_satp_msa loaded successfully



Sample sysboot.log

sysboot: Loading VMkernel Module 'vmw_satp_svc'

Module vmw_satp_svc loaded successfully

sysboot: Loading VMkernel Module 'vmw_satp_alua_cx'

Module vmw_satp_alua_cx loaded successfully

sysboot: Loading VMkernel Module 'vmw_psp_rr'

Module vmw_psp_rr loaded successfully

sysboot: Loading VMkernel Module 'vmw_psp_mru'

Module vmw_psp_mru loaded successfully

sysboot: psa-nmp-pre-claim-config ...

sysboot: Loading NMP SATP rules

sysboot: Executing 'esxcli nmp boot restore'

sysboot: psa-load-rules ...

sysboot: Executing 'esxcli corestorage claimrule load'

sysboot: psa-nmp-post-claim-config ...


vmkernel Log

Located in /var/log directory

Contains all events generated by vmkernel

vmkwarning log is a subset of the vmkernel log and

contains only the warning events

Rotated with a numeric extension. The current log

without extension and the next newest one with “.1”

extension

All events since last vmkernel load are also in

memory in /proc/vmware/log


VMware ESXi 4.0 Specific Log Files

/var/log

• messages

• Equivalent to /var/log/vmkernel on ESX 4.0

• sysboot.log

• Same as ESX 4.0

• The rest of boot logs do not apply to ESXi 4.0

/var/log/vmware

• Same as VMware ESX 4.0


Messages Log Files (VMware ESXi 4.0)

Located in /var/log directory

Contains all events generated by vmkernel

Rotated and compressed with a numeric extension.

The current log without extension and the next

newest one with “.0.gz” extension


SCSI Errors Verbose logging

VMware ESX and VMware ESXi 4.0 are enabled by default for

reporting the SCSI Error Strings “Scsi.LogCmdErros” (set to “1”)

To increase the verbosity of MultiPath events logging, ESX and

ESXi 4.0 advanced vmkernel option “Scsi.LogMPCmdErrors” is

enabled by default (set to “1”)

It is recommended that you redirect the logs to a vMA appliance

or a syslog host to collect historic logs and (for ESXi 4.0) to

preserve them between reboots


Redirecting VMware ESX/ESXi Logs to vMA

vMA (vSphere Management Assistant) is a Virtual

Appliance for managing multiple VMware ESX hosts

from a single Service Console

To enable log forwarding to vMA use:

sudo vifp addserver <server-name or ipAddr>

vilogger enable --server <VMware ESX name>


Redirecting VMware ESX/ESXi 4.0 Server Logs to vMA

Location of forwarded logs on vMA

/var/log/vmware/<ESX-Hostname>

Notice that the log files are all suffixed with “.log”

extension


Redirecting VMware ESXi 4.0 Logs to a syslog server

Log on to VMware ESXi directly using vSphere Client

Configuration Advanced Options Syslog Remote

Enter the syslog server‟s host name


vmkernel Log - Components

System

Date/Time

Jun 19 09:12:54 giza vmkernel: 14:22:31:50.009 cpu3:1033)scsi-qla0: Scheduling SCAN for new luns....

Host

name

Message

source

uptime

Device

MessageCPU:World ID


Rescan Event - LUN Discovery – ESX 4.0

LUN 21 on target 0 on vmhba2 channel 0 discovered.

Storage Vendor’s ID, Array Model and Microcode Rev.

Reported ANSI version is 4 = SCSI-3

TPGS (Target Port Group Support) is off (0)

If the Target supports ALUA, this would be on (1, 2 or 3)

NMP Plugin claimed the path. If 3rd party MPIO is installed, it

would claim paths on its configured arrays instead of NMP

ScsiScan: 839: Path 'vmhba2:C0:T0:L21': Vendor: 'EMC ' Model: 'SYMMETRIX ' Rev: '5772'

ScsiScan: 842: Path 'vmhba2:C0:T0:L21': Type: 0x1f, ANSI rev: 4, TPGS: 0 (none)

ScsiPath: 3685: Plugin 'NMP' claimed path 'vmhba2:C0:T0:L21'


Rescan Event – LUN Discovery - ESX 4.0

Inquiry Commend returns Device ID

A logical device representing the discovered LUN is registered.

Notice the LUN ID is part of the logical device name

Native Multipathing Plugin (NMP) is use here

Device type 0 means “Block Device”

VMWARE SCSI Id: Id for vmhba2:C0:T0:L21

0x01 0x90 0x10 0x05 0x60 0x53 0x30 0x30 0x32 0x32 0x46 0x53 0x59 0x4d 0x4d 0x45 0x54

ScsiDevice: 1756: Successfully registered device "naa.60060480000190100560533030323246" from plugin "NMP" of type 0


Rescan Event – LUN Discovery - ESX 4.0Device Type Codes

Code Description

00h Direct access block device

01h Sequential-access device (e.g. Tape Drive)

02h Printer device

03h Processor device

04h Write-once device

05h CD/DVD device

08h Tape Library


LUN Discovery - ESX 4.0 - More Examples

LUN on an EMC Clariion with ALUA not enabled

Tape Drive “IBM: ULT3580-TD4”

Notice type 0x1

Fibre Attached Tape Library “IBM:3573-TL”

Notice type 0x8

Path 'vmhba2:C0:T1:L0': Vendor: 'DGC ' Model: 'RAID 5 ' Rev: '0428„

Path 'vmhba2:C0:T1:L0': Type: 0x0, ANSI rev: 4, TPGS: 0 (none)

Path 'vmhba2:C0:T2:L0': Vendor: 'IBM' Model: 'ULT3580-TD4' Rev: '89B2„


Path 'vmhba2:C0:T2:L1': Vendor: 'IBM' Model: '3573-TL ' Rev: '7.30„



vmkernel log entries - Understanding SCSI Error Strings

Format:

Device/Host Sense_buffer[2] 12 13

Abbrev: D/H S ASC ASCQ

Extended:

“Device Status”/”Host Status” “Sense Key” “Additional Sense

Code” “Additional Sense Code Qualifier”

Example:

2/0 0x6 0x29 0x0


vmkernel log entries - Understanding SCSI Error Strings

New format (Extended):

Host Status: <hex> Device Status: <hex> Plugin Status: <hex>

Example:

H:0x1 D:0x0 P:0x0

Host No_Connect, Device OK, Plugin OK

Possible sense data: 0x0 0x0 0x0

No Sense Data


Understanding SCSI Error Strings – Device Status

Code Meaning

0 No errors

2 Check Condition

8 Device Busy

24 Reservation Conflict

Device Status: (Displayed in decimal values)


Understanding SCSI Error Strings – Host Status

Host Status (displayed in decimal values)

Code Meaning

0 Host_OK

1 Host No_Connect

2 Host_Bus_Busy

3 Host_Timeout

4 Host_Bad_Target

5 Host_Abort


Understanding SCSI Error Strings –Host Status (cont.)

Host Status (displayed in decimal or Hex values) listed here

in decimal

Code Meaning

6 Host_Parity

7 Host_Error

8 Host_Reset

9 Host_Bad_INTR

10 Host_PassThrough

11 Host_Soft_Error


Understanding SCSI Error Strings – Sense Key

Sense Key (displayed in hex)

Code Meaning

0x0 No Sense Information

0x1 Last command completed but used error correction

0x2 Unit Not Ready

0x3 Medium Error

0x4 Hardware error

0x5 ILLEGAL_REQUEST (Passive SP)

0x6 LUN Reset


Understanding SCSI Error Strings –Sense Key – (cont.)

Sense Key (displayed in hex)

Code Meaning

0x7 Data_Protect – Access to data is blocked

0x8 Blank_Check – Reached an unexpected region

0xa Copy_Aborted

0xb Aborted_Command – Target aborted command

0xc Comparison for SEARCH DATA unsuccessful

0xd Volume_Overflow – Medium is full

0xe Source and Data on Medium do not agree


Understanding SCSI Error Strings – ASC/ASCQ

ASC and ASCQ are always in pairs (in hex) (ASQ usually 0)

Code Meaning

0x4 Unit Not Ready

0x3 Unit Not Ready – Manual Intervention Required

0x2 Unit Not Ready - Initializing Command Required

0x29 Device Power on or SCSI Reset

0x8b ASC_QUIESCENCE_HAS_BEEN_ACHIEVED (IBM FAStT)

0x94 ASC_Invalid_Req_due_To_Current_LU_Ownership (IBM FAStT)

0x01 ASCQ_Invalid_Req_due_To_Current_LU_Ownership (IBM FAStT)

0x02 ASCQ_QUIESCENCE_HAS_BEEN_ACHIEVED (IBM FAStT)


Understanding SCSI Error Strings - Examples (4.0)Plugin Code Translation

Code Translation Meaning

0x0 No Error OK

0x1 VMK_SCSI_PLUGIN_TRANSIENT transient plugin failure, retry.

(e.g. out of memory)

0x2 VMK_SCSI_PLUGIN_SNAPSHOT Device is a deactivated

Snapshot

0x3 VMK_SCSI_PLUGIN_RESERVATION_LOST Host lost a reservation of a

previously reserved LUN


Understanding SCSI Error Strings - Examples (4.0)

H:0x1 D:0x0 P:0x0 (Host No_Connect/Device OK/Plugin OK)

Possible sense data: 0x0 0x0 0x0. (No sense data)

Physical Path: vmhba0:C0:T1:L1

nmp_CompleteCommandForPath: Command 0x2a (0x410004237c00) to NMP device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.



H:0x0 D:0x2 P:0x0 (Host OK/Device Check Condition/Plugin OK)

Valid sense data: 0x6 0x29 0x0. (LUN Reset)


NMP: nmp_CompleteCommandForPath: Command 0x25 (0x410004134a80)to NMP device "naa.60060160b4111600826120bae2e3dd11" failed onphysical path "vmhba0:C0:T1:L0" H:0x0 D:0x2 P:0x0 Valid sensedata: 0x6 0x29 0x0.

NMP: nmp_CompleteCommandForPath:Command 0x25 to device "naa.60060160b4111600826120bae2e3dd11"failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.



H:0x0 D:0x8 P:0x0 (Host OK/Device Busy/Plugin OK)

Possible sense data: 0x0 0x0 0x0. (No sense date)

Device name: naa.6006016004351f00bc913175af58de11

ScsiDeviceIO: 747: Command 0x16 to device "naa.6006016004351f00bc913175af58de11" failed H:0x0 D:0x8 P:0x0 Possible sense data: 0x0 0x0 0x0.



H:0x0 D:0x2 P:0x0 (Host OK/Device Check Condition/Plugin OK)

Valid sense data: 0x6 0x3f 0xe. (LUN Removed) (Clariion)


Device name: naa.60060160e6341f00988aef0a4f4fde11

NMP: nmp_CompleteCommandForPath: Command 0x25 (0x4100051f7bc0) toNMP device " naa.60060160e6341f00988aef0a4f4fde11" failed onphysical path "vmhba0:C0:T1:L2" H:0x0 D:0x2 P :0x0 Valid sensedata: 0x6 0x3f 0xe.



I/O Failover for device naa.600601600ade2200aa3f43e90e7dde11

failed

Reason: Device not found

Cause: All Paths are Dead (APD)

NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.600601600ade2200aa3f43e90e7dde11" - failed to issue command due to Not found (APD), try again...


Identifying path state

From ESX console, vMA or vCLI, run:# esxcli nmp path list

NOTE: vm-support collects esxcli-nmp-paths.<pid>.txt which is the same as

above

For a specific device:# esxcli nmp path list -d <device-name>

# esxcli nmp path list -d naa.600601608cc01c002d45db8b0fb8dd11

fc.2001001b32351707:2101001b32351707-fc.50060160c1e04536:5006016841e04536-

naa.600601608cc01c002d45db8b0fb8dd11

Runtime Name: vmhba2:C0:T1:L3

Device: naa.600601608cc01c002d45db8b0fb8dd11

Device Display Name: DGC Fibre Channel Disk (naa.600601608cc01c002d45db8b0fb8dd11)

Group State: standby

Storage Array Type Path Config:

Path Selection Policy Path Config: {non-current path}


Identifying Path State - cont.

fc.2001001b32351707:2101001b32351707-fc.50060160c1e04536:5006016141e04536-

naa.600601608cc01c002d45db8b0fb8dd11




Group State: active


Path Selection Policy Path Config: {current path}

fc.2000001b32151707:2100001b32151707-fc.50060160c1e04536:5006016041e04536-

naa.600601608cc01c002d45db8b0fb8dd11




Group State: active


Path Selection Policy Path Config: {non-current path}


Any Questions?


Extra Slides


Appendix -ESX 4.0 Failover Logs With Additional Logging

14:13:47:59.878 cpu7:4109)NMP: nmp_HasMoreWorkingPaths: STANDBY path(s) only to device

"naa.60060160432017005c97aea1b32fdc11".

14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_PspSelectPathForIO: Plugin VMW_PSP_MRU selectPath() returned

path "vmhba0:C0:T1:L1" for device "naa.60060160432017005c97aea1b32fdc11" which is in state standby

instead of ON. Status is Bad parameter

14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_SelectPathAndIssueCommand: PSP select path "vmhba0:C0:T1:L1"

in a bad state on device "naa.60060160432017005c97aea1b32fdc11".

14:13:47:59.887 cpu7:4374)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004237c00) to NMP device

"naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0

Possible sense data: 0x0 0x0 0x0.

14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_DeviceRetryCommand: Device

"naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O

blocked...

14:13:47:59.887 cpu7:4374)WARNING: NMP: nmp_DeviceStartLoop: NMP Device

"naa.60060160432017005c97aea1b32fdc11" is blocked. Not starting I/O from device.

14:13:48:00.069 cpu7:4109)NMP: nmp_DeviceUpdatePathStates: Activated path "vmhba0:C0:T1:L1" for NMP

device "naa.60060160432017005c97aea1b32fdc11".

14:13:48:00.888 cpu1:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device

"naa.60060160432017005c97aea1b32fdc11" - issuing command 0x410004237c00

14:13:48:00.888 cpu2:4373)WARNING: NMP: nmp_CompleteRetryForPath: Retry command 0x2a (0x410004237c00) to

NMP device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x0

D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.

14:13:48:00.888 cpu2:4373)WARNING: NMP: nmp_CompleteRetryForPath: Retry world restored device

"naa.60060160432017005c97aea1b32fdc11" - no more commands to retry

14:13:48:00.888 cpu2:4373)ScsiDeviceIO: 746: Command 0x2a to device

"naa.60060160432017005c97aea1b32fdc11" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.


Last slide magnified

NMP: nmp_HasMoreWorkingPaths: STANDBY path(s) only to device "naa.60060160432017005c97aea1b32fdc11“

NMP: nmp_PspSelectPathForIO: Plugin VMW_PSP_MRU selectPath() returned path "vmhba0:C0:T1:L1" for device "naa.60060160432017005c97aea1b32fdc11" which is in state standby instead of ON. Status is Bad parameter

cpu7:4374)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004237c00) to NMP device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba0:C0:T1:L1" H:0x1 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

NMP: nmp_DeviceRetryCommand: Device "naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O blocked...


Appendix - ESX 4.0 Failover Logs – FC cable unplugged14:13:32:16.716 cpu3:4099)<6>qla2xxx 003:00.1: LOOP DOWN detected mbx1=2h mbx2=5h mbx3=0h.

14:13:32:24.425 cpu6:4195)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004286980) to NMP

device "naa.60060160432017005c97aea1b32fdc11" failed on physical path "vmhba1:C0:T0:L1" H:0x5 D:0x0

P:0x0 Possible sense data: 0x2 0x3a 0x0.

14:13:32:24.425 cpu6:4195)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device

"naa.60060160432017005c97aea1b32fdc11" state in doubt; requesting fast path state update...

14:13:32:24.425 cpu6:4195)ScsiDeviceIO: 746: Command 0x2a to device

"naa.60060160432017005c97aea1b32fdc11" failed H:0x5 D:0x0 P:0x0 Possible sense data: 0x2 0x3a 0x0.

14:13:32:26.718 cpu4:4198)<3> rport-4:0-0: blocked FC remote port time out: saving binding

14:13:32:26.718 cpu4:4198)<3> rport-4:0-1: blocked FC remote port time out: saving binding

14:13:32:26.718 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x410004286980) to NMP


P:0x0 Possible sense data: 0x0 0x0 0x0.

14:13:32:26.718 cpu5:4101)WARNING: NMP: nmp_DeviceRetryCommand: Device

"naa.60060160432017005c97aea1b32fdc11": awaiting fast path state update for failover with I/O

blocked...

14:13:32:26.718 cpu5:4101)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4100042423c0) to NMP


P:0x0 Possible sense data: 0x0 0x0 0x0.

14:13:32:26.718 cpu3:4281)WARNING: VMW_SATP_CX: satp_cx_otherSPIsHung: Path "vmhba1:C0:T1:L1" MODE

SENSE PEER SP command failed 0/1 0x0 0x0 0x0.

14:13:32:26.719 cpu1:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device

"naa.60060160432017005c97aea1b32fdc11" - issuing command 0x410004286980

14:13:32:26.752 cpu2:4237)NMP: nmp_CompleteRetryForPath: Retry world recovered device

"naa.60060160432017005c97aea1b32fdc11"


Messages Log (VMware ESX 4.0)Jul 24 19:13:33 giza sshd[18915]: Connection from 10.16.112.24 port 1396

Jul 24 19:13:36 giza sshd[18915]: Accepted password for root from 10.16.112.24 port 1396 ssh2

Jul 24 19:13:36 giza sshd(pam_unix)[18915]: session opened for user root by (uid=0)

Jul 29 01:01:03 giza iscsid[32725]: cannot make connection to 10.16.95.161:3260: No route to host

Console events

Logon events


hostd.log

Located in /var/log/vmware

Sym-linked to the current rotated hostd log file

Hostd events

VI Client communications when directly connected toVMware ESX

Events done on behalf of

VPXA

System Services

Firewall System

HA services

VMware Converter

vm-support collects hostd-support*.tgz files


hostd-support dump

vm-support collects hostd-support*.tgz files located

in /var/log/vmware directory

Generated by /usr/sbin/vmware-hostd-support script

Rotated up-to 4 copies (blank, -1, -2 and -3 suffix)

Extracted to hostd-support-<pid> directory which includes

Config

Content of /etc/vmware/hostd directory

Logs

/var/log/vmware/hostd-*.log file with current hostd’s PID

/var/log/vmware/hostd-trace.log file if exists

Runtime

Content of /var/lib/vmware/hostd/journal directory

Content of /var/lib/vmware/hostd/stats directory


hostd.log

ESX Version reported

Log compression listed

VMFS volume access events logged and reported

vSphere Distributed Switch discovery events

Log for VMware ESX, pid=3049, version=4.0.0, build=build-140815, option=Release, section=10

Compression of /var/log/vmware//hostd-3.log to /var/log/vmware//hostd-3.log.gz took 167 ms

[2009-04-27 08:39:13.157 F624BB90 info 'ha-eventmgr'] Event 3558 : Lost access to volume 492c94b9-7b4f6ee8-bb0b-001a645dba02 (PSE-i144-100gb) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

[2009-04-27 08:36:14.927 F56EBB90 info 'ha-eventmgr'] Event 3549 : Successfully restored access to volume 492c94b9-7b4f6ee8-bb0b-001a645dba02 (PSE-i144-100gb) following connectivity issues.

[2009-04-27 08:36:15.307 F6188B90 verbose 'DvsTracker'] FetchSwitches: added 0 items

[2009-04-27 08:36:15.307 F6188B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items

[2009-04-27 08:36:34.081 F61C9B90 verbose 'PropertyProvider'] RecordOp 1: latestPage[3540], session[524622d3-a4f2-0704-e090-fedc59dbc2f5]52553360-a9c2-63b0-7138-0486ae7d4b0f


oldconf Files

Backup copies of /etc/vmware/esx.conf file

Located in /var/log/oldconf directory

Created prior to updating the existing file

Only when changes done via VMware vCenter, vSphere Client

or esxcfg-* scripts

Date and time of backup used as the extension of the file name

esx.conf.2009-03-11_16:39:06 esx.conf.2009-04-14_10:01:57

vsphere4 storage

Documents