ser1289be vsphere troubleshooting tips and … 1 use of the vmware vsphere command-line interface...
TRANSCRIPT
Javier Menendez & Francis Daly
SER1289BE
#VMworld #SER1289BE
vSphere Troubleshooting Tips and Tricks
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
#SER1289BE CONFIDENTIAL 2
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1Use of the VMware vSphere Command-line Interface (vCLI),
the ESXi Shell, and Log Files to Diagnose and Correct Problems
2 Troubleshooting Networking Issues
3 Troubleshooting Storage Issues
4 Troubleshooting vSphere HA Cluster and Performance Issues
3#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
4Certain troubleshooting topics are addressed in the VMware vSphere: Fast Track [6.5] course.
Scope
This Presentation covers portions of the Troubleshooting Workshop:
For more information, go to http://vmware.com/education
VMworld 2017 Content: Not fo
r publication or distri
bution
Troubleshooting Overview
VMworld 2017 Content: Not fo
r publication or distri
bution
Troubleshooting Process
The troubleshooting process consists of the following tasks:
6
Defining the problem
• Identifying symptoms
• Gathering information
Identifying the cause of the problem
• Identifying possible causes
• Determining the root cause
Solving the problem
• Identifying possible solutions
• Implementing the best solution
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Solving the problem … Can I do it with the Web Client or should I use the esxi shell?
7
What are my choices:
• Can I fix the problem with the Web, Host or vSphere Client?
• Do I want or need to use the command line utilities?
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Troubleshooting Tools and Choices
VMworld 2017 Content: Not fo
r publication or distri
bution
Command-line Troubleshooting Choices
Choices to obtain command-line access on a VMware® ESXi™ host:
9
vSphere Management Assistant (VMA)
VMware vSphere® ESXi™ Shell
• Can be access via:
• 1. The DCUI (if enabled via ALT + F1)
• 2. Via putty after enabling ssh.
• Appliance deployed via ovf
• Contains vCLI package
• Does NOT use port 22.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
The esxcli Command
The esxcli command
offers the following
namespaces,
as well as many new esxcli functions added in
the vSphere 6.5.
If working with vSAN,
check out the new esxcli
vsan debug namespace
10#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Example: The esxcli vsan Namespace
The vSAN related
commands are found in
esxcli vsan
11#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
New commands added with every new vSphere release.
Ex: The esxcli vsan debug Namespace
The new vSAN debug
related commands are
amazing to discover the
state of your objects, vms,
etc.
12#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
esxcli esxcli command list
displays all the command
options.
To quickly find the command
needed, combine it with the
grep command
Tip: Remember the esxcli esxcli command list command
13#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
The VMA’s vicfg-* Commands
14
vicfg- [tab] [tab]
• Commands with the vicfg- prefix enable you to manage
your storage, network, and host configuration.
• Type vicfg- followed by the TAB key twice so see all the available commands.
• For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
The esxi shell’s esxcfg- Commands are still there
15
esxcfg- [tab] [tab]
• Commands with the esxcfg-
prefix enable you to manage
your storage, network, and
host configuration.
• Type esxcfg- followed by the TAB key twice so see all the available commands.
• Remember the –h (help) option to master these commands.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Use the –h option (for help)
to see options for this
particular command and in
some cases, even
examples on how to use the
command.
How many times did you
struggle to create a
vmkernel port with the cli?
Tip: Remember the “-h” option for esxcfg-* commands
#SER1289BE CONFIDENTIAL 16
VMworld 2017 Content: Not fo
r publication or distri
bution
Manipulating virtual machines with the cli from the V.M.A.?
17
• vmware-cmd
– The vmware-cmd command is exclusively used for virtual machines.
vmware-cmd
• The vmware-cmd command is used to interact with virtual machines.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Example: Listing your Virtual Machines
18
vmware-cmd -l
• Lists the virtual machines that are located on the target host. Lists virtual machines by path to the .vmx file.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Learn More On Your Own
19
Type vmware-cmd --help to
see what the command allows
you to do.
vmware-cmd allows you to
start a vm, take snapshots,
register a vm, etc.
The esxi host has a similar
command to manipulate vms
called vim-cmd vmsvc.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Manipulating virtual machines from the esxi shell
20
vim-cmd vmsvc can
perform similar
tasks from the
esxi shell.
Wondering what
the destroy
option does?
Better try it on
a test vm.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Location of VMware vCenter Server 6.5 Log Files
21
The VMware vCenter Server
6.5 logs are located in the
%ALLUSERSPROFILE%\VM
Ware\vCenterServer\logs
folder.
The VMware vCenter Server
Appliance 6.5 logs are
located in the
/var/log/vmware folder.
Recommendation
See http://kb.vmware.com/kb/2110014 and 1021806 for additional details.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Useful ESXi Host Logs for Troubleshooting
ESXi hosts write to multiple log files, depending on which action is being performed.
22
Log file Purpose
hostd.log Host Management service logs
syslog.logManagement service initialization, watchdogs, scheduled
tasks, and DCUI use
vmkernel.log
Core VMkernel logs, including device discovery, storage
and networking device and driver events, and virtual
machine startups
vmkwarning.logA summary of warning and alert log messages excerpted
from the VMKernel logs
vmksummary.log
A summary of ESXi host startup and shutdown, and an
hourly heartbeat with uptime, number of virtual machines
running, and service resource consumption
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Other Critical Logs to Be Aware Of and how to view them
23
/var/log/fdm.log displays HA
cluster related events and
notifications.
/var/log/shell.log displays
commands executed inside
of the esxi shell.
/var/log/clomd.log displays vSAN
health related information for
devices among the output
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Remember /var/log
24
Lots of logs available in
this directory
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Collecting Diagnostic Data for
VMware Technical Support
25
Methods for collecting diagnostic information to send to VMware Technical Support include the following:
• Use the GUI to export files to a log bundle.
− vSphere Client or vSphere Web Client
• Use the vm-support command to collect
information from an individual ESXi host.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Running vm-support when engaging support
26#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Network Troubleshooting Commands
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Useful Networking Related Commands
28
Use esxcfg-vmknic –l or
vicfg-vmknic -l to view your
vmkernel port related
information. Are they still there
and enabled?
Use esxcfg-vswitch –l or
vicfg-vswitch –l to view virtual
switch related information. Is
the switch connected to at
least one uplink?
Use esxcfg-nics –l or vicfg-
nics –l to view uplink related
information. Are the nics up or
down?
esxcli also has options to
perform these tasks.
For example:
esxcli network nic list is similar
to esxcfg-nics -l
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Somebody could accidentally
down a nic?
If so, type esxcli network nic up
–n vmnic2.
Tip: Did You Know That…
29#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Somebody could disable a vmkernel port
with commands like esxcfg-vmknic -D?
If so, type esxcfg-vmknic -e to re-
enable the vmkernel port.
Tip: Did You Know That…
30#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Somebody could accidentally
disconnect an uplink from a
virtual switch?
ffaaa
Tip: Rebuilding the Management Network from scratch
31#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
There is a built-in packet
sniffer called pktcap-uw
and an older one called
tcpdump-uw.
Need a packet sniffer? Remember pktcap-uw
32#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Storage Troubleshooting
VMworld 2017 Content: Not fo
r publication or distri
bution
You can see how many dastastores exist and how full
the datastores are? This command displays vmfs, nfs,
vsan and vvol datastores. There is also a vdf –h
command.
Tip: Did You Know That…
34#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
You can easily view all
sorts of information
with esxcli?
Tip: Did You Know That…
35#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Use this command to
retrieve data about a
specific SSD device.
Critical for vSAN,Flash
Read Cache or any
datastore using SSDs.
vSphere 6.5
Includes
Tip: Use esxcli Storage Core Device Smart for SSD information
36 36
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Be familiar with partedUtil
37
partedUtil is the command
that partitions luns. It can also
create, resize and delete
partitions.
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Suspect metadata corruption?
38
voma is the on-disk metadata
analyzer for vmfs datastores.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Troubleshooting Clusters and Performance
VMworld 2017 Content: Not fo
r publication or distri
bution
vSphere HA
40
vCenter
Server
ESXi host (slave)
FDM
ESXi host (master)
FDM
ESXi host (slave)
FDM
vpxd
hostdhostdhostd
Management
Network
vpxa vpxa vpxa
Heartbeat
Datastores
A reliable network connection between the hosts and VMware® vCenter Server™
is essential for enabling vSphere HA.
VMworld 2017 Content: Not fo
r publication or distri
bution
There is a log for HA that
you can’t see with the Web
or vSphere Client?
Tip: Remember F.D.M
41#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
You can stop, start or restart the HA service from the cli if needed?
Tip: Did You Know That…
42#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: You can restart all services at once if needed …
43
Remember services.sh restart
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Did You Know That…
44
esxtop is an amazing tool for
performance troubleshooting?
esxtop has options for CPU
(c), RAM (m), Disk (d,u,v)
and Network (n) related
statistics?
esxtop has an option for vSAN
(-x option)?
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
esxtop has built-in help?
Tip: Did You Know That…
45#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
You can add more
output to esxtop by
pressing “f”?
For example, you can
add NUMA related
information (G Option)
Tip: Did You Know That…
46#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Learn the vsish command and its capabilities
47
vsish has hundreds of
subcommands.
Just type the command and
take a look …
You can type commands like
cd to change directories and
get to view information.
#SER1289BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Tip: Learn the vsish command and its capabilities
vsish has hundreds of
subcommands.
Just type the command and
take a look …
#SER1289BE CONFIDENTIAL 48
VMworld 2017 Content: Not fo
r publication or distri
bution
Additional Education Resources
At VMworld 2017 Europe
• Education & Certification Lounge: VMvillage, Hall 6
• Certification Exam Center: CC5
Online
• VMware Training: www.vmware.com/education
• VMware Certification: www.vmware.com/certification
Global Support Services
Learn more about how VMware is radically transforming Customer Support through VMware Skyline™ technology.
• Visit the demo booth within the Solutions Exchange
• Sign up for a Meet the Experts session in the Content Catalogue
• Visit www.vmware.com/support/service/skyline
Save 50% off VCP & VCAP
exams at
VMworld 2017
#SER1289BE CONFIDENTIAL 49
VMworld 2017 Content: Not fo
r publication or distri
bution
Prove Your Expertise with VMware Digital Badges
What are VMware Digital Badges?
• Single source that combines your credential and
a complete overview of your skills
• Way for you to easily share your accomplishments
in social media
• Provides employers with easy, valid verification of
VMware credentials
Digital Badges Online
• vmware.com/go/vSAN2017badge
• vmware.com/go/VxRail2017badge
VMware vSAN 2017 Specialist
Dell EMC VMwareCo-Skilled - VxRail 2017
NEW!VMware Digital Badges
Prove your expertiseon vSAN and
vSAN HCI environments
#SER1289BE CONFIDENTIAL 50
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution