ser2905be uncovering esxtop or distribution for ......this screen displays vm-centric storage...

Jagadeesh Devaraj & Ramesh Nataraja

SER2905BE

#VMworld SER2905BE

Uncovering ESXTOP

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2#SER2905BE CONFIDENTIAL



bution

Agenda

1 What is ESXTOP

2 ESXTOP Screen

3 CPU, Memory, Power, Storage, VM screen

4 Storage adapter, Storage device, VSAN screen

5 Network

6 Data collection




bution

What is esxtop?

• ESXTOP is a built-in command line tool that gives real time information on

– Resources used (cpu, memory, network, storage, power)

– Understand the resource fault trending

– Help in hardware scalability assessment

4

How do you run ESXTOP?

Run esxtop from the ESXi console with command

“esxtop”

#SER2905BE CONFIDENTIAL



bution

esxtop screen

Key strokes to navigate

5

CPU c

Memory m

Power Management p

Storage VM v Network n

Storage Device u

Storage Adapter d

vSAN x

Interrupts i




bution

‘c’ for CPU

6

Current timeUptime of the hostNumber of userworlds runningNumber of VMs runningNumber of vCPUs assigned to all running VMs

CPU load average in

1 minute, 5 minutes, and 15 minutes

based on 6-second samples

The percentage CPU usage per Physical CPU, and its average over all Physical CPUsThe percentage of unhalted CPU cycles per Physical CPU, and its average over all

Physical CPUsID of the worldID of the groupName of the sched groupNumber of worlds in the groupThe percentage physical CPU time accounted to the world.

%USED = %RUN + %SYS - %OVRLPThe percentage of time the virtual machine is consuming CPU resourcesThe percentage of time spent by system services on behalf of the world

The percentage of time the world spent in wait state.

The world is waiting for some vmkernel resourceRepresents the total percentage of time the World spent in a blocked state waiting for events. Amount of time the virtual machine was ready to run, waiting in a queue to be scheduled on the PCPUThe percentage of time the vCPU world is in an idle loop.

The percentage of time spent by system services on behalf of other worldsAmount of time an SMP vm was ready to run, but

incurred delay due to co-vCPU scheduling contentionPercentage of time the world was ready to run but deliberately wasn't

scheduled because that would violate the "CPU limit" settings

The percentage of time the world is waiting for the vmkernel swapping memory

%SWPWT time is included in the %WAIT time




bution

‘m’ for Memory

7

PMEM /MB - This line displays the physical memory statistics for the server

total is for the total amount of physical memory in the server

vmk is for the amount of physical memory being used by the vmkernel

other is for the amount of physical memory being used by everything other than the vmkernel

free is for the amount of physical memory that is free

VMKMEM/MB - This line displays the machine memory statistics for the vmkernel

managed is for the total amount of physical memory managed by the vmkernel

minfree is for the minimum amount of physical memory that the vmkernel would like to keep free

rsvd is for the amount of physical memory that is currently reserved

ursvd is for the amount of physical memory that is

currently unreserved and

state is for the memory state as reported by the vmkernel.

high(300%)|clear(64%)|soft(32%)|hard(16%)|low(0%)

NUMA /MB - This line displays the ESXi NUMA statistics.

This line is only displayed if ESXi is running on a NUMA

server

The first statistic is the total amount of physical memory in

the NUMA node that is managed by ESXi.

The second statistic, that is displayed within round brackets, is the amount of physical memory in the node that is currently free




bution

‘m’ for Memory

PSHARE/MB This line displays the ESXi page-sharing

statistics.

shared - is for the amount of physical memory that is

being shared

common - is for the amount of physical memory that is

common across World(s)

saving - is for the amount of physical memory that is

saved due to page-sharing

SWAP /MB This line displays the ESXi swap usage statistics

curr is for the current swap usage

rclmtgt is for where ESXi expects the reclaimed memory to be,

using swapping and compression

r/s is for the rate at which memory is swapped in by ESXi from

disk

w/s is for the rate at which memory is swapped to disk by the

ESXi Server

ZIP /MB - This line displays the ESXi memory compression

statistics

zipped - is for the total compressed physical memory

saved - is for the saved memory by compression

MEMCTL/MB This line displays the memory balloon statistics

curr is for the total amount of physical memory reclaimed using

the vmmemctl modules

target is for the total amount of physical memory ESXi would

like to reclaim using the vmmemctl modules

max is for the maximum amount of physical memory ESXi can

reclaim using the vmmemctl modules

#SER2905BE CONFIDENTIAL 8



bution

‘m’ for Memory

Resource Pool ID / World idResource Pool nameThe amount of physical memory allocated to a vmThe amount of machine memory the ESXi VMkernel wants to allocate to vmThe amount of machine memory consumed by a vmThe amount of physical memory ESXi VMkernel wants to allocate to a vmThe working set estimate for a vmThe write working set estimate for a vmCurrent swap usage by vmWhere ESXi expected the Swap usage by the vm to be

Rate at which memory is being swapped in by ESXi from disk for the vmRate at which vm memory is being swapped to disk by ESXiRate at which memory is faulted from host cacheRate at which memory is written to the host cache from various sourcesCurrent space overhead for the user worldCurrent space overhead for vmMaximum space overhead that may

be incurred by vm

#SER2905BE CONFIDENTIAL 9



bution

‘p’ for Power Management

10

This screen displays CPU Power utilization statistics. On this screen statistics are arranged per PCPU

-

-.

Power Usage The current total power usage in Watts

Power Cap The total power cap in Watts

%USED The percentage of PCPU nominal frequency that was used since

the last screen update

%UTIL The raw PCPU utilization, that is the percentage of real time that

PCPU was not idle

%Cx The percentage of time the PCPU spent in C-State 'x'

%Px The percentage of time the PCPU spent in P-State 'x'.

%Tx The percentage of time the PCPU spent in T-State 'x'




bution

‘p’ for Power Management

When to use Power Management?

11

Sluggish Application

Performance on a vm

Verify if there is a discrepancy

between PCPU Used% and PCPU Util%

Verify if there is CPU contention

(high %RDY)

Divide %RDY value by the number of

vCPUs assigned

Isolate if excess vCPU is

the cause of low application performance




bution

This screen displays VM-centric storage statistics. On this screen, statistics are aggregated on a per-resource-poolbasis by default. One VM has one corresponding resource pool, so they are equivalent to per-VM statistics

12

‘v’ for Storage VM

The GroupID of VM/Resource PoolThe name of VMThe name of the virtual deviceThe number of vscsi devicesThe number of commands issued per secondThe number of read commands issued per secondThe number of write commands issued

per secondThe megabytes read per secondThe megabytes written

per second

The average latency (millisecs)

per readThe average latency (millisecs)

per write




bution

‘u’ for Storage Device

13

Name of the storage devicePath nameWorld IDPartition IDCurrent device queue depth of the storage deviceWorld queue depthNo. of commands in the vmkernel that are

currently activeNo. of commands in the ESXi VMkernel that

are currently queued

Percentage of the queue depth used by

vmkernel active commands

%USD = ACTV / QLEN * 100%

Ratio of vmkernel active commands plus vmkernel

queued commands to queue depthNo. of commands issued per secondNo. of read commands issued per

second

No. of write commands issued per

secondMegabytes read per secondMegabytes written per second

Average device latency per

command in milliseconds

Average ESXi VMkernel latency

per command in milliseconds

Average guest operating system latency

per command in milliseconds

GAVG = KAVG + DAVG

Average queue read latency per read

operation in milliseconds




bution

‘d’ for Storage Adapter

14

This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per

storage adapter by default

The name of the storage adapterThe storage pathnameThe number of pathsThe number of commands

completed per secondThe number of read commands

completed per second

The number of write commands

completed per secondThe megabytes read per secondThe megabytes written per

second

The average device latency

(millisecs) per command

The average vmkernel

latency (millisecs) per

commandThe average Guest OS latency (millisecs) per commandThe average queue latency (millisecs) per command




bution

‘x’ for vSAN

15

VSAN Enabled Whether VSAN is currently enabled on the host

ROLE The name of VSAN DOM Role

READS/s Number of read operations completed per second

MBREAD/s Megabytes read per second

WRITES/s Number of write operations completed per second

MBWRITE/s Megabytes written per second

RECOWR/s Number of recovery write operations completed per second

MBRECOWR/s Megabytes written for recovery per second

SDLAT Standard deviation of latency in millisecs for read, write and recovery write

AVGLAT Average latency in millisecs for read, write and recovery write

This screen displays VSAN statistics




bution

‘n’ for Network

16

PORT-ID The virtual network device port id

UPLINK Y, implies the corresponding port is uplink. N, implies it is not

UP Y, implies the corresponding link is up. N, implies it is not

SPEED The link speed in MegaBits per second

FDUPLX Y, implies the corresponding link is operating at full duplex. N, implies it is not

USED-BY The virtual network device port user

TEAM-PNIC The physical NIC name for the team uplink

DNAME The virtual network device name

PKTTX/s The number of packets transmitted per second

PKTRX/s The number of packets received per second

MbTX/s The MegaBits transmitted per second

MbRX/s The MegaBits received per second

PSZTX The average packet size transmitted in bytes

PSZRX The average packet size received in bytes

%DRPTX The percentage of transmit packets dropped

%DRPRX The percentage of receive packets dropped

ACTN/s Number of actions per second

PKTTXMUL/s The number of multicast packets transmitted per second

PKTRXMUL/s The number of multicast packets received per second

PKTTXBRD/s The number of broadcast packets transmitted per second

PKTRXBRD/s The number of broadcast packets received per second




bution

Data Collection

ESXTOP results could be collected using Batch mode. The output can be redirected to a .csv file, this file could be opened through MS Excel or Perfmon tool

Below is the syntax for generating esxtop in batch mode

-b : for batch mode -d : for delay -n : for no. of iterations-a : for collection of all metrics

In the above example esxtop will log all metrics for 500 seconds and compress the results in zipped format to the specified location




bution



bution

Questions?




bution



bution

ser2905be uncovering esxtop or distribution for ......this screen displays vm-centric storage...

Documents