how to get the most out of - geekboy.pro

#vmworld

HBI2333BU

How to Get the Most Out of vSphere vMotion

Niels Hagoort, VMware, Inc.Arunachalam Ramanathan, VMware, Inc.

#HBI2333BU

VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Disclaimer

This presentation may contain product features or functionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

2

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc.

Agenda

3

How does vMotion work?

How to scale vMotion performance?

How to tune vMotion concurrent limits?

Troubleshooting vMotion

Q&A


©2019 VMware, Inc. 4

One of the most momentous game-changers in the IT industry!

History of vMotion


5©2019 VMware, Inc.

How Does vMotion Work?



Start a live-migration

vMotion Process



Quiesce VM on Source

Copy Memory

vMotion Workflow

Create VM on Destination1

2

3

Transfer Device State

Resume VM on Destination

Power Off VM on Source

4

5

6

vMotion Network

Datastore

SourceESX Host

DestinationESX Host

Execution Switchover

Time of 1 sec



What happens when you initiate a live-migration?

vMotion Process




vMotion Process

Compatibility specification• Versions• Available resources

• For virtual machine• To support vMotion process




vMotion Process

Migration specification• The virtual machine that is being live-migrated• Configuration of that virtual machine (virtual

hardware, VM options, etc.)• Source ESXi host• Destination ESXi host• vMotion network details

Compatibility specification• Versions• Available resources

• For virtual machine• To support vMotion process




vMotion Process



How is memory copied?

vMotion Process

Source VM Memory

Destination VM Memory

• Phase 0: Copy the VM’s 24GB of memory, trace pages. As we send that memory, the VM dirties 8GB

• Phase 1: Retransmit the dirtied 8GB. In the process, the VM dirties another 3GB

• Phase 2: Send the 3GB. While that transfer is happening, the VM dirties 1GB

• Phase 3: Send the remaining 1GBVMworld 2019 Content: Not for publication or distribution


What if Guest writes to memory during live-migration? Page Tracing

vMotion Process



Iterative memory pre-copy

vMotion Process



Switchover phase

vMotion Process



How to Scale vMotion Performance?Saturating the given network


©2019 VMware, Inc.

Agenda

17

Saturating link speed

• Multiple vMotion vmkernel NICs

• Scaling single vMotion vmkernel NIC

• Auto scaling vMotion vmkernel NIC



Multiple vMotion VMkernel InterfacesFor high speed links



vMotion VMkernel interface

Standard vMotion Configuration



How many treads / helpers are used by vMotion?

Streams and Threads



Multi-NIC vMotion helped to saturate multiple 1-10GbE NICs

Streams and Threads



The challenge with > 25GbE NICs

Streams and Threads



Scale bandwidth utilization by adding vMotion VMkernel interfaces!

Streams and Threads



Instantiate multiple vMotion streams

Streams and Threads



Using multiple CPU cores = Higher transfer rates

Streams and Threads



A single vMotion stream bandwidth utilization capability of ~15 GbE

25 GbE : 1 stream = ~15 GbE

40 GbE : 2 streams = ~30 GbE



vMotion stream performance

Streams and Threads



Scaling Single vmknic to Link SpeedTune vMotion streams and vmknic RX queues



Create multiple vmknics on single uplink

• Requires multiple IP addresses

• Each vmknic will have

– One vMotion stream

– One receive queue

• 25 GbE will require 2 vmknics

– 1 vmknic gets 15 Gbps

• Management overhead

– IP per vmknic

Multiple vmknics

Scaling vMotion to Link Speed

Physical NIC

vMotion vmknic

vMotion stream vMotion stream

Hardware RX queue

vmknic RX queues

Crypto

Helper

Stream

Helper

Completion

Helper

Crypto

Helper

Stream

Helper

Completion

Helper

25 GbE NIC

vMotion vmknic



What if a single vmknic could have

• Multiple vMotion streams

• Multiple receive dispatch (RX) queues

Still need the NIC to scale out the flows

• Receive Side Scaling

– Dynamic RSS

– Maps each vMotion stream to a NIC hardware ring

Tune vMotion streams and RX queues

Scaling Single vMotion vmknic

Physical NIC

vMotion vmkernel NIC


vmknic RX queues

Crypto

Helper

Stream

Helper

Completion

Helper

Crypto

Helper

Stream

Helper

Completion

Helper

25 GbE NIC

Hardware RX queue



1. Scale vmknic RX queue

/net/tcpip/defaultNumRxQueue

vsish -e set /net/tcpip/defaultNumRxQueue 2

2. Scale vMotion streams

/config/Migrate/intOpts/VMotionStreamHelpers

vsish -e set /config/Migrate/intOpts/VMotionStreamHelpers 2

Tune vMotion streams and RX queues

Scaling Single vMotion vmknic

Physical NIC

vMotion vmkernel NIC


vmknic RX queues

Crypto

Helper

Stream

Helper

Completion

Helper

Crypto

Helper

Stream

Helper

Completion

Helper

25 GbE NIC

Hardware RX queue



Auto Scaling vMotion

1. Determine uplink speed at vMotion start

2. Determine Scale factor

Uplink speed/15 Gbps

3. Scale vMotion

vmknic’s RX queues

4. Start required no. of

vMotion streams

No tuning required

Dynamically scale

• Works out of the box

– Avoids need for manual tuning

• ESX network stack

– Provides ability to dynamically scale vmknic

• vMotion

– Can dynamically start the required no. of streams



Tuning vMotion for Hybrid CloudLong distance vMotion



Scaling Long Distance vMotion

1. Determine Network Latency

2. Determine Network

Bandwidth

3. Compute BDP

4. Set TCP Socket buffer

size

Saturating available network bandwidth

To saturate network

• Size TCP socket buffer

– With bandwidth delay product (BDP)

– BDP = latency x network bandwidth



1. Expected Bandwidth

2. Max Socket Buffer Size

vMotion doesn’t determine bandwidth

• Takes uplink speed as bandwidth

• Doesn’t work for long distances

• So 1 GbE is default bandwidth

Default MAX is 16 MB

• 10 GbE at 150 ms requires ~186 MB

vsish –e set

/config/Migrate/intOpts/NetExpectedLineRateMBps

10000

vsish –e set

/net/tcpip/instances/defaultTcpipStack/sbMax

195035136

(186 MB in bytes)

Manual Scaling for Long Distance vMotion Tuning line rate and socket buffer

Configure Why?How?

ESX Host: vmkernel config options



• At vMotion start compute latency

1. Measure latency

• If latency > 4ms then compute BW

2. Measure bandwidth

• Compute Bandwidth delay product (BDP)

3. BDP = BW x latency

• Adjust socket buffer (SB) Max to BDP

4. If BDP > SB MAX

• Adjust socket buffer size to BDP

5. Start pre-copy

Determining expected bandwidth

Auto Scaling Long Distance vMotion



Tuning Concurrent vMotion LimitsvMotion Per Resource limits



Concurrent vMotion LimitsvMotion cost per resource

vMotion Cost

• Network Cost = 1

• Storage Cost = 1

• Host Cost = 1

vMotion limits

• 4 per 1 GbE NIC

• 8 per 10 GbE NIC

• 8 per ESX Host

• 128 per datastore

MAX Cost: 4

1 GbE

MAX Cost: 8

10/40/100 GbE

Datastore

MAX Cost: 128

ESX Host

MAX Cost: 8



Single vMotion can saturate given link speed

• 1/10/40/100 GbE Nics

Concurrent vMotions => Longer vMotion duration

– Multiple vMotions share vMotion network

– Longer time put host into maintenance mode

Reason for these limits => Historical

• Limits defined when vMotion couldn’t saturate link speed

• Increase network cost of vMotion

vCenter Advanced Config option

config.vpxd.ResourceManager.networkCostPerVmotion

• Default cost is 1

• For 1 vMotion at any given time

– Increase cost to 4 for 1 GbE

– Increase cost to 8 for 10 GbE

Why change default concurrency limits?

Limit Concurrency

Tuning Concurrent vMotionsHow to limit concurrency?



Concurrent Storage vMotion LimitsStorage vMotion cost per resource

Storage vMotion Cost



• Host Cost = 4

Storage vMotion Limits

• 2 per ESX Host

• 8 per datastore

Datastore

MAX Cost: 128

ESX Host

MAX Cost: 8



Copy one disk at a time for a given VM

– For the given source destination datastore pair

VMX Config option

• svmotion.maxSimultaneousDiskCopy

• Eg. For a VM with 3 disks on NVMe datastore

• svmotion.maxSimultaneousDiskCopy = 3

Increase/Decrease storage vMotion cost

• Maximize datastore throughput and minimize latency

– Preserve VM requirements

vCenter Advanced Config option

• config.vpxd.ResourceManager.CostPerEsx6xSVmotion

• Default Cost is 16

• Eg. Reduce cost to 8 to double concurrency

– For high perf NVMe storage

Tuning simultaneous disk copy Tuning Concurrency

Tuning Storage vMotionsConcurrency based on storage array



Concurrent vMotion without Shared Storage LimitsvMotion cost per resource

vMotion Cost



• Host Cost = 4

vMotion Limits

• 4 per 1 GbE NIC

• 8 per 10 GbE NIC

• 2 per ESX Host

• 8 per datastore

MAX Cost: 4

1 GbE

MAX Cost: 8

10/40/100 GbE

Datastore

MAX Cost: 128

ESX Host

MAX Cost: 8



• Follow storage vMotion tuning guidelines

• Tune simultaneous disk copy

• Tune concurrent limits

• Tune network cost

• Check if disk copy is remote

– Src ESX host cannot access destination storage

• Check if disk copy can saturate vMotion network

• If it can limit concurrent vMotion to 1

– By tuning the network cost

Moving storage dominates vMotion time

First tune for Storage then Network

Tuning Concurrent vMotions without Shared StorageConcurrency based storage array performance

90%

10%

vMotion Time

1 2



Troubleshooting vMotionNavigating vSphere logs



vMotion LoggingvSphere logging

vSphere Components 5 Processes 5 Log Files

ESX

vCenter Server

vpxd

vpxa

hostd

VMX

vmkernel

vpxd.log

vpxa.log

Hostd.log

vmware.log

vmkernel.logVMworld 2019 Content: Not for publication or distribution


Processes

• vCenter vpxd

• vCenter agent vpxa

• Host Daemon hostd

Operation ID

• opID=1807e8fa-3b9d-453e-8dd4-79f2d0ac91ca-327

Hostd

• opID to migration ID mapping

Processes

• VMX

• Vmkernel

Migration ID

• 1439244808967130

Management/Control Plane Data Plane

Tracking vMotionManagement and data plane



Associating logs

vMotion Tracing Logs

Log files (vmkernel and VM log files)• grep “VMotion” keyword in vmkernel log files (/var/log/vmkernel*)

# grep VMotion /var/log/vmkernel*

2015-08-10T22:13:40.958Z cpu89:2237036)VMotionUtil: 3995: 1439244808967130 S: Stream connection 1 added.

2015-08-10T22:13:41.164Z cpu89:2237036)VMotion: 6834: 1439244808967130 S: Detected 50Ms round-trip latency to remote host.

2015-08-10T22:13:41.266Z cpu89:2237036)XVMotion: 3284: 1439244808967130 S: Starting XVMotion stream.

2015-08-10T22:16:37.556Z cpu20:2237038)VMotion: 4873: 1439244808967130 S: Estimated network bandwidth 112.070 MB/s during disk copy.

• grep migration id in VM log files (vmware*log files in VM home directory)

# grep 1439244808967130 $(VM_HOME)/vmware*

VPXD Logs (vCenter Server)• Find the “Operation ID” of vMotion

# grep "relocate" /var/log/vmware/vpxd/vpxd-*.log | grep BEGIN

2015-08-10T22:13:28.847Z info vpxd[7FD6AED5A700] [Originator@6876 sub=vpxLro opID=1807e8fa-3b9d-453e-8dd4-79f2d0ac91ca-327-ngc-bf] [VpxLRO] --BEGIN task-55 -- vm-46 -- vim.VirtualMachine.relocate --

Hostd Logs (ESX Server)• Find the Migration ID of vMotion from the Operation ID

# grep 1807e8fa-3b9d-453e-8dd4-79f2d0ac91ca-327 /var/log/hostd.log | grep -i migrate

2015-08-10T22:13:39.568Z info hostd[2FCC2B70] [Originator@6876 sub=Vcsvc.VMotionSrc (1439244808967130) opID=1807e8fa-3b9d-453e-8dd4-79f2d0ac91ca-327-ngc-bf-40-86-7fb8 user=vpxuser:VSPHERE.LOCAL\Administrator] VMotionEntry: migrateType = 1



vMotion Failures

• vMotion network connectivity issues

– ESX hosts cannot ping or timeout of 20 secs

– MTU mismatch with vmknic and network layer i.e switches and routers

• Storage

– Datastore unreachable or APD

– IOs timeout of 20 secs or more

• vMotion successful but guest VM issues

– VM network is not reachable – No L2 level connectivity on Dest ESX host

• Resource overcommit

– Cannot allocate memory for a long time

– Swapping takes a long time leading to vMotion timeout

Common failures and patterns

vMotion Failures



vMotion process overview

Scaling vMotion performance for high speed links

• Multiple vmknics

• Scaling single vmknic

• Auto scaling vMotion to high speed links

Scaling long distance vMotion

• Tuning socket buffers

• Auto scaling vMotion to size socket buffer

Tuning vMotion concurrent limits

• Tuning vMotion network and storage costs

Key takeaways

Summary



[UX70007U] Managing workloads at scale

• Feedback about vMotion UX, bulk migration etc

Design Studio Session

Managing workloads at scale



ASSESSMENTLOUNGE

Moscone West Level 3 Lobby

Hyper-convergedInfrastructure (HCI)

vSphere OptimizationAssessment (VOA)

Virtual NetworkAssessment (VNA)

HOURS

Sunday, Aug. 25 8:00 AM – 6:00 PM

Monday, Aug. 26 10:30 AM – 6:30 PM *

Tuesday, Aug. 27 10:30 AM – 6:30 PM *

Wednesday, Aug. 29 8:00 AM – 5:00 PM

Thursday, Aug. 30 9:00 AM – 3:00 PM

Closed for keynote 9:00 AM -10:30 AM

vSphere AssessmentTool (vSAT)


how to get the most out of - geekboy.pro

Documents