five practices to optimize vmware capacity

© 2009 Systar, Inc.

http://www.systar.com/solutions/virtualization_management

1

Managing Capacity in VMware® Environments

Part II – Five Practices that will Optimize your VMware

Capacity and Result in a Standing Ovation

May 2009



2

Executive Summary

Nearly every large enterprise IT shop has virtualized some portion of their infrastructure. By

producing nearly 30% hardware cost savings annually, their IT executives are applauding the

initial success. But, can they do better?

While new virtualization platforms proved to be stable, and consolidation of physical resources

was achieved during the initial deployments, the reality of the situation is that VMware

infrastructure remains bloated. Systar has spoken with hundreds of IT executives, virtualization

architects and capacity managers at leading enterprises who have all admitted to only achieving

post-virtualization capacity utilization rates of 10 – 20%, while their objectives were to safely

reach 50% - 60%. Reaching the >50% objective cannot be accomplished with default VMware

settings, and requires a new understanding of virtualized capacity.

In this paper, Systar explores five practices for optimizing the utilization of virtualized capacity

without increasing risk to service quality. The practices are:

• Managing at the cluster level

• Sizing objects correctly

• Placing workloads carefully

• Optimizing DRS

• Minimizing impact of HA

As companies prepare to expand their virtualized environments, Systar sees an opportunity for

IT organizations to improve their understanding of virtualized capacity, reduce new hardware

spending significantly, and meet their utilization objectives safely. By applying the practices

discussed in this paper, Systar sees the initial round of applause transforming into a standing

ovation as virtualization expands across the enterprise while safely meeting the >50% objective.



3

Executive Summary............................................................................................................. 2

Improve Quality and Contain Costs .................................................................................... 4

The Many Dimensions of VMware Capacity....................................................................... 4

Managing at the Cluster Level ............................................................................................ 5

Sizing Objects Correctly ...................................................................................................... 7

Placing Workloads............................................................................................................... 9

Optimizing DRS.................................................................................................................. 11

Minimizing Impact of High Availability ............................................................................. 12

Summary ........................................................................................................................... 13

Glossary............................................................................................................................. 14



4

Improve Quality and Contain Costs

The vast majority of today’s VMware infrastructure capacity is bloated. Where physical capacity

was over-provisioned on average by a factor of 10, consolidation to virtualized environments

has reduced the footprint of IT infrastructure, but not optimized computing capacity. On

average, virtualized capacity is over-provisioned by a factor of 4 including peak headroom –

representing millions of dollars in over-spending as corporate IT budgets continue to tighten.

A proven way to combat over-provisioning and reduce unnecessary virtualization expenditures

is through improved capacity management practices. According to a Forrester report titled The

Capacity Planning Software Market1, “As data centers struggle with server consolidation and

server virtualization, capacity planning becomes the key to maintaining or improving service

quality while containing costs”.

In theory, optimizing the utilization of IT infrastructure capacity via consolidation to virtualized

environments is easy, but in practice it can be difficult without the right information. In order to

maximize the utilization of virtualized infrastructure, the proper visibility, control, processes and

expertise are needed. Yet each path to optimizing virtual capacity is met with its own challenge.

For example, IT organizations must:

• Maximize the aggregate utilization of virtualized resources, but respect the

tolerance for risk of the organization

• Take advantage of the advanced capabilities of the virtualized infrastructure, but be

able to quantify its available effective capacity, workload patterns, migration

tendency and patterns, and HA capability

• Maintain the flexibility to respond to unexpected workload variances with virtual

machine migrations, but strive to minimize migrations through careful placement

• Take advantage of cost-effective failover capabilities, but minimize capacity

dedicated to support failovers

• Establish processes to automate the efforts mentioned above ensuring their burden

does not outweigh their benefit as the virtualized environment scales

The Many Dimensions of VMware Capacity

Virtualization changes many IT functions but it changes capacity management more than most.

Where capacity in the physical world often focused on a single machine hosting a single

application, VMware clusters – made up of ESX Hosts and Virtual Machines – are the new

“computer” and capacity must be managed accordingly. VMware’s CTO, Steven Herrod,

recently pointed to this notion when he commented that “virtualization is the mainframe for the

21st

century”.2

1 The Capacity Planning Software Market: Sustaining Application Performance by Evelyn Hubbert and Jean-Pierre

Garbani with Thomas Mendel, Ph.D. 2 VMware’s vSphere Introduction, Conference Call (2009)



5

Understanding capacity of VMware environments, starting at the cluster level, is a

multidimensional concept. Cluster capacity is first determined by the size and number of ESX

Hosts belonging to the cluster. Each ESX Host will support a number of VMs that have sizing

properties like Reservations and calculated Entitlements. Each Reservation should account for

the typical workload including some headroom for demand spikes. Next, we need to

understand potential workload resource utilization contention on each ESX Host due to its

resource sharing between all VMs in addition to the hypervisor and whitespace3. Then, as

contention for resources builds, load balancing and policy enforcement performed by VMware’s

Distributed Resource Scheduler (DRS) can migrate VMs from one host to another within the

cluster. Finally and of equal importance, cluster capacity needs to account for headroom

requirements of High Availability (if enabled).

Although VMware capacity is complex in nature, optimizing capacity utilization in these

environments does not have to be difficult. In fact, Systar has successfully led and enabled

many organizations through optimizing their virtualized capacity. Based on many years of

experience in optimizing physical and virtualized infrastructure, Systar recommends increasing

your understanding and adoption of five proven practices:

• Managing at the cluster level

• Sizing objects correctly

• Placing workloads carefully

• Optimizing DRS

• Minimizing impact of HA

Let’s explore some of the multidimensional aspects of cluster capacity while discussing practices

that can be applied to improve their management.

Managing at the Cluster Level

During the initial rollout of VMware more concern is typically placed on sizing VMs correctly and

placing workloads carefully, but the approach taken to achieve this is often rudimentary. We

have spoken with many organizations that ignore proper VM sizing and workload placement

methods in favor of placing common Operating Systems on the same machine and using a core-

to-VM ratio rule.

As deployments expand across the enterprise and DRS and HA enabled clusters enter into the

equation, managing capacity of the cluster becomes paramount. This sentiment is echoed in

Gartner, Inc.’s Data Center Conference Survey4, which states “In the long term, Gartner believes

that capacity-planning tools and processes will have to shift their orientation to focusing less on

a single VM or physical server to assist with the sizing of resource pools and clusters.”

3 Whitespace: capacity on a host that cannot be utilized due to alignment of resource requirements, or resources

that cannot be used since they are too small support a whole VM. 4 Data Center Conference Survey: Addressing the Operational Challenges of Virtual Server Management, by Cameron

Haight (February 2008)



6

Balancing workload among hosts is an important aspect of managing cluster capacity. If the

load of an individual host is too high, a DRS enabled cluster will try to rebalance that load.

Although this automated load balancing is available, it does create overhead in the

environment. Simply relying on DRS to balance all workloads, all of the time, will lead to

unnecessary migration overhead in the system and is not considered a best practice. By

monitoring migrations and VM tracks, you can begin to determine if excess balancing is

occurring or if VMs are wandering around like lost souls. With careful balancing of loads

upfront, you can preempt capacity loss as a result of excess migrations.

Next, we will look at Entitlements5. The general measure of the capacity and health of a cluster

is the ability of the cluster to deliver the entitled resources to all VMs. If VM Entitlements total

more than the total capacity of the cluster, the cluster is undersized or improperly balanced.

Within the cluster itself, a good measure of a host’s ability to provide expected capacity is the

measure of its total Entitlements vs. total capacity. Unlike Reservations6, the cluster and its

hosts will not identify a violation when Entitlements exceed capacity.

By understanding the sum of all VM Entitlements on a host and within a cluster, VMware

architects and administrators will have a clear picture of the resources being made available to

meet demands on their capacity.

Another practice to consider is the calculation of target headroom7 within a cluster plus its high

availability (HA) failover capacity. The sum of these elements can be used to establish an

“effective capacity” of the cluster. The image below describes the effective capacity of the

cluster in terms of the percentage of CPU and memory available.

Figure 1. Systar’s OmniVision VMware Capacity Cluster Reports automatically

calculate effective capacity of clusters by percentage, normalized MIPS, and number

of CPUs. Daily, weekly and monthly report perspectives are available.

5 Entitlements are the computed result of configurations, reservations, limits, and shares used to establish the

resource allocation given to each VM for its operation. The Entitlement will always fall between the Reservation and

the Limit, based upon its Share. 6 A Reservation is the amount of vCPU and memory (in absolute units) that a VM is guaranteed should it need it.

7 Target headroom = demand spike headroom (which depends on workload profiles and risk tolerance + whitespace

(5% for large hosts, 10% for smaller ones); then, add in HA space of 15% per host in an 8-host cluster.



7

Effective capacity is only one measure of the cluster that should be considered. Another

recommended element of this practice is to not only measure the available capacity of the

cluster in terms of resources like CPU and memory, but also in terms of VMs that can be added.

It is important to understand that raw computing resource sizes do not take into account

whitespace sizes (resources that cannot be used since they are too small support a whole VM).

When calculating the number of VMs that can be added to a cluster, it is important to first

define differing VM template sizes (e.g., small, medium and large). The definition can start with

a simple average VM reservation for each resource per template size. For a more accurate

picture of where VMs should be placed within a cluster, you can expand the calculation to

consider maximum and minimum VM sizes and workload type such as sustained or peaky.

The diagram below provides a calculated assessment of how many medium-sized VMs can be

added to each cluster on the basis of CPU and memory requirements for that VM template.

Figure 2. Systar’s OmniVision VMware Capacity Cluster Reports have calculated

that on the basis of CPU requirements 23 additional medium-sized VMs can be

added to cluster “clsioub19”. The same cluster shows available memory for 53 new

VMs. When determining the number of new VMs that can be hosted, it is

recommended to use the lesser of the Effective CPU and memory metrics as a

guide.

Sizing Objects Correctly

The next practice we will explore is the importance of sizing objects correctly. The key to

successful VMware capacity optimization is to set the Reservation property for each VM. The

Reservation determines the amount of resources that a VM can receive before it begins

competing with other VMs for the remaining shared resources. The Reservation also

determines the size of the VMKernel swap file for the VM’s memory and impacts HA and DRS

calculations.

Reservations are used by VMware Admission Control to prevent placing too many VMs on a host

based on resources. A VM can only be powered on if there are adequate unreserved resources

available on the host to satisfy that VM’s Reservation requirement. If all VMs have the default

Reservation setting of zero, then there is no effective Admission Control and VMs can be loaded



8

on hosts until overhead space finally runs out. This approach totally defeats Admission Control

and makes it very hard for both the VMKernel and DRS to manage resources optimally.

When HA is enabled, Reservations are used to calculate the amount of space (slots) needed to

meet the Failover Level Policy in effect. HA will calculate the maximum of all Reservations and

then based on the Failover Level Policy in effect, it will set aside space to provide the designated

number of VMs with sufficient capacity to operate if trouble strikes. Without Reservations in

place, HA must use an input parameter that defaults to 256MHz/256MB slots sizes, which is not

optimal.

Reservations are also used as one of the triggers for DRS VM migrations at Periodic Invocation

time (along with load balancing and other mandatory moves). If the sum of the Reservations on

a host exceeds its capacity then a VM is selected for migration to correct the situation.

Now that we have established the importance of Reservations when managing capacity, we will

provide some guidance on selecting its correct size. From our experience in assisting large

organizations with their VMware environments, a common practice is to set the Reservation in

the range of the 45th to 60th percentile for the VM’s historical resource utilization. Beyond this

setting, allow the VM to compete with other VMs based on Shares and Entitlements during

periods of greater load.

Reservation too low

VM Workload Profiles

Reservation too high

Reservation low side Reservation high side

Reservation too low

VM Workload Profiles

Reservation too high

Reservation low side Reservation high side

Figure 3. Systar’s OmniVision Workload Profile Reports display minimum (light blue), average

(blue) and peak (orange) resource usage hour-by-hour for CPU, I/O and Memory. The measures

are calculated from data collected every 15 seconds. Profiles are available of daily, multi-day,

weekly and monthly views. Average and maximum usage profiles can be used to accurately

assign VMware Reservations.



9

Once you have established the Reservations and Shares (Entitlements are established indirectly

based on total capacity, number of VMs, Limits and Shares) you need to track them against the

VM’s resource utilization over time. Usage should stay below the Reservation 50-60% of the

time and below the Entitlement 90% of the time or more. Where utilization is not matched well

to these settings, you can improve service delivery by providing more resources or reclaim

capacity by assigning fewer resources to the VM.

Once the reservations have been set, there is an opportunity to set Shares that represent the

VM’s business priority. Shares will determine resource Entitlement which the VMKernel will try

to provide when needed (e.g., handling peak workload periods).

Placing Workloads

The third practice to optimize your VMware capacity is placing workloads effectively. Capacity is

determined to some extent by how well a workload behaves with other workloads in the shared

resource environment.

If a number of workloads peak at similar times within an ESX Host, the result may be degraded

service or restricted capacity for other workloads needing access to the remaining resources.

Multiple peaking workloads can be more troublesome if their behavior pattern is unpredictable,

making them more challenging to manage. Peaky workloads like user-generated transactions

must be studied more carefully to see which match up best for resource sharing. On the other

hand, sustained workloads such as batch can be safely stacked, resulting in high capacity

utilization, because their peak resource requirements are well known. It would make sense, for

example, to place transaction workloads that peak during working hours with batch processes

that run throughout the night to provide a balanced use of the resources available. To

accomplish this type of safe-stacking requires a keen understanding of workload behaviors,

including average and peak usage, over a period of time. Most experts would request

monitoring of the workload behaviors for a minimum of one month. Of course, placement

accuracy will increase by gathering additional workload data points over an extended period of

time.

A good rule is to produce a stacked chart of all workloads targeted for a resource (e.g., CPU,

memory) belonging to the ESX Host. Flatter curves or sets of bars over time indicate a better

workload fit. A highly variable curve or set of bars means that the peaks may be coinciding.

Coinciding workloads increase the risk of resource contention, or wasted resources, and the

need to separate those workloads on different hosts or clusters.



10

Figure 4. Systar’s OmniVision Workload Profile Reports utilize pie and stacked bar

charts to represent the combined behavior of multiple VMs running on a host. The

stacked bar chart in the top-right corner shows a period between 12pm and 2pm

where multiple workloads are peaking at the same time and contention for

resources may be affecting quality of service as response times slow. Additionally,

the chart shows that up to 6 CPUs are under-utilized 22 hours of the day.

Another benefit of careful workload placement is to minimize DRS migrations. Although DRS

can automatically place a new VM and load balance existing ones, it only considers overall

resource utilization of the hosts and not workload profiles. Therefore, if a VM is migrated to a

host where workloads peak shortly after the move, DRS will trigger another migration. For

example, the figure below shows a stacked bar chart of VM Memory Resource use on an ESX

host. Each color represents memory use of a different VM. As you can see, memory use

declines on the Host from 2am to 1pm and then peaks. DRS could easily migrate a memory

intensive VM to this Host at 11am, but would then have to migrate it once again at 2pm if

memory contention reaches an unacceptable level. When workloads like this are not placed

effectively, it can result in continually wandering VMs.

Figure 5. Systar’s OmniVision shows a stacked bar of VM memory usage

for an ESX Host over a one-day period.



11

When considering placement of critical applications, we suggest placing the workload manually

in order to ensure the best fit with minimal migrations. Although the dynamic nature of

VMware workload migrations theoretically provides greater availability for critical VMs, best

practices recommend minimal shifting of these workloads. You may even want to set the VM to

manual migration for very critical workloads.

DRS migrations via vMotion are very efficient but still require some overhead. DRS migrations

should not be approached haphazardly (e.g., stacking VMs at random and letting DRS determine

how to best balance the workloads). Peaky workloads that are not placed effectively may result

in excessive migrations, causing what has come to be known as “vMotion sickness”. In general,

setting anti-affinity rules within VMware is not recommended, but can be very helpful for

workloads that are variable and may peak simultaneously.

Matching workload patterns is one of many considerations when stacking VMs on a host.

Affinity rules, geographical constraints, organizational alignment, compliance issues and other

factors will play into best practice guidelines for where workloads are permitted to be placed.

Optimizing DRS

As a reminder from Part I of this white paper series, Distributed Resource Scheduling (DRS)

provides a watchful eye over VMs in clustered environments. With an intention to provide each

VM its required resources, DRS observes resource utilization on each host within a cluster.

When unsatisfactory conditions are observed within one host, DRS assesses other hosts within

the cluster where conditions may be more attractive. If DRS finds a suitable location, it then

facilitates a VM move known as a VM migration.

Our fourth practice points to the need to optimize DRS. Not all application environments need

or are suited to its workload balancing features. For some sets of applications (VMs) it makes

sense not to use DRS. For example, horizontally scaled applications like web servers are already

load balanced.

In other instances, the cluster may be hosting hundreds of tier 2, non-critical business

applications that roughly demonstrate the same resource consumption. This environment may

be best suited to setting DRS to “Auto” and the default level to “Aggressive” for all VMs. Auto

settings allow for the initial placement of a VM inside the cluster to be automated and the

automatic execution of migration recommendations. Aggressive migration thresholds will

trigger movements that promise even a slight improvement in the cluster’s load balance.

Where sets of workloads take on the opposite profile of the example above - becoming less

homogeneous and more critical in nature - you will want to consider less aggressive migration

thresholds and either partially-automated or manual placement and migration settings.

The vCenter screen shot below shows a DRS enabled cluster with 4 ESX Hosts, of which only 2

are active. These systems are hosting 36 VMs and show 190 migrations. In this instance,

workloads are clearly not balanced properly and excessive migrations are occurring.



12

Figure 6. VMware’s vCenter shows 4 hosts, of which 2 are active. In this case,

190 Migrations were observed. (Source: VMware)

As we discussed above in the Placing Workloads section, accomplishing safe-stacking requires a

keen understanding of workload behaviors, including average and peak usage, over a period of

time. The more variable the workload, the harder it is to determine proper placement for new

VMs. If DRS migrations are occurring frequently, the workload profiles of new VMs will be

difficult to match to existing VMs on a host. Based on our extensive experience in profiling

workloads and recommending VM placement, the goal of the optimizing DRS practice is to

minimize migrations and use the capability as a last resort.

Minimizing Impact of High Availability

Our final practice is centered on high availability (HA) within DRS-enabled clusters. HA is a very

cost-effective capability built into the DRS-enabled cluster. However, there is a tradeoff and its

cost is twofold:

• HA’s strict Admission Control is very conservative and wastes a great deal of capacity

• It is difficult to understand whether an application can be restarted in a given cluster

state

The current method of calculating VMware’s HA failover capacity is complicated (and too

lengthy to share here). However, in short, HA’s strict Admission Control uses the maximum

Reservation size in the cluster as a slot size for all calculations. In fact, many users report seeing

a message that there are “Insufficient resources to satisfy configured failover level for HA”,

when attempting to configure their HA environment.

Many sites we talk to limit resource utilization far below what they might need to restart the

critical VMs on a host. And many of these same sites do not set their HA restart priority. We

recommend setting and optimizing the restart priority around two points: minimizing capacity

loss, and ensuring critical VMs restart immediately while low priority VMs restart when possible.



13

In order to minimize capacity loss from poor HA configurations, we recommend the following

approaches:

• Turn off strict HA Admission Control; VMware admits this is a very conservative

approach (i.e., it wastes a lot of capacity).

• Set the restart priority of all VMs very carefully, usually according to the policies defined

in your DR plan.

• Take the max of the sum of the Reservations for your High Priority VMs on any host.

Subtract that sum plus an additional 5% for overhead from the aggregate capacity.

Subtract the remainder from your required headroom which is based on the peakiness

of the workloads and your risk tolerance. Manage the cluster to the remaining

“effective capacity”.

VMware is planning to change its HA approach in ESX 4, and once we have sufficient experience

with that release, this section of the paper will be revised.

Summary

As your VMware environment continues to expand and the pressure to reduce costs continues

to increase, applying the five practices recommended in this paper will provide greater control

over new virtualization spending and improve the quality of services delivered. Systar is

confident that by following these practices, your organization will be able to safely maximize the

utilization of your VMware capacity above the 50% mark.

Applying virtualization-aware management solutions, processes, and best practices is key to

achieving results that deliver standing ovations. Optimizing capacity utilization is not only a

great practice, but can result in substantial savings. According to a recent IDC report8, “an

optimally managed or ‘advanced virtualization’ infrastructure (described as an infrastructure

that includes penetration of virtualized servers of more than 25%, storage virtualization, and the

use of systems management tools) can deliver a total [cost] reduction of up to 52% per user per

year”.

Your standing ovation awaits.

8 Business Value of Virtualization: Realizing the Benefit of Integrated Solutions, July 2008



14

Glossary

The concepts in this paper apply to most virtual environments – however, we use VMware VI 3.x to

illustrate our points.

• Admission Control – if on, will not allow the power on of a new VM on a host if there is not enough

unreserved resources available for the VM’s specified reservation

• Workload – the basic unit of work, a Virtual Machine (VM).

• Capacity – the aggregate capacity of a host or cluster.

• Cluster – a group of hosts managed as an aggregate computing resource, VMware DRS-enabled

cluster

• Configured size – the number of vCPUs and the number of MB of memory that represent the size of

the physical machine the VM is presented with.

• Entitlement - the computed result of configurations, reservations, limits, and shares used to establish

the resource allocation given to each VM for its operation. The Entitlement will always fall between

the Reservation and the Limit, based upon its Share.

• Effective capacity – the amount of that capacity that can be used given workload mix, high availability

requirements and white space.

• Host – a server that supports multiple workloads and the ESX server.

• Limit – this property serves as a hard cap on resource allocation for a VM. If Limit is not specified,

then the configuration size is the Limit.

• Reservation – the amount of CPU MHz and MB of memory (in absolute units) that a VM is guaranteed

should it need it.

• Shared resources CPU and memory resources that are actively managed by the hypervisor

• Shares – relative units that determine a VM’s priority among sibling VMs, used to determine resource

allocation under contention.

• White space – capacity on a host that cannot be utilized due to alignment of resource requirements,

or resources that cannot be used since they are too small support a whole VM.



15

About Systar

Systar is a leading worldwide provider of performance management software. Systar’s OmniVision

product suite enables customers to achieve the optimal alignment between IT resources and business

requirements in both distributed and virtualized server environments. Systar’s proven capacity

management solutions deliver the full benefits of virtualization by enabling customers to gain visibility

into these complex environments, tune for optimal capacity, and move business-critical applications into

production with full confidence.

United States

8618 Westwood Center Dr.

Suite 240

Vienna, VA 22182

Tel. +1 703-556-8400

Fax +1 703-556-8430

[email protected]

France

171 bureaux de la Colline

92213 Saint-Cloud Cedex

Tel. +33 (0) 1 49 11 45 00

Fax +33 (0) 1 49 11 45 45

[email protected]

United Kingdom

Ground Floor Left

3 Dyer’s Buildings

London EC1N 2JT

Tel. +44 2072 692 799

Fax +44 2072 429 400

[email protected]

Germany

Mergenthallerallee 79-81

D-65760 Eschborn

Tel. +49 211 598 8520

[email protected]

Spain

Centro de Negocios Eisenhower

C/ Cañada Real de las Merinas, 17

Edificio 5 - 1º D

28042 Madrid

Tel. +34 91 747 88 64

Fax +34 91 747 54 35

[email protected]

Systar, BusinessBridge, OmniVision, BusinessVision, ServiceVision, WideVision and Systar’s logo are registered

trademarks of Systar. All other brand names, product names and trademarks are the property of their respective

owners. Copyright 2009.

five practices to optimize vmware capacity

Documents