Understanding VMware Capacity - Measuring Memory Capacity (6 of 10)

Today we'll take a look at measuring memory capacity, below is a typical graph we might use to understand how memory is being used by a VM. 

The VM has been granted about 4GB RAM. That’s what the OS thinks it has to use.

The host memory in use, shows how much physical RAM is currently being used by the VM. We can see that at 09:00 this increases and at the same time shared memory reduces.

Memory Used is our term for Active memory, and remains steady throughout, as does the Memory Overhead on the host used to support this VM. What we can see is that only about 400MB of memory is being accessed on a regular basis. Between 1.5GB and 2GB of memory is not unique to this VM and is shared with others.

Balloon Driver

The balloon driver takes about 2.5GB of space in the memory of the VM. This will cause the OS to swap out to disk until it cannot do any more, and then the hypervisor starts to swap out memory also. At that point, performance is likely to be impacted.

Here's an example of the balloon driver in use.

Where ballooning is persistent (rather than an occasional spike), then changes should be made in order to ensure there is enough RAM available for the VMs in the cluster. Ballooning itself has an overhead on the hypervisor and as such there is the potential to impact performance for the host. Changes don’t necessarily have to be more RAM installed. The very first thing to consider is if any VMs have been created and “forgotten” about. On Monday I'll take a look at Cluster memory.

Phil Bell

Consultant

Understanding VMware Capacity - Measuring Memory Capacity (5 of 10)

Memory capacity is generally the limiting factor in a VMware cluster. This is improving as hardware specifically designed to be a host has become available with plenty of RAM able  to be installed, but clusters still generally have less Memory headroom than CPU. Memory on VMware is not just a case of monitoring % used. Again like CPU we want to be able to compare memory utilizations in comparable metrics, so that’s MB or GB. Then we move on to the specific VMware type metrics.

Reservations

A VM or a resource pool can have a reservation. Which means if it needs that memory, it gets to use it - no questions asked. However you cannot reserve more memory than exists in the host server.

Limits. Just because a VM has been configured to have 8GB RAM, doesn’t mean you can’t set a lower limit of say 4GB RAM. At which point, if the VM tries to use more than the limit, then some data will be placed into the Swap file of the OS or VMware may balloon some of the VMs memory to free up pages that the OS can use.

Ballooning

When VMware wants to allocate memory to a VM, but there is a shortage of Memory, then a balloon may be inflated inside the memory of one (or more) VMs. This balloon pins itself into RAM and cannot be swapped out, thus forcing the OS to swap some memory out to disk. The pages pinned into memory are not all actually stored in memory on the Host, as it knows the contents of the balloon are not important. This then frees up pages, that the Host can allocate the VM it wanted to give memory too. One of the reasons this happens is that an OS will use a page to store some data, then later the program will no longer need that data, and the OS puts that page on the free list. However, the hypervisor has no idea the page is no longer needed. The data in the page has remained unchanged. In order to identify free pages when the balloon expands and the OS put it into the free pages, the hypervisor can

identify the balloon memory pages, and therefore the pages that the OS is not using for other processes.

Shared pages

Shared Pages are pages in memory that are identical. Rather than store duplicates of the same page, the hypervisor will store a single copy, and point the appropriate VMs to it. This works well where servers are running the same OS and doing the same job, and therefore much of the memory in use is identical.

Active Memory

Active memory is the amount of memory that has been accessed in the last 20 seconds. Having sufficient memory to contain the active memory of the VMs is crucial to performance.

Memory Available for VMs

Memory available to the VMs shows what it says. There is some memory in the cluster that VMs cannot use as it’s being used by the Hypervisor to support the VMs. The memory left over is available.

On Friday I'll talk about understanding how memory is being used by a VM. In the meantime don't forget to sign up for my next webinar 'Top 5 VMware Tips for Performance and Capacity'

https://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Phil Bell

Consultant

Virtualization Oversubscription - What’s so scary? Memory stats(10 of 20)

Let’s look at some stats for a single Virtual Machine

This is a 4GB VM, but it’s only accessing about 400 MB on a regular basis.  It’s got 2.6GB of memory that’s unique to itself, and 1.4GB that’s shared with other VMs.

So at least one other VM is likely to be sharing at about 1.4GB memory as well.  Given there are a lot of windows VMs in that cluster it’s likely a lot of them have similar amounts of shared memory.  If there are 10 VMs on that host then that’s about 15GB or RAM that you don’t have to have installed or rather, a few more VMs that will fit on the host.

There are also a couple of hours where the balloon driver steals some memory from the VM.  Only about 50MB and given the VMs only accessing 4 to 500MB of RAM, out of the 2.6GB that it’s using, the OS probably just released some cache to satisfy that request.

On Friday we'll be looking at Reservations and how they work.

Don't forget that registration is open for our next webinar 'Hardware's a commodity - Why bother managing capacity? http://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Phil Bell

Consultant

Virtualization Oversubscription - What’s so scary? Memory Oversubscription (6 of 20)

As stated in my previous blog having set out a few ground rules we can now look at memory oversubscription.

So how can we over-subscribe memory?

Free Space - Just like disks have a lot of free space, servers typically run with free space in memory. 

Then there are a number of tricks that the hypervisor can do to find even more savings.

·        Page Sharing (Deduplication)

·        Balloon Driver (VMware)

·        Reservations

·        Shares

We'll start with a look at:

Transparent Page Sharing

When two or more Virtual Machines have the same pages of data in memory, VMware can store a single copy and present it to all the VMs.  Should a VM alter a shared memory page, a copy will be created by VMware and presented to that VM.

An Example is shown in the graphic below:

VM1 starts and allocates some unique memory.

VM2 starts and allocates some unique memory.

VM1 allocates memory for a standard windows dll.

VM2 also allocates memory for the same standard windows dll.

VMware maps both systems memory to the same page in RAM.

On Wednesday I'll explain more about the VMware Balloon Driver.

In the meantime register for access to some of our great Resources, white papers, on-demand webinars and more.

http://www.metron-athene.com/_resources/login.asp

Phil Bell

Consultant

Virtualization Oversubscription - What’s so scary? VMware CPU and Memory Maximums (5 of 20)

CPU and Memory are the main items people consider for Virtualized systems, so let’s lay down the maximums.

•        Virtual Machine Maximum

–       128 vCPUs per VM

•        Host CPU maximums

–       Logical CPUs per host 480 (Logical CPUs being simultaneous threads so that might be 240 hyper-threaded cores.)

–       Virtual machines per host 1024

–       Virtual CPUs per host 4096

–       Virtual CPUs per core 32

•        Maximum with a caveat: The achievable number of vCPUs per core depends on the workload and specifics of the hardware. For more information, see the link below for the latest version of Performance Best Practices for VMware vSphere

https://www.vmware.com/pdf/vsphere6/r60/vsphere-60-configuration-maximums.pdf

This raises 2 points.

·        Clearly it’s ok to oversubscribe CPUs 

·        There is no set number to tell you how much oversubscription is OK.

Memory VMware Maximums

Memory is a lot simpler.

•        6TB per Host

–       Well 12TB on specific hardware

•        4TB per VM

Having set out those few ground rules we can now look at memory oversubscription and I'll be doing just that on Monday.

There are still a few places left on our VMware vSphere Capacity & Performance Essentials online workshop so don't forget to book your place.

http://www.metron-athene.com/services/online-workshops/capacity-management-workshops.html

Phil Bell

Consultant

Virtualization Oversubscription (What’s so scary?) - What can be oversubscribed? (4 of 20)

Today I'll deal with what can be oversubscribed.

What can be oversubscribed?

•        CPUs

•        Memory

•        Disk

•        NICs

In our virtual world, now that we have broken the link between the OS and the hardware, we can over provision all sorts of things.

CPU, Memory, Disk (as we mentioned) and NICs are all “Oversubscribed”.

Disk we have already looked at, Memory and CPU I’ll go into in more detail on later but I thought it was worth mentioning NICs here. 

Typically people seem to be running with 10 - 15 VMs on a single host which will have significantly fewer NICs installed. A Server typically wouldn’t use all the bandwidth of its NIC so that unused bandwidth is like the unused space on disk.

When the VMs talk to other VMs on the same Host that’s not generating traffic though the physical NICs, so we might consider that as the equivalent of de-duplication.

In the next few blogs I'll be looking at CPU and Memory.

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Cluster Memory

As I mentioned on Monday the next place to look at for memory issues is at the Cluster.

It is useful to look at:

–       Average memory usage of total memory available

–       Average amount of memory used by memory control

–       Average memory shared across the VM’s

–       Average swap space in use

In the graph below we can see that when the shared memory drops the individual memory usage increases.

In addition to that swapping and memory control increased at the same time.

If it isn’t memory that people are talking to us about then it's storage and that's what we'll take a look at on Friday.

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Monitoring Memory

Memory still seems to be the item that prompts most upgrades, with VM’s running out of memory before running out of vCPU.

It’s not just a question of how much of it is being used as there are different ways of monitoring it. Some of the things that you are going to need to consider are:

–       Reservations

–       Limits

–       Ballooning

–       Shared Pages

–       Active Memory

–       Memory Available for VMs

VM Memory Occupancy

In terms of occupancy the sorts of things that you will want to look at are:

–       Average Memory overhead

–       Average Memory used by the VM(active memory)

–       Average Memory shared

–       Average amount of host memory consumed by the VM

–       Average memory granted to the VM

In this instance we can see that the pink area is active memory and we can note that the average amount of host memory used by this VM increases at certain points in the chart.

VM Memory Performance

It's useful to produce a performance graph for memory where you can compare:

–       Average memory reclaimed

–       Average memory swapped

–       Memory limit

–       Memory reservation

–       Average amount of host memory consumed.

As illustrated below.

In this instance we can see that this particular VM had around 2.5gb of memory ‘stolen’ from it by the balloon driver (vmmemctrl), at the same time swapping was occurring and this could cause performance problems.

The next place to look at for memory issues is at the Cluster and I'll deal with this on Wednesday.

In the meantime don't forget to book your place on our VMware vSphere Capacity & Performance Essentials workshop taking place in December http://www.metron-athene.com/services/online-workshops/index.html

Phil Bell

Consultant

VMware Capacity Management

VMware is the go-to option for virtualization for many organizations, and has been for some time.
The longer it's been around, the more focus there is on making efficiency savings for the organization. This is where the Capacity Manager really needs to understand the technology, how to monitor it, and how to decide what headroom exists.

I'm running a VMware Capacity Management webinar this Wednesday May 25 (8am PDT,  9am MDT,  10am CDT,  11am EDT, 4pm UK,  5pm CEST) where I'll be taking a look at some of the key topics in understanding VMware Capacity.

Topics will include:

• Why OS monitoring can be misleading
• 5 Key Metrics
• Measuring Processor Capacity
• Measuring Memory Capacity
• Calculating Headroom in VMs

Register for your place now http://www.metron-athene.com/services/webinars/index.html

Look forward to seeing you there.

Dale Feiste

Principal Consultant

5 Top Performance and Capacity Concerns for VMware - Monitoring Memory

Memory still seems to be the item that prompts most upgrades, with VM’s running out of memory before running out of vCPU.

It’s not just a question of how much of it is being used as there are different ways of monitoring it. Some of the things that you are going to need to consider are:

–       Reservations

–       Limits

–       Ballooning

–       Shared Pages

–       Active Memory

–       Memory Available for VMs

VM Memory Occupancy

In terms of occupancy the sorts of things that you will want to look at are:

–       Average Memory overhead

–       Average Memory used by the VM(active memory)

–       Average Memory shared

–       Average amount of host memory consumed by the VM

–       Average memory granted to the VM

In this instance we can see that the pink area is active memory and we can note that the average amount of host memory used by this VM increases at certain points in the chart.

VM Memory Performance

It is useful to produce a performance graph for memory where you can compare:

–       Average memory reclaimed

–       Average memory swapped

–       Memory limit

–       Memory reservation

–       Average amount of host memory consumed.

As illustrated below.

In this instance we can see that this particular VM had around 2.5gb of memory ‘stolen’ from it by the balloon driver (vmmemctrl), at the same time swapping was occurring and this could cause performance problems.

Cluster Memory

The next place to look at for memory issues is at the Cluster.

It is useful to look at:

–       Average memory usage of total memory available

–       Average amount of memory used by memory control

–       Average memory shared across the VM’s

–       Average swap space in use

In the graph below we can see that when the shared memory drops the individual memory usage increases. 

In addition to that swapping and memory control increased at the same time.

On Friday join me as I discuss storage latency. In the meantime we've got some great white papers and webinars on VMware Capacity Management - join our Community and get free access to them http://www.metron-athene.com/_resources/index.html

Phil Bell

Consultant