Virtualization Oversubscription - What’s so scary? Memory Oversubscription (6 of 20)

As stated in my previous blog having set out a few ground rules we can now look at memory oversubscription.

So how can we over-subscribe memory?

Free Space - Just like disks have a lot of free space, servers typically run with free space in memory. 

Then there are a number of tricks that the hypervisor can do to find even more savings.

·        Page Sharing (Deduplication)

·        Balloon Driver (VMware)

·        Reservations

·        Shares

We'll start with a look at:

Transparent Page Sharing

When two or more Virtual Machines have the same pages of data in memory, VMware can store a single copy and present it to all the VMs.  Should a VM alter a shared memory page, a copy will be created by VMware and presented to that VM.

An Example is shown in the graphic below:

VM1 starts and allocates some unique memory.

VM2 starts and allocates some unique memory.

VM1 allocates memory for a standard windows dll.

VM2 also allocates memory for the same standard windows dll.

VMware maps both systems memory to the same page in RAM.

On Wednesday I'll explain more about the VMware Balloon Driver.

In the meantime register for access to some of our great Resources, white papers, on-demand webinars and more.

http://www.metron-athene.com/_resources/login.asp

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Cluster Memory

As I mentioned on Monday the next place to look at for memory issues is at the Cluster.

It is useful to look at:

–       Average memory usage of total memory available

–       Average amount of memory used by memory control

–       Average memory shared across the VM’s

–       Average swap space in use

In the graph below we can see that when the shared memory drops the individual memory usage increases.

In addition to that swapping and memory control increased at the same time.

If it isn’t memory that people are talking to us about then it's storage and that's what we'll take a look at on Friday.

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Monitoring Memory

Memory still seems to be the item that prompts most upgrades, with VM’s running out of memory before running out of vCPU.

It’s not just a question of how much of it is being used as there are different ways of monitoring it. Some of the things that you are going to need to consider are:

–       Reservations

–       Limits

–       Ballooning

–       Shared Pages

–       Active Memory

–       Memory Available for VMs

VM Memory Occupancy

In terms of occupancy the sorts of things that you will want to look at are:

–       Average Memory overhead

–       Average Memory used by the VM(active memory)

–       Average Memory shared

–       Average amount of host memory consumed by the VM

–       Average memory granted to the VM

In this instance we can see that the pink area is active memory and we can note that the average amount of host memory used by this VM increases at certain points in the chart.

VM Memory Performance

It's useful to produce a performance graph for memory where you can compare:

–       Average memory reclaimed

–       Average memory swapped

–       Memory limit

–       Memory reservation

–       Average amount of host memory consumed.

As illustrated below.

In this instance we can see that this particular VM had around 2.5gb of memory ‘stolen’ from it by the balloon driver (vmmemctrl), at the same time swapping was occurring and this could cause performance problems.

The next place to look at for memory issues is at the Cluster and I'll deal with this on Wednesday.

In the meantime don't forget to book your place on our VMware vSphere Capacity & Performance Essentials workshop taking place in December http://www.metron-athene.com/services/online-workshops/index.html

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Ready Time

As I mentioned on Wednesday there are 3 states which the VM can be in:

Threads – being processed and allocated to a thread.

Ready – in a ready state where they wish to process but aren’t able to.

Idle – where they exist but don’t need to be doing anything at this time.

In the diagram below you can see that work has moved over the threads to be processed and there is some available headroom. Work that is waiting to be processed requires 2 CPU’s so is unable to fit and creates wasted space that we are unable to use at this time.

We need to remove a VM before we can put a 2 CPU VM on to a thread and remain 100% busy.

In the meantime other VM’s are coming along and we now have a 4vCPU VM accumulating Ready Time.

2 VM’s moves off but the 4vCPU VM waiting cannot move on as there are not enough vCPU’s available.

It has to wait and other work moves ahead of it to process.

Even when 3vCPU’s are available it is still unable to process and will be ‘queue jumped’ by other VM’s who require less vCPU’s.

Hopefully that is a clear illustration of why it makes sense to reduce contention by having as few vCPUs as possible in each VM.

Ready Time impacts on performance and needs to be monitored. On Monday I'll be dealing with Monitoring Memory.

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Ready Time

Imagine you are driving a car, and you are stationary, there could be several reasons for this.  You may be waiting to pick someone up, you may have stopped to take a phone call, or it might be that you have stopped at a red light.  The first two of these (pick up, phone) you have decided to stop the car to perform a task.  In the third instance the red light is stopping you doing something you want to do.  In fact you spend the whole time at the red light ready to move away as soon as the light turns to green.  That time is ready time.

When a VM wants to use the processor, but is stopped from doing so.  It accumulates ready time and this has a direct impact on performance.

For any processing to happen all the vCPUs assigned to the VM must be running at the same time.  This means if you have a 4 vCPU all 4 need available cores or hyperthreads to run.  So the fewer vCPUs a VM has, the more likely it is to be able to get onto the processors. 

To avoid Ready Time

You can reduce contention by having as few vCPUs as possible in each VM.  If you monitor CPU Threads, vCPUs and Ready Time you’ll be able to see if there is a correlation between increasing vCPU numbers and Ready Time in your systems.

Proportion of Time: 4 vCPU VM

Below is an example of a 4vCPU VM, each doing about 500 seconds worth of real CPU time and about a 1000’s worth of Ready Time.

For every 1 second of processing the VM is waiting around 2 seconds to process, so it’s spending almost twice as long to process than it is processing. This is going to impact on the performance being experienced by the end user who is reliant on this VM.

Now let’s compare that to the proportion of time spent processing on a 2 vCPU VM. The graph below shows a 2 vCPU VM doing the same amount of work, around 500

seconds worth of real CPU time and as you can see the Ready Time is significantly less.

There are 3 states which the VM can be in and we'll take a look at these on Friday.

Don't forget to book on to our VMware vSphere Capacity & Performance Essentials workshop starting on Dec 6 http://www.metron-athene.com/services/online-workshops/index.html

Phil Bell

Consultant

5 Top Performance and Capacity Concerns for VMware - Time Slicing

As I mentioned on Friday the large difference between what the OS thinks is happening and what is really happening all comes down to time slicing.

In a typical VMware host we have more vCPUs assigned to VMs than we do physical cores. 

The processing time of the cores has to be shared among the vCPUs. Cores are shared between vCPUs in time slices, 1 vCPU to 1 core at any point in time.

More vCPUs lead to more time slicing. The more vCPUs we have the less time each can be on the core, and therefore the slower time passes for that VM.  To keep the VM in time extra time interrupts are sent in quick succession.  So time passes slowly and then very fast.

More time slicing equals less accurate data from the OS.

Anything that doesn’t relate to time, such as disc occupancy should be ok to use.

On Wednesday I'll be dealing with Ready Time, you've still got time to register to come along to my webinar 'VMware and Hyper-V Virtualization over-subscription(What's so scary?) taking place on October 12. http://www.metron-athene.com/services/webinars/index.html

Phil Bell

Consultant

VMware Capacity Management

VMware is the go-to option for virtualization for many organizations, and has been for some time.
The longer it's been around, the more focus there is on making efficiency savings for the organization. This is where the Capacity Manager really needs to understand the technology, how to monitor it, and how to decide what headroom exists.

I'm running a VMware Capacity Management webinar this Wednesday May 25 (8am PDT,  9am MDT,  10am CDT,  11am EDT, 4pm UK,  5pm CEST) where I'll be taking a look at some of the key topics in understanding VMware Capacity.

Topics will include:

• Why OS monitoring can be misleading
• 5 Key Metrics
• Measuring Processor Capacity
• Measuring Memory Capacity
• Calculating Headroom in VMs

Register for your place now http://www.metron-athene.com/services/webinars/index.html

Look forward to seeing you there.

Dale Feiste

Principal Consultant