Virtualization Oversubscription - What’s so scary? 18 of 20

Happy New Year!

If you’ve been following my series then you’ll know that just before the Holidays I said that I’d deal with what’s the worst that can happen when you oversubscribe.

So what’s the worst that can happen?

Well if you push things too far, all those things that the Hypervisor can do to try and keep things running will eventually be overwhelmed.

If you try to use too much memory you’ll start to see ballooning on a consistent basis, then swapping.  At that point performance will degrade rapidly.  Watch active memory values and take ballooning increasing as the indication things are getting tight.

CPU is as always a more gentle decay in performance.  CPU also has it’s indicators that the limits are being approached.  CPU Ready and Co-Stop are indicators that VMs are finding it tricky to find CPUs when they want to do some processing.

The reason CPU degrades differently to Memory is that it’s used differently.  A process is in memory all the time, but only uses a CPU when it needs so CPU busy is dictated by how frequently the CPU is required and for how long.  The performance of a transaction will be dictated by the ‘chance’ that a CPU will not be available when the transaction arrives.  If all the CPUs are busy it’ll enter a queue and this is where queueing theory comes in.

Contention and Queuing

Any system has a finite set of resources.  If you only have a single user trying to use one workstation then there is no contention for the use of that workstation.  As soon as you have more than one user then there is a chance that they will want to use the workstation at the same time.  That’s contention.  It’s perfectly normal and happens inside every OS all the time.  There are lots more process threads than there are CPUs, and when there is contention, then the processes queue.  Poor performance only occurs when queueing becomes excessive.

On Friday I'll go in to more detail about the basic ideas of queuing. In the meantime register for our first webinar of 2017 'Performance Management made easy'

http://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Phil Bell

Consultant

Virtualization Oversubscription - What’s so scary? VMWare vCPU Co-Scheduling & Ready Time (14 of 20)

Today I’ll explain the effect of what is happening inside the host to schedule the physical CPUs/cores to the vCPUs of the VMs.  Clearly most hosts have more than 4 consecutive threads that can be processed but let’s keep this simple to follow.

·        VMs that are “ready” are moved onto the Threads.

·        There is not enough space for all the vCPUs in all the VMs so some are left behind.  (CPU Utilization = 75%, capacity used = 100%)

·        If a single vCPU VM finishes processing, the spare Threads can now be used to process a 2 vCPU VM. (CPU Utilization = 100%)

·        A 4 vCPU VM needs to process.

·        Even if the 2 single vCPU VMs finish processing, the 4 vCPU VM cannot use the CPU available and while it’s accumulating Ready Time, other single vCPU VMs are able to take advantage of the available Threads

·        Even if we end up in a situation where only a single vCPU is being used, the 4 vCPU VM cannot do any processing. (CPU Utilization = 25%)

As mentioned when we discussed time slicing, improvements have been made in the area of co-scheduling with each release of VMware.  Among other things the time between individual CPUs being scheduled onto the physical CPUs has increased, allowing for greater flexibility in scheduling VMs with large number of vCPUs.  Acceptable performance is seen from larger VMs.

Along with Ready Time, there is also a Co-Stop metric.  Ready Time can be accumulated against any VM.  Co-Stop is specific to VMs with 2 or more vCPUs and relates to the time “stopped” due to Co-Scheduling contention.  E.g. One or more vCPUs has been allocated a physical CPU, but we are stopped waiting on other vCPUs to be scheduled.

Imagine the bottom of a “ready” VM displayed, sliding across to a thread and the top sliding across as other VMs move off the Threads, so the VM is no longer rigid it’s more of an elastic band.  

VMs and Resource Pools can be allocated Reservations, Shares and Limits and I'll be taking a look at these on Monday.

If you haven't already done so don't forget to sign up to get free access to our Resources, there are some great VMware white papers and on-demand webinars on there.

http://www.metron-athene.com/resources/login.asp

Phil Bell

Consultant