Top 5 z System Capacity Issues

As Information Technology continues to evolve within enterprises, such as cloud and virtualization, there is one constant in the IT infrastructure that doesn’t change.  That constant is the z Systems^® mainframe. 

Many organizations continue to rely on the mainframe for the heavy lifting of data in various business applications.  The subsystem components of CICS, DB2, MQ Series and IMS continue to feed not only legacy applications but also new multi-tier applications.  As the mainframe operating system continues to develop, you are seeing more functionality added to provide a better user experience. 

With all these changes and innovations, there are challenges that need consideration relating to Capacity Management on the mainframe.

I'll be running a webinar on September 13 which will provide information to assist you, discussing  the Top 5 Performance and Capacity Management Challenges for IBM z Systems^®.

Some of the topics I'll be covering are:

• What has happened in the past 12 months with the mainframe?
• How are organizations managing the mainframe capacity?
• The Top 5 performance and capacity z Systems® challenges facing organizations
• The impact of the Cloud on mainframe
• What is new with the z14

Register for your place now
https://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Charles Johnson
Principal Consultant

VMware Capacity Planning - When a Peak is Not a Peak

Performance metrics collected from VMware’s API, which includes data stored in the vCenter database, reports on Minimum/Maximum (Peak) values for selected counters. For example, CPU usage and Usage Mhz.

Peak values are of interest to capacity planners as it's their job to account  for worst case scenarios. 

Fortunately, VMware provides such metrics but unfortunately, these values are not real peaks (or maximums) they are averages for the collection interval, which is typically 5 minutes.

The following example of CPU utilization shows a worse-case scenario illustrating why this can be “bad” (well, at least problematic for the capacity planner)

This table shows 20 second averages (as reported by the realtime monitor). The entire table represents the 5 minute interval on which the API reports.

Products which use the API or database will show the 5 minute interval as having a maximum value of 62.5% utilization. This isn't the true peak, it's an average over the interval. If you were to look at this number you'd draw the conclusion that the system is healthy - after all, there's plenty of headroom.

This is simply not the case:  ¾ of the time the CPU is maxed out and there is no headroom. Unfortunately, you didn’t see it as the true peak is hidden by averaging of the values.

That 3 minute period where utilization is at 100% is a long time for an application to be starved of CPU resources. Bad things can happen: queues can form, applications can slow to a crawl or completely fall over.And no one would be any the wiser as to why.

Capacity planners must account for worst case scenarios in order to meet application demands and meet service level agreements (SLAs). An average in this case just doesn’t fulfill the requirement.

Consider a service provider (internal or external) supporting an application. The client (internal or external) receives reports or a dashboard illustrating application and system performance. The application or service has issues, yet the information the client is provided shows no issues - after all, there is plenty of headroom. So what’s the problem? And who is responsible?

Ultimate responsibility falls back to the service provider (and the capacity planner) who must diagnose and correct the issue.  After the fact. And must (embarrassingly no doubt) report back to the client what happened and why it was not provided for. Not good.

Our athene® Capacity Management software leverages the real time feed from VMware to report actual peaks. These can be incorporated into a reporting regimen to give capacity planners, and their customers, proper insight into a system’s behavior to account for the demands of an application. This is all done with no additional impact on the vCenter.

What the capacity planner needed to account for was the 100% utilization.

Our Resources has a great selection of VMware Capacity Management videos and white papers, go take a look. 

https://www.metron-athene.com/resources/index.asp

Don Fadel

Regional Director

Maturing your Capacity Management process - Summary ( 11/11)

As mentioned at the beginning of my blog series, the implementation of a mature business focused capacity process is a long term project that requires a good deal of effort.

In this series  I’ve given you a basic beginner’s guide,  something to present to the business to gauge the potential requirements and interest.

In summary and to provide some guiding principles for anyone looking to achieve a more mature implementation of capacity management, the following are the 5 key areas that should be the focus throughout your project:

·         Involve all aspects of the business as soon as possible.  Early involvement will pay dividends throughout the project with getting buy-in and building communication interfaces

·         Be selective with the technical information being captured and processed.  Across the enterprise there are likely to be hundreds (if not thousands) of metrics available, but it’s likely that only a small subset will be useful.  Remember the more data captured, the more likely you are to miss the important information.

·         Be sensible with the scope.  Start on a service that is important to the business, but isn’t so complex or sensitive that you won’t be able to make any progress.  Review the incident or change records to see if any of the services are suffering large numbers of capacity outages or late change requests for upgrades.

·         Aim for the quick wins where possible and strive to demonstrate value/progress as quickly as possible.

·         Be cautious with tooling.  Get the process configured and ensure that the needs are understood before starting any tooling evaluations.

I hope you’ve enjoyed the series and if you need expert guidance with maturing your capacity management process check out our Consulting Services.

https://www.metron-athene.com/services/consulting/capacity-management-consulting.html

Jamie Baker

Principal Consultant

Maturing your Capacity Management processes - Demonstrate the value (10 of 11)

Sometimes getting started and maintaining capacity management can be a hard sell into an organization, especially where the common misconception is that just increasing the available capacity is cheap. 

There are a number of methods that can be used to demonstrate the progress being made and the value to the business:

·         Ensure all presentations and reports contain the right level of data.  Whether it’s for the business or a technical department, ensure that it’s written in a way that means something to them

·         In the initial audit or implementation phase there will be objectives set by the business about what they want to see from the capacity process.  Ensure these are understood and at least some of the KPIs are built around those objectives.

·         Carefully monitor incidents and problems, ensure all capacity related tickets are captured and ultimately reduced over time.

Continual Service Improvement (CSI) is a “reasonably new” concept within ITIL, but capacity management has had an iterative cycle of “Monitor – Analyze – Tune – Change” since at least version 2.  This cycle of improvement is critical in optimizing the IT environment and is one of the key parts of the process for creating value. 

By optimizing the environment the process can demonstrate the following:

o   Release of spare capacity

o   Deferred or optimized expenditure

o   Ensure service level targets are met, whilst reducing costs

On Wednesday I’ll summarize in the final part of my blog series.

If you missed part 3 of our Capacity Management Maturity series - Defined to Managed you can catch it on demand  in our Resources section. 

https://www.metron-athene.com/resources/index.asp

Jamie Baker

Principal Consultant

Maturing your Capacity Management processes - Keeping the momentum going ( 9 of 11 )

As with any implementation, once the initial project has been concluded, all the initial objectives have been met and the quick wins achieved, the difficulty is maintaining the momentum.

Having this momentum is essential for business capacity management and the following techniques are very effective:

Intranet/Newsletters

Most organizations have an Intranet or similar that is used to send out information.  Where possible, try to get capacity staff mentioned or get capacity management successes published.  These can be in the form of selling the benefits and ensuring people are aware of the process and the value it provides.  Where possible, mention successes, achieving targets, reducing expenditure etc.  In short, promote and sell the work being done.

Road shows

An extension of the Intranet option, if there is the time and opportunity, mini presentations on capacity management to key stakeholders, different departments  and locations can be an excellent source of information.

These also serve to raise the profile of capacity management and ensure people are aware of what’s happening.  These can be done on their own or perhaps as part of a wider Service Management initiative.

The content would normally be at a high level (not too many graphs) and focus on the following:

·         Progress so far

·         Successes and failures

·         What’s next and how people can get involved

All of this momentum will help you to demonstrate value and that’s what I’ll be discussing on Monday. 

Check out our on-demand Capacity Management Maturity workshop  

https://www.metron-athene.com/services/online-workshops/capacity-management-workshops.html

Jamie Baker

Principal Consultant

Maturing your Capacity Management processes - People( 8/11)

The number of people used to support the capacity management process within your organization will obviously vary depending on budget, scope, IT landscape etc. 

Throughout the industry there is no set pattern on the number of people associated with capacity management with a wide variance in the number of people deployed regardless of the size of the estate.

If you are starting the journey of implementing capacity management the decision on experience and how many people to recruit will largely depend on the factors outlined above. 

Whether as a business you recruit an experienced capacity manager, contractor or a consultancy company, the one key people element is the utilization of capacity champions. 

A capacity champion is a person that is an expert in their respective area, has an interest in capacity management (they’re probably doing it already in their own silo) and is a good communicator/facilitator.  

The recruitment of champions provides you with the following benefits:

·         Easier to obtain relevant data

·         More detailed expertise and knowledge

·         Improved integration with the business

·         Easier to raise and maintain the profile of the capacity process

The capacity management champion will also be crucial in what I’ll be looking at next time, keeping the momentum going.

Don't forget to register for our live Capacity Management Maturity series webinar taking place today

https://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Jamie Baker

Principal Consultant

Maturing your Capacity Management processes - Information and Governance ( 7 of 11 )

With the complexity of today’s modern IT environments and the multitude of delivery mechanisms available, it’s safe to say that organizations are swamped with data. 

The challenge for implementing capacity management is being able to distil this wealth of information into a useable form.  Rather than concentrate on specific platforms or metrics, I have found the following to be a useful guide for filtering information:

·         If implementing for the first time, follow the usual ITIL guidance of starting small and building up.  For capacity management this could be starting with a particular platform, service or application. 

·         Work with technical teams to understand which metrics are available for each of the platforms.  This is especially important when dealing with virtualized platforms as these tend to have various levels e.g. cluster, host, guest etc which are key for capacity planning

·         Work with the application and service support teams to determine all valid metrics and importantly how these can be correlated with the underlying infrastructure data

Remember, the more data captured, the more you need to store, manage and understand.  As well as the obvious capacity implications of storing more and more data, you are also increasing the likelihood of missing key alerts and data trends.

On Wednesday I’ll be looking at your most important asset, People. 

In the meantime why not find out where you are on the Capacity Management Maturity Scale and receive your free 20 page report.

www.metron-athene.com/survey/

Jamie Baker

Principal Consultant

Maturing your Capacity Management processes – Tooling ( 6 of 11)

Given the complexity of today’s IT environments, the challenge in obtaining good and reliable capacity management data has never been more difficult.

 

Most organizations utilize a wide range of technologies to deliver their services and each has its own range of metrics, interface methods and technical nuances.  There are a wide range of tools available for the capture and collation of data and all have their strengths and weaknesses.

There are also options for free trials with some companies offering the ability to download the software or better still, as a preconfigured virtual machine.  Rather than list them individually, it is probably more useful to list the sort of functionality that a solid capacity management tool should be able to provide:

·       Capture across the enterprise.  Multiple point solutions are inherently harder to manage and collate the data

·        The ability to bring in additional non-native data sources.  This could be via frameworks, partner products or more traditional sources such as SNMP, application API’s etc

·       Provide the capability for modelling.  Certainly basic techniques such as trending have their place, but realistically, to truly understand the relationship between the utilization, the response time and how this will change over time you will need to use more advanced methods such as analytical or simulation modelling.

     Good reporting provides the foundation to any mature capacity process so any tool chosen should have the ability to deliver reports in a variety of ways e.g. web, email etc and support the correlation of data.

The important thing to remember is that all organizations are different and have specific requirements.  To get the best results, utilize the experience of the Capacity Management SIG and spend plenty of time producing the Request for Proposal (RFP) document to ensure it covers all of your requirements.

I’ll be taking a look at Information and Governance on Monday but in the meantime don't forget to register for our Capacity Management Maturity series webinar  -  Defined to Managed taking place on August 16.  

https://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Jamie Baker

Principal Consultant

Maturing your Capacity Management processes – Beginning the journey ( 5 of 11 )

Now that the Gap analysis or process audit is complete you’re ready to take the next step on the journey and begin by building your community.

Starting the process of education by involving key business stakeholders throughout the GAP analysis will have laid firm foundations for this implementation stage. 

It varies between organizations, but a solid first step is to provide a high-level overview of what capacity management is, its requirements and ultimately the benefits to the business.  This should cover the basic principles, the information interfaces and the value it will provide.  The intended audience should be similar to that of the GAP analysis; the key requirement is to include a mixture of people and roles. 

By putting business, capacity and technical staff, (people that wouldn’t inherently mix) together it starts to build the necessary relationships and provide the flow of information required for a business focused process.  In addition, it provides everyone with a common language when talking about capacity management and the required information.

The next requirement, again depending on existing maturity, may be a more intensive look at capacity management, usually spread over a number of days and covering the whole process. 

This sort of course will normally cover all aspects of capacity management e.g. methods, techniques etc and involve staff that are likely to be more hands on or at least have key involvement.  The people attending this course will form the basis of the Special Interest Group for capacity management and on completion of this course can work with the process manager to define the goals, objectives and outputs for capacity management.

At the lowest level of granularity, more technically focused capacity management courses dealing with key elements of a particular technical domain can be provided.  Looking at the requirements of managing a particular domain e.g. vmware, Unix etc; can be useful in providing support personnel with an additional set of skills for front line incident analysis and ensure that everyone is talking the same language.

The next step is of course to automate and on Friday I’ll be talking about tooling. 

If you'd like help with a gap analysis or other capacity management projects contact us. 

https://www.metron-athene.com/services/consulting/capacity-management-consulting.html

Jamie Baker

Principal Consultant

Maturing your Capacity Management process – Interviews/Workshops ( 4 of 11)

The real information and business integration starts here. 

The individual interviews are essential in obtaining the complex information about a particular department or business area; the reality being that most people are far more willing to talk openly about particular issues when meeting individually.

It also allows for a degree of knowledge transfer, in particular some business areas or departments who might not have any prior knowledge of capacity management and provides an overview of what the project objectives are e.g. what are we trying to achieve?

One of the prime advantages with this approach is the early involvement of all key areas of the business.

This is especially true with the workshop format. Bringing together groups of people that may not normally work that closely together forges more of a co-operative working party, where ideas can flow and people can properly challenge others in a constructive, facilitative way.  These people may also become capacity management champions and form the beginnings of a capacity management community or capacity management special interest group.

So the GAP analysis/process audit has been completed and should have documented the current situation, the desired situation and a series of prioritized next steps for getting there.

Much as before, there are lots of things that will be required to implement or mature a business focused capacity process.  I’ll start to take a look at the “key next steps” and an approach that has been successfully used in a number of organizations on Wednesday. 

Jamie Baker

Principal Consultant

Maturing your Capacity Management process - The Questionnaire (3 of 11)

Questionnaires tend to be a standard tool for capacity management maturity exercises with most organizations providing on-line variants or a standardized set of questions. 

This approach can be very useful as a starting point but their generic nature tends to mean there are lots of questions that are irrelevant to particular people or departments. 

While it’s true that most organizations face similar challenges, these challenges can be caused by different sets of circumstances. The other issue is language or jargon, in an ideal world everyone would talk the same language, the reality is that most organizations still have terminology or processes unique to them and a questionnaire is unlikely to align with the way things are done within your world.

Having said all that, questionnaires are very useful as a facilitator for future interviews and workshops. They begin the process of getting people to start thinking along the same lines.  By providing a selection of “standard” templates, people can begin to understand what will be required from them or their particular area.

The next step is the interviews and workshops themselves which I’ll discuss on Monday. In the meantime why not take our Capacity Management Maturity survey and find out where you are on the Maturity Scale

www.metron-athene.com/survey/

Jamie Baker

Principal Consultant

Maturing your Capacity Management process - Situation Appraisal( 2 of 11)

Before you can set about implementing or maturing any ITIL process, the first step is to conduct a process audit or GAP analysis.  

This can obviously be done internally, but it’s unlikely to provide an objective and honest appraisal.  A better option is to use some targeted consultancy that will provide the necessary expertise, knowledge and importantly, objectivity.  Whether it’s delivered internally or via a third party the following approach has always proved successful.

Rather than using a prescriptive approach each time, a combination of the following will yield the relevant (and appropriate) information. 

The roles or groups will alter slightly depending on the organization but the following areas should be engaged to ensure the project is successful.

·         Key business stakeholders

·         Service Managers or Service Owners

·         Technical domain experts

·         Tooling Personnel – Monitoring, alerting etc

·         Existing capacity staff

·         Service Management personnel

Every environment is unique; however there are a number of key questions that should be asked. By thinking these through it will help to ensure you create a Strategy for Capacity Management, and if you need to de-scope elements to start building momentum, you will at least have a clear sight of your end objectives.

     ·         What information channels/interfaces exist currently?               
   ·         What reports are available and the associated delivery mechanism?

         ·         What metrics are captured and how are their thresholds derived?
         ·         What tooling is available?
         ·         What KPIs are being used?
         ·         What capacity deliverables are being provided?
         ·         Is the business suffering from capacity outages?
         ·         Are upgrades performed in a timely fashion?                            

I’ll be looking at questionnaires and their relevance on Friday. Register for our webinar ' Capacity Management Maturity series - Defined to Managed'

https://www.metron-athene.com/services/webinars/capacity-management-webinars.html

Jamie Baker

Principal Consultant