Monitoring VCSA volumes with vRealize Operations

This question seems to come up quite frequently… What’s the best way to monitor my vCenter Server Appliance volumes? This question is usually a follow up to VCSA problems due to one of it’s many volumes filling up. I’ve seen some customers run scripts such as the one provided in VMware KB2058187 as cron jobs to send e-mails when a volume fills. But running custom scripts on a vCenter Server Appliance won’t necessarily get you the nod from VMware support (unless it’s implemented exactly as prescribed in a KB). In this post, I’ll show you a better way for monitoring VCSA volumes with vRealize Operations Manager!

vRealize Operations Holds The Key

Take a look at the metrics available for your VCSA and you’ll see that vRealize Operations is already aware of all the appliance’s volumes. As you can see in the example above, vROps shows my root partition at about 60% utilization and the PuTTy session verifies this is true. We’re off to a great start!

Now that all the hard work is done for you out of the box, all we need to do is tell vROps to e-mail us when any of our VCSA’s volumes start to fill up. Thankfully vROps includes an alert for when a guest volume is running out of space.

By default, this alert is triggered by one of two symptoms. When a volume reaches 90% utilization it will trigger a warning. When that volume reaches 95% it is considered critical.

These definitions can be modified if needed or new symptoms can be created if you’re looking for a little more lead time. For this example, we’ll leave them at their defaults.

Keep in mind we haven’t had to do anything to vROps up until this point. All we need to do is configure the notifications.

Monitoring multiple vCenter Server Appliances

If you only have a single vCenter server, feel free to skip this section if you want. But if you plan to deploy more vCenters in the near future, it may be a good idea to follow this section and lay down some important groundwork. To monitor multiple vCenters, we can either set up notification rules for each, or better yet we could create an application group!

To create an application group, click Environment, Applications, and then click the green plus sign to create a new application.

We’re going to create a custom application.

Give your application a name and click the green plus sign to create a new application tier.

Give the tier a name and click Update.

At this point, you can either create a single Tier for all your VCSA’s and external PSC’s (if applicable), or you could decide to create a separate tier for each. It’s entirely up to you.

Select the tier that you created and in the lower left hand pane select Object Type > Virtual Machine. Or you could just type the name of your vCenters and PSC’s in the filter box.

Locate each of your your vCenter virtual machines and drag them up into the Tier Objects pane. 

Your application should look something like this. Note that I’m using internal PSC’s so I only needed to add my VCSA’s. Click Save.

Create the notification

Click on the Alerts heading, select Notification Settings, and click the green plus sign to create a new notification.

Give your notification rule a name and select the Standard Email Plugin.

If you haven’t set up SMTP notifications before, you’ll need to add a new SMTP instance now by clicking the Add button. Enter your SMTP info and click save.

Enter the recipient e-mail address, the number of times you want vRealize Operations Manager to notify the recipients, the maximum number of notifications (optional), how many minutes to wait between notifications, and finally a description if you’re so inclined.

If you just have a single vCenter Server Appliance you wish to monitor, then set the scope to Object and select your VCSA. 

If you’re using multiple vCenters, here’s where we’ll use our custom group. Set the scope to Applications and select your new group.

Set the notification trigger to alert definition.

Enter “guest file” in the filter field and select the “One of more virtual machine guest file systems are running out of disk space” definition. Click Select.

Select the appropriate criticalities. By default we only have warning and critical definitions.

Save your new rule.

Testing it all out

Our final step is to test our new alert. DO NOT perform the following on a production vCenter Server Appliance!

To test in my lab environment, I’ll go ahead and create an 8GB file in my log volume. This will bring the storage utilization up to 100%.

A few minutes later, I received the following alert. SUCCESS!!!!

New alert was generated at Wed Aug 22 22:38:06 EDT 2018:
Info:VC1 VirtualMachine is acting abnormally since Wed Aug 22 22:38:06 EDT 2018 and was last updated at Wed Aug 22 22:38:06 EDT 2018

Alert Definition Name: One or more virtual machine guest file systems are running out of disk space
Alert Definition Description: One or more guest file systems of the virtual machine are running out of disk space. Check the symptom details to identify the affected guest file system
Object Name : VC1
Object Type : VirtualMachine
Alert Impact: health
Alert State : critical
Alert Type : Virtualization/Hypervisor
Alert Sub-Type : Capacity
Object Health State: critical
Object Risk State: info
Object Efficiency State: info
Control State: Open
Symptoms:
SYMPTOM SET – self

Symptom Name	Object Name	Object ID	Metric	Message Info
Guest file system space usage at critical level	VC1	bcadd517-4b72-4d1d-8a6f-ed493996f964	guestfilesystem:/storage/log|percentage	98.46 > 95.0
Guest file system space usage at warning level	VC1	bcadd517-4b72-4d1d-8a6f-ed493996f964	guestfilesystem:/storage/log|percentage	98.46 > 90.0

Recommendations: 
– Add a new virtual hard disk or expand the existing disk of the virtual machine. You must first remove all snapshots before expanding an existing disk. After completing this, use a guest OS specific procedure to expand the file system on the new or expanded disk.
Notification Rule Name: VCSA volume is filling up
Notification Rule Description: VCSA volume is filling up
Alert ID : c5ecc99a-e0f7-4440-8678-67cb396dfd37
VCOps Server – vRealizeClusterNode

Next, I deleted the 8GB file and a few minutes later I recieved the following alert cancellation e-mail.

Alert was cancelled at Wed Aug 22 23:15:06 EDT 2018:
Info:VC1 VirtualMachine is acting abnormally since Wed Aug 22 22:38:06 EDT 2018 and was last updated at Wed Aug 22 22:38:06 EDT 2018

Alert Definition Name: One or more virtual machine guest file systems are running out of disk space
Alert Definition Description: One or more guest file systems of the virtual machine are running out of disk space. Check the symptom details to identify the affected guest file system
Object Name : VC1
Object Type : VirtualMachine
Alert Impact: health
Alert State : critical
Alert Type : Virtualization/Hypervisor
Alert Sub-Type : Capacity
Object Health State: critical
Object Risk State: info
Object Efficiency State: info
Control State: Open
Recommendations: 
– Add a new virtual hard disk or expand the existing disk of the virtual machine. You must first remove all snapshots before expanding an existing disk. After completing this, use a guest OS specific procedure to expand the file system on the new or expanded disk.
Notification Rule Name: VCSA volume is filling up
Notification Rule Description: VCSA volume is filling up
Alert ID : c5ecc99a-e0f7-4440-8678-67cb396dfd37
VCOps Server – vRealizeClusterNode

In summary, vRealize Operations Manager has everything you need to monitor you VCSA volumes right out of the box. All you need to do is create the notification rule for your individual VCSA(s) or create a custom group and base your rule off that.