Have you ever seen a datastore that’s larger than its underlying LUN?
Recently, we’ve been working with another team of administrators to migrate some critical applications to VMware. Because of the unique storage requirements around one application, we’ve ended up with many datastores to support those VM’s. This week while troubleshooting a Commvault issue we discovered one of the datastores was reporting nearly double the capacity of its principal volume.
The datastore properties showed a 1TB volume
The storage device was only 600GB.
Maybe the datastore had been extended to another LUN that was later removed. But that would destroy the underlying file system, wouldn’t it? No! This sounds more like a UI error, I thought.
Get-Datastore showed the same information.
What was confusing was the fact that vCenter was implying there were 1,000 GB in use, even though du was telling a different story.
DU reported only 500GB were in use.
I took a look at the VMDK settings and the maximum size seemed to line up with the LUN size.
This was interesting because there was a 98.79 GB difference between the VMDK size and the maximum size, which was equal to the free space being reported.
One of the great gizmos in the VMware toolkit is vmkfstools; there isn’t much this command can’t do when it comes to storage. To get a detailed look at what made this datastore tick I used vmkfstools –queryfs.
Vmkfstools verified this datastore was made up of two LUNS, one of which was offline.
While cleaning up old LUNs my colleagues accidentally removed one of the extents, but the VM kept running because it needed less than the 600GB that was available. Since there’s no way to shrink a datastore, the VMDK needed to be migrated to a new datastore. You can storage vMotion individual disks by going into the advanced migration settings (Migrate -> Change Datastore ->Advanced).
This was an interesting discovery and is something that I will look out for in the future. This may even call for another script!
