Server 2012 VMs don’t boot after a restart

It’s patch week again which means many guest reboots in our VMware environment. Lately, we’ve been receiving alerts on many of our Server 2012 VM’s, and upon investigating we’ve found these guests to be hung at the Windows boot screen. After a few hours they might come up, but more often than not a hard reset is required to bring them back.

It turns out this is a known issue and VMware has released KB2092807 to address it. Each processor has a TimeStampCounter (TSC) which is essentially an odometer that counts how many cycles the processor has run since its last reboot. When Windows reboots, it resets this counter if the value is equal to or larger than 18,014,398,509,481,984. That is to say, the VM has been running for about two months, or longer, without being powered off. The problem is that Windows only resets the counter on CPU0, and a discrepancy between multiple vCPU’s can cause the OS to hang at boot up. This issue only occurs on version 10 virtual hardware due to the hardware clock mechanism. Version 11 hardware is not impacted.

They do offer a fix, but this requires a slight change of each VM. You must add an advanced parameter that will reset the TSC of all vCPU’s on a soft reset. What they don’t tell you is if you want to use the GUI (Web or C# Client) to make the change, you’ll have to power down the VM first. Otherwise the option to edit advanced settings is greyed out.

• If you have the time and flexibility to power down each VM, simply navigate to theadvancedparameters of your VM’s settings…
  • Web Client: Settings -> VM Options -> Advanced -> Edit Configuration
  • C# Client: Edit Settings -> Options -> General -> Configuration Parameters
• Add a new row with the following
  • Name: monitor_control.enable_softResetClearTSC
  • Value: True

If you’re like the rest of us, you can’t afford to bring down each VM and edit it manually. Thankfully, VMware has offered a great PowerCLI script to get the job done. The script adds the advanced option to all affected VM’s. Don’t worry about the line where it searches for guestid like windows8*Guest. Server 2012 VM’s show up as windows8Server64Guest. The script then performs a stationary migration (the VM doesn’t change hosts) which applies the new configuration.

Note: VMware removed line 13 from their KB because they discovered that self-migration didn’t always perform the Fast Suspense Resume (FSR) required to activate the changes. Instead they recommend vMotioning your VM’s. 

ForEach ($vm in (Get-VM)){
$vmv = Get-VM $vm | Get-View
$name = $vmv.Name
$guestid = $vmv.Summary.Config.GuestId
$state = $vmv.Summary.Runtime.PowerState
$vmx = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmx.extraConfig += New-Object VMware.Vim.OptionValue
$vmx.extraConfig[0].key = "monitor_control.enable_softResetClearTSC"
$vmx.extraConfig[0].value = "TRUE"
if ($guestid -like "windows8*Guest") {
($vmv).ReconfigVM_Task($vmx)
if ($state -eq "poweredOn") {
# $vmv.MigrateVM_Task($null, $_.Runtime.Host, 'highPriority', $null)
}
}
}