In my previous post on VMware vSphere thin provisioning, I pointed out the new datastore alarm feature. You can take advantage of this feature to respond to a sudden storage demand and automatically take action before end users notice.
When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script. Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.
Storage VMotion to the Rescue
When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore. VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users. Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.
Let’s take a look at a functional prototype PowerCLI PowerShell script:
Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1
$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1
if ($destDS) {
move-vm -VM $vmToMove -Datastore $destDS -RunAsync
}
This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.
Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.
Create the Datastore Alarm
Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:
On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.
Run PowerShell Directly from vCenter Server
For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished. As a workaround, it can be launched from cmd.exe as long as it receives something on standard input. To do all that, the necessary code looks like this:
"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"
For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.
On the Actions tab, add a “Run a command” action and supply the appropriate command. You also need to decide whether to run one time or repeat the action.
Action!
To test the alarm, either fill up the datastore or temporarily lower the alarm threshold. When the alarm fires, a Storage VMotion should be seen in the vSphere Client:
Note the “Initiated by” column — that’s the machine account for this vCenter Server. The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.
Additional information is available by looking at the Tasks & Events tab for the datastore. Here you can see a sample sequence of events, newest on top:
The Last Resort
This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore. It should not take the place of more proactive storage management, but it sure beats VM downtime.
In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion. Just another reason to choose VMware vSphere — as if you needed another reason.
Have you used vCenter alarms to automate any recovery processes in your environment?
Pingback: Tweets that mention Use VMware vCenter alarms with PowerShell to automatically migrate virtual machines | VCritical -- Topsy.com
That is amazing, very cool. Great use of PowerCLI and Alarms!
Thanks, Alan! Coming from you that means a lot.
Eric
Eric,
Great write up, I was just discussing my fears of going thin with a fellow co-worker. This addresses our fears of a “storage emergency” situation. Going to give it a shot once I get through all of our upgrades.
Thanks!
Scott
Scott,
That’s good news. If you devise a more robust VM selection/migration script, please consider sharing.
Eric
Pingback: How to recover when a VMware ESX datastore runs out of space | VCritical
Eric,
Here is an article I wrote that I would like to share with anyone that comes across the thread.
http://www.virtualinsanity.com/index.php/2009/10/12/get-thin-provisioning-working-for-you-in-vsphere/
Thanks!
Scott
Pingback: VMTN Podcast number 70, Storage… » Yellow Bricks
Pingback: Provision a Thin Provisioned Standby LUN For vSphere Thin Provisioning | VM /ETC
Pingback: ben.neise.co.uk » Datastore alerts via Twitter
I’ve gone through the process of trying to figure out a better VM selection process.. I pulled the gut of the VM Disk usage from another source but the rest of tathered together by me. The script will search suspect datastore for the smallest VM based on disk utilization and then check if there are any snapshots.. if there are snapshots for that particular VM they get removed the the VM gets moved to an evacuation datastore of your choosing.
Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost
$vmToMove = Get-View -ViewType VirtualMachine | %{
New-Object PSObject -Property @{
Name = $_.Name
Host = (Get-View $_.Summary.Runtime.Host).Name
Datastore = [system.String]::Join(“,”,($_.Storage.PerDatastoreUsage | %{Get-View $_.Datastore} | %{$_.Name}))
Size = ($_.Storage.PerDatastoreUsage | Measure-Object -Property Committed -Sum).Sum
}
} | Sort Size | ?{$_.DataStore -eq $ENV:VMWARE_ALARM_TARGET_NAME} | Select Name, Datastore -first 1
$destDS = Get-Datastore yourevacuationdatastore
$snapshot = get-vm $vmtomove.name | get-snapshot
If ($snapshot -eq $NULL)
{
write-host “No Snapshots Present”}
else { write-host “Snapshots Found and Removing”
get-vm $vmtomove.name | get-snapshot | remove-snapshot -Confirm:$false }
if ($destDS) {
move-vm -VM $vmToMove.Name -Datastore $destDS -RunAsync
}
Send-MailMessage -To “whereveryouwanttogo” -From “somewhereitcamefrom” -Subject “VM Successfully Moved to Evacuation Datastore” -Body “$vmToMove has been moved to $destDS from $env:VMWARE_ALARM_TARGET_NAME” -SMTPServer “yoursmptserver”
This is working for me right now in POC.
Thank you,
Adam Savage
Awesome, Adam! Thanks for sharing the enhancement.
No problem at all.. I also had some difficulties with getting the alarm to launch the script correctly.. Turned out that virtualcenter was launching powershell from C:\Windows\SysWOW64\WindowsPowershell\v1.0\powershell.exe instead of the normal c:\windows\system32.. So i had the execution policy set on the x86 version but not the x64 version.. food for thought.
Thanks!
Adam Savage
Pingback: Protecting your Virtual Machines From “That Guy”
Is there any to create an alarm for datacenter using VI API ?
If yes, please help.
Hi
Could you please tell me what’s this ‘$env:VMWARE_ALARM_TARGET_NAME’??
Pingback: VCE-101 Thin Provisioning Part 1 – The Basics - Vaughn Stewart