PowerShell Prevents Datastore Emergencies

In my previous post on VMware vSphere thin provisioning, I pointed out the new datastore alarm feature.  You can take advantage of this feature to respond to a sudden storage demand and automatically take action before end users notice.

When triggered, vCenter Server alarm actions allow several options, including the ability to run an arbitrary command such as a VMware PowerCLI PowerShell script.  Please see Carter Shanklin’s in-depth article for more details on how this works — note that he uses a different technique to launch the scripts.

Storage VMotion to the Rescue

When a datastore is about to run out of space, the fastest resolution may be to simply migrate virtual disks to another datastore.   VMware Storage VMotion provides that capability with zero downtime for VMs and no disruption to end users.  Fortunately, PowerCLI can perform this feat with ease, thanks to the Move-VM cmdlet.

Let’s take a look at a functional prototype PowerCLI PowerShell script:

Add-PSSnapin VMware.Vimautomation.Core
Connect-VIServer localhost

$vmToMove = get-vm -Datastore $env:VMWARE_ALARM_TARGET_NAME | select-object -first 1

$destDS = Get-Datastore | where {$_.FreeSpaceMB -gt 50000 -and $_.Accessible -eq $true} | select-object -first 1

if ($destDS) {
	move-vm -VM $vmToMove -Datastore $destDS -RunAsync

This script is a proof-of-concept that is not ready for your production environment as it is — it just picks an arbitrary VM from the nearly-full datastore, finds another datastore with at least 50GB free, and moves the VM disks. More comprehensive selection logic and error checking are needed for a critical task like this.

Save your script on the vCenter Server system somewhere, such as C:\scripts\datastore.ps1.

Create the Datastore Alarm

Create a new alarm at an appropriate level in the vCenter hierarchy, such as a datacenter, and configure like this:

Datastore Alarm

On the Triggers tab, add a “Datastore Disk Usage (%)” trigger to alert at a reasonable percentage — I opted for 93.

Run PowerShell Directly from vCenter Server

For whatever reason, PowerShell.exe does not do well when launched directly by another process — it tends to hang instead of exiting when it is finished.  As a workaround, it can be launched from cmd.exe as long as it receives something on standard input.  To do all that, the necessary code looks like this:

"c:\windows\system32\cmd.exe" "/c echo.|powershell.exe -nologo -noprofile -noninteractive c:\scripts\datastore.ps1"

For an alternate approach, take a look at the intermediate batch file solution described by Carter Shanklin in the link above.

On the Actions tab, add a “Run a command” action and supply the appropriate command.  You also need to decide whether to run one time or repeat the action.

Datastore alarm running a PowerShell script


To test the alarm, either fill up the datastore or temporarily lower the alarm threshold.  When the alarm fires, a Storage VMotion should be seen in the vSphere Client:

Storage VMotion in progress

Note the “Initiated by” column — that’s the machine account for this vCenter Server.  The PowerCLI script is kicked off from vpxd.exe, which is running as LocalSystem.

Additional information is available by looking at the Tasks & Events tab for the datastore.  Here you can see a sample sequence of events, newest on top:

Datastore emergency events

The Last Resort

This automated Storage VMotion recovery alarm is a safety valve that could help you avoid suddenly running out of space on a datastore.  It should not take the place of more proactive storage management, but it sure beats VM downtime.

In case you are wondering: No, you can’t do the same thing with Hyper-V because Hyper-V does not have zero-downtime Storage VMotion.  Just another reason to choose VMware vSphere — as if you needed another reason.

Have you used vCenter alarms to automate any recovery processes in your environment?

(Visited 5,231 times, 1 visits today)
This entry was posted in Virtualizationism and tagged , , , , , , , , . Bookmark the permalink.

17 Responses to PowerShell Prevents Datastore Emergencies

  1. Pingback: Tweets that mention Use VMware vCenter alarms with PowerShell to automatically migrate virtual machines | VCritical -- Topsy.com

  2. Alan Renouf says:

    That is amazing, very cool. Great use of PowerCLI and Alarms!

  3. Eric Gray says:

    Thanks, Alan! Coming from you that means a lot.


  4. Scott Sauer says:

    Great write up, I was just discussing my fears of going thin with a fellow co-worker. This addresses our fears of a “storage emergency” situation. Going to give it a shot once I get through all of our upgrades.

  5. Pingback: How to recover when a VMware ESX datastore runs out of space | VCritical

  6. Scott Sauer says:

    Here is an article I wrote that I would like to share with anyone that comes across the thread.


  7. Pingback: VMTN Podcast number 70, Storage… » Yellow Bricks

  8. Pingback: Provision a Thin Provisioned Standby LUN For vSphere Thin Provisioning | VM /ETC

  9. Pingback: ben.neise.co.uk » Datastore alerts via Twitter

  10. Adam says:

    I’ve gone through the process of trying to figure out a better VM selection process.. I pulled the gut of the VM Disk usage from another source but the rest of tathered together by me. The script will search suspect datastore for the smallest VM based on disk utilization and then check if there are any snapshots.. if there are snapshots for that particular VM they get removed the the VM gets moved to an evacuation datastore of your choosing.

    Add-PSSnapin VMware.Vimautomation.Core
    Connect-VIServer localhost

    $vmToMove = Get-View -ViewType VirtualMachine | %{
    New-Object PSObject -Property @{
    Name = $_.Name
    Host = (Get-View $_.Summary.Runtime.Host).Name
    Datastore = [system.String]::Join(“,”,($_.Storage.PerDatastoreUsage | %{Get-View $_.Datastore} | %{$_.Name}))
    Size = ($_.Storage.PerDatastoreUsage | Measure-Object -Property Committed -Sum).Sum
    } | Sort Size | ?{$_.DataStore -eq $ENV:VMWARE_ALARM_TARGET_NAME} | Select Name, Datastore -first 1

    $destDS = Get-Datastore yourevacuationdatastore
    $snapshot = get-vm $vmtomove.name | get-snapshot
    If ($snapshot -eq $NULL)
    write-host “No Snapshots Present”}
    else { write-host “Snapshots Found and Removing”
    get-vm $vmtomove.name | get-snapshot | remove-snapshot -Confirm:$false }

    if ($destDS) {
    move-vm -VM $vmToMove.Name -Datastore $destDS -RunAsync
    Send-MailMessage -To “whereveryouwanttogo” -From “somewhereitcamefrom” -Subject “VM Successfully Moved to Evacuation Datastore” -Body “$vmToMove has been moved to $destDS from $env:VMWARE_ALARM_TARGET_NAME” -SMTPServer “yoursmptserver”

    This is working for me right now in POC.

    Thank you,

    Adam Savage

    • Eric Gray says:

      Awesome, Adam! Thanks for sharing the enhancement.

      • Adam says:

        No problem at all.. I also had some difficulties with getting the alarm to launch the script correctly.. Turned out that virtualcenter was launching powershell from C:\Windows\SysWOW64\WindowsPowershell\v1.0\powershell.exe instead of the normal c:\windows\system32.. So i had the execution policy set on the x86 version but not the x64 version.. food for thought.


        Adam Savage

  11. Pingback: Protecting your Virtual Machines From “That Guy”

  12. ap says:

    Is there any to create an alarm for datacenter using VI API ?

    If yes, please help.

  13. boraq says:


    Could you please tell me what’s this ‘$env:VMWARE_ALARM_TARGET_NAME’??

  14. Pingback: VCE-101 Thin Provisioning Part 1 – The Basics - Vaughn Stewart

Comments are closed.