Clearly, the capability introduced with VMware vSphere 4 that allows VMware ESX 4 to virtualize itself is a real crowd-pleaser.
However, one limitation that some have discovered while using this lab-testing technique is the lack of ability to use snapshots with virtual ESX systems. In fact, after taking a snapshot of a virtual ESX VM, you will see the system boot into the recovery shell like so:
Ouch.
Can we fix it?
I asked some super-smart engineers inside VMware about the issue, and of course they know all about it and how to make it work.
The solution is to enable an undocumented advanced configuration option on the physical ESX host like so:
In other words:
- Log into the console of the physical VMware ESX 4 host
- Run the following command: esxcfg-advcfg -s 1 /COW/PermitVmfsOnRedoHierarchy
- Verify the setting with the “get” flag: esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy
Do I even need to say that this stuff is not supported by VMware Global Support Services (GSS)? Well, I just did.
Yes we can!
After executing that one command on your physical machine, take a snapshot and boot up your virtual ESX:
Everything works just like you need it to now — you can even create a snapshot hierarchy:
Now what?
Were you constrained by the lack of virtual ESX snapshots? What interesting things will you use this for?
I guess ESXi is also concerned ?
Hmm. I have not taken a look at ESXi snapshots in a VM. I suspect that since ESXi does not have the same VMFS architecture as ESX, it might just work as is.
Pingback: uberVU - social comments
This is good to know. But this also brings up a big question on how we use ESX as a VM. When I do it, I don’t use the local datastore of the guest ESX host. I always point to a defined iSCSI resource that is constant, regardless of the host state. Further I maintain snapshots of the guests outside of a virtualized ESX host state.
Maybe I go about it differently, but this lets me keep images and VMs constant in spite of what host hopping and chopping I do.
I, too am interested on how this applies to ESXi – as most of my nested virtualization does use ESXi.
Thanks as always for another good post, Eric.
As Eric mentioned, the advanced host configuration does not apply to ESXi.
I’ve tried this in the past and saw some interesting results when snapshotting a vESXi host while it was offline.
I just tried on a powered on ESXi host and basically took a snapshot while it was on, then created a dummy vSwitch and renamed the datastore. I then reverted the snapshot and only the vSwitch was removed, the renamed datastore persisted even after a revert. I need to do more testing but I think with ESXi, it might not work 100%.
Keep you guys posted when I can test further with both online/offline snapshots for vESXi
—William
I just wanted to share my experience. But with little difference… I’m using Workstation 7 and virtual ESXi as a VM. Snapshots works fine. I keep my snapshot files in the same directory of my individual linked clones.
I did a snapshot test with Powered On vESXi 4, and then revert back.. everything worked. Even password change between those 2 snapshots.
Thanks
Vladan
@Vladan: Does Workstation let you put x64 VMs as a guest on the vESXi server? Or does it have the same hurdle as ESX in this regard.
Rick,
you’re right. Just verified. If I want to put x64 VM on the virtual ESXi 4, there is a message saying that the destination does not support 64bit hosts. (fyi. tried to use vmware converter to “send” the VM there, not a direct install of x64 guest…)
The same trouble on the W7.
Vladan
@Vladan Cool – Thanks for verifying. Saved me some time.
Eric,
not a question about snapshots or ESXi, but what applications do you use that creates the torn edge on the images?
David
It looks like snag-it, I know it can do tears and other type of borders
–William
You got it.
Thanks William & Eric
This also affects virtual hosts that are moved with storage vmotion. This makes sense as the svmotion process uses snapshots, though I find it curious as I moved the VM when it was powered down.
I ended up just reinstalling ESX as I want to keep our hosts in line with support.
Actually, it looks like it is the simple fact that the new data store is NFS instead of block iSCSI. I was unable to reinstall ESX on the NFS VMDK, but I was able to reinstall by creating a new VMDK on iSCSI. Moving the original failed virtual ESX host back to iSCSI resulted in an inability to boot.
Hi,
[~]# cat /etc/vmware-release
VMware ESX 4.1 (Kandinsky)
[~]# esxcfg-advcfg -g /COW/PermitVmfsOnRedoHierarchy
Exception occured: Unable to find option PermitVmfsOnRedoHierarchy
Is this feature named differently in ESX 4.1 (VMKernel Release Build 235786) ?
thx,
az
hello
I was having trouble running a esx 4.0 on a esxi 4.1, it worked previously, but after doing a cold migration to another host it did not work anymore.
I’ve found that if you disable ctk ( ctkEnabled=false for both the vm and the scsi devices) going to advanced properties on the vm hosting esx it does work again.
hope that helps
Hi Miguel,
thanks a lot for sharing your experience, I confirm that I was able to resolve the boot problem simply changing the ctkEnabled=false for both the vm and the scsi devices, without any action on the physical ESX host.
Pingback: VCAP-DCA Study notes – 3.1 Tune and Optimize vSphere Performance | www.vExperienced.co.uk
Is there a way to get this working on ESXi 5?