NEW: VMware vSphere 5 makes this even easier and supports nested 64-bit guests.
Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.
But what if you prefer to run ESX on ESX instead of Workstation?
You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host. You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!
VMware vSphere 4.1 UPDATE: VMware ESXi 4.1 has a keyboard issue when virtualized on an ESX 4.0 host. In order to virtualize ESXi 4.1, the underlying host must be 4.1. However, ESX 4.1 classic will work on ESX 4.0.
The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.
After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:
Create a new VM with the following guidance (choose “Custom”):
- Virtual Machine Version 7
- Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
- 2 VCPUs, 2GB RAM
- 2 NICs – e1000
- LSI Logic Parallel
- New disk – reasonable size greater than 10GB for ESX classic
After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.
After installation, add the new virtual ESX to vCenter 4 and create a new VM.
If you do not need to run VMs on your virtual ESX, you can stop there. However, if you try to power on that nested VM, you will see the following error:
To prevent this, just one tweak is needed:
- Shut down the virtual ESX VM
- Click Edit Settings
- Click the Options tab
- Click Advanced / General / Configuration Parameters…
- Click Add Row
- For the Name/Value enter: monitor_control.restrict_backdoor / TRUE
The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.
Now you are ready to power your virtual ESX VM back on, as well as the nested VMs. This capability should come in handy as you start investigating the new features of vSphere 4.
UPDATE: In order to take snapshots of a VM running ESX, a quick configuration change is necessary on the physical host .
You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.
Eric: Good stuff. I think this is a great way for administrators to get started with the base functionality of the product without allocating a tremendous amount of hardware. This won’t help too much with upgrades, but will definitely help with the base product!
Cheers.
Thx you very much . it was very use full for me .. i don’t find any Youtube video on this ..
This was very helpful stuff especially for testing environment. I actually had two ESXi 4.0 running as Guess within ESXi 4.0 on the physical machine.
Thanks, this works well. I do have a problem loading the tools on the vm’s I create on the virtual esx. Does anyone have any ideas.
Thanks
i’ve also noticed that you can’t clone a vm from one virtual esx to another virtual esx.
You should be able to come up with a way to do this, but there are some issues — such as duplicate MAC addresses — that need to be dealt with.
Feel free to chime in if anyone out there has some tips for Mike.
Pingback: VMware ESX 4 can even virtualize itself « DeinosCloud
Eric,
Great post. I’ve been struggling to achieve this since long time with no luck. Unfortunately though its not working on the ESX 4.0 RC, so I guess I’ll have to wait for the GA. Besides, I believe you need to add this inside the nested VMs config: mx.allowNested = “TRUE” ? or it’s not required any more?
thanks
Eric, great post!
Rick, you are probably right about not helping with upgrades from ESX 3.x, but going forward this environment could be used to test the ESX 4 patches and upgrades. Actually, I’m told that ESX 3 will also run in a VM like this but it is hardware dependent, and I have not tried it myself.
Hany, that additional configuration element is no longer needed — it was in the past for certain configurations. By the way, internally VMware has been running ESX on ESX for a long time. A special build was required, though.
Anton, spasebo vam! It’s very cool to see my articles translated into Russian. Especially if you take a look at my background on the About page…
Hi Eric,
I am using esx on esx. I have a number of w2k3 vm’s running under it. I’m having problems loading the tools. It says Vmware tools should be installed inside a virtual machine.
Any ideas? Is there another switch I need to set.
Thanks
Mike
Pingback: New vSphere feature allows ESX as a VM - in ESX - Virtualization Pro
Eric,
Would you mind testing out installing ESX 3.5 as a VM on ESX4? This would be huge for us here!
Pingback: Finally! Running ESX as a VM inside a physical ESX host!| Virtualization .. and every thing around it| HyperViZor
Great post. I would have posted something similiar myself but I was told I would be in breach of the NDA agreement I’d signed…
Justin, you can run ESX 3.5 in a VM on certain hardware — AMD Barcelona and Shanghai, probably Intel Nehalem.
Pingback: Welcome to vSphere-land! » Running ESX/ESXi in a VM Links
Pingback: ESX as a VM - nothing new @ A. Mikkelsen
Pingback: Cheap vSphere Server « Virtualization, Windows, Infrastructure and all that “stuff” in-between
Eric,
I’m a bit confused here. I know that the ESX VM is 4.0, but what about the underlying physical ESX host, does it have to be ESX 4.0 also, or it can be ESX 3.5?
I just tired the ESX 4.0 GA as a VM on ESX 3.5U3 but it didn’t work for me, the VM starts then a I get a blank screen and nothing happens, few seconds later the VM shutdown by itself and no errors are given.
Any ideas?
Thanks
In my example, the physical host is also running ESX 4.
Hi Michael,
In my testinstallation, vESX4.0(GA,RC1) is running on physical ESX3.5U4(but I have trouble with vswif0 after installation), and on W2K3+VMWorkstation 6.5.2.
Eric,
I try this with a ESXi4 final, but I can install and boot only Esxi3.5(very unstable).
I can not install ESXi4, ESX3.5 and ESX4(Final)(PSODs and more PSODs).
Where I download this version (GA)?
My hardware is a whitebox(Athlon64 X2 7750 extreme)
Sorry for my english, thanks.
thanks Yue Yao for the note, in my case the vESX4.0 boots/run fine, and i have no issues whatsoever with the networking, it’s just the VM running within the vESX4.0 that is not work for me, I’m using physical ESX3.5U3, so I’ll upgrade to U4 and see how it goes…
I just hope Eric won’t kick us out of here :p
Pingback: 1 week of G/A of vSphere 4.0 : getting started | Virtualfuture.info
Astor: by GA I am referring to the official (final) release — not beta/RC.
Unfortunately, it’s hard to say why folks are experiencing mixed results without digging in deeper.
To get ESX 4 hosts working as VMs within your ESX 3.5 environment do the following:
1) Create the VMs to run ESX4 on an ESX 3.5 host.
2) Add these VMs to the inventory on an ESX4 host.
3) Build the virtual ESX 4 hosts as above.
4) Once built, these VMs should run in the ESX3.5 environment.
Sadly, vmware tools do not work. Any ideas?
Pingback: Twitted by dwaler
Hi,
Can you start a nested windows x64 VM under the vESX 4.0 (VM) ?
Rene, I do not believe 64-bit nested VMs are possible at the present.
Can you verify this for me? Would like to know if window 2008 x64 would run (without hyper-v)
Pingback: Running a HA / DRS ESX 4 cluster on a single HP ML110 G5 | blog.verlande.nl
rene,
I can verify x64 VM’s aren’t running within the vESX 4.0 (VM). Only 32 bit hosts boot. More info can be found here. Hope this answers your question.
I came a bit lat to this post because I was away of my Island at holidays in France. It sounds great all this.
In the pas Xtravirt published a white paper for ESX 3.0 and 3.5. I could learn a lot from this. Now They did the same thing again… For ESX 4.0.
http://www.vladan.fr/vsphere-4-in-vmware-workstation/
Heiko, thanks for pointing that out for Rene.
Welcome back, Vladan.
Pingback: vSphere ESXi as a VM – VMKernel Traffic Not Working « Virtualization, Windows, Infrastructure and all that “stuff” in-between
Great info, I was able to get 4.0 working in 3.5U4, once I modified the vSwitch.
If anyone is interested, I got it working with 768 MB of memory (instead of 2GB)- documented here http://infinitedisorder.com/?p=158
Akabdog – cool, thanks for the contribution.
http://www.jhangra.com/blog/2009/07/how-to-install-esxvsphere-4-0-in-vmware-workstation-6-5-2/
But has a video which helped me quite a bit.
Hi,
I installed ESXi 4.0 base host & couple of ESXi servers in there as a VM.
Now I am unable to run nested windows 32 bit VM’s on those ESXi servers.
All I get is a blank console while booting the nested VMs.
I even tried deploying a VM from Marketplace, however, the VM gets deployed, gets poweredON, but only a blank console to look at.
No error messages as such while powering ON the nested VM.
Thanks in advance.
Ali
Hi,
Great article. I’ve been able to setup a complete vSphere test rig on a single €200 ML115.
One issue I have is that I can’t add hosts to a dvSwitch. I have available NICs but the Add Hosts list is blank. Has anyone had any luck with this?
Thanks,
Neil.
Hi,
I just answered my own question. dvSwitches are only available with the Enterprise Plus and evaluation licenses, so NFR licenses don’t work as they are just Enterprise.
Neil
Pingback: Il mondo è bello perchè è virtuale! @ Marco’s Informatik Blog
eric: I was successfull installing virtual ESX 4.0 inside physical ESX 4.0. But I am not able to power on VM’s inside Virtual ESX. I had did the configuration changes as you have mentioned above. Could you please help me in this?
Rahul, I suspect it is your CPU. What is the exact vendor and model info?
Eric: I tried it in Dell power edge server and it worked….Thanks a lot…
keep posting this kinda stuff….its really helpfull.
Good to know, thanks.
Hi Eric,
Great Web site and articles. I was going to buy more hardware to build a test environment. Now I am going to try to create another ESX 4.0 server in my physical ESX 4.0. I hope I will be able to test vMotion with this setup.
Thanks for the great work.
Regards,
Shan
Ali – I am having the same problems with the blank/black console from vSphere (running on the Host. The issue persists in both ESX and ESXi 4.0. I suspect that it may be a Windows 7 issue (host). Like you, I have tried an appliance (2 actually, a Turnkey LAMP app and VA-ntop).
Next I plan to try:
Import a Win XP VM and try to RDP to it from within Win7.
Build a pWinXP, install vSphere and try that.
Lastly (is that a word?) Install Wkstn on the pXP machine, install vESX and try to get a console from pWin7.
I will report back, but it might be a week or 2 as I have several pESX boxes to build this week.
Hello
Has anyone been able to successfully view the console of a nested VM regardless of OS. My VMs also have blank/black consoles…
cheers
gogogo5
Yes …I had installed WIN2K3 R2 as a VM GOS inside my virtual ESX 4.0 . I didnt find any issue as you are facing. I was able to use my nested WIN2K3 VM console.
Pingback: VMworld 2009 Virtual Infrastructure Design – Lab Manager vPODS Enable Conference Cloud | VM /ETC
Hi Eric,
please could you talk a bit about your hardware setup and the performance? Are you able to run a cluster of two virtualized ESX to make some VMotion, HA etc.?
Sven,
You can absolutely exercise most of the features of vSphere with a 2-node virtual ESX cluster. VMotion, HA, Host Profiles, vDS — everything should work the same way. Performance of the nested VMs is not stellar, but you can at least get a feel for all of the functionality. Check out the various posts linked in these comments to see what others are doing with virtual ESX systems.
Eric
Hi Eric,
I have 2 x ESX4 hosts running on the same ESX3.5 AMD Dell box. I have configured iSCSI and vmotion on the 2 ESX vm’s. I can successfully see the shared storage on the 2 boxes, and created a vm that is runnign w2k3. Problem i have is that they will not see each other via the service console or vmption network. Do you know of any further network tweaks? I have already set the vswitch promiscuous mode to accept.
Thanks
Deon, it’s too hard to say for sure. Interesting that you are using ESX 3.5 as the host, did you try with ESX 4?
Pingback: IGT Part 7: Virtual VMware ESX 4 in high demand | VCritical
Hi,
Have try this install on a HP DL360 G4, ESX 4 is running fine.
But when I try to start the virtual ESX4 I receive the following error:
“This version of ESX requires a CPU capable of 64-bit operation.”
CPU’s are EM64T as the ESX4 hardware install is working.
Hi Goris,
I know I am too late in answerin this Question. But I think rest users would be able to get the benefit out of the same.
Once u create the VM…go to the Option …….check in the option where in you get the BIOS screen at Power on.
2- Now go the advanced tab in the processor you have 1.4 & 1.1. select 1.1 version….
press F10..you are ready to go.
It worked for me(MY White box is Q9550 + S3210 SHLC+ 8 GB).
Hi sripati,
I tried this with my DELL PE 2850 and the host esx is working with ESX 4, and when i followed ur steps above i get the same error still “This version of ESX requires a CPU capable of 64-bit operation.”.
Then i down graded the esc to 3.5 U5 and still i get the error.
But I can run ESX4 on the DELL tin still what do i need to do please let me know anyone so I can start working on my lab and get my VCP soon.
many thanks
DA
have run “CPU Identification Utility” from vmware.
It seems to the CPU’s can not run 64bit as virtual 🙁
Supports 54-bit Longmode: Yes
Supports 64-bit VMware :No
Hi,
I cannot run VMs on “virtual” ESX4, blank console and 100% CPU,
is-it related to AMD opteron ? I just bought a poweredge sc1435 for a lab … 🙁
Nicolas
I have seen this issue with not enabling Virtualization Technology on the AMD processor in the BIOS – double check and let me know if that works!
I’m running 7 ESXi 4 to test HA stuff…
Thanks again for this good post Eric
Eric,
I am running VM in ESX 4.0 Evaluation version on my desktop, but unable to power on the VM in it. I just get a black/blank screen and never get any screen even though it says that the VM is powered on. I have an ISO for windows 2003 for which I dont get any installation wizard.
Guru.
Hi Eric
Do you have an Intel or Amd ?, you MUST have virtualisation instructions enabled ( VT / in the bios).
Nicolas
Pingback: My Home Lab Setup « Rick Vanover's Blog
Nicolas,
I have an AMD 64 bit, enabled the bios setting (NX for AMD) for virtualization.
Guru
Ok, on amd there is a special generation level you must have !!, even with VT enabled it does’t work.
You can search if you have a second generation or a third one this link, you MUST have at least a third one !!
http://www.cpu-world.com/CPUs/K10/AMD-Third%20Generation%20Opteron%208356%20-%20OS8356WAL4BGD.html
Thanks for your contribution, Nicolas.
I have an Athlon 64.. Not sure about the generation.
Guru
Eric, This is great info.
I had discovered this for 3.5 last year, Check posts from vmware Academy Partners,
One added issue we have been having with our dell 2950’s is that the install would hang on formatting the vmfs.
We have been able to resolve this by using a RDM for the disk of the esx VM.
Pete
Eric,
I have a HP DL385 G2 with AMD Opteron Processor. I have enabled the Virtualization in BIOS. This machine has enough RAM and DASD.
I was able to install ESX 4 on the bare metal machine
I am trying to get more familar with vSphere and to get familar with other features I needed another ESX server so I followed your post and was able to successfully install a virtual ESX 4i instance within the ESX 4 server. I have successfuly added both the ESX servers in the test cluster in vCenter.
I have also added a bare metal pentium 4 OpenFiler machine which is working as iSCSI storage. I have successfully configured and openfiler storage is visible to the ESX4 and virtual ESX4i instance
virtual ESX4i instance allows me to create virtual machines. However when I power them on I am not able to see anything in the console. When I power on the same machine and migrate it to the physical ESX host and I can see the console.
I am a virtualization newbie who is trying to learn … any assistance will be greatly appreciated.
Thanks!
Hi Eric… Thx for this post!
I executed the 2 tweaks and was able to install an ESX3.5 server and an ESX 4 server
The ESX4 server fails after a reboot. it fails on vsd-mount ….
Any suggestions?
The ESX3.5 runs fine
Thanks
Fred
It looks like we need another tweak for snapshot usage….. because that was the issue.
The ESX 3.5 reboots fine (with snapshot in use) but the ESX4 fails on vsd-mount
Any suggestions?
Tx
fred
I’m seen the same issue when I create snapshots on Virtual ESX 4s. I think it has to do something with the fact that the COS is on the same vmfs disk as the one you use for regular vmstorage. I’ve Google and found the issue, but I’m not sure if anybody has found a workaround yet.
FYI, I had problems getting ESX 4.0u1 to install under ESX 4.0 on an HP DL680c G5. When the install would start formatting the disks, vmkfstools would fail with status 65280, print the vmkfstools command options, then say “Error: not supported”. It turns out I only had the problem when the guest disk was on a volume on a NetApp filer via NFS. Putting the volume on direct-attach disk led to a successful installation.
Pingback: Take snapshots of VMware ESX 4 running inside a VM with this undocumented setting | VCritical
The snapshot issue may be addressed with an undocumented configuration setting. Please see https://vcritical.com/2010/02/taking-snapshots-of-vmware-esx-4-running-in-a-vm/
Thanks for this advanced configuration setting Eric!
My Physical ESX host is ESX4i and sadly it is an unknown Option in there…
C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
vh10 –username root -g /COW/PermitVmfsOnRedoHierarchy
Enter password:
Option /COW/PermitVmfsOnRedoHierarchy not found.
C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
vh10 –username root -s 1 /COW/PermitVmfsOnRedoHierarchy
Enter password:
Option /COW/PermitVmfsOnRedoHierarchy not found.
Since this is an unsupported/undocumented configuration setting, it cannot be controlled via remote API. The good news is that you should be able to just log into the “tech support mode” console of ESXi and run the same command that works on the service console.
In other words, go to the ESXi console, press ALT+F1, type in the word “unsupported” (blindly, as it is not echoed to the screen), enter your password when prompted, and then run the esxcfg-advcfg command.
Thanks again Eric! i went to “unsupported “service console” on the physical ESX4i and execu ted esxcfg-advcfg -s 1 /COW/PermitVmfsOnRedoHierarchy
made a snapshot and could startup without any problem.
Pingback: Link – Running VMware ESX 4.0 on itself. | TechHead.co.uk
Pingback: VMware ESX 4 can even virtualize itself | VCritical | scandals of an ITian
Hi Eric,
I have installed Vmware Vsphere on vmware workstation and I am able to create the HA & DRS cluster with FREENAS Shared storage but my problem is when I create the VM and try to power on the VM with the ISO or any other medium, I am getting blank black screen and I am not getting any kind of error but just the blank screen.
I even changed the points like monitor_control.restrict_backdoor = TRUE etc.
When I poer on the ESX host VM I am still getting the error that I can’t run the NESTED VMs.
Prashant,
Could it be that you have an older AMD CPU? If you notice, some of the comments above have had similar issues and one reader felt it was AMD-specific. I have used AMD Shanghai CPUs for this and everything worked properly.
Eric
Eric,
Once again — good stuff. I set up ESXi V4 on a Dell R900 W/128GB Ram and 8 CPUs. We have 15 or so test VMs running on it.
With that said I downloaded the 60 trial of ESX 4.0.u1 and vSphere. Installed a 3 host cluster and turned on HA and DRS. Added 6 guest VMs on the virtualized ESX and all works great, systems are a bit slow but that’s what we get for multilayer hypervisors.
My question is — has anyone got Fault Tolerance to work? When I turn it on I get a hardware compatibility error. It seems that FT wants VT turned on in the bios, but the VMware bios of the guest ESX server goes not have this option. The backend storage is IB so there is no delay there. Any ideas?
I need to give a presentation demo to the higher ups and they are interested in FT.
Hi Eric,
Thanks for this post!
I have installed ESX 4 on a white-box (Dell 9200) and it works like a charm.
Now I’ve followed yours instructions to install ESXi4 as a VM on it.
And I got a PSOD “No place on disk to dump data”.
Any suggestions ?
Thanks
Hi,
I have two esx servers as guest os on my physical esx host but the two esx guest are unable to communicate each other.
Please help me in acheiving this feature.
Thanks
Bhaskar.
Bhaskar — you need to change the Security on the Service Console of your physical ESX host AND the VM Traffic Port on your physical ESX host to allow Promiscuous Mode (change the setting to Accept instead of the default of Reject). That will then allow communication between your ESX hosts.
Datto
Hi Eric we have the same name! Anyway, if you need a good company to buy any VMware vSphere 4 products, a good place is my company ITO Solutions. Here is a link to our site for VMware ESX http://www.itosolutions.net/Server_Datacenter_Virtualization_s/523.htm
Hi Eric, do you have any clue on how to run 64bit VM inside the ESX that is running over a virtual machine?
Currently not possible…
Hi,
I have installed ESX 4.0 on workstation 7. the ESX has 22 gig VMFS datastore. I am trying to create a new VM in ESX and dont know how to boot from ISO to install the OS. After creating the VM when I go to connect cd/dvd it did not detect the OS in dvd drive. Do i have to copy the ISO to datastore first if yes then how i can do that or I can only create VM using the VC 4.
any help appreciated
Ok, so I have a problem with my esxi 4 “update 2” guests running under the same esxi 4 version for the host. The problem is that the guest esxi servers do not have correct time despite having ntp setup just like the host.
The host esxi server has been pointed to pool.ntp.org and shows the correct time via the vshpere client. The guest/s esxi servers are pointed to the same exact ntp server “pool.ntp.org” but after restarting the service just like I did on the host the time does not change and is off. I manually set the time on the guests to the correct time but after 8-10 hours the time is faster on the guests by about 2 minutes and some change. Are others experiencing time difference on there nested esx/i vm’s? I figured it would use the ntp server and adjust it time correctly just like the host did. I also verified they could resolve and communicated with the ntp server …so that can be taken out of the equation.
Thanks,
Mike
Pingback: Why we are using VMware Lab Manager. | Planetchopstick, dont ask wont tell
Great article, help me established a test ENV for Lab manager, thanks!
Hello Eric
My Host OS is windows 07, on which I have installed VMware server and ESX 4.0 inside it
Following is the order
Win07Ultimate (Host OS)–> VMware-server-2.0.2-203138 –>ESX4.0 –> win 2008 VM
I tried above solution of setting up the Vswitch in promiscuous mode and adding the line “monitor_control.restrict_backdoor / TRUE”
But still I get the error message “You may not power on a virtual machine in a virtual machine”
Is there any solution if the ESX is not directly installed on the hardware as the host OS
Regards,
Anantha
Hello all,
I tried to install the ESX v4 server on ESX v4 server as described in the beginning of this article. The installation cannot even take place.
Error code:
CosSup: 406: vmkernel not loaded: cannot PSOD – Basically not supported microcodes.
My cluster configuration is with EVC AMD Opteron Generation 3. The exact processor type is Ams Opteron Quad-Core 8389.
Installation of ESX v3.5 is working, but after reboot I have the same error all the time even with different VM settings.
Error code:
…….initialization of vmkernel failed, status 0xbad001e – Microcodes problem again.
Can someone help?
Thanks.
Does any one tried to move ESX 4.0 Host VMs to Another ESX4.0 Host.
I hve Dell 2950 2.6 GHz 8x Core, want to shift my ESX 4.0 Linux VMs from this one to DELL 2950 2.0 GHZ processor.
Any idea?
I have VCenter VM also on the same ESX Host1.
Thanks for advise in advance.
MAZ
MAZ – You should have no problems moving VM’s. My questions would be is how exactly are you thinking of doing this? Are you wanting to use vMotion? Are these host in a HA/DRS cluster? Answering these questions would help me better understand what answer your looking for.
Thanks!
Pingback: Kasraeian
Just wanted to note that anyone “experiencing” issues with blank screens and such on AMD platform CPUs be sure to check Vritulization Enabled on the Processor settings in the BIOS. I know this may sound kind of insulting but believe it or not I missed this on my system and yes I felt like a total newbie. I could easily run VM’s in workstations but had random issues with nested VM’s. Just double check and make sure that you weren’t pulling a “chad” and NOT saving the BIOS configuration…. would’ve saved me a headache..
Hi,
I have some problem while installing ESX 4.0 Server in Vmware Work Station 6.5 in Home. Problem is they need any External Network Connection is Failed.
So Any one of them give me some solutions
Pingback: Virtualizzazione “indentata” « ILSimo's Blog
Pingback: How-to run XenServer on top of vSphere | PlanetVM
I cannot create VM’s on Virtual ESX 4.1 server. I followed the notes to create virtual ESX 4.1 on the physical ESX 4.1. When, I try to create VM’s inside virtual ESX 4.1, I get this error –
host cpu is incompatible with the virtual machine’s requirements at cpuid level register edx
Any help appreciated.
Tks / RL
you can only do BT in a nested ESX, try 32bit VM 😉
Hello people
I’m trying to attach an external USB drive on the ESXi host. I can see the USB drive but cannot do much with.
Is there away to have a USB drive working as an external storage attached to the ESX?
thanks
Louis
Just a slight aside here, I have LabManager running inside LabManager at the moment, just thought I’d show off! 😉 We’re running multiple test environments to play with, and one happened to be a LabManager implementation. No modification needed, just flew straight up!
Inception.
Eric,
I am using vESXi to run our VMware Academy. The one issue I can’t seem to resolve is getting NTP to work with the vESXi. It did work when I used vESX however the trend from VMware is towards ESXi. I believe this is creating an issue for the vCenter to use AD for authentication. I can get the VM that is running the VC to join the AD but the VC won’t since the hosts are not in sync.
I’ve also noticed your comments on 64bit OSs, that are “currently” not possible. Are you aware of any potential fix for this?
Thanks again for all the posts.
Hello,
I have vSphere cluster consist of three hosts. Can I setup vESXi4.1 as VM to run on ESX4.0?
Thanks,
Hussain
Most likely you will run into problems with the keyboard and won’t be able to install ESXi 4.1 on an ESX 4.0 host.
Hello,
Yes, I faced this issue. Any trick for it?
Another question; i have successfully installed vESX4.1 on pESX4.0 server. And I managed to run nested 32bit VM on the vESX4.1 hosts.
How can I install the VMWare Tools inside the nested guest OS?
Thanks
Habibalby – I was able to install vESX4.1 inside pESX4.1 however, I am unable to run nested 32bit VM’s inside vESX4.1. How did you do? Appreciate your input pls.
pESX4.1 is Dell E6400 Laptop Intel VT with 8GB Mem.
Thanks
hi Ram,
you need to enter this line in the Advanced Configuration “monitor_control.restrict_backdoor / TRUE”
Hi,
Can someone advise me if it is possible to install ESX 3.5 as a VM on ESX4? I have a client that has a software product that is only supported esx 3.5 but they platform they have is vsphere 4.0?
@spice : yes it’s possible and it works 🙂
Thanks NitRo – How stable is it? would you build for a production environment?
Hello once again,
I’m finding this lab is very interested and questions keep raising:)
What about VMTools on the vESX Server? Do I have to install it? If yes, how to?
Thanks,
Worked well until I tried to enable either HA or EVC in a cluster. Nor could I start the VMWare Data Recovery appliance on the nested VMs. All of which worked without a problem when done on the bare metal ESXi 4.1 hypervisor.
Messing with the CPUID would allow the VDR VM boot to start but it wouldn’t get much further into the boot process than a few seconds.
A little annoying when studying for the VCP not being able to play with those 3 major components. Still not the end of the world.
I tell a lie, increased the memory from 2GB to 3GB and the HA agents were able to start and HA works.
Since you’re just looking at using your nested ESX setup for training, you might look at using the 1.0.x version of VDR under a nested ESXi 4.1 as described similarly by LifeGood in this forum post:
http://communities.vmware.com/thread/290907
Datto
Thanks!
For our VMware Academy, I am planning to use a hybrid approach, since vSphere4.1 has hosed several of the Labs: 64bit Data Recovery VM, HA, plus the others I’ve mentioned earlier.
The hybrid approach is by using SuperMicro’s X8SLI-F motherboard. It is totally compatible with esx/esxi 4.1, and available for <$200. It even includes IPMI.
I still will use the nested approach for most of the labs, but I thought you might like to know of this mb.
It should be SuperMicro; X8SIL-F
Matt:
Good on Ya, as they say downunder.
I bumped the memory up on the nested ESXi hosts then and HA now functions.
This buys me a some time since I don’t feel as stressed to implement the hybrid setup. To me though the writing is on the wall that unless there is some way to enable VT in the vESX BIOS the requirement for 64bit OS is going to, as Pink Floyd would say, the Wall.
It is definitely an advantage in having the vESX hosts which is why I still plan to run the training as a hybrid. Being able to build Physical hosts under $1k that can be used as a home or small business Cloud server is tough to beat.
BTW: There is also an issue with the version of the X8SIL, make sure it is v1.02. This allows for i5 intel processors to work. I’ve have read that it will even work with i7s but I can not confirm this. I’m using neither since the x3400 series will allow upto 32G RAM. You need to verify the RAM is compatible.
Hi Eric,
I’m so glad I stumbled upon your site! I almost bought a 2nd Server for a home lab setup to test out the HA, DRS, vMotion, vDS, etc. Your article just might have saved me thousands (literally)! I initially tried to test the Enterprise features on a virtual ESXv4 running on a Workstation product hosted on a desktop w/ Intel Core 2 Quad Q6600; however, I quickly realized that even w/ the EVC enabled, I could not do vMotion, etc.
In any case, here is my current config:
– 24GB Mem, 1 Quad Core Xeon, 2x2TB HDD, 4 x Gigabit NICs
– ESXi v4 on internal SD/Flash
Motivation: Learning/Hobby/Home Virtual Infrastructure (i.e.: Not production)
Here are my questions:
– Ideally I’d like to do ESXi on ESXi installation; however, I only have one internal SD
module w/ 1GB media. Is it possible to do two ESXi installation on ONE 2GB SD media?
This way, I can boot to Host ESXi from SD as it is currently setup. Then, after I have
booted, power-on the 2nd instance of ESXi that is also hosted on the same SD card.
– If the above is not possible or recommended, then I would most likely do ESX on ESX
installation. Are there any gotchas with this setup with regards to the vDS (virtual
distributed switch), networking in general, 32-bit/64-bit guest OS support?
– Also, I’m looking to configure VLAN networking across two (physical & virtual) ESX/i
hosts. Any suggestions/recommendations/pointers on how I can do VLAN 801q
tagging for all VLANs through the virtual distributed switch only (i.e.: without a physical
switch that supports VLAN)? If so, are there any articles on howto setup/configure
VLANs on ESX?
Sorry for bombarding you with all the questions, but my recent interest in the ESX product/features has got me too excited.
Again, I can’t thank you enough for posting some of the most valuable articles on the ‘net (e.g: ESX on ESX, P2V VMotion, etc.).
Sincerely,
Jia
This is a very helpful post. I have one issue though. I was able to install a Windows 2003 server VM inside an ESX VM but the Windows VM can receive IP address via dhcp but cannot ping LAN IP or connect to the internet. I have disabled firewall settings still got the same result.
Please help.
Thank you
Dave — you need to engage “Promiscuous Mode – Acceot” on a) the Service Console Management Port Group as well as engage it on b) the Virtual Machine port group.
Assuming your using lab equipment and not production servers, click on the Properties of your VSwitch in the VIC, select the port group and choose Edit, find the security tab, check the Promiscuous Mode checkbox for Accept/Reject and choose Accept. Then OK out of the dialogs. Then do the same for the other port group/VSwitch (if the VSwitch is different than the VSwitch used for the Service Console).
That will then allow traffic out from the nested VMs running on nested ESX 4.x hosts.
Datto
Pingback: Virtualization test lab tricks with SSDs | TechRepublic
I was able to do this, but I have an interesting issue when I try to VMotion.
The physical hosts are Intel Xeon Quad Core CPU’s with Virtualization Technology turned on in the BIOS.
When I create a VM on the virtualized ESX Hosts, it says I need to enable the Virtualization Technology on my virtualized hosts. Kind of confusing.
3 Physical 4.1 ESX Hosts (in a cluster)
3 Virtual 4.1 ESX Hosts (in a cluster)
1 Virtual Machine on the Virtual 4.1 ESX Host cluster (giving the warning about enabling Virtualization on the host)
Jordan,
In most cases, the vMotion issue can be solved by changing the CPU/MMU Virtualization option on the virtual host. In my test environment running ESXi 4.1 Update 1, I was successful using the 3rd option “Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization”. Your CPU may allow or require a different choice. Try all the available options and test the system’s reaction to each one.
The virtualised Hypervisors no longer have access to the Hardware Virtualisation instructions (i.e. Intel VT and AMD-V) in the bios/cpu since they are only seeing vCPUs presented from the hypervisor on the baremetal. You could try messing with the CPUID Mask in your VM but your best option would probably be to enable Enhanced VMotion Compatibility (EVC) on the cluster.
http://www.vmware.com/files/pdf/vmotion_info_guide.pdf
I understand the issue that they no longer have direct physical access to the hardware anymore. I forgot to mention I did try to enable EVC Intel on the cluster but got the error:
“EVC cannot be enabled because the cluster contains hosts that lack EVC-capable hardware”
Which also makes sense.
There are some VMs that require access to the hardware virtualised instructions, could that be the issue? I ran into that issue when trying to use the VDR VM, if that is the VM you are trying to run then read back a few posts Datto had a good suggestion. (figured this was worth a mention since its a little unclear from your posts if the error is occurring when you created the VM or during a vmotion within the virtualised cluster)
Other than that all I’ve got are rhetorical questions, sorry if you’d run though this already: Did EVC work in your Physical hypervisor cluster (i.e. are the physical CPUs happy to work together even before they were virtualised)? Were all the VMs shutdown in the cluster you enabling EVC? Does altering the affected VM’s CPUID Mask change anything?
Decided to do a bit of reading after I checked my lab and found the same issue with EVC. Seems EVC has a requirement of having hardware virtualisation enabled in the bios so its not going to work for a cluster with the nested hosts.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003212
This of any benefit for the vMotion issue?
http://vinf.net/2009/06/12/vsphere-esxi-as-a-vm-vmkernel-traffic-not-working/
For those with smaller lab budgets —
Just wanted to mention that this morning I was able to consolidate eight nested Classic ESX 4.0 hosts each with three lightly used Win2003 VMs as well as two nested XenServer 5.x hosts each with two lightly used CentOS 5.2 VMs on a single physical ESX host. For the physical host I used a 1U Dell SC1435 server with 2x Opteron 2356 CPUs and 16GB of physical memory running ESXi 4.1 from a USB flash drive. Storage for VMs was from one of my Fibre SANs.
My total cost for the Dell SC1435 server with parts from eBay for it was $512.70 (detailed costs were 2x Opteron quad 2356 CPUs were $84.26 delivered, the Dell SC1435 server was $222.44 delivered and I inventoried the parts not used from this server, 16GB ECC REG memory was $160 delivered, the QLogic PCI-e HBA was $40 delivered and the flash drive was about $6 delivered). I didn’t use the 2x hard drives delivered with the Dell SC1435 server nor the existing DC CPUs that came with the Dell SC1435 server since I’d separately bought two 3rd generation 2356 Opterons for nesting purposes and intended to run the VMs from SAN. I could have just as easily run the nested VMs off the local hard drives and not have needed the Fibre HBA.
Performance inside the nested VMs running on the nested ESX/nested XenServer hosts appears to me to be the same as when those same VMs ran on nested ESX hosts/nested XenServer hosts that previously ran on three different physical 1 CPU 8GB ESX white box servers. The physical Dell SC1435 server currently is running about 32% for overall CPU, 88% for memory use with about 2GB for vSWP in use (all the nested ESX/nested XenServer VMs get 4GB memory assigned while the nested ESX servers get 1 vCPU and the nested XenServer servers get 2x vCPUs).
If I’d have used ESXi 4.1 for the nested ESX hosts instead of the Classic ESX 4.0 hosts that would have saved me a bit more memory use but I’d already had the pre-built nested Classic ESX 4.0 VMs built from a different project.
Great setup, thanks for sharing!
Hi Eric,
Very good post, especially the part on adjusting the vSwitch and enable promiscuous mode. It saved my day.
ESXi on ESXi works well without this change. But after installing ESX on ESXi, trouble begun and could only be fixed by changing to promiscuous mode. At this moment the theory behind this change is not completely clear to me…
Regards,
Paul
Has anyone experienced problems with untagged traffic? All my other VLANs / Portgroups work great on both the physical host and nested VMs but I have one untaged portgroup and the VMs are only able to communicate if they are on the physical host. I’m using a distributed switch and have promiscuous mode enabled on all the portgroups.
Has anyone else run into issues when cloning ESXi VMs?
It seems to work fine initially, but the vCenter managing the virtual ESX hosts gets confused about the datastores. The local datastore on each virtual ESX has the same UUID, leading to problems in vCenter.
Anyone else seen or solved this? How are the disk UUIDs generated?
because you’ve cloned the esx. I’m facing the same issue on vmware workstation
Im trying to install ESX4.0 on the new VM already running in ESX4.0 connect with SAN Storage. The new VM used for the ESX installation space allocated from the SAN Datastore. I need to know during the ESX 4.0 setup on VM ver7.0 would it format only the space allocated to the VM disk or the entire SAN LUN which provisioned to the VM. (Currently I’ve 2 running VM’s in the Datastore) Thanks
vSphere 5 makes this easier by adding a guest OS type for ESX/ESXi. It also supports nested 64-bit guests.
Check out https://vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/
Is it possible to install vsphere 5 with 2 esxi in a vsphere 4.1.xxx environnement.
Should i follow the same instruction as in here or is it different for the guest os.
also is the promiscuous mode really important?
lafrenal@crim.ca
I use VMs for separating my development environment from my testing environment and use Oracle’s (formerly, Sun’s) VirtualBox. It meets my need though it isn’t stable enough to run on a server. I did run VirtualBox within a Hyper-V virtualized Windows server install and was able to run a VM within a VM, though I used two different products for it.
Just finished getting my single new host on ESXi 4.1 U1 up and running. Connected it to my Iomega ix4-200d, and now using iSCSI I have a vCenter VM and two ESXi 4.1 U1 hosts one one layer, and then the second layer I have the virtualized hosts running in a full HA/DRS cluster sharing two iSCSI LUNS. Your original post and the comments here helped me get this thing going – very much appreciated!
Pingback: VMware vSphere can virtualize itself + 64-bit nested guests
I made an ESX vm okay but no matter what I do I can’t connect to it with the VI client and I can’t add it as a host in Virtual Centre. It’s on the same vswitch as the other vms, so basically it’s on the same network as the VirtualCentre box as far as I know, and obviously I can connect to that and that can also connect to the Internet. So I don’t know what’s up. I tried giving it different ip addresses and using a DHCP setting. The virtual machine network is on one physical nic and the management network on the other. Someone suggested moving all the machines over to the management network but I haven’t been able to do it.
sounds like might have overlooked the promiscuous mode on the vswitch as per the original article.
I added the “monitor_control.restrict_backdoor / TRUE line” like you said but I still get the error “Running VMware ESX in a virtual machine requires the outer virtual machine to be configured for running a VMware ESX guest operating system. You may not power on a virtual machine until the outer virtual machine is reconfigured.” when I try to power on the nested virtual machine.
Richard,
Just to verify a few points.
The “monitor_control.restrict_backdoor” parameter is added not to the nested VM, but to the Advanced configuration of the VM that contains your virtual host.
The Parameter name is “monitor_control.restrict_backdoor” and the Value is “TRUE”. Check to make sure there are no leading or trailing spaces – this can happen a lot if you copy/paste from the article. Also, make sure to NOT use the “/” character.
If this all checks out and you still have this problem, download and open the .VMX file of the virtual host in a text editor like Wordpad. The line should look like this –
monitor_control.restrict_backdoor = “TRUE”
Pingback: Kasraeian
I have imported a virtual machine(win xp 32 bit) to the virtual ESX host, but the network connection of the win xp is not available, I try many ways, could any one please tell me how to create valid network connection with other machines?
Thanks in advance
Jack,
Not sure exactly the symptoms from your description, but I’ll try.
First, you need to treat the virtual host as though it is a normal ESX host. This means you need to create any port groups on this server with the proper VLAN IDs and Port Group name. The port group name must be the same as what is defined on the other systems.
Next, the uplinks – the vmnics – on your virtual host need to have all of your VLANs trunked to it. The easiest way to accomplish this is to use promiscuous mode. You need to set this on the physical host.
Example – host ESX01 is the physical host, ESX02 is running inside a virtual machine on ESX01. You would create a vSwitch (vSwitch1 for example) on ESX01 and put it in promiscuous mode. The VM that runs ESX02 would have one or more virtual NICs tied to ESX01 – vSwitch1. At ESX02, you would make sure that your vSwitch is joined to vmnic0 and create a VM port group on the proper VLAN for your XP VM.
Go back and review that section of the posting. You may have used promiscuous mode on the wrong switch.
Pingback: Building a Multi-Tenant Veeam Replication Target
Pingback: How To Install The Cisco Nexus 1000V on vSphere 5 - Ray Heffer