VMware ESX 4 can even virtualize itself

ESX on ESX

NEW: VMware vSphere 5 makes this even easier and supports nested 64-bit guests.

Running VMware ESX inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. It is pretty common to do this on VMware Workstation nowadays — the first public documentation of this process that I know of was published by Xtravirt a couple of years ago.

But what if you prefer to run ESX on ESX instead of Workstation?

You may be pleased to know that the GA build of ESX 4 allows installing ESX 4 as a virtual machine as well as powering on nested virtual machines — VMs running on the virtual ESX host.  You can even VMotion a running virtual machine from the physical ESX to a virtual ESX — on the same physical server!

VMware vSphere 4.1 UPDATE: VMware ESXi 4.1 has a keyboard issue when virtualized on an ESX 4.0 host.  In order to virtualize ESXi 4.1, the underlying host must be 4.1.  However, ESX 4.1 classic will work on ESX 4.0.

The extra tweaks to make it all work are minimal, and I will show you how without even opening up a text editor.

After installing ESX 4 onto your real hardware, configure as desired and enable promiscuous mode on a vSwitch:

vswitch0 promisc

Create a new VM with the following guidance (choose “Custom”):

  • Virtual Machine Version 7
  • Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
  • 2 VCPUs, 2GB RAM
  • 2 NICs – e1000
  • LSI Logic Parallel
  • New disk – reasonable size greater than 10GB for ESX classic

After you have the VM ready, simply attach a VMware ESX 4 ISO image, power on, and install ESX as a guest OS.

Virtual ESX 4

After installation, add the new virtual ESX to vCenter 4 and create a new VM.

vSphere Client with virtual ESX

If you do not need to run VMs on your virtual ESX, you can stop there.  However, if you try to power on that nested VM, you will see the following error:

You maynot power on a virtual machine inside a VM.

To prevent this, just one tweak is needed:

  • Shut down the virtual ESX VM
  • Click Edit Settings
  • Click the Options tab
  • Click Advanced / General / Configuration Parameters…
  • Click Add Row
  • For the Name/Value enter: monitor_control.restrict_backdoor / TRUE

Editing VM configuration.

The above procedure is just an alternative to hand-editing the .vmx file — if you prefer to do it that way, feel free.

Now you are ready to power your virtual ESX VM back on, as well as the nested VMs.  This capability should come in handy as you start investigating the new features of vSphere 4.

UPDATE:  In order to take snapshots of a VM running ESX, a quick configuration change is necessary on the physical host .

You may be interested in this related post where a VM is migrated between the physical and virtual ESX hosts.

(Visited 16,162 times, 1 visits today)
This entry was posted in Virtualizationism and tagged , , , . Bookmark the permalink.

159 Responses to VMware ESX 4 can even virtualize itself

  1. Rick Vanover says:

    Eric: Good stuff. I think this is a great way for administrators to get started with the base functionality of the product without allocating a tremendous amount of hardware. This won’t help too much with upgrades, but will definitely help with the base product!

    Cheers.

    • PAramesh says:

      Thx you very much . it was very use full for me .. i don’t find any Youtube video on this ..

    • Mike says:

      Thanks, this works well. I do have a problem loading the tools on the vm’s I create on the virtual esx. Does anyone have any ideas.

      Thanks

      • Mike says:

        i’ve also noticed that you can’t clone a vm from one virtual esx to another virtual esx.

        • Eric Gray says:

          You should be able to come up with a way to do this, but there are some issues — such as duplicate MAC addresses — that need to be dealt with.

          Feel free to chime in if anyone out there has some tips for Mike.

  2. Pingback: VMware ESX 4 can even virtualize itself « DeinosCloud

  3. Hany Michael says:

    Eric,

    Great post. I’ve been struggling to achieve this since long time with no luck. Unfortunately though its not working on the ESX 4.0 RC, so I guess I’ll have to wait for the GA. Besides, I believe you need to add this inside the nested VMs config: mx.allowNested = “TRUE” ? or it’s not required any more?

    thanks

  4. Eric Gray says:

    Rick, you are probably right about not helping with upgrades from ESX 3.x, but going forward this environment could be used to test the ESX 4 patches and upgrades. Actually, I’m told that ESX 3 will also run in a VM like this but it is hardware dependent, and I have not tried it myself.

    Hany, that additional configuration element is no longer needed — it was in the past for certain configurations. By the way, internally VMware has been running ESX on ESX for a long time. A special build was required, though.

    Anton, spasebo vam! It’s very cool to see my articles translated into Russian. Especially if you take a look at my background on the About page…

    • Mike says:

      Hi Eric,

      I am using esx on esx. I have a number of w2k3 vm’s running under it. I’m having problems loading the tools. It says Vmware tools should be installed inside a virtual machine.

      Any ideas? Is there another switch I need to set.

      Thanks
      Mike

  5. Pingback: New vSphere feature allows ESX as a VM - in ESX - Virtualization Pro

  6. Justin says:

    Eric,
    Would you mind testing out installing ESX 3.5 as a VM on ESX4? This would be huge for us here!

  7. Pingback: Finally! Running ESX as a VM inside a physical ESX host!| Virtualization .. and every thing around it| HyperViZor

  8. Great post. I would have posted something similiar myself but I was told I would be in breach of the NDA agreement I’d signed…

  9. Eric Gray says:

    Justin, you can run ESX 3.5 in a VM on certain hardware — AMD Barcelona and Shanghai, probably Intel Nehalem.

  10. Pingback: Welcome to vSphere-land! » Running ESX/ESXi in a VM Links

  11. Pingback: ESX as a VM - nothing new @ A. Mikkelsen

  12. Pingback: Cheap vSphere Server « Virtualization, Windows, Infrastructure and all that “stuff” in-between

  13. Hany Michael says:

    Eric,

    I’m a bit confused here. I know that the ESX VM is 4.0, but what about the underlying physical ESX host, does it have to be ESX 4.0 also, or it can be ESX 3.5?

    I just tired the ESX 4.0 GA as a VM on ESX 3.5U3 but it didn’t work for me, the VM starts then a I get a blank screen and nothing happens, few seconds later the VM shutdown by itself and no errors are given.

    Any ideas?

    Thanks

  14. Yue Yao says:

    Hi Michael,
    In my testinstallation, vESX4.0(GA,RC1) is running on physical ESX3.5U4(but I have trouble with vswif0 after installation), and on W2K3+VMWorkstation 6.5.2.

  15. Astor says:

    Eric,
    I try this with a ESXi4 final, but I can install and boot only Esxi3.5(very unstable).
    I can not install ESXi4, ESX3.5 and ESX4(Final)(PSODs and more PSODs).
    Where I download this version (GA)?
    My hardware is a whitebox(Athlon64 X2 7750 extreme)
    Sorry for my english, thanks.

  16. Hany Michael says:

    thanks Yue Yao for the note, in my case the vESX4.0 boots/run fine, and i have no issues whatsoever with the networking, it’s just the VM running within the vESX4.0 that is not work for me, I’m using physical ESX3.5U3, so I’ll upgrade to U4 and see how it goes…

    I just hope Eric won’t kick us out of here :p

  17. Pingback: 1 week of G/A of vSphere 4.0 : getting started | Virtualfuture.info

  18. Eric Gray says:

    Astor: by GA I am referring to the official (final) release — not beta/RC.

    Unfortunately, it’s hard to say why folks are experiencing mixed results without digging in deeper.

  19. TimB says:

    To get ESX 4 hosts working as VMs within your ESX 3.5 environment do the following:
    1) Create the VMs to run ESX4 on an ESX 3.5 host.
    2) Add these VMs to the inventory on an ESX4 host.
    3) Build the virtual ESX 4 hosts as above.
    4) Once built, these VMs should run in the ESX3.5 environment.

    Sadly, vmware tools do not work. Any ideas?

  20. Pingback: Twitted by dwaler

  21. Rene says:

    Hi,

    Can you start a nested windows x64 VM under the vESX 4.0 (VM) ?

  22. rene says:

    Can you verify this for me? Would like to know if window 2008 x64 would run (without hyper-v)

  23. Pingback: Running a HA / DRS ESX 4 cluster on a single HP ML110 G5 | blog.verlande.nl

  24. rene,

    I can verify x64 VM’s aren’t running within the vESX 4.0 (VM). Only 32 bit hosts boot. More info can be found here. Hope this answers your question.

  25. Vladan says:

    I came a bit lat to this post because I was away of my Island at holidays in France. It sounds great all this.

    In the pas Xtravirt published a white paper for ESX 3.0 and 3.5. I could learn a lot from this. Now They did the same thing again… For ESX 4.0.

    http://www.vladan.fr/vsphere-4-in-vmware-workstation/

  26. Eric Gray says:

    Heiko, thanks for pointing that out for Rene.

    Welcome back, Vladan.

  27. Pingback: vSphere ESXi as a VM – VMKernel Traffic Not Working « Virtualization, Windows, Infrastructure and all that “stuff” in-between

  28. akabdog says:

    Great info, I was able to get 4.0 working in 3.5U4, once I modified the vSwitch.
    If anyone is interested, I got it working with 768 MB of memory (instead of 2GB)- documented here http://infinitedisorder.com/?p=158

  29. Eric Gray says:

    Akabdog – cool, thanks for the contribution.

  30. Ali says:

    Hi,

    I installed ESXi 4.0 base host & couple of ESXi servers in there as a VM.
    Now I am unable to run nested windows 32 bit VM’s on those ESXi servers.
    All I get is a blank console while booting the nested VMs.
    I even tried deploying a VM from Marketplace, however, the VM gets deployed, gets poweredON, but only a blank console to look at.
    No error messages as such while powering ON the nested VM.

    Thanks in advance.
    Ali

  31. Neil says:

    Hi,

    Great article. I’ve been able to setup a complete vSphere test rig on a single €200 ML115.

    One issue I have is that I can’t add hosts to a dvSwitch. I have available NICs but the Add Hosts list is blank. Has anyone had any luck with this?

    Thanks,

    Neil.

  32. Neil says:

    Hi,

    I just answered my own question. dvSwitches are only available with the Enterprise Plus and evaluation licenses, so NFR licenses don’t work as they are just Enterprise.

    Neil

  33. Pingback: Il mondo è bello perchè è virtuale! @ Marco’s Informatik Blog

  34. rahul says:

    eric: I was successfull installing virtual ESX 4.0 inside physical ESX 4.0. But I am not able to power on VM’s inside Virtual ESX. I had did the configuration changes as you have mentioned above. Could you please help me in this?

  35. Eric Gray says:

    Rahul, I suspect it is your CPU. What is the exact vendor and model info?

  36. Rahul says:

    Eric: I tried it in Dell power edge server and it worked….Thanks a lot…
    keep posting this kinda stuff….its really helpfull.

  37. Shan N says:

    Hi Eric,
    Great Web site and articles. I was going to buy more hardware to build a test environment. Now I am going to try to create another ESX 4.0 server in my physical ESX 4.0. I hope I will be able to test vMotion with this setup.

    Thanks for the great work.

    Regards,
    Shan

  38. Mark W says:

    Ali – I am having the same problems with the blank/black console from vSphere (running on the Host. The issue persists in both ESX and ESXi 4.0. I suspect that it may be a Windows 7 issue (host). Like you, I have tried an appliance (2 actually, a Turnkey LAMP app and VA-ntop).
    Next I plan to try:
    Import a Win XP VM and try to RDP to it from within Win7.
    Build a pWinXP, install vSphere and try that.
    Lastly (is that a word?) Install Wkstn on the pXP machine, install vESX and try to get a console from pWin7.

    I will report back, but it might be a week or 2 as I have several pESX boxes to build this week.

  39. gogogo5 says:

    Hello

    Has anyone been able to successfully view the console of a nested VM regardless of OS. My VMs also have blank/black consoles…

    cheers
    gogogo5

  40. rahul says:

    Yes …I had installed WIN2K3 R2 as a VM GOS inside my virtual ESX 4.0 . I didnt find any issue as you are facing. I was able to use my nested WIN2K3 VM console.

  41. Pingback: VMworld 2009 Virtual Infrastructure Design – Lab Manager vPODS Enable Conference Cloud | VM /ETC

  42. Sven says:

    Hi Eric,

    please could you talk a bit about your hardware setup and the performance? Are you able to run a cluster of two virtualized ESX to make some VMotion, HA etc.?

    • Eric Gray says:

      Sven,

      You can absolutely exercise most of the features of vSphere with a 2-node virtual ESX cluster. VMotion, HA, Host Profiles, vDS — everything should work the same way. Performance of the nested VMs is not stellar, but you can at least get a feel for all of the functionality. Check out the various posts linked in these comments to see what others are doing with virtual ESX systems.

      Eric

  43. Deon says:

    Hi Eric,

    I have 2 x ESX4 hosts running on the same ESX3.5 AMD Dell box. I have configured iSCSI and vmotion on the 2 ESX vm’s. I can successfully see the shared storage on the 2 boxes, and created a vm that is runnign w2k3. Problem i have is that they will not see each other via the service console or vmption network. Do you know of any further network tweaks? I have already set the vswitch promiscuous mode to accept.

    Thanks

  44. Pingback: IGT Part 7: Virtual VMware ESX 4 in high demand | VCritical

  45. Goris Christophe says:

    Hi,

    Have try this install on a HP DL360 G4, ESX 4 is running fine.

    But when I try to start the virtual ESX4 I receive the following error:

    “This version of ESX requires a CPU capable of 64-bit operation.”

    CPU’s are EM64T as the ESX4 hardware install is working.

    • sripati says:

      Hi Goris,

      I know I am too late in answerin this Question. But I think rest users would be able to get the benefit out of the same.

      Once u create the VM…go to the Option …….check in the option where in you get the BIOS screen at Power on.
      2- Now go the advanced tab in the processor you have 1.4 & 1.1. select 1.1 version….
      press F10..you are ready to go.
      It worked for me(MY White box is Q9550 + S3210 SHLC+ 8 GB).

      • Denesh Ashok says:

        Hi sripati,

        I tried this with my DELL PE 2850 and the host esx is working with ESX 4, and when i followed ur steps above i get the same error still “This version of ESX requires a CPU capable of 64-bit operation.”.

        Then i down graded the esc to 3.5 U5 and still i get the error.

        But I can run ESX4 on the DELL tin still what do i need to do please let me know anyone so I can start working on my lab and get my VCP soon.

        many thanks

        DA

  46. have run “CPU Identification Utility” from vmware.

    It seems to the CPU’s can not run 64bit as virtual 🙁

    Supports 54-bit Longmode: Yes
    Supports 64-bit VMware :No

  47. Nicolas says:

    Hi,

    I cannot run VMs on “virtual” ESX4, blank console and 100% CPU,
    is-it related to AMD opteron ? I just bought a poweredge sc1435 for a lab … 🙁

    Nicolas

    • Chadwick King says:

      I have seen this issue with not enabling Virtualization Technology on the AMD processor in the BIOS – double check and let me know if that works!

  48. NiTRo says:

    I’m running 7 ESXi 4 to test HA stuff…
    Thanks again for this good post Eric

  49. Guru says:

    Eric,

    I am running VM in ESX 4.0 Evaluation version on my desktop, but unable to power on the VM in it. I just get a black/blank screen and never get any screen even though it says that the VM is powered on. I have an ISO for windows 2003 for which I dont get any installation wizard.

    Guru.

  50. Pingback: My Home Lab Setup « Rick Vanover's Blog

  51. Guru says:

    Nicolas,

    I have an AMD 64 bit, enabled the bios setting (NX for AMD) for virtualization.

    Guru

  52. Nicolas says:

    Ok, on amd there is a special generation level you must have !!, even with VT enabled it does’t work.

    You can search if you have a second generation or a third one this link, you MUST have at least a third one !!

    http://www.cpu-world.com/CPUs/K10/AMD-Third%20Generation%20Opteron%208356%20-%20OS8356WAL4BGD.html

  53. Guru says:

    I have an Athlon 64.. Not sure about the generation.

    Guru

  54. V.Peter.S says:

    Eric, This is great info.
    I had discovered this for 3.5 last year, Check posts from vmware Academy Partners,
    One added issue we have been having with our dell 2950’s is that the install would hang on formatting the vmfs.

    We have been able to resolve this by using a RDM for the disk of the esx VM.

    Pete

  55. Rafi says:

    Eric,

    I have a HP DL385 G2 with AMD Opteron Processor. I have enabled the Virtualization in BIOS. This machine has enough RAM and DASD.

    I was able to install ESX 4 on the bare metal machine

    I am trying to get more familar with vSphere and to get familar with other features I needed another ESX server so I followed your post and was able to successfully install a virtual ESX 4i instance within the ESX 4 server. I have successfuly added both the ESX servers in the test cluster in vCenter.

    I have also added a bare metal pentium 4 OpenFiler machine which is working as iSCSI storage. I have successfully configured and openfiler storage is visible to the ESX4 and virtual ESX4i instance

    virtual ESX4i instance allows me to create virtual machines. However when I power them on I am not able to see anything in the console. When I power on the same machine and migrate it to the physical ESX host and I can see the console.

    I am a virtualization newbie who is trying to learn … any assistance will be greatly appreciated.

    Thanks!

  56. Fred says:

    Hi Eric… Thx for this post!
    I executed the 2 tweaks and was able to install an ESX3.5 server and an ESX 4 server

    The ESX4 server fails after a reboot. it fails on vsd-mount ….

    Any suggestions?

    The ESX3.5 runs fine

    Thanks
    Fred

  57. Fred says:

    It looks like we need another tweak for snapshot usage….. because that was the issue.
    The ESX 3.5 reboots fine (with snapshot in use) but the ESX4 fails on vsd-mount

    Any suggestions?

    Tx
    fred

  58. Rick says:

    I’m seen the same issue when I create snapshots on Virtual ESX 4s. I think it has to do something with the fact that the COS is on the same vmfs disk as the one you use for regular vmstorage. I’ve Google and found the issue, but I’m not sure if anybody has found a workaround yet.

  59. Steve Shockley says:

    FYI, I had problems getting ESX 4.0u1 to install under ESX 4.0 on an HP DL680c G5. When the install would start formatting the disks, vmkfstools would fail with status 65280, print the vmkfstools command options, then say “Error: not supported”. It turns out I only had the problem when the guest disk was on a volume on a NetApp filer via NFS. Putting the volume on direct-attach disk led to a successful installation.

  60. Pingback: Take snapshots of VMware ESX 4 running inside a VM with this undocumented setting | VCritical

  61. Eric Gray says:

    The snapshot issue may be addressed with an undocumented configuration setting. Please see https://vcritical.com/2010/02/taking-snapshots-of-vmware-esx-4-running-in-a-vm/

  62. Fred says:

    Thanks for this advanced configuration setting Eric!
    My Physical ESX host is ESX4i and sadly it is an unknown Option in there…

    C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
    vh10 –username root -g /COW/PermitVmfsOnRedoHierarchy
    Enter password:
    Option /COW/PermitVmfsOnRedoHierarchy not found.

    C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-advcfg.pl –server nl01sol
    vh10 –username root -s 1 /COW/PermitVmfsOnRedoHierarchy
    Enter password:
    Option /COW/PermitVmfsOnRedoHierarchy not found.

    • Eric Gray says:

      Since this is an unsupported/undocumented configuration setting, it cannot be controlled via remote API. The good news is that you should be able to just log into the “tech support mode” console of ESXi and run the same command that works on the service console.

      In other words, go to the ESXi console, press ALT+F1, type in the word “unsupported” (blindly, as it is not echoed to the screen), enter your password when prompted, and then run the esxcfg-advcfg command.

  63. Fred says:

    Thanks again Eric! i went to “unsupported “service console” on the physical ESX4i and execu ted esxcfg-advcfg -s 1 /COW/PermitVmfsOnRedoHierarchy
    made a snapshot and could startup without any problem.

  64. Pingback: Link – Running VMware ESX 4.0 on itself. | TechHead.co.uk

  65. Pingback: VMware ESX 4 can even virtualize itself | VCritical | scandals of an ITian

  66. Prashant M M says:

    Hi Eric,

    I have installed Vmware Vsphere on vmware workstation and I am able to create the HA & DRS cluster with FREENAS Shared storage but my problem is when I create the VM and try to power on the VM with the ISO or any other medium, I am getting blank black screen and I am not getting any kind of error but just the blank screen.

    I even changed the points like monitor_control.restrict_backdoor = TRUE etc.

    When I poer on the ESX host VM I am still getting the error that I can’t run the NESTED VMs.

    • Eric Gray says:

      Prashant,

      Could it be that you have an older AMD CPU? If you notice, some of the comments above have had similar issues and one reader felt it was AMD-specific. I have used AMD Shanghai CPUs for this and everything worked properly.

      Eric

  67. Tom Galat says:

    Eric,
    Once again — good stuff. I set up ESXi V4 on a Dell R900 W/128GB Ram and 8 CPUs. We have 15 or so test VMs running on it.
    With that said I downloaded the 60 trial of ESX 4.0.u1 and vSphere. Installed a 3 host cluster and turned on HA and DRS. Added 6 guest VMs on the virtualized ESX and all works great, systems are a bit slow but that’s what we get for multilayer hypervisors.
    My question is — has anyone got Fault Tolerance to work? When I turn it on I get a hardware compatibility error. It seems that FT wants VT turned on in the bios, but the VMware bios of the guest ESX server goes not have this option. The backend storage is IB so there is no delay there. Any ideas?
    I need to give a presentation demo to the higher ups and they are interested in FT.

  68. tommelise says:

    Hi Eric,

    Thanks for this post!

    I have installed ESX 4 on a white-box (Dell 9200) and it works like a charm.
    Now I’ve followed yours instructions to install ESXi4 as a VM on it.
    And I got a PSOD “No place on disk to dump data”.
    Any suggestions ?

    Thanks

  69. Bhaskar says:

    Hi,

    I have two esx servers as guest os on my physical esx host but the two esx guest are unable to communicate each other.
    Please help me in acheiving this feature.

    Thanks
    Bhaskar.

    • Datto says:

      Bhaskar — you need to change the Security on the Service Console of your physical ESX host AND the VM Traffic Port on your physical ESX host to allow Promiscuous Mode (change the setting to Accept instead of the default of Reject). That will then allow communication between your ESX hosts.

      Datto

  70. Eric Gray says:

    Hi Eric we have the same name! Anyway, if you need a good company to buy any VMware vSphere 4 products, a good place is my company ITO Solutions. Here is a link to our site for VMware ESX http://www.itosolutions.net/Server_Datacenter_Virtualization_s/523.htm

  71. Fabiano Silos says:

    Hi Eric, do you have any clue on how to run 64bit VM inside the ESX that is running over a virtual machine?

  72. VM_newbie says:

    Hi,
    I have installed ESX 4.0 on workstation 7. the ESX has 22 gig VMFS datastore. I am trying to create a new VM in ESX and dont know how to boot from ISO to install the OS. After creating the VM when I go to connect cd/dvd it did not detect the OS in dvd drive. Do i have to copy the ISO to datastore first if yes then how i can do that or I can only create VM using the VC 4.

    any help appreciated

  73. mike says:

    Ok, so I have a problem with my esxi 4 “update 2” guests running under the same esxi 4 version for the host. The problem is that the guest esxi servers do not have correct time despite having ntp setup just like the host.
    The host esxi server has been pointed to pool.ntp.org and shows the correct time via the vshpere client. The guest/s esxi servers are pointed to the same exact ntp server “pool.ntp.org” but after restarting the service just like I did on the host the time does not change and is off. I manually set the time on the guests to the correct time but after 8-10 hours the time is faster on the guests by about 2 minutes and some change. Are others experiencing time difference on there nested esx/i vm’s? I figured it would use the ntp server and adjust it time correctly just like the host did. I also verified they could resolve and communicated with the ntp server …so that can be taken out of the equation.

    Thanks,

    Mike

  74. Pingback: Why we are using VMware Lab Manager. | Planetchopstick, dont ask wont tell

  75. SargentTi says:

    Great article, help me established a test ENV for Lab manager, thanks!

  76. Anantha says:

    Hello Eric

    My Host OS is windows 07, on which I have installed VMware server and ESX 4.0 inside it

    Following is the order

    Win07Ultimate (Host OS)–> VMware-server-2.0.2-203138 –>ESX4.0 –> win 2008 VM

    I tried above solution of setting up the Vswitch in promiscuous mode and adding the line “monitor_control.restrict_backdoor / TRUE”

    But still I get the error message “You may not power on a virtual machine in a virtual machine”

    Is there any solution if the ESX is not directly installed on the hardware as the host OS

    Regards,
    Anantha

  77. Onder says:

    Hello all,

    I tried to install the ESX v4 server on ESX v4 server as described in the beginning of this article. The installation cannot even take place.

    Error code:
    CosSup: 406: vmkernel not loaded: cannot PSOD – Basically not supported microcodes.

    My cluster configuration is with EVC AMD Opteron Generation 3. The exact processor type is Ams Opteron Quad-Core 8389.

    Installation of ESX v3.5 is working, but after reboot I have the same error all the time even with different VM settings.

    Error code:
    …….initialization of vmkernel failed, status 0xbad001e – Microcodes problem again.

    Can someone help?

    Thanks.

  78. MAZ says:

    Does any one tried to move ESX 4.0 Host VMs to Another ESX4.0 Host.
    I hve Dell 2950 2.6 GHz 8x Core, want to shift my ESX 4.0 Linux VMs from this one to DELL 2950 2.0 GHZ processor.

    Any idea?
    I have VCenter VM also on the same ESX Host1.
    Thanks for advise in advance.

    MAZ

    • Chadwick King says:

      MAZ – You should have no problems moving VM’s. My questions would be is how exactly are you thinking of doing this? Are you wanting to use vMotion? Are these host in a HA/DRS cluster? Answering these questions would help me better understand what answer your looking for.

      Thanks!

  79. Pingback: Kasraeian

  80. Chadwick King says:

    Just wanted to note that anyone “experiencing” issues with blank screens and such on AMD platform CPUs be sure to check Vritulization Enabled on the Processor settings in the BIOS. I know this may sound kind of insulting but believe it or not I missed this on my system and yes I felt like a total newbie. I could easily run VM’s in workstations but had random issues with nested VM’s. Just double check and make sure that you weren’t pulling a “chad” and NOT saving the BIOS configuration…. would’ve saved me a headache..

  81. anand says:

    Hi,

    I have some problem while installing ESX 4.0 Server in Vmware Work Station 6.5 in Home. Problem is they need any External Network Connection is Failed.

    So Any one of them give me some solutions

  82. Pingback: Virtualizzazione “indentata” « ILSimo's Blog

  83. Pingback: How-to run XenServer on top of vSphere | PlanetVM

  84. ram says:

    I cannot create VM’s on Virtual ESX 4.1 server. I followed the notes to create virtual ESX 4.1 on the physical ESX 4.1. When, I try to create VM’s inside virtual ESX 4.1, I get this error –
    host cpu is incompatible with the virtual machine’s requirements at cpuid level register edx

    Any help appreciated.

    Tks / RL

  85. Hello people

    I’m trying to attach an external USB drive on the ESXi host. I can see the USB drive but cannot do much with.

    Is there away to have a USB drive working as an external storage attached to the ESX?

    thanks

    Louis

  86. Iain says:

    Just a slight aside here, I have LabManager running inside LabManager at the moment, just thought I’d show off! 😉 We’re running multiple test environments to play with, and one happened to be a LabManager implementation. No modification needed, just flew straight up!

  87. Pete says:

    Eric,
    I am using vESXi to run our VMware Academy. The one issue I can’t seem to resolve is getting NTP to work with the vESXi. It did work when I used vESX however the trend from VMware is towards ESXi. I believe this is creating an issue for the vCenter to use AD for authentication. I can get the VM that is running the VC to join the AD but the VC won’t since the hosts are not in sync.

    I’ve also noticed your comments on 64bit OSs, that are “currently” not possible. Are you aware of any potential fix for this?

    Thanks again for all the posts.

  88. habibalby says:

    Hello,

    I have vSphere cluster consist of three hosts. Can I setup vESXi4.1 as VM to run on ESX4.0?

    Thanks,
    Hussain

    • Eric Gray says:

      Most likely you will run into problems with the keyboard and won’t be able to install ESXi 4.1 on an ESX 4.0 host.

    • habibalby says:

      Hello,

      Yes, I faced this issue. Any trick for it?

      Another question; i have successfully installed vESX4.1 on pESX4.0 server. And I managed to run nested 32bit VM on the vESX4.1 hosts.

      How can I install the VMWare Tools inside the nested guest OS?

      Thanks

  89. ram says:

    Habibalby – I was able to install vESX4.1 inside pESX4.1 however, I am unable to run nested 32bit VM’s inside vESX4.1. How did you do? Appreciate your input pls.

    pESX4.1 is Dell E6400 Laptop Intel VT with 8GB Mem.

    Thanks

    • habibalby says:

      hi Ram,

      you need to enter this line in the Advanced Configuration “monitor_control.restrict_backdoor / TRUE”

  90. spice says:

    Hi,

    Can someone advise me if it is possible to install ESX 3.5 as a VM on ESX4? I have a client that has a software product that is only supported esx 3.5 but they platform they have is vsphere 4.0?

  91. NiTRo says:

    @spice : yes it’s possible and it works 🙂

  92. spice says:

    Thanks NitRo – How stable is it? would you build for a production environment?

  93. habibalby says:

    Hello once again,

    I’m finding this lab is very interested and questions keep raising:)

    What about VMTools on the vESX Server? Do I have to install it? If yes, how to?

    Thanks,

  94. Matt says:

    Worked well until I tried to enable either HA or EVC in a cluster. Nor could I start the VMWare Data Recovery appliance on the nested VMs. All of which worked without a problem when done on the bare metal ESXi 4.1 hypervisor.

    Messing with the CPUID would allow the VDR VM boot to start but it wouldn’t get much further into the boot process than a few seconds.

    A little annoying when studying for the VCP not being able to play with those 3 major components. Still not the end of the world.

  95. Pete says:

    For our VMware Academy, I am planning to use a hybrid approach, since vSphere4.1 has hosed several of the Labs: 64bit Data Recovery VM, HA, plus the others I’ve mentioned earlier.

    The hybrid approach is by using SuperMicro’s X8SLI-F motherboard. It is totally compatible with esx/esxi 4.1, and available for <$200. It even includes IPMI.

    I still will use the nested approach for most of the labs, but I thought you might like to know of this mb.

  96. Pete says:

    It should be SuperMicro; X8SIL-F

  97. Pete says:

    Matt:

    Good on Ya, as they say downunder.

    I bumped the memory up on the nested ESXi hosts then and HA now functions.

    This buys me a some time since I don’t feel as stressed to implement the hybrid setup. To me though the writing is on the wall that unless there is some way to enable VT in the vESX BIOS the requirement for 64bit OS is going to, as Pink Floyd would say, the Wall.

    It is definitely an advantage in having the vESX hosts which is why I still plan to run the training as a hybrid. Being able to build Physical hosts under $1k that can be used as a home or small business Cloud server is tough to beat.

    BTW: There is also an issue with the version of the X8SIL, make sure it is v1.02. This allows for i5 intel processors to work. I’ve have read that it will even work with i7s but I can not confirm this. I’m using neither since the x3400 series will allow upto 32G RAM. You need to verify the RAM is compatible.

  98. Jia Ji says:

    Hi Eric,

    I’m so glad I stumbled upon your site! I almost bought a 2nd Server for a home lab setup to test out the HA, DRS, vMotion, vDS, etc. Your article just might have saved me thousands (literally)! I initially tried to test the Enterprise features on a virtual ESXv4 running on a Workstation product hosted on a desktop w/ Intel Core 2 Quad Q6600; however, I quickly realized that even w/ the EVC enabled, I could not do vMotion, etc.

    In any case, here is my current config:
    – 24GB Mem, 1 Quad Core Xeon, 2x2TB HDD, 4 x Gigabit NICs
    – ESXi v4 on internal SD/Flash

    Motivation: Learning/Hobby/Home Virtual Infrastructure (i.e.: Not production)

    Here are my questions:
    – Ideally I’d like to do ESXi on ESXi installation; however, I only have one internal SD
    module w/ 1GB media. Is it possible to do two ESXi installation on ONE 2GB SD media?
    This way, I can boot to Host ESXi from SD as it is currently setup. Then, after I have
    booted, power-on the 2nd instance of ESXi that is also hosted on the same SD card.
    – If the above is not possible or recommended, then I would most likely do ESX on ESX
    installation. Are there any gotchas with this setup with regards to the vDS (virtual
    distributed switch), networking in general, 32-bit/64-bit guest OS support?
    – Also, I’m looking to configure VLAN networking across two (physical & virtual) ESX/i
    hosts. Any suggestions/recommendations/pointers on how I can do VLAN 801q
    tagging for all VLANs through the virtual distributed switch only (i.e.: without a physical
    switch that supports VLAN)? If so, are there any articles on howto setup/configure
    VLANs on ESX?

    Sorry for bombarding you with all the questions, but my recent interest in the ESX product/features has got me too excited.

    Again, I can’t thank you enough for posting some of the most valuable articles on the ‘net (e.g: ESX on ESX, P2V VMotion, etc.).

    Sincerely,

    Jia

  99. Dave McLeod says:

    This is a very helpful post. I have one issue though. I was able to install a Windows 2003 server VM inside an ESX VM but the Windows VM can receive IP address via dhcp but cannot ping LAN IP or connect to the internet. I have disabled firewall settings still got the same result.

    Please help.

    Thank you

    • Datto says:

      Dave — you need to engage “Promiscuous Mode – Acceot” on a) the Service Console Management Port Group as well as engage it on b) the Virtual Machine port group.

      Assuming your using lab equipment and not production servers, click on the Properties of your VSwitch in the VIC, select the port group and choose Edit, find the security tab, check the Promiscuous Mode checkbox for Accept/Reject and choose Accept. Then OK out of the dialogs. Then do the same for the other port group/VSwitch (if the VSwitch is different than the VSwitch used for the Service Console).

      That will then allow traffic out from the nested VMs running on nested ESX 4.x hosts.

      Datto

  100. Pingback: Virtualization test lab tricks with SSDs | TechRepublic

  101. Jordan says:

    I was able to do this, but I have an interesting issue when I try to VMotion.

    The physical hosts are Intel Xeon Quad Core CPU’s with Virtualization Technology turned on in the BIOS.

    When I create a VM on the virtualized ESX Hosts, it says I need to enable the Virtualization Technology on my virtualized hosts. Kind of confusing.

    3 Physical 4.1 ESX Hosts (in a cluster)
    3 Virtual 4.1 ESX Hosts (in a cluster)
    1 Virtual Machine on the Virtual 4.1 ESX Host cluster (giving the warning about enabling Virtualization on the host)

    • T-Bone says:

      Jordan,
      In most cases, the vMotion issue can be solved by changing the CPU/MMU Virtualization option on the virtual host. In my test environment running ESXi 4.1 Update 1, I was successful using the 3rd option “Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization”. Your CPU may allow or require a different choice. Try all the available options and test the system’s reaction to each one.

  102. Matt says:

    The virtualised Hypervisors no longer have access to the Hardware Virtualisation instructions (i.e. Intel VT and AMD-V) in the bios/cpu since they are only seeing vCPUs presented from the hypervisor on the baremetal. You could try messing with the CPUID Mask in your VM but your best option would probably be to enable Enhanced VMotion Compatibility (EVC) on the cluster.

    http://www.vmware.com/files/pdf/vmotion_info_guide.pdf

    • Jordan says:

      I understand the issue that they no longer have direct physical access to the hardware anymore. I forgot to mention I did try to enable EVC Intel on the cluster but got the error:

      “EVC cannot be enabled because the cluster contains hosts that lack EVC-capable hardware”

      Which also makes sense.

      • Matt says:

        There are some VMs that require access to the hardware virtualised instructions, could that be the issue? I ran into that issue when trying to use the VDR VM, if that is the VM you are trying to run then read back a few posts Datto had a good suggestion. (figured this was worth a mention since its a little unclear from your posts if the error is occurring when you created the VM or during a vmotion within the virtualised cluster)

        Other than that all I’ve got are rhetorical questions, sorry if you’d run though this already: Did EVC work in your Physical hypervisor cluster (i.e. are the physical CPUs happy to work together even before they were virtualised)? Were all the VMs shutdown in the cluster you enabling EVC? Does altering the affected VM’s CPUID Mask change anything?

  103. Datto says:

    For those with smaller lab budgets —

    Just wanted to mention that this morning I was able to consolidate eight nested Classic ESX 4.0 hosts each with three lightly used Win2003 VMs as well as two nested XenServer 5.x hosts each with two lightly used CentOS 5.2 VMs on a single physical ESX host. For the physical host I used a 1U Dell SC1435 server with 2x Opteron 2356 CPUs and 16GB of physical memory running ESXi 4.1 from a USB flash drive. Storage for VMs was from one of my Fibre SANs.

    My total cost for the Dell SC1435 server with parts from eBay for it was $512.70 (detailed costs were 2x Opteron quad 2356 CPUs were $84.26 delivered, the Dell SC1435 server was $222.44 delivered and I inventoried the parts not used from this server, 16GB ECC REG memory was $160 delivered, the QLogic PCI-e HBA was $40 delivered and the flash drive was about $6 delivered). I didn’t use the 2x hard drives delivered with the Dell SC1435 server nor the existing DC CPUs that came with the Dell SC1435 server since I’d separately bought two 3rd generation 2356 Opterons for nesting purposes and intended to run the VMs from SAN. I could have just as easily run the nested VMs off the local hard drives and not have needed the Fibre HBA.

    Performance inside the nested VMs running on the nested ESX/nested XenServer hosts appears to me to be the same as when those same VMs ran on nested ESX hosts/nested XenServer hosts that previously ran on three different physical 1 CPU 8GB ESX white box servers. The physical Dell SC1435 server currently is running about 32% for overall CPU, 88% for memory use with about 2GB for vSWP in use (all the nested ESX/nested XenServer VMs get 4GB memory assigned while the nested ESX servers get 1 vCPU and the nested XenServer servers get 2x vCPUs).

    If I’d have used ESXi 4.1 for the nested ESX hosts instead of the Classic ESX 4.0 hosts that would have saved me a bit more memory use but I’d already had the pre-built nested Classic ESX 4.0 VMs built from a different project.

  104. Paul Grevink says:

    Hi Eric,

    Very good post, especially the part on adjusting the vSwitch and enable promiscuous mode. It saved my day.
    ESXi on ESXi works well without this change. But after installing ESX on ESXi, trouble begun and could only be fixed by changing to promiscuous mode. At this moment the theory behind this change is not completely clear to me…

    Regards,
    Paul

  105. Ryan says:

    Has anyone experienced problems with untagged traffic? All my other VLANs / Portgroups work great on both the physical host and nested VMs but I have one untaged portgroup and the VMs are only able to communicate if they are on the physical host. I’m using a distributed switch and have promiscuous mode enabled on all the portgroups.

  106. Thom says:

    Has anyone else run into issues when cloning ESXi VMs?

    It seems to work fine initially, but the vCenter managing the virtual ESX hosts gets confused about the datastores. The local datastore on each virtual ESX has the same UUID, leading to problems in vCenter.

    Anyone else seen or solved this? How are the disk UUIDs generated?

  107. govin says:

    Im trying to install ESX4.0 on the new VM already running in ESX4.0 connect with SAN Storage. The new VM used for the ESX installation space allocated from the SAN Datastore. I need to know during the ESX 4.0 setup on VM ver7.0 would it format only the space allocated to the VM disk or the entire SAN LUN which provisioned to the VM. (Currently I’ve 2 running VM’s in the Datastore) Thanks

  108. Eric Gray says:

    vSphere 5 makes this easier by adding a guest OS type for ESX/ESXi. It also supports nested 64-bit guests.

    Check out https://vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/

    • Alain says:

      Is it possible to install vsphere 5 with 2 esxi in a vsphere 4.1.xxx environnement.
      Should i follow the same instruction as in here or is it different for the guest os.

      also is the promiscuous mode really important?

      lafrenal@crim.ca

  109. I use VMs for separating my development environment from my testing environment and use Oracle’s (formerly, Sun’s) VirtualBox. It meets my need though it isn’t stable enough to run on a server. I did run VirtualBox within a Hyper-V virtualized Windows server install and was able to run a VM within a VM, though I used two different products for it.

  110. Jeff Messeer says:

    Just finished getting my single new host on ESXi 4.1 U1 up and running. Connected it to my Iomega ix4-200d, and now using iSCSI I have a vCenter VM and two ESXi 4.1 U1 hosts one one layer, and then the second layer I have the virtualized hosts running in a full HA/DRS cluster sharing two iSCSI LUNS. Your original post and the comments here helped me get this thing going – very much appreciated!

  111. Pingback: VMware vSphere can virtualize itself + 64-bit nested guests

  112. I made an ESX vm okay but no matter what I do I can’t connect to it with the VI client and I can’t add it as a host in Virtual Centre. It’s on the same vswitch as the other vms, so basically it’s on the same network as the VirtualCentre box as far as I know, and obviously I can connect to that and that can also connect to the Internet. So I don’t know what’s up. I tried giving it different ip addresses and using a DHCP setting. The virtual machine network is on one physical nic and the management network on the other. Someone suggested moving all the machines over to the management network but I haven’t been able to do it.

  113. Matt1 says:

    sounds like might have overlooked the promiscuous mode on the vswitch as per the original article.

  114. I added the “monitor_control.restrict_backdoor / TRUE line” like you said but I still get the error “Running VMware ESX in a virtual machine requires the outer virtual machine to be configured for running a VMware ESX guest operating system. You may not power on a virtual machine until the outer virtual machine is reconfigured.” when I try to power on the nested virtual machine.

  115. Tom Maloney says:

    Richard,
    Just to verify a few points.
    The “monitor_control.restrict_backdoor” parameter is added not to the nested VM, but to the Advanced configuration of the VM that contains your virtual host.
    The Parameter name is “monitor_control.restrict_backdoor” and the Value is “TRUE”. Check to make sure there are no leading or trailing spaces – this can happen a lot if you copy/paste from the article. Also, make sure to NOT use the “/” character.
    If this all checks out and you still have this problem, download and open the .VMX file of the virtual host in a text editor like Wordpad. The line should look like this –

    monitor_control.restrict_backdoor = “TRUE”

  116. Pingback: Kasraeian

  117. Jack says:

    I have imported a virtual machine(win xp 32 bit) to the virtual ESX host, but the network connection of the win xp is not available, I try many ways, could any one please tell me how to create valid network connection with other machines?

    Thanks in advance

    • Tom Maloney says:

      Jack,

      Not sure exactly the symptoms from your description, but I’ll try.

      First, you need to treat the virtual host as though it is a normal ESX host. This means you need to create any port groups on this server with the proper VLAN IDs and Port Group name. The port group name must be the same as what is defined on the other systems.
      Next, the uplinks – the vmnics – on your virtual host need to have all of your VLANs trunked to it. The easiest way to accomplish this is to use promiscuous mode. You need to set this on the physical host.
      Example – host ESX01 is the physical host, ESX02 is running inside a virtual machine on ESX01. You would create a vSwitch (vSwitch1 for example) on ESX01 and put it in promiscuous mode. The VM that runs ESX02 would have one or more virtual NICs tied to ESX01 – vSwitch1. At ESX02, you would make sure that your vSwitch is joined to vmnic0 and create a VM port group on the proper VLAN for your XP VM.
      Go back and review that section of the posting. You may have used promiscuous mode on the wrong switch.

  118. Pingback: Building a Multi-Tenant Veeam Replication Target

  119. Pingback: How To Install The Cisco Nexus 1000V on vSphere 5 - Ray Heffer

Comments are closed.