VMware vSphere can virtualize itself + 64-bit nested guests

Running VMware ESXi inside a virtual machine is a great way to experiment with different configurations and features without building out a whole lab full of hardware and storage. Virtualization enthusiasts everywhere have benefited from the ability to run ESXi on ESXi, first introduced with the vSphere 4 release.

VMware vSphere 5 makes it easier than ever to virtualize hypervisor hosts. With new capabilities to run nested 64-bit guests and take snapshots of virtual ESXi VMs, the sky is the limit for your cloud infrastructure development lab. Heck, you can even run Hyper-V on top of vSphere 5 — not that you’d want to.

Physical Host Setup

The physical host running VMware ESXi 5 requires just a few configuration changes; here is a guide:

Install VMware ESXi 5 on a physical host and configure networking, storage, and other aspects as needed
Configure a vSwitch and/or Port Group to have Promiscuous Mode enabled
Create a second Port Group named “Trunk” with VLAN ID All (4095) if you want to use VLANs on virtual hypervisors
Log in to Tech Support Mode (iLO or ssh) and make the following tweak to enable nested 64-bit guests
```
echo 'vhv.allow = "TRUE"' >> /etc/vmware/config
```

Virtual VMware ESXi Machine (vESXi) Creation

For various reasons, it’s not feasible to clone virtual ESXi VMs. As an alternative, create a fully-configured shell VM to use as a template — it can be cloned before ESXi is installed.

Create a new VM with the following guidance:

Guest OS: Linux / Red Hat Enterprise Linux 5 (64-bit)
2 virtual sockets, 2+ GB RAM
4 NICs — connect NIC 1 to the management network and the rest to the “Trunk” network:
Thin provisioned virtual disks work fine
Finish creating the VM, then edit the following settings
- Options/General Options: change Guest Operating System to Other – VMware ESXi 5.x
- CPU/MMU Virtualization: Use Intel VT … EPT… ( bottom radio button)
Don’t power this VM on — keep it to act as a template
Clone and install VMware ESXi via ISO image or PXE boot
Add to vCenter and configure virtual ESXi hosts for action

Nested 64-bit Guests

With the release of VMware vSphere 5, nested guests can be 64-bit operating systems. Just be sure to make the change to /etc/vmware/config on the physical host as indicated above.

Nested guests can be migrated with vMotion between virtual or physical VMware ESXi hosts; this requires a vMotion network and shared storage.

Nested Hyper-V Virtual Machines

It is possible to run other hypervisors as vSphere virtual machines, and even power on nested VMs. Here you can see Hyper-V running a CentOS virtual machine — all on VMware ESXi. Talk about disrupting the space-time continuum!

A couple of extra tweaks are needed to enable this, and performance is not great. Nevertheless, an amazing feat of engineering from VMware!

Do the following to enable Hyper-V on VMware ESXi:

Add hypervisor.cpuid.v0 = FALSE to the VM configuration

Add —-:—-:—-:—-:—-:—-:–h-:—- to the CPU mask for Level 1 ECX (Intel)

For another take, check out William Lam’s post on this topic.

Parting Thoughts

Given the right hardware, it is possible to create a fully-functional VMware test lab that is completely virtual. Go ahead and experiment with the Distributed Virtual Switch, vShield, vCloud Director, and everything else without deploying a ton of servers and storage.