AWS recently announced support for nested virtual machines within virtualized EC2 instances running KVM or Hyper-V. A long-awaited feature by the community, the new option enables use cases such as app emulation and hardware simulation on supported C8i, M8i, and R8i instances.
Developers can use this feature to run mobile app emulators, simulate in-car automotive hardware, and use Windows Subsystem for Linux (WSL) on Windows workstations. According to the documentation, the Nitro System exposes processor features such as Intel VT-x to the instance, allowing it to run virtual machines inside it.
The architecture for nested virtualization has three layers: the physical AWS infrastructure and Nitro hypervisor (L0), your EC2 instance running its own hypervisor (L1), and the virtual machines running inside that instance (L2). The new feature supports KVM and Hyper-V on C8i, M8i, and R8i instances. Ioannis Aslanidis, director of engineering at AWS, writes:
This capability empowers developers and enterprises to build flexible, nested environments with a wide range of standard virtual EC2 instance types.
The feature has been a long-term requirement from the community, with a developer writing in 2018 on Reddit:
In Azure, I can run KVM in my virtual machine, a technique known as nested virtualization. Has Amazon made any progress in allowing HyperV/VMware/KVM in EC2?
Most comments on the new feature echo that sentiment: Rolf Neugebauer, Production Engineer at Meta, writes, “What took you so long?” while Igor Sfiligoi adds: “This really was not possible until now?” In a popular Hacker News thread, Michael Boulos, software engineer at Google, writes:
I feel vindicated :). We put in a lot of effort with great customers to get nested virtualization running well on GCE years ago, and I’m glad to hear AWS is coming around. You can tell people to just do something else, there’s probably a separate natural solution, etc. but sometimes you’re willing to sacrifice some peak performance just have that uniformity of operations and control.
Anurag Goel, founder and CEO of Render, adds:
This is a big deal because you can now run Firecracker/other microVMs in an AWS VM instead of expensive AWS bare-metal instances.
Guillermo Ruiz, senior solution architect at AWS, tested running Firecracker microVMs inside an instance, promising future benchmarks. Ruiz writes:
Interesting angle for anyone looking at isolation, Lambda-style runtimes, or just comparing microVMs vs containers from a performance and density perspective.
In the past, the only option to run nested virtualization on EC2 was to use bare metal instances, physical servers without a hypervisor. Many developers resorted to EC2 Mac instances, the smallest and cheapest bare metal option, as a workaround. Enabling nested virtualization is now an API option set at launch, for example:
aws ec2 run-instances
--image-id ami-0abcdef1234567890
--instance-type r8i.4xlarge
--cpu-options "NestedVirtualization=enabled"
--key-name my-key-pair
--placement "Tenancy=host"
AWS still suggests that customers running workloads that require hardware virtualization, are sensitive to performance, or require low latency should continue using bare-metal instances rather than nested virtualization.
Nested virtualization is available in all regions on C8i, M8i, and R8i instance types; Graviton instances are not currently supported. KVM and Hyper-V are the only supported L1 hypervisors.
