NVIDIA engineer Ankit Agrawal today posted a request for comments (RFC) patch series working on Linux virtualization support for Extended GPU Memory (EGM).
Extended GPU Memory functionality is found with their Grace Hopper and Grace Blackwell systems so the GPU can access system memory allocations whether they be on the same node or even different sockets or entirely different nodes of multi-node clusters. The work happening now for the Linux kernel is allowing NVIDIA EGM to work in the context of virtualization.
Ankit Agrawal explained with today’s RFC patch series for the Linux kernel:
“Grace Hopper/Blackwell systems support the Extended GPU Memory (EGM) feature that enable the GPU to access the system memory allocations within and across nodes through high bandwidth path. This access path goes as: GPU <–> NVswitch <–> GPU <–> CPU. The GPU can utilize the system memory located on the same socket or from a different socket or even on a different node in a multi-node system. This feature is being extended to virtualization.
…
EGM when enabled in the virtualization stack, the host memory is partitioned into 2 parts: One partition for the Host OS usage called Hypervisor region, and a second Hypervisor-Invisible (HI) region for the VM. Only the hypervisor region is part of the host EFI map
and is thus visible to the host OS on bootup. Since the entire VM sysmem is eligible for EGM allocations within the VM, the HI partition is interchangeably called as EGM region in the series. This HI/EGM region range base SPA and size is exposed through the ACPI DSDT properties.
…
Applied over v6.17-rc4 and using qemu repository. Tested on the Grace Blackwell platform by booting up VM, loading NVIDIA module and running nvidia-smi in the VM to check for the presence of EGM capability.To run CUDA workloads, there is a dependency on the Nested Page Table patches being worked on separately.”
These RFC patches introduce “nvgrace-egm” as a new VFIO driver for handling this Extended GPU Memory virtualization setup.
