In addition to NVIDIA improving peer-to-peer (P2P) DMA for block devices in Linux 6.19, NVIDIA also led an effort providing DMA-BUF support for VFIO PCI devices for opening up some interesting new cases moving forward. As part of the VFIO pull request this new functionality has landed for Linux 6.19.
DMA-BUF support for VFIO PCI devices has been merged thanks to the work of NVIDIA’s Leon Romanovsky and Jason Gunthorpe plus Intel’s Vivek Kasireddy. This week’s VFIO pull request explained of the new code:
“Introduce dma-buf support for vfio-pci devices, allowing MMIO regions to be exposed through dma-buf objects with lifecycle managed through move operations. This enables low-level interactions such as a vfio-pci based SPDK drivers interacting directly with dma-buf capable RDMA devices to enable peer-to-peer operations. IOMMUFD is also now able to build upon this support to fill a long standing feature gap versus the legacy vfio type1 IOMMU backend with an implementation of P2P support for VM use cases that better manages the lifecycle of the P2P mapping.”
The patch series for the code further elaborated:
“This series extends the VFIO PCI subsystem to support exporting MMIO regions from PCI device BARs as dma-buf objects, enabling safe sharing of non-struct page memory with controlled lifetime management. This allows RDMA and other subsystems to import dma-buf FDs and build them into memory regions for PCI P2P operations.
The series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device’s doorbell using PCI P2P.
However, as a general mechanism, it can support many other scenarios with VFIO. This dmabuf approach can be usable by iommufd as well for generic and safe P2P mappings.
In addition to the SPDK use-case mentioned above, the capability added in this patch series can also be useful when a buffer (located in device memory such as VRAM) needs to be shared between any two dGPU devices or instances (assuming one of them is bound to VFIO PCI) as long as they are P2P DMA compatible.
…
The series includes significant refactoring of the PCI P2PDMA subsystem to separate core P2P functionality from memory allocation features, making it more modular and suitable for VFIO use cases that don’t need struct page support.”
Linus Torvalds merged the VFIO changes on Friday for Linux 6.19.
