Building off the existing Linux support for GPU Direct RDMA / Peer-To-Peer DMA functionality, a set of patches were posted by NVIDIA today enabling this P2P DMA support to also work for device-private pages.
There’s been a lot of Linux work done for P2PDMA for avoiding system memory copies between devices/accelerators. The patches posted this afternoon are focused on providing GPU Direct RDMA (P2P DMA) for device private memory pages.
NVIDIA engineer Yonatan Maman explained in the patch series:
“This patch series aims to enable Peer-to-Peer (P2P) DMA access in GPU-centric applications that utilize RDMA and private device pages. This enhancement is crucial for minimizing data transfer overhead by allowing the GPU to directly expose device private page data to devices such as NICs, eliminating the need to traverse system RAM, which is the native method for exposing device private page data.”
Besides the infrastructure changes to the Linux memory management code and Heterogeneous Memory Management (HMM), the patch series also adds code to the NVIDIA Mellanox MLX5 driver for optimizing PCIe peer-to-peer private device pages and then also to the Nouveau driver for P2P DMA support. The Nouveau driver support then allows to “handle P2P page operations seamlessly” — not that you’d really be using the Nouveau open-source DRM driver for any very demanding workloads at this point. Adding it to Nouveau is done for demonstrating the functionality using fully open-source drivers and to demonstrate an open-source user per upstream kernel policies.
See this patch series for the GPU Direct RDMA proposal for device private pages.
