While the Linux kernel itself may not be often viewed as a bottleneck to typical high performance computing (HPC) workloads, optimizing the Linux kernel with Profile Guided Optimizations (PGO) can prove worthwhile for those seeking maximum performance potential. A presentation this past weekend at FOSDEM 2025 is highlighting around a 3% performance gain for HPC software compiled with PGO enabled.
Alex Domingo with the HPC team at Vrije Universiteit Brussel presented at FOSDEM 2025 on the effect of Linux kernel optimizations on HPC workload performance. While HPC engineers are often after achieving maximum performance, they do not want to sacrifice stability either. Many in the HPC world are against using out-of-tree patches or other non-standard kernel features while the HPC team in Belgium decided to evaluate HPC cluster performance by optimizing the Linux kernel using compiler optimization techniques with an otherwise stock kernel.
Those involved found that compiler optimizations of the kernel typically didn’t have any significant impact but with the exception of employing Profile Guided Optimizations (PGO). PGO yielded a “small but significant” performance improvement around 3%. There is also room for more improvement with the likes of AutoFDO, BOLT, and other modern compiler features. But with Profile Guided Optimizations requiring a profile of the workload to be run on the HPC cluster, the build/setup process is lenghtier and not trivial.
In any event it’s interesting to see their 3% benefit to PGO’ing the Linux kernel for HPC clusters. Where every bit counts, it’s certainly something for more HPC administrators to consider. Those wanting to learn more about this research can find all the presentation details on FOSDEM.org.
