The Linux 6.15 kernel has just merged a fix for the big performance regression I spotlighted yesterday on Phoronix with a huge hit to the Nginx HTTPS web server performance that could see a 3x regression from the in-development Linux 6.15 kernel code. It turns out other workloads/applications also were negatively impacted by this regression. While a stumper at first even with the bisected commit, the issue was luckily resolved very quickly.
In less than 24 hours from shining attention on this new performance regression introduced in the Linux 6.15 kernel, Linus Torvalds has now pulled in the patch to address the unexpected and quite severe performance regression.
The kernel mailing list discussion began rather unexpectedly with the author of the bisected commit causing the regression commenting: “3x regression? wow. Thanks for heads up. I’m staring at the patch and don’t see it. Adding more experts.“
Upstream kernel developers at first had trouble reproducing the performance regression but fortunately, overnight SUSE Linux engineer Vlastimil Babka had widdled down the path of the issue thanks to his work on the kernel’s slab code. With a newer GCC compiler also provided an important clue and ultimately a patch was figured out. Testing it confirmed the Nginx regression reported was resolved.
Linux developer Alexei Starovoitov also discovered the troubled commit caused a 3x performance regression for Netperf and asked Linus Torvalds to pull the fix directly given the severity of the regression.
Linus Torvalds has merged the patch by Vlastimil Babka that takes care of the performance regression.
I’ve tested the patch multiple times and indeed confirming the huge hit to the performance on Linux 6.15 Git is resolved by today’s patch. Since yesterday’s article spotlighting the regression, I was also able to confirm several other workloads were negatively impacted by this same regression.
Ethr running on the localhost saw a massive hit to the network TCP throughput on this AMD EPYC server…
With this patch, in some Ethr localhost tests the patched kernel is even performing slightly better for throughput and latency on this 5th Gen AMD EPYC server.
Even the OpenFOAM computational fluid dynamics software saw its performance impaired by the problematic commit.
The ClickHouse database server did too albeit to a lesser extent than some of the other workloads.
PostgreSQL was also impacted to a lesser extent.
Memcached though was another heavy-hitter with some benchmark configurations seeing tremendous performance loss on the bisected commit.
Thankfully this regression is now corrected in Linux 6.15 Git. Now onto other Linux hardware testing and other Linux 6.15 performance benchmarking. Thanks to Phoronix supporters for making this testinb/siecting possible — especially those who subscribe to Phoronix Premium or tips via PayPal or Stripe or at least not using any ad-blockers on this site. And thanks to AMD for speedy EPYC processors that make light work out of bisecting the Linux kernel and other large codebases.
If you enjoyed this article consider joining Phoronix Premium to view this site ad-free, multi-page articles on a single page, and other benefits. PayPal or Stripe tips are also graciously accepted. Thanks for your support.
