For those building your own server around the new Intel Xeon 6 “Granite Rapids” or AMD EPYC 9005 “Turin” platforms and using DDR5-6000/DDR5-6400 memory (or the interesting MRDIMM-8000 memory with Xeon 6 P), one of the factors you need to be much more mindful about than in the past is that at least the initial generation of DDR5-6000+ memory is running much hotter than prior server memory. There’s been guidance from Intel and AMD as well as sever vendors about the thermal considerations with DDR5-6000/DDR5-6400 and extra precautions. Here’s a look at the DDR5-6000 thermals on a recent AMD EPYC 9655 Turin server build and then taming the DDR5 server memory modules using Corsair Vengeance Airflow Memory Cooling Fans.
When building out an AMD 5th Gen EPYC Turin 1P server a few weeks ago with the Supermicro H13SSL-N motherboard in a 4U rackmount chassis, one of the very early (immediate) observations was the much higher memory temperatures with the DDR5-6400 (operating at DDR5-6000) memory modules compared to the much more comfortable temperatures with DDR5-4800 server memory or prior. But it wasn’t unexpected given my hands-on with other Intel Xeon 6 and AMD EPYC 9005 series servers and hearing from vendors: these faster DDR5 server memory modules are significantly hotter than DDR5-4800 ECC RDIMMs of prior generation servers or DDR4 server memory.
If buying a pre-assembled server it’s less of a concern given the ODM has factored the memory cooling into the equation, but for those assembling your own server, this article is also to serve as a public service announcement (PSA) to draw more awareness.
This AMD EPYC 96-core Turin server build was within a Rosewill RSV-L4500U 4U rackmount chassis. I’ve used about three dozen of these Rosewill 4U cases of similar model over the years and this was the first time finding the triple 120mm intake fans (and 3~4 80mm exhaust fans) to be inadequate for keeping the system memory operating efficiently. Even with this Turin server build using a SilverStone XE360-SP5 AIO liquid cooler for Socket SP5 processors, the memory temperatures were running too hot. The AMD EPYC 9655 1P server with the Supermicro H13SSL-N motherboard was using 12 x 64GB DDR5-6000MT/s Micron MTC40F2046S1RC64BDY QSFF memory modules and the BMC was confirming the memory operating too warm and above the platform thresholds.
The BMC on this Supermicro motherboard exposes the RAM temperatures as “DIMMA~F Temp” and “DIMMG~L Temp” that could easily run above 90 degrees C under load in this 4U chassis. Even running all of the fans at full speed still was showing the DDR5 temperatures spiking. So immediately I opted for some active RAM cooling and since then the Micron DDR5 memory temperatures have been back under control. For cooling the Micron DDR5-6000 (DDR5-6400) modules I picked up two Corsair Vengeance Airflow Memory Cooling Fan CMYAF coolers.
The Corsair Vengeance Airflow Memory Cooling Fan has an adapter for fitting atop DDR3/DDR4/DDR5 memory and equipped with a 60mm PWM fan. While more designed for desktop systems, the Corsair Vengeance Airflow Memory Cooling Fan is cheap at around $25 USD each (Amazon.com; affiliate link) and works fine for helping cool DDR5 server memory especially with this server just being a build for more benchmarking and other hardware testing at Phoronix as opposed to a commissioned production server.
As part of this PSA and showing the Corsair Vengeance Airflow Memory Cooling Fan performance for DDR5-6000/DDR5-6400 server memory, I ran some benchmarks on this AMD EPYC Turin 1P 4U server without the RAM fans while having the 3 x 120mm intake fans and 3 x 80mm exhaust fans at full speed and with the SilverStone XE360-SP5 AIO liquid cooling. And then the same exact configuration but having a Corsair Vengeance Airflow Memory Cooling Fan installed on each set of the six DDR5 memory DIMMs. From there the memory temperature (using the BMC-reported “DIMMA~F Temp”) was looked at as well as the impact of the system performance when the DDR5 memory was throttling in the original configuration. Again, this article is namely for reference/PSA purposes.
