One of the latest Linux kernel optimizations being worked on by Meta’s large kernel engineering team is making reading of /proc/interrupts less costly. Due to monitoring of their servers frequently reading /proc/interrupts, it’s actually become a noticeable cost over time with their massive fleet of systems.
Dmitry Ilvokhin with Meta has optimized /proc/interrupts by optimizing the decimals printing by adapting the use of the seq_printf() function and also adding a helper routine for putting space and decimal numbers without the overhead of rich formatting of printf().
While it may seem like a mundane optimization, these print changes for /proc/interrupts resulted in a 29% relative speed-up in the time elapsed for printing /proc/interrupts. On a massive fleet of servers routinely capturing the interrupts output, the savings is very much worthwhile.
“Monitoring tools periodically scan /proc/interrupts to export metrics as a timeseries for future analysis and investigation.
In large fleets, /proc/interrupts is polled (often every few seconds) on every machine. The cumulative overhead adds up quickly across thousands of nodes, so reducing the cost of generating these stats does have a measurable operational impact. With the ongoing trend toward higher core counts per machine, this cost becomes even more noticeable over time, since interrupt counters are per-CPU. In Meta’s fleet, we have observed this overhead at scale.
Although a binary /proc interface would be a better long-term solution due to lower formatting (kernel side) and parsing (userspace side) overhead, the text interface will remain in use for some time, even if better solutions will be available. Optimizing the /proc/interrupts printing code is therefore still beneficial.
Function seq_printf() supports rich format string for decimals printing, but it doesn’t required for printing /proc/interrupts per CPU counters, seq_put_decimal_ull_width() function can be used instead to print per CPU counters, because very limited formatting is required for this case. Similar optimization idea is already used in show_interrupts().
As a side effect this aligns the x86 decriptions with the generic interrupts event descriptions.
…
Relative speed up in time elapsed is around 29%.”
This latest optimization patch is now under review on the Linux kernel mailing list.
