Sent out in original patch form this past week and already iterated to a second version this Sunday, a new proposal is underway to introduce “hung_task_detect_count” as a convenient means of tracking the number of times hung tasks are detected since boot.
Via /proc/sys/kernel/hung_task_warnings is already the ability to read the number of hung task warnings and the like. With a set of two patches now under review, /proc/sys/kernel/hung_task_detect_count would be added to report the total number of hung tasks detected since boot time.
Lance Yang who sent out the patches proposing the “hung_task_detect_count” counter explained in the patch cover letter:
“This patchset adds a counter, hung_task_detect_count, to track the number of times hung tasks are detected.
IHMO, hung tasks are a critical metric. Currently, we detect them by periodically parsing dmesg. However, this method isn’t as user-friendly as using a counter.
Sometimes, a short-lived issue with NIC or hard drive can quickly decrease the hung_task_warnings to zero. Without warnings, we must directly access the node to ensure that there are no more hung tasks and that the system has recovered. After all, load average alone cannot provide a clear picture.
Once this counter is in place, in a high-density deployment pattern, we plan to set hung_task_timeout_secs to a lower number to improve stability, even though this might result in false positives. And then we can set a time-based threshold: if hung tasks last beyond this duration, we will automatically migrate containers to other nodes. Based on past experience, this approach could help avoid many production disruptions.
Moreover, just like other important events such as OOM that already have counters, having a dedicated counter for hung tasks makes sense ;)”
Adding this functionality to the Linux kernel for this hung task detected counter is just 18 lines of new code and under review on the Linux kernel mailing list. Given its small size and shouldn’t be really controversial, hopefully it will end up in one of the mainline kernel releases soon.
