A Phoronix reader pointed out a bug report from 2015 now getting renewed interest… Linux software RAID via MD RAID, DRBD, LVM RAID, and similar software-based solutions can be broken from user-space around O_DIRECT usage. The issue is that these RAID arrays can be put silently into an inconsistent state across disks.
The 2015 bug report provided a sample script for showing how MD RAID and similar can be put into an inconsistent state across disks from a user-space program poorly abusing O_DIRECT. As Wolfgang Bumiller later summed up in the thread with the heart of the issue:
“This is not at all about the contents of the data. It is expected that garbage is written to the disks, but each disk making up the raid will contain different garbage, which means the disks are out of sync, iow. the raid is “broken”. This in turn means the user space can “break” the raid.
The problem is that with O_DIRECT the the user space pointer is passed to the block drivers for the underlying layers making up the raid, and they all read from it independently. Any user who can run a program where they can use O_DIRECT on a file on a raid can break the raid.
It is expected that garbage is written to the disk, but the whole point of a raid is that each disk should contain the *same* garbage. Keep the garbage consistent… or something.”
That was ten years ago and this bug report started by Stanislav German-Evtushenko remains open.
In the years since, discussion around the bug has come up occasionally, most recently in the context if this bug could surface in the context of VM live migration.
The only Linux file-systems reported to not be affected by this O_DIRECT bug with software RAID are OpenZFS and Bcachefs.
Those curious can find out all the details around this software RAID inconsistent issue via the Kernel.org BugZilla.
