XTX Markets as one of the largest algorithmic trading firms that handles $250 billion in daily traded volume and relies on around 650+ petabytes of storage for its price forecasts and other algorithmic trading data has open-sourced its Linux file-system. XTX developed TernFS for distributed storage after they outgrew their original NFS usage and other file-system alternatives.
TernFS is a distributed file-system for dealing with reading and writing large immutable files — immutable in the context that they are typically not modified after being created. And large in the context of being at least a few megabytes. TernFS is designed for XTX data center needs of maxing out at around 10EB of logical file storage, around one trillion files and 100 billion directories with around one million clients. All running atop commodity hardware and Ethernet networking.
TernFS is also able to span across multiple geographic regions, store data redundantly to protect against drive failures, no single point of failure for metadata, and other features for the demanding needs of XTX Markets.
TernFS has been in development for over three years and for over the past year is being used by all of the XTX machine learning efforts. XTX Markets reports that “to this day, we haven’t lost a single byte” on TernFS.
Those wishing to learn more about this Linux-native TernFS file-system that is now open-source can do so via the XTX Markets blog. TernFS is available under GPLv2+ and Apache 2.0 licenses. The TernFS file-system is open-source on GitHub.
