LinkedIn today announced Northguard, a scalable log storage system that replaces Kafka, and Xinfra, a virtualized Pub/Sub layer. Northguard delivers sharded data & metadata, log striping, strong consistency, and self-balancing clusters at a larger scale than Kafka, while Xinfra enables seamless migration and unified access across Kafka and Northguard.
According to LinkedIn’s engineers, Kafka had become increasingly difficult to manage at LinkedIn’s scale (32T records/day, 17 PB/day, 400K topics, 150 clusters). Northguard’s architecture, consisting of shared data & metadata, decentralized coordination, and minimal global state, eliminates Kafka’s single-controller and partition-based limitations.
Northguard’s data model organizes logs into records, segments, ranges, and topics. Records (key, value, headers) are written into segments — immutable units of replication. Segments form ranges, which represent a contiguous keyspace slice and support dynamic splitting and merging for scaling and ordering. Topics, on the other hand, are collections of ranges that cover the entire keyspace, with flexible storage policies for replication and retention.
A Northguard Range containing three segments (source)
This fine-grained structure enables a balanced load, high availability, and seamless scaling. Brokers naturally self-balance as producers produce new segments, eliminating the need for expensive rebalancing or data movement when brokers are added or fail.
New segments get added to the range and get assigned to potentially new brokers (source)
Compared to traditional indexed partitions, ranges provide a more flexible scaling mechanism. Range splits only interrupt clients’ writing to the affected range (not the entire topic) while maintaining total ordering guarantees. This difference reduces disruption and avoids costly “stop-the-world” synchronization. Additionally, aligned ranges across topics simplify stream processing joins by reducing the need for shuffles.
Northguard’s metadata model uses sharded, Raft-backed replicated state machines (DS-RSM), distributed across vnodes. Each vnode manages metadata for topics, ranges, and segments — tracking state changes (e.g., splits, merges, sealing), replica sets, and retention policies. By sharding metadata via consistent hashing and using decentralized coordination, Northguard avoids Kafka’s controller bottlenecks and supports millions of replicas with strong consistency and high availability.
LinkedIn optimized Northguard’s protocols for performance and durability. Metadata operations such as create, delete, and query use unary request/response calls routed to vnode leaders. Produce, consume, and replication flows are sessionized streaming protocols with pipelining and windowing to maximize throughput and minimize latency.
Producers write to active segment leaders, receiving acknowledgments only after fsync on all replicas, ensuring strong durability. Consumers use a similar streaming model with client-controlled flow, supporting efficient high-throughput reads. Active segment replication and sealed segment replication utilize the same efficient streaming architecture, resulting in Northguard’s high performance and self-healing capabilities.
Migrating from Kafka to Northguard at LinkedIn’s scale required seamless, zero-downtime transitions for thousands of mission-critical apps. To support this, LinkedIn built Xinfra, a virtualized Pub/Sub layer that abstracts away physical clusters. Xinfra enables topics to span Kafka and Northguard through dual-write mechanisms, allowing live migrations without client changes.
An example use case where a consumer subscribes to three topics under the same virtual cluster, with each topic located in different clusters (source)
Segment storage in Northguard is pluggable, with the default “fps-store” implementation optimized for durability and latency. It uses a write-ahead log (WAL), creates one file per segment, applies Direct I/O to bypass OS buffering, and maintains a sparse index in RocksDB. Batches of records are flushed and fsynced across replicas within milliseconds, ensuring durability even under failure scenarios. This design avoids cache inconsistencies, supports efficient reads from old segments, and allows predictable performance as clusters grow.
To ensure reliability at scale, Northguard undergoes rigorous testing under deterministic simulation. Entire clusters and clients run in a single-threaded, controlled environment where faults — such as broker shutdowns, network partitions, disk errors, and rolling upgrades — are injected and replayed. This method allows LinkedIn to simulate years of activity daily, catch edge cases early, and continuously validate correctness under complex failure scenarios.
LinkedIn engineers state that they “have successfully migrated thousands of topics from Kafka to Northguard, accounting for trillions of records per day” and that over 90% of applications at LinkedIn are already running Xinfra clients.
InfoQ reached out to LinkedIn regarding intentions to open-source Northguard and Xinfra. LinkedIn stated that they “are focused on finalizing the implementation of Northguard and Xinfra within our internal systems, and as we continue to build, learn, and iterate on these tools, we’ll explore the possibilities of open-sourcing them.”
