Apache Kafka has reached a significant milestone with the release of version 4.0, a major update that introduces a host of new features and improvements, most notably the default operation in KRaft mode, which, according to Confluent’s documentation, eliminates the dependency on Apache ZooKeeper..
For over a decade, ZooKeeper has served as the backbone of Kafka, and the community has expressed gratitude for its contributions. However, the move to KRaft by default in Kafka 4.0 streamlines deployment and management by removing the need to maintain a separate ZooKeeper ensemble.
(Source: Confluent documentation)
Lalit Moharana, an AWS Community Builder, posted on LinkedIn:
ZooKeeper is stepping aside as Apache Kafka adopts KRaft with the upcoming Kafka 4.0 release, marking the end of a 14-year partnership. This shift simplifies Kafka’s architecture by ditching the separate ZooKeeper system, boosting scalability, and paving the way for a self-sufficient future – all thanks to KRaft’s Raft protocol magic.
In addition:
Why the Change? ZooKeeper’s overhead and limits (think 100,000+ partitions) couldn’t keep up with Kafka’s growth. And:
KRaft Benefits: One system, millions of partitions, faster recovery – Kafka’s ready to soar!
Beyond the architectural shift, Kafka 4.0 brings the general availability of KIP-848, which introduces a next-generation consumer group protocol. This new protocol is designed to dramatically improve rebalance performance, reducing downtime and latency for consumer groups, especially in large-scale environments. By minimizing “stop-the-world” rebalances, Kafka aims to provide a more stable and responsive data streaming experience. The new protocol is enabled by default on the server side, with consumers needing to opt in by setting group.protocol=consumer.
In a Hacker News thread, a respondent commented:
One thing I immediately noticed after switching from SNS/SQS to Kafka was its speed. Messages seem to get sent/received almost immediately.
Furthermore, Kafka 4.0 offers early access to Queues for Kafka (KIP-932). This feature introduces the concept of “share groups” to enable cooperative consumption using regular Kafka topics, effectively allowing Kafka to support traditional queue semantics. While not a direct addition of a “queue” data structure, this enhancement expands Kafka’s versatility, making it suitable for a broader range of messaging use cases, particularly those requiring point-to-point messaging patterns akin to durable shared subscriptions.
In a LinkedIn post, Govindan Gopalan, an AI & Data Engineering Leader at IBM, concluded:
Early queue support (KIP-932) introduces point-to-point messaging, expanding Kafka’s use cases beyond traditional publish-subscribe workflows.
This major release marks a significant step forward in platform modernization. As part of its evolution, Kafka 4.0 has removed APIs deprecated for at least 12 months. Furthermore, it updates the minimum Java requirements, with Kafka Clients and Kafka Streams now requiring Java 11, and Kafka Brokers, Connect, and Tools requiring Java 17. This move encourages the adoption of newer Java features and aligns Kafka with more current technology stacks. The release also updates the minimum supported client and broker versions (KIP-896) and defines new baseline requirements for supported upgrade paths, as detailed in KIP-1124.
