Apache Kafka – Do we need zookeeper

Do we need Zookeeper for a Kafka application?

The short answer is: It depends on the version of Kafka you are using.

Kafka Versions Prior to 2.8.0

For versions of Apache Kafka prior to 2.8.0, Zookeeper is required. Kafka relies on Zookeeper for the following tasks:

1. Managing Cluster Metadata: Zookeeper is used to store and manage metadata about the Kafka brokers, topics, and partitions.
2. Leader Election: Zookeeper helps with leader election for Kafka brokers and partitions. Each Kafka partition has a leader broker that is responsible for handling read and write operations. If the leader fails, Zookeeper helps elect a new leader.
3. Consumer Offset Management: Zookeeper is used for storing the offsets of consumers, so they can keep track of the messages they have consumed across restarts.
4. Cluster Coordination: Zookeeper coordinates the communication between Kafka brokers and helps maintain the Kafka cluster state.

Kafka 2.8.0 and Later

Since Kafka 2.8.0, Zookeeper is no longer a strict requirement for Kafka. Kafka introduced the KRaft mode (Kafka Raft Metadata mode), which eliminates the need for Zookeeper by using the Kafka Raft Consensus Protocol to manage metadata and handle leader election.

In KRaft mode (introduced as an experimental feature in Kafka 2.8.0 and gradually being improved), Kafka brokers can manage their own metadata and coordinate cluster activities without relying on an external Zookeeper service.

However, KRaft mode is still evolving and may not yet be suitable for all production use cases, depending on your specific requirements. It is recommended for newer versions of Kafka, and it works for simplified use cases or in testing environments.

So, Do You Need Zookeeper?

• Kafka versions 2.7.x and earlier: Yes, you need Zookeeper to run Kafka.
• Kafka version 2.8.0 and later (using KRaft mode): No, you can run Kafka without Zookeeper, but only if you’re using the KRaft mode, which may not be fully feature-complete for all scenarios yet.

How to Run Kafka Without Zookeeper (KRaft mode)

If you’re using Kafka 2.8.0 or later and want to run it without Zookeeper, follow these steps:

• Start Kafka in KRaft Mode : You must configure Kafka to run in KRaft mode by setting the process.roles and metadata.log.dir properties in the server.properties file:

# server.properties for KRaft mode (no Zookeeper) 
process.roles=broker,controller 
metadata.log.dir=/tmp/kraft-metadata 
# Disable Zookeeper 
zookeeper.connect=  

• Start Kafka as usual using the following command:

bin/kafka-server-start.sh config/server.properties

• KRaft Mode Advantages:
  • Simplifies Kafka deployment since there’s no need to manage Zookeeper separately.
  • Provides stronger consistency guarantees for managing metadata, as the leader election and metadata handling are built into Kafka itself.

Conclusion

• For Kafka versions 2.7.x and below, you need Zookeeper.
• For Kafka 2.8.0 and above, you can run Kafka without Zookeeper if you enable KRaft mode. However, this mode is still maturing, and Zookeeper is still the recommended option for production environments in some use cases.

If you are running Kafka in a production environment or require a stable, well-tested deployment, it’s generally recommended to use Zookeeper (as of Kafka 2.7.x and below) unless you specifically want to experiment with KRaft mode or your use case fits well with it.