Apache Kafka – Cluster Architecture

This explanation and provides a clearer understanding of how Kafka components interact and function.

Components and Descriptions

1. Broker
   A Kafka cluster typically consists of multiple brokers to ensure load balancing. Kafka brokers are stateless, relying on ZooKeeper to manage their cluster state. Each broker is capable of handling hundreds of thousands of reads and writes per second and can store terabytes of messages without any performance degradation. ZooKeeper also facilitates leader election among brokers, ensuring that the cluster remains resilient and balanced.
2. ZooKeeper
   ZooKeeper is a distributed coordination service used for managing and orchestrating Kafka brokers. It plays a key role in notifying producers and consumers about changes in the Kafka system, such as the addition of new brokers or the failure of existing ones. Upon receiving notifications from ZooKeeper regarding broker changes, producers and consumers adjust their operations accordingly, ensuring that data flows seamlessly between brokers.
3. Producers
   Producers are responsible for pushing data to Kafka brokers. When a new broker is added to the Kafka cluster, producers automatically detect it and begin sending messages to that broker. Kafka producers are designed for high throughput, meaning they do not wait for acknowledgments from brokers and send messages as quickly as the brokers can handle, optimizing performance and throughput.
4. Consumers
   Consumers read data from Kafka brokers. Since Kafka brokers are stateless, consumers track which messages they have consumed using partition offsets. When a consumer acknowledges a specific message offset, it indicates that all preceding messages have been consumed. Consumers make asynchronous pull requests to brokers to retrieve a buffer of messages. They can also rewind or skip to any point within a partition by specifying an offset value. The consumer offset is stored and managed by ZooKeeper, ensuring that consumers can resume processing from the correct point even in the event of a failure.