Apache Kafka – Broker Setup

To set up an Apache Kafka broker, we need to install and configure Kafka on a server or set of servers. Kafka brokers are the heart of the Kafka cluster, and they handle storing data, managing topics, and serving producers and consumers.

In this guide, we will walk through the steps to set up a Kafka broker using Apache Kafka 2.8.0+ on a local machine or server. This example assumes that Apache Zookeeper is either set up separately or used within the Kafka broker configuration (for versions before KRaft mode was introduced in Kafka 2.8.0, Zookeeper is required).

Prerequisites:

1. Java: Kafka runs on Java, so you need to install Java 8 or later on your machine.
2. Apache Kafka: Download and install Apache Kafka.
3. Apache Zookeeper: Kafka uses Zookeeper (in traditional Kafka setup) to manage brokers and partitions. If using KRaft mode (Kafka without Zookeeper), this step can be skipped.

Step 1: Install Java

Kafka requires Java to run. You can download the latest JDK from Oracle’s official website or use OpenJDK.

To verify Java installation:

java -version

openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7)
OpenJDK 64-Bit Server VM (build 11.0.12+7, mixed mode)

It should output something like:

Step 2: Download and Install Apache Kafka

1. Download Kafka from the official website:
   • Kafka Download Link
2. Extract Kafka to a folder:

tar -xvf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0

Alternatively, if you’re on Windows, you can use a ZIP version and extract it to your desired location.

Step 3: Start Zookeeper (if applicable)

Kafka relies on Zookeeper for managing the cluster. If you’re using an older version of Kafka or have not set up KRaft mode (Kafka without Zookeeper), you need to run Zookeeper first.

1. Start Zookeeper Server : Kafka comes with a built-in Zookeeper, so you can use the default zookeeper.properties file to start Zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties 

This will start Zookeeper on port 2181 by default.

Step 4: Start Kafka Broker

Once Zookeeper is running, you can start the Kafka broker.

1. Start Kafka Broker : Kafka’s broker configuration is found in the config/server.properties file. By default, it is set to use Zookeeper running on localhost:2181.To start Kafka, use the following command :

bin/kafka-server-start.sh config/server.properties 

This will start the Kafka broker on localhost:9092 by default.

Configuring the Kafka Broker:

• listeners : The Kafka broker will listen for incoming connections here.
• zookeeper.connect :vKafka connects to Zookeeper using this property (if using Zookeeper mode).
• log.dirs : This is the directory where Kafka stores message logs.

Example server.properties :

# Kafka Broker Configuration

# Kafka will listen on port 9092 (default)
listeners=PLAINTEXT://localhost:9092

# Zookeeper connection string for cluster management (only needed for older versions)
zookeeper.connect=localhost:2181

# Directory where Kafka stores its logs (this can be changed)
log.dirs=/tmp/kafka-logs

# Broker ID (can be any unique integer for each broker in the cluster)
broker.id=1

# Number of partitions per topic (adjust according to your needs)
num.partitions=3

# Replication factor (number of replicas for partitions)
default.replication.factor=1

Step 5: Verify Kafka Broker is Running

Once you start Kafka using the above command, you should see logs indicating that the broker is running. You can verify by checking the logs or querying the Kafka broker.

To verify the broker is running, you can use the following command to list topics:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

If the Kafka broker is running properly, it will show an empty list of topics initially.

Step 6: Create a Kafka Topic

Now that your Kafka broker is up and running, you can create a topic.

bin/kafka-topics.sh --create --topic insurance_policies --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1

• insurance_policies : The name of the topic.
• –partitions 3 : The number of partitions for this topic.
• –replication-factor 1 : Number of replicas of the partitions.

This creates a topic with 3 partitions and 1 replica.

Step 7: Produce Messages to the Topic

You can now send messages to this topic using the Kafka producer.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic insurance_policies

After running this command, you can start typing your messages. Each message will be sent to the insurance_policies topic.

For example:

{"policyId": "P12345", "customer": "John Doe", "coverage": "Health", "amount": 50000}

Press Enter after typing each message to send it to Kafka.

Step 8: Consume Messages from the Topic

To consume messages from the insurance_policies topic:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic insurance_policies --from-beginning

• –from-beginning : This option ensures the consumer reads all messages from the start of the topic (not just the latest ones).
• The consumer will display each message along with the offset.

Step 9: Scaling Kafka Brokers (Optional)

If you want to scale Kafka to handle more traffic or provide redundancy, you can set up multiple Kafka brokers.

To scale Kafka:

1. Configure additional brokers by changing the broker.id and updating the listeners property in the server.properties for each new broker.
2. Make sure each broker connects to the same Zookeeper instance and uses unique ports and directories.

You can then run these additional broker instances using the same bin/kafka-server-start.sh command for each broker, pointing to their respective configurations.

Conclusion

You’ve now successfully set up a Kafka broker and a topic with multiple partitions. You can produce and consume messages using the Kafka Producer and Kafka Consumer tools.

In this setup, we used Zookeeper to manage the Kafka cluster. However, starting from Kafka 2.8.0, KRaft mode allows you to run Kafka without Zookeeper. If you want to set up Kafka without Zookeeper (using KRaft mode), you need to configure the Kafka broker accordingly.