Apache Kafka – Installation

Apache Kafka – Installation and Setup Guide

Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications. Below is a step-by-step guide to install and set up Kafka on a local machine (Linux/macOS/Windows).

Prerequisites

Before you install Kafka, make sure you have the following installed:

• Java: Kafka requires Java to run, so ensure that Java 8 or later is installed.
• Check Java version : java -version
• If Java is not installed, you can install it as follows:
• Ubuntu/Debian:

sudo apt update 
sudo apt install openjdk-11-jdk

• macOS : brew install openjdk@11

• Windows: Download and install Java from the official website: https://www.oracle.com/java/technologies/javase-jdk11-downloads.html

Apache Zookeeper: Kafka uses Zookeeper for managing and coordinating brokers.

• Kafka requires Zookeeper to function correctly, but starting from Kafka 2.8, there is an option to run Kafka without Zookeeper in KRaft mode. For simplicity, we will cover the standard Zookeeper setup.

Step 1: Download Apache Kafka

1. Go to the official Kafka website: https://kafka.apache.org/downloads.
2. Download the latest stable version of Kafka (e.g., 3.x.x).
3. Extract the downloaded tar file to your desired directory.

For example:

tar -xvzf kafka_2.13-3.x.x.tgz
cd kafka_2.13-3.x.x

Step 2: Start Apache Zookeeper

Kafka requires Zookeeper for managing cluster metadata and leader election between brokers. The Zookeeper instance is included in the Kafka download.

1. Start Zookeeper: This command will start the Zookeeper server on port 2181 by default. Keep this terminal window open.

bin/zookeeper-server-start.sh config/zookeeper.properties 

Step 3: Start Apache Kafka Broker

Once Zookeeper is up and running, you can start the Kafka broker. The Kafka broker will manage the messages sent by producers and consume messages by consumers.

1. Start Kafka: Open another terminal window and run the following command to start the Kafka broker :

bin/kafka-server-start.sh config/server.properties

This will start the Kafka broker on port 9092 by default. You can customize this configuration by modifying the server.properties file.

Step 4: Create a Kafka Topic

Kafka organizes messages into topics. You can create a topic by running the following command:

bin/kafka-topics.sh --create --topic my_topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

–topic: Specify the topic name (e.g., my_topic).
–partitions: Number of partitions for the topic (e.g., 1).
–replication-factor: Replication factor for the topic (e.g., 1 for single node setup).

Step 5: Start Kafka Producer

To start sending messages to Kafka, you can use the Kafka producer.

1. Start a Kafka producer:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic 

After running this command, you can start typing messages, and they will be sent to the my_topic topic.

Step 6: Start Kafka Consumer

To consume messages from a Kafka topic, use the Kafka consumer.

1. Start a Kafka consumer :

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my_topic --from-beginning

The consumer will start consuming messages from the my_topic topic. The –from-beginning flag ensures that the consumer reads all the messages from the beginning.

Step 7: Verify the Setup

1. Send some messages to the producer terminal.
2. Check the consumer terminal to verify that the messages are being consumed.

Optional: Configure Kafka to Run as a Service (Linux)

If you want to set up Kafka to run as a background service (using systemd on Linux), you can follow these steps:

• Create a Kafka service file at /etc/systemd/system/kafka.service with the following content:

[Unit] 
Description=Apache Kafka Server 
Documentation=http://kafka.apache.org/ 
After=zookeeper.service 

[Service] 
Type=simple 
User=kafka 
ExecStart=/path/to/kafka/bin/kafka-server-start.sh /path/to/kafka/config/server.properties 
ExecStop=/path/to/kafka/bin/kafka-server-stop.sh 
Restart=on-failure 

[Install] 
WantedBy=multi-user.target

• Reload systemd and start the Kafka service:

sudo systemctl daemon-reload 
sudo systemctl start kafka 
sudo systemctl enable kafka

Conclusion

This setup guide outlines the basic installation and configuration steps for setting up Apache Kafka with Zookeeper. For production environments, you should configure multiple brokers, replication, and proper fault tolerance, as well as manage Zookeeper and Kafka configurations to meet your performance and scalability needs.